Planet Crustaceans

This is a Planet instance for community feeds. To add/update an entry or otherwise improve things, fork this repo.

May 06, 2021

asrpo (asrp)

Instruction level just-in-time programming May 06, 2021 08:39 PM

Just-in-time programming is a workflow for creating a program top-down, while a program is running. This is typical in Smalltalk environments like Squeak.

In this post, I want to describe an instruction level variant of this.

I think the best way to describe instruction level just-in-time programming (IL-JIT programming) is by showing how it works. I'll start with the classic Fibonacci example (Rosetta code has many implementations). We'll implement Fibonacci while evaluating fib(3).

Robin Schroer (sulami)

Onboarding Across Timezones May 06, 2021 12:00 AM

Even in a fully distributed organisation, teams are often clustered in timezones to facilitate collaboration. Another model is the deliberate spreading of teams to enable Follow the Sun workflows, which can also improve pager rotations.

In a distributed team, onboarding new team members can be especially difficult. New onboardees lack the context and institutional knowledge required to work effectively in an independent manner. Here are some strategies which I have found ease this process, specifically in the field of software engineering.

Onboarding Buddies

It is standard practice to designate an onboarding buddy for a new onboardee, a person in the same team who can take care of them during the first few weeks, answer questions, and pair a lot. It is not always possible to have this onboarding buddy in the same timezone, and in these cases it is a good idea to have an existing employee in the same timezone as a designated point of contact, regardless of team affiliations.

Making the Most of the Overlap

When onboarding someone in different timezones, everything has to be planned around the overlap in office hours. Start by clearly identifying this overlap. It is most important for the onboardee to be able to onboard and work effectively outside of this overlap.

To optimise the value gotten out of the overlap, it is best spent with synchronous conversations. Other tasks such as code reviews or planning work should be moved outside the overlap unless there is value in doing it synchronously, for example by sharing additional context.

If the overlap is very small, short screencasts can be medium-cost, high-bandwidth method to transfer context asynchronously. These can be recorded and watched outside the overlap, but convey much of the same context as pairing could, albeit without the ability to actually interject questions.

Enabling Self-Directed Learning

To enable the onboardee to learn on the job, there needs to be a pool of work items for them to pick up tasks from. These items need be narrowly scoped and include much more detail and context than they ordinarily would, to reduce the risk of the onboardee getting stuck.

Tasks should be sized “just right,” that is not larger than two or three days, but also not shorter than a single day. Very small tasks encourage the onboardee to start working on several tasks in parallel while waiting for code reviews, increasing cognitive load.

A library of self-directed training material is a must-have for every organisation, distributed or not, but can also serve as a fallback if the onboardee gets stuck on their current task and has to wait for the next overlap.

Just Because You Can Does Not Mean You Should

While asking colleagues for ideas & feedback, a sentiment I have heard more than once was “if you can, avoid scattering your team.” Just because you can spread a team literally around the globe does not mean you should do this. At my current day job we have employees around the globe, but generally try to keep teams within two timezone regions, for example within the North American east coast and Europe, or the west coast and APAC, so that we have an overlap of at least two to three hours.

May 05, 2021

Mark Fischer (flyingfisch)

Is a bell noisy if there is no bike to attach it to May 05, 2021 08:24 PM

There was once a young lady named Bella
Who fancied herself at Coachella
She wanted a blog post
Instead got a blog roast
I don’t know how to end this poem wella.

Mark J. Nelson (mjn)

Academic ancestor graphs from Wikidata May 05, 2021 12:00 PM

Note: A version of this blog post is available as an interactive Jupyter Notebook hosted on Google Colab.

The Wikidata relation doctoral advisor (P184) links researchers to the advisor or advisors who supervised their Ph.D. Note that coverage is currently a bit spotty. But, it's at least a superset of the data in the Mathematics Genealogy Project (because MGP data was imported into Wikidata), plus some from various other sources, such as parsing Wikipedia infoboxes, and manual additions (I've added quite a bit myself).

A nice thing about Wikidata is that it's queryable with SPARQL. Here is a query that finds all my academic "ancestors":

SELECT ?ancestorLabel WHERE {
  wd:Q65921654 wdt:P184+ ?ancestor.
  ?ancestor wdt:P31 wd:Q5.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }

Brief explanation:

  • Q65921654 is the Wikidata ID for me (Mark J. Nelson).
  • P184+ means one or more applications of the doctoral advisor relation.
  • ?ancestor is a variable
  • We only consider possible ancestors who are (P31) humans (Q5). This shouldn't really be necessary, but I sometimes get spurious results if it's omitted.
  • The wikibase:label service adds some magic that grabs the labels for variables: for any variable ?foo, ?fooLabel will be Wikidata's label for that item (in this case in English). That's so we get people's names back instead of Qxxxx codes.

You can edit and run queries like this interactively over at Wikidata, or programmatically from any language that can make HTTP requests.

Academic ancestor tree

That's a flat list of ancestors. But how do they relate? It might be nice to draw out the tree. To do that, we need to save a little more data: who's connected to who. (The query below uses some techniques adapted from this Stackoverflow answer by Joshua Taylor.) Specifically, we want every link in the ancestry tree. In the query below, ?ancestor2 is a direct advisor of ?ancestor1, and ?ancestor1 is either myself, or one of my academic ancestors:

SELECT ?ancestor1Label ?ancestor2Label WHERE {
  wd:Q65921654 wdt:P184* ?ancestor1.
  ?ancestor1 wdt:P184 ?ancestor2.
  ?ancestor1 wdt:P31 wd:Q5.
  ?ancestor2 wdt:P31 wd:Q5.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
As before, this can be run interactively. But it's probably more useful to do it programmatically, since we want to graph the results. For examples of doing that in Python, see either the Jupyter notebook version of this post, or the standalone Python script.

Either way, we now we have enough information to draw an ancestor tree using graphviz!

Mine ends up being pretty giant, so I won't show it here. One of my advisors, Charles Isbell, has an absolutely huge ancestor tree that trails off into long chains of medieval mathematicians that the Mathematics Genealogy Project has meticulously chronicled. But here's the other half of my academic ancestor tree, the one starting at my advisor Michael Mateas:

Common academic ancestor tree

So far we've been querying ancestors of a specific person. There are of course a lot more ways to slice and dice this big doctoral-advisor graph in Wikidata. Another one I find interesting: do two people have a common ancestor?

Here's one way to pull that out of SPARQL, again borrowing an idea from something Joshua Taylor posted on Stackoverflow. It's a bit hairier than the previous queries.

SELECT ?ancestor1aLabel ?ancestor2aLabel ?ancestor1bLabel ?ancestor2bLabel WHERE {
  # ancestors of the first person leading to a common ancestor (or ancestors)
  wd:Q65921654 wdt:P184* ?ancestor1a.
  ?ancestor1a wdt:P184 ?ancestor2a.
  ?ancestor2a wdt:P184* ?common_ancestor.
  # ancestors of the second person leading to a common ancestor (or ancestors)
  wd:Q105669257 wdt:P184* ?ancestor1b.
  ?ancestor1b wdt:P184 ?ancestor2b.
  ?ancestor2b wdt:P184* ?common_ancestor.
  # stop at the common ancestor(s) rather than retrieving their own ancestors
    wd:Q65921654 wdt:P184* ?intermediate_ancestor.
    wd:Q105669257 wdt:P184* ?intermediate_ancestor.
    ?intermediate_ancestor wdt:P184 ?common_ancestor.
  ?ancestor1a wdt:P31 wd:Q5.
  ?ancestor2a wdt:P31 wd:Q5.
  ?ancestor1b wdt:P31 wd:Q5.
  ?ancestor2b wdt:P31 wd:Q5.
  ?common_ancestor wdt:P31 wd:Q5.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
This particular query (interactive version) looks for a common ancestor between myself and Amy Hoover.

The result:

We weren't sure we had one, but it turns out that we do have a common academic ancestor, Roger Schank.

Naturally, all these queries depend on the data being in Wikidata. Anyone can add data there, so if you or your advisor are missing (or are in Wikidata but the advisor link isn't there), it's possible to go add it.

* * *

I've written some command-line Python scripts implementing the above queries:

May 03, 2021

Ponylang (SeanTAllen)

Last Week in Pony - May 3, 2021 May 03, 2021 04:27 PM

Version 0.40.0 of ponylang/ponyc has been released! Also, the ‘Add String.add iso proposal’ RFC is now in final comment period.

May 02, 2021

Derek Jones (derek-jones)

Claiming that software is AI based is about to become expensive May 02, 2021 09:59 PM

The European Commission is updating the EU Machinery Directive, which covers the sale of machinery products within the EU. The updates include wording to deal with intelligent robots, and what the commission calls AI software (contained in machinery products).

The purpose of the initiative is to: “… (i) ensuring a high level of safety and protection for users of machinery and other people exposed to it; and (ii) establishing a high level of trust in digital innovative technologies for consumers and users, …”

What is AI software, and how is it different from non-AI software?

Answering these questions requires knowing what is, and is not, AI. The EU defines Artificial Intelligence as:

  • ‘AI system’ means a system that is either software-based or embedded in hardware devices, and that displays behaviour simulating intelligence by, inter alia, collecting and processing data, analysing and interpreting its environment, and by taking action, with some degree of autonomy, to achieve specific goals;
  • ‘autonomous’ means an AI system that operates by interpreting certain input, and by using a set of predetermined instructions, without being limited to such instructions, despite the system’s behaviour being constrained by and targeted at fulfilling the goal it was given and other relevant design choices made by its developer;

‘Simulating intelligence’ sounds reasonable, but actually just moves the problem on, to defining what is, or is not, intelligence. If intelligence is judged on an activity by activity bases, will self-driving cars be required to have the avoidance skills of a fly, while other activities might have to be on par with those of birds? There is a commission working document that defines: “Autonomous AI, or artificial super intelligence (ASI), is where AI surpasses human intelligence across all fields.”

The ‘autonomous’ component of the definition is so broad that it covers a wide range of programs that are not currently considered to be AI based.

The impact of the proposed update is that machinery products containing AI software are going to incur expensive conformance costs, which products containing non-AI software won’t have to pay.

Today it does not cost companies to claim that their systems are AI based. This will obviously change when a significant cost is involved. There is a parallel here with companies that used to claim that their beauty products provided medical benefits; the Federal Food and Drug Administration started requiring companies making such claims to submit their products to the new drug approval process (which is hideously expensive), companies switched to claiming their products provided “… the appearance of …”.

How are vendors likely to respond to the much higher costs involved in selling products that are considered to contain ‘AI software’?

Those involved in the development of products labelled as ‘safety critical’ try to prevent costs escalating by minimizing the amount of software treated as ‘safety critical’. Some of the arguments made for why some software is/is not considered safety critical can appear contrived (at least to me). It will be entertaining watching vendors, who once shouted “our products are AI based”, switching to arguing that only a tiny proportion of the code is actually AI based.

A mega-corp interested in having their ‘AI software’ adopted as an industry standard could fund the work necessary for the library/tool to be compliant with the EU directives. The cost of initial compliance might be within reach of smaller companies, but the cost of maintaining compliance as the product evolves is something that only a large company is likely to be able to afford.

The EU’s updating of its machinery directive is the first step towards formalising a legal definition of intelligence. Many years from now there will be a legal case that creates what later generation will consider to be the first legally accepted definition.

Gustaf Erikson (gerikson)

14,000 dead in Sweden May 02, 2021 02:23 PM

May 01, 2021

Jan van den Berg (j11g)

2020 Music Discoveries May 01, 2021 07:52 PM

Remember 2020? Yeah, I know. Well here is the list you’ve been waiting for.

Previous lists are here: 2019, 2018 part 1, 2018 part 2, 2017, 2016 and 2015

As usual: a you get couple of words and a YouTube video. All killer, no filler. I promise.

Note: I only noticed when I compiled the list that a lot of songs are about death…

Let’s kick it off with Molchat Doma. What’s that? Glad you asked! Molchat Doma is russian doomer music and the perfect soundtrack for 2020. Their album Etazhi is probably my most played album of 2020. It’s a rare album in its complete tonal consistency and overall freshness. This next track Volny is one of the highlights. The video alone should receive all sorts of prizes. Enjoy, and don’t worry there is more Molchat Doma on the list.

Small song, country feel. 272 views. Steph Copeland ft. Chris Wong. When I lay my head to rest, there will grow a garden from my chest.

This next song is quite the trip! If the first few notes don’t grab you by the throat, maybe Ben’s performance will. This is actually three songs into one. Take your time, this is a very intense performance. And do stick around for what happens after four minutes.

I like me some Bruce and you know it. Bruce is like a machine and delivers, always. And the title track for the docu/album is as Springsteen as they come. Banger!

While we’re on the subject of old rockers with guitars, let me just share this rendition of an old folk song. I can picture this song to be the last song in every Irish pub before closing time (you know, pre-covid etc.). You cannot not love this.

Ok last old guy with a guitar. I can’t quite put my finger on it but the remaster of Pocahontas is much better than the original?? It’s the tempo, maybe. I don’t know. Anyway here is Neil Young singing he wished he had a thousand pelts so he could sleep with Pocahontas (?!), read the wiki if you want to know more. It’s quite the trip.

Lets pick up the tempo. I am a sucker for urgent sounding songs like this. The pacing here is terrific. And only 278 views. How?! This song is STRAIGHT FIRE. Time! Let the room untouched!

Talking about urgency, Fontaines D.C. know everything about that. The band with multiple entries on last year’s list just keeps producing one great track after. This band is everything a band should be. And if this next track doesn’t convince you, I am sorry for you.

One of the few women on the list. Katie Pruitt. Poppy but she has skills. I like it! The Fleetwood Mac is strong in this one.

Back to the UK. If you are still wondering where Arctic Monkeys got their inspiration, look no further. I was thoroughly surprised when I found out this song actually predates the Arctic Monkeys because this song is MOST definitely the inspiration for Teddy Picker. Not the other way around. Alex Turner you cheeky bastard!

Let’s stay in the UK shall we. Bands like the Enemy, with clear tracebacks to the Jam can only spring from the UK. A quintessentially British song.

Here is a very special song by Empress Camelian. This song has everything you would want from a laptop artist. Or any artist for that matter. And this is exactly the kind of song I hope to find when I scour the internet. Intimate and melodic. You’ll be surprised.

Ok I lied, here is another old guy with a guitar. Bob Dylan. Of course. Dylan released his 39th studio album in 2020 Rough and Rowdy Ways. And a few alternate takes and reissues surfaced. Good stuff for fans of the man that’ll turn 80 (!) soon. But the one song that had everybody talking, because of its length and content, was Murder Most Foul. There is a lot to unpack here, because not only is it a Bob Dylan song, it’s also almost 17 minutes. Clear your schedule please.

Ok so this is a Dutch track, but I can’t not include this. These boys hit all the right notes. The guitar is definitely a callback to best Dutch popbands of the 80s. But don’t be fooled, there is more here than what it looks like, and it already looks just terrific.

Paul Simon is and will always be one of my favorite songwriters. This song was written just after Nixon got elected and is probably one of his most political songs. And I only recently learned what makes the melody for American Tune so great. Can you hear it?

I am sorry what’s that? You want sci-fi m83-ish sounds? Say no more.

I promised a little bit more of Molchat Doma. Well here it is, for the acquired taste palette. This track is just completely bonkers, but in a good way, image what that sounds like live.

Oh, Bright Eyes how I love thee. This guy can hardly do wrong and just keeps churning out instant classics. And this is a fantastic live rendition of a fantastic song. Handsdown the best Covid performance I saw this year. For the daily double bonus: can you tell who the drummer is?

This track was removed from Youtube. So I’ll share another rather similar one.

You know what, that is a GREAT question Jason Isbell?! What have *I* done.

Declan McKenna was on my list in 2015, when he was just a 16 year old kid, because the talent was so obviously apparent. This next track — 5 years on — proves that his talent has not been wasted. He already has quite the following and is on his way to becoming a bona fide popstar. This is his 2020 hit. Poppy, sure. Delicious, yes. You can’t hide talent like this.

Save the best for last. This is hands down the best song of 2020 for me (Molchat Doma takes home the album award). Wolf Parade. The video, well it’s… something and the artist went ALL out. The song itself is a grower. And I admit It didn’t click on my first listen. But hidden in this song is something quite remarkable. There is no greater build up to an outro break in all of 2020 — or maybe even the decade — than the 3:28 mark in this video. Just when you start wondering where this song is going, the essence of this track lays it soul bare. It is astounding and it’s the absolute pinnacle of an already absolutely incredible song. One for the ages. Turn it up to 11. Please do also check out the some of the live versions. The live performances really punctuate what this song is about. Drums, keys, guitar and a singer who doesn’t hold back. A delight to watch. This is what it is about. To say that he loves you, would be unfair towards you.

The post 2020 Music Discoveries appeared first on Jan van den Berg.

Pierre Chapuis (catwell)

What I do at Inch May 01, 2021 06:00 PM

I have been working at Inch for about a year and a half now, so I thought it was a good time to write a bit about what I do there.

First, a bit of context: Inch is SaaS software for property managers that operates in the French (and Belgian) market. Many of our customers are co-ownership trustees, which is a much bigger deal in France than in most countries, but we also serve other professionals such as rental managers and social housing landlords.

The software can be seen as a mix between a CRM and a ticketing solution. The code base dates back to 2013. The backend is a Rails monolith and the frontend is a React application. There were two other developers on the team when I joined, now we are five, of which I am the most senior. We all directly report to the founder responsible for product, but I am sometimes presented as the CTO to customers because from their point of view I act as such (more on that later).

Now, what do I do there in practice? As you may guess if you know me, a few different things.

Dealing with integrations

That is my main role, and the reason I was hired in the first place. See, our software is used by our customers to manage the relationship with their own customers and suppliers, but they invariably already have pre-existing sotware with overlapping data: their accounting software or ERPs. Some have built them in-house, but most picked from a very fragmented market of industry-specific software which we have to integrate with. So I maintain somewhere between 20 and 30 different integrations with those pieces of software you have probably never heard about.

You may not understand the complexity until you realize that none of those integrations looks the same, because the software itself doesn't. Some is cloud-based, some is on-prem. In some cases they provide APIs or flat-file data export mechanisms, but sometimes we have to go get the data directly in their database. Of course, we almost never have technical documentation for their data model, and the databases themselves are all different. I have code for over 10 different major DBMS, not all of them using SQL, and not all of them running on typical OSs (hello AS400, hello SCO...).

Add to that integration with a few APIs we use, the maintenance of our own APIs so others can integrate with us and a few custom features for large customers, and you will have an idea of the scope. It is a role that requires domain knowledge, technical insight about many systems and the ability to understand new ones quickly. And maybe most importantly, I have to model our own software and systems so that they can support all of this. If you think about it it's all about synchronizing data between systems, so I am not all that far from my specialty after all. :)

This role means that when we talk to large customers or prospects I am often the person representing the technical side of the company (hence the CTO thing).

Operations and security

You know the pattern now: I arrive at a startup with a small team of developers where nobody has significant operations experience and I end up becoming responsible for them. It never bothered me, quite the opposite actually: I have been doing systems administration for 20 years and I have always liked it.

In this case, there had been some people who knew what they were doing on the team before, so it was not that bad. For instance, they had a reasonable backup strategy already (yay!).

One of the first initiatives I took was look at spending, which was too high, rationalize a few things and get the hosting bills down. Also, Inch had four main hosting providers for historical reasons: AWS, GCP, Digital Ocean and OVH. The plan was to move everything to GCP, for simplification and compliance reasons. I'm still not done but I got out of Digital Ocean and reduced our usage of AWS by a lot.

We still had three servers left at OVH when their DCs burned last March. We lost two which served as integration gateways, and I took the opportunity to move them to GCP. Losing those servers has made me very busy in the last two months because their IP adresses were all over our customers' and partners' config, so I had to get in touch with them all to fix it, since there was sadly no way we could get back the IPs from OVH in time. That won't happen again now, I used reallocatable IPs this time. The third server will remain at OVH for the time being because it is a very special machine with very specific hardware hosting needs.

At small companies like this, operations also come with a lot of security-related responsibilities. Security is pretty important for Inch, which deals with a lot of personal data. So far, I have worked on improving our operational practices, added a team password manager, rationalized internal auth{n,z} around the GCP tools, and written some documentation. I have also upgraded several important software components and reviewed / fixed all the usual configuration (systems, TLS, CORS, CSP...). I have also started working on the Web application security itself, I have less experience in that space - especially regarding the frontend - than in systems security but I am learning and we have a few experienced Web developers on the team to help.

As usual, nothing is ever done in that space and I still have a lot of work coming up, but at least I believe we've significantly improved since I joined.

Fullstack Web development

Finally, the third leg of my role is the same as every developer at Inch: working on the product.

I know the "fullstack" wording makes some people cringe. Of course we all have our favorite part of the stack but I think its is extremely important, especially on small teams, that everyone be able to write or at least understand a feature in its entirety.

With the other two roles taking much of my time, I don't do as much feature work as the rest of the team but I still try to implement features myself regularly. I had never done frontend stuff seriously before Chilli and it was a different stack so I learn a lot about React and its ecosystem. I do my share of performance and bug fixes too. On the backend especially, I also do quite a lot of architecture and code reviews.

Other things

Besides those three main parts of my role, there are of course other things I take part in. Inch encourages its employees to participate in many cross-cutting activities.

An important one is that we all do support in rotation, which means the whole team ends up knowing the customers and identifying the parts of the product that need improvement. This also means everyone can, and does, participate in product decisions.

But that doesn't stop at product. Inch founders are very transparent about the business, our finances and future plans. Everyone is also encouraged to chime in on strategic decisions for the company.

This article is already long enough so I'll stop here. I hope that gives you an idea of what I've been doing at work since November 2019. If any of this interests you in any way, don't be shy, get in touch. :)

Pages From The Fire (kghose)

Got rid of more social media May 01, 2021 11:15 AM

I went off facebook a few years ago and then I went off Twitter. No one harassed me, no one said mean things to me. It was just that it was all so depressing for me. There were very few posts that I actually found useful. For me, Twitter was full of strangers announcing job… Read More Got rid of more social media

April 30, 2021

Yann Esposito (yogsototh)

Static Blog Builder April 30, 2021 10:00 PM

As someone on the Internet said not so far ago. Building its own static building system is a rite of passage for many developers. It has a lot of nice features. It gives a goal with a feeling of accomplishment. It is simple enough so most developers could build their own system. But it could also become very complex when you go down the rabbit hole.

Along the years I used different tools and used and wrote of few static website systems:

So if you look at the progression, I first used nanoc because I used ruby and it was a very new solution, the website looked really great. Also the main developer Denis Defreyne was really helpful. Ruby was really great at dealing with regular expressions for hacking my documents.

Then I was interested in Haskell, and I switched to a Haskell-made solution. I used hakyll, and I wrote a bit about it in Hakyll Setup. As a side note, the author of Hakyll Jasper Van der Jeugt is apparently a friend of the author of nanoc. They both wrote a static site generators with their preferred programming language. I added a lot of personal features to my own site builder. It was a nice toy project.

Then, due to a major disruption in my professional and private life I stopped to take care of my website.

And a few years ago, I wanted to start a new website from scratch. In the meantime I switched my editor of choice from vim to Emacs. I started to work in Clojure and emacs is generally a natural choice because you can configure it with LISP. I discovered org-mode (I don't think the homepage of org mode makes justice to how incredible it is). So org-mode comes with an export system. Thus I switched to org-publish. Again I wrote a bit about it.

It was nice, but very slow. I improved a few things like writing a short script to Generate RSS from a tree of html files. But I still had the feeling it was too slow.

Static site building is a specific usage of a build system. And as I knew I could use pandoc to build HTML out of org-mode files and still versed in the Haskell culture I decided to try shake. You can learn more by reading this excellent paper about it, I think all developer should read it: Build System à la carte.

As a bonus, pandoc is written in Haskell. I could then directly use the pandoc library in my build program. It worked like a charm and it was very fast as compared to other solutions I tried. So really let me tell you shake is a great build system.

But it was not perfect. While it was very fast, and I was able to use pandoc API directly. It made me dependent on Haskell. The best way I found to have Haskell reproducible build environment is to use nix. This was great until the Big Sur update. To keep it short, nix stopped working on my computers after I upgraded my to Big Sur. Gosh, it was painful to fix.

Concurrently I discovered gemini and wanted to duplicate my website into gemini sphere. So I tried to update my build system but my code was to oriented to use pandoc and it was painful to have gemini in the middle of it. Particularly, generating a gemini index file. My main goal was to have gemini file that could only be linked from withing gemini sphere. Because gemini is a lot smaller web where you could feel a bit more protected from what the Web has become along the years. Whatever, in the end, I just had two problems to tackles.

  1. Haskell became difficult to trust as very stable tool. Stable in the sense that I would not have any support work to do in order to keep just using it and not fixing/tweaking it.
  2. Simplify the overall system to have a simpler build description

So a very stable tool that I am pretty sure will still work almost exactly as today in 10 years is make (more precisely gnumake). I expected a lot of people had already come to the same conclusion and wrote about it. To my great surprise, I found very few article about generating static website with make. I only found solutions a bit too specific for my need. This is why I would like to give you a more generic starting point solution.

The Makefile

Instead of copy/pasting my current Makefile entirely let me give you a more generic one. It should be a great start.

The first part will be used to simply copy the files from src/ to _site/.

all: website

# directory containing my org files as well as my assets files
SRC_DIR ?= src
# directory where I will but the files for my website (HTML + assets)
DST_DIR ?= _site

# list all files in src
# if you want to exclude .org files use the exclude from the find command
SRC_RAW_FILES := $(shell find $(SRC_DIR) -type f)
# generate all file that should be copied in the site
# For my site, I want to publish my source files along the HTML files
DST_RAW_FILES   := $(patsubst $(SRC_DIR)/%,$(DST_DIR)/%,$(SRC_RAW_FILES))
ALL             += $(DST_RAW_FILES)

# COPY EVERYTHING (.org file included)
$(DST_DIR)/% : $(SRC_DIR)/%
    mkdir -p "$(dir $@)"
    cp "$<" "$@"

This part is about running the pandoc command for all org files in src/ so they generate a html file in _site/.

# ORG -> HTML, If you prefer markdown replace .org by .md
EXT := .org
# all source file we'll pass to pandoc
SRC_PANDOC_FILES ?= $(shell find $(SRC_DIR) -type f -name "*$(EXT)")
# all destination files we expect (replace the extension by .html)
DST_PANDOC_FILES ?= $(subst $(EXT),.html, \
                        $(subst $(SRC_DIR),$(DST_DIR), \
ALL              += $(DST_PANDOC_FILES)

# use a template (you should use one)
TEMPLATE ?= templates/post.html
# URL of the CSS put yours
CSS = /css/y.css
# The pandoc command to run to generate an html out of a source file
PANDOC := pandoc \
            -c $(CSS) \
            --template=$(TEMPLATE) \
            --from org \
            --to html5 \

# Generate all html if the org file change or the template change
$(DST_DIR)/%.html: $(SRC_DIR)/ $(TEMPLATE)
    mkdir -p $(dir $@)
    $(PANDOC) $< \
        --output $@

A missing part is often the part where you would like to generate an index page to list the latest posts. Here you are a bit alone, you need to make one yourself. There is not generic way to do this one.

# Generating an index page is not difficult but not trivial either
HTML_INDEX := $(DST_DIR)/index.html
MKINDEX := engine/
    mkdir -p $(DST_DIR)

Finally, a few useful make commands. make clean and make deploy.

# make deploy will deploy the files to my website write your own script
deploy: $(ALL)

website: $(ALL)

.PHONY: clean

    -rm -rf $(DST_DIR)/*

Limitation: make is old. So it really does not support spaces in filenames. Take care of that.

But let me tell you. While this is quite a minimalist approach (<100 lines) it is nevertheless very fast. It will only generate the minimal amount of work to generate your website. I have a nice watcher script that update the website every time I save a file. It is almost instantaneous.

The only risky dependencies for my website now is pandoc. Perhaps, they will change how they generate an HTML from the same org file in the future. I still use nix to pin my pandoc version. But the static site builder itself is very simple, very stable and still very efficient.

As a conclusion, if you want to write your own static site builder that's great. There are plenty of things to learn along the way. Still if you want something stable for a long time, with a minimal amount of dependencies, I think this Makefile is really a great start.

April 28, 2021

Sevan Janiyan (sevan)

Served a Gemini page to myself April 28, 2021 10:17 PM

I’ve been looking at Gemini recently, having tried out various clients such as Lagrange on macOS and Bullox on the shell, I thought I’d try serving. There are many projects for servers written in different languages, I picked The Unsinkable Molly Brown from the list for the technical reason of great name 🙂

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — Conclusion & Bibliography April 28, 2021 09:00 PM

The circle and the point: the circle is the symbol of eternity. The point is the symbol of the concentration of time in the moment

  • Internet: Medium For Communication, Medium For Narrative Control
  • Conclusion & Bibliography
Table Of Content

Our mini internet study has come to an end.
In this series we’ve seen the new artifacts and spaces introduced by the internet, the actors using them, from the new economies, to netizens in between, to state actors. We also reflected and tried to understand why we can be susceptible to biases and why we have so much difficulties with online interactions. Next we’ve looked at the big picture by diving into subjects such as paralysis, neoliberalism, the truth and trust crises, and a future glance at mass hypnosis and psychosis. Finally, in the last part of the series, we’ve seen four type of solutions: market and economy or laissez-faire, legal path with governments being involved for transparency and accountability, technical software solutions, and web literacy as education and maturity to learn to live in the information society — a post-modern or meta-modern society.

I hope you’ve learned as much as I did during this series. We’ve broadly covered quite a lot of topics, from sociology, psychology, computer science, art, memetics, history, warfare studies, politics, and more.

Let me know if it has been as helpful as it has been for me.

This series can also be found in PDF booklet format.


Table Of Content

Attributions: Philotheus, Symbola Christiana, Frankfurt, 1677

April 27, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — Adapting: Education, Literacy, and Reality April 27, 2021 09:00 PM

Perpetual motion. In order to understand it, one must think of it in motion

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 5 — Adapting
  • Section 4 — Education, Literacy, and Reality
Table Of Content
  • Web and internet literacy
  • Biases and Human Nature
  • Our Online Social Lives
  • Disconnecting to Better Reconnect
  • Learning to Live in The Information Era

We can attempt to patch things, use tools and software as countermeasures, to add laws and regulations, or to let the market decide, but in the end we are at the center of everything. If we are facing difficulties with the medium it’s because we haven’t matured enough to handle it properly. Right now we’re still in the process of trying to grasp how it works, in an apprenticeship stage.

Knowing our tools, how to use the internet medium properly, is a foundational skill, as important as any other in our information society. This goes under the name of digital, internet, or web literacy: the ability to read, write, and participate properly on the medium, in its full extent. It’s an application of the broader information literacy.

Literacy is important because it shapes societies. The degree of literacy of people affects directly what they’re able to imagine doing and actually do on a medium. By providing these skills to more people, teaching them how to use the internet, it would make the access to participation more equitable. It would ensure that there would be more representation of diverse ideas, geographies, languages, and cultures. Consequently, it would reduce the digital divide and increase the cognitive diversity of the netizens.

At the moment, as we’ve seen before, some topics are reserved to the elites. For example, we’ve dived into privacy as a luxury in one of the previous sections. Bringing internet-literacy to everyone, as a human right, would bridge the class gaps.
In 2016 the UN started considering the right to connect as a human right and some countries have adopted it in their laws. However, education on how to use the medium is as important but often dismissed.

Different organizations have thought this through and came up with pedagogical curriculums for the digital literacy core skills needed.
These are skills that are part of information literacy in general but applied to the internet medium. They fall in these categories:

  • Seeking information
  • Verifying information
  • Using information
  • Collaborating and participating
  • Ethics
  • Security and protection

The Mozilla foundation has created a web literacy curriculum that sets the base for the creation of courses that educators can give. They have three broad sections: write, read, and participate. These are subdivided into particular digital skills such as: designing, coding, composing, revising, remixing, connecting, protecting, open practices, contributing, sharing, evaluating, synthesizing, navigating, and searching.

Instead of jumping without knowing — letting users discover everything by themselves — this curriculum could be taught in schools, or as part of trainings easily available to everyone willing to learn.
Such digital citizenship curriculum would help everyone navigate the internet effectively, communicate on it using a variety of methods and tools to a range of different audiences, and have the adequate critical thinking to be able to evaluate information and arguments, identify patterns and connections, and construct meaningful knowledge that can be applied online and in the real world.

Additionally, this curriculum could have an emphasis on understanding the dynamics of the environment, namely: the actors, economies, and algorithms we’ve seen through this series.
Awareness is key to empower us against the influential algorithms, the curation engines of the internet, so that we can use them to our advantage.

Inherently part of this dynamic are the last point above of security and protection, which rhyme with anonymity, lack of trust, and the truth crisis we’ve seen before.

Security also goes along with privacy, implying tackling social cooling problems. Teaching people about cyberbullying, digital footprints, e-safety, and cyber hygiene as soon as possible is a must to protect ourselves online. As we said, people can, after getting a basic overview of the topics, follow recommendations and news from the Electronic Frontier Foundation (EFF) and other non-profit organizations.

One of the most important aspect of information literacy on the internet is making sure of the veracity of what we are consuming. Some organizations and institutions related to libraries, scholars, and media literacy as a whole have put forward good practices that can be taught to everyone to critically assess and navigate the internet space.
The IFLA (International Federation of Library Associations and Institutions) has redacted a document on how to spot fake news and published it initially on, to later translate it into 45 languages for accessibility.
Similarly, the Center For Media Literacy has also put forward key questions to ask every time we encounter dubious messages.

A critical part of fact checking is to know our own biases, to understand our own vulnerabilities.
Being aware of the existence and inner workings of the techniques that can be used to abuse our biases is the best defence against them. It provides the basis to quickly notice when we’re being tricked by messages and take some distance. Be it propaganda or marketing schemes, an intimate knowledge of our automatic impulses, training to be primed to know the coercive nature of these messages, make us proactive instead of reactive.

Outrage and shock are two emotional reactions that are employed to get us caught in the tornado of rampage. Noticing how anger doesn’t make our opinion valid, nor more righteous, is essential. Our emotions and opinions shouldn’t always mix.

Many of these biases are temporal and only work for a short period of time, this is why they often seek immediacy, scarcity, and speed. Slowing down media sharing and consumption practices is a good method to avoid falling for them.
The slow thinking way of processing information, compared with fast thinking, would also counter the speed property of the internet medium.

The author Hans Rossling has written about the declinism mindset, how the media change our perceptions, and how to fight these biases in his book Factfulness. The book takes three sides: realization that we do not see the world as is, awareness and identification of our instincts, and how to fight and control them by changing the way we see things.
Documents and teaching materials for educators can be found on the gapminder’s website.

Biases and instincts are short-term, on the long-term we should consider habits related to our surroundings, how we participate online and create communities. Just like it’s essential to learn the dynamics of the environment, it’s also essential to understand and learn how social interactions work on the internet.

This includes grasping what social biases we are prone to — the influence of our virtual surroundings — a good calibration of what this surrounding consists of, and how they are linked.

To calibrate, we can deliberately determine and examine the gatekeeping level of the community we are part of, or the ones we intend on joining or creating. This can be used to bring back a certain quality and trust.

The calibration can come in the form of segregation or separation of our online interactions and profiles. Instead of partaking on platforms that force us to integrate all of our lives, we have to select the ones that help us distinguish between the multiple facets and activities we do.
In the real world we go to different places with different people to do different things, we can do the same virtually. This would help mitigate the phenomena of private life becoming public life, social cooling, and overall paralysis.

The influence of our digital surrounding comes from various sides, all of them wanting to put us in a box, as a well-defined templatized individual. This could be because of marketing, the recommendation and curation engines, or simply the influencers and people we keep up-to-date with.

As with most of what we’ve mentioned so far, knowledge is the best way to avoid being manipulated to fit a narrative. We could train ourselves in metamemetics thinking to be able to recognize the content of the memeplexes, take some distance to criticise them. This would include stopping seeing memes as infotainment and to see them in their real light.
Above all, we need to learn to separate our definition of self from what we consume and share.

We could take this further and distance ourselves completely, to disconnect as a way of reconnecting.

Many are seeking interactions on the internet as an escapism from the difficulties of their lives. To feel safe between the walls of filter bubbles.
Some of these malaises emerge from the cultural insecurities and lacuna we’ve dived into in other sections.

Yet, some studies have found that these insecurities are reduced, along with fewer biases and reaction to outrage, and better communication, when we pull the plug for a while to do offline activities.

In particular, the most effective studies have looked at performing offline pro-social activities such as helping NGOs. Keep in mind that this should be sustained long enough so that it isn’t linked to any online social justice, mob-justice, internet karma points, or the virtue signalling we discussed earlier.

On one side, it works because we get far away from outrage and the negativity that is prominent online. It brings a more positive balance to what we feel instead of being consumed by our screens and their constantly urgent and pressing issues that we sense we are forced to react to.

On another side, it works because it takes us back to raw reality, with its sense of community away from neoliberalism and individualism. Maybe it shows how real human connection cannot be replicated online, but only the chemical reactions.
Nearly everything distinctive about human nature can be seen as a form of cooperation, and taking part in offline activities brings back our humanity.

Ironically, this is the same mechanism that is used to deprogram people from cults or addictive habits. To create a support group around them, with real connection, bonding, support, and orientation.

It could also be because taking time offline and doing non-profit work changes our neoliberal views, if our societies are inundated with them. We can think again of society as an organism which we are part of, with its flaws, and accepting each member with their minute differences.

This last point is probably what triggers our issues on the internet, the malaise and cultural gaps we talked about but that are not addressed.
Learning to live in the information era and the societies shaped around it needs the acceptance of multi-culturality and ambiguity. It requires a post-modern mindset, or even a meta-modern mindset, while most are equipped with a modern one.

For that, we have to make peace with post-modernism and what it implies, without reverting, because of our insecurities, to a cozy pre-modern mindset of tribalism. Some call this transition meta-modernism, or neo-modernism and post-postmodernity.
This means mitigating absolutist thinking, mitigating our inner urge to grasp for reassurances that would reinforce our beliefs when we find ourselves in the absence of certainty. The post-modern world doesn’t owe us certainty, on the contrary, it disintegrates absolutisms, annihilate dichotomy thinking (black-and-white mindset), removes categorical imperatives, and all the expectations on oneself and others. For post-modernity and meta-modernity, these are obstacle to our growth, they are our immunity to change and it makes us uncomfortable when we have to let go.
Absolutist thinking can significantly contribute to disorders of mood and affect, which can, in turn, negatively impact our quality of life because the world itself isn’t absolute and doesn’t bend to our preconceived notions. They are considered cognitive distortions in modern psychology since most of reality and everyday life takes place in the gray area, in between extremes.

This also applies to the trend of over-rationality and over-objectivism, where everything has metrics and is calculated. This over-emphasis on clear definitions and delineations, with no in-betweens, is another kind of absolutist thinking.

Navigating risk and uncertainty is a never ending endeavor. No amount of reassurance will ever quell all of the anxieties we have in our lifetimes.

This transition from modernity isn’t new, it’s been happening for almost a hundred years now and is only accelerated by the new attributes the internet brings.
This is clearly seen through some Western and Eastern art movements.

Modernism was about individualism, the conquest of the single person against the world, independence, personal ownership, but also a general malaise about our place in the world, nihilism, existentialism, fordism, seeing humans as tools, consumption, the psychoanalysts such as Freud, and the marketers such as Bernays.
In this transition, initially out of the absurdity of consumption, sprung art movements such as Dada. It had objects at its center and rejected all logic and reasons, the non-sense of all the machineries.

This extended into movements such as surrealism, shinkankakuha, and abstract expressionism that now had in the center the internal human experience and its reflection on the world. It portrayed a space between dreams and reality, using psychic automatism as a tool to extract thoughts, an inner discovery of our own senses and connection with our environment. The Surrealists and the shinkankakuha movement were fascinated by dreams, desire, magic, sexuality, and the revolutionary power of artworks to transform how we understand the world.

All of these movements were prequels to today’s post-modernism and meta-modernism. Our malaise was put unto art that expressed our infinite smallness, squeezing out our unconscious thought for us to admire and discover new visions, always reinterpreting, always breaking assumptions.
They express the tension we have between structure and non-structure, in any domain, even vis-à-vis language with topics such semiotics.

Presently, we have neural networks dreaming, algorithms that go into inceptions called deep dreams. We are the spectators of this space, the guides which gets back the reflection of their own dreams through a machine.
Yet, dreams rely on associations, even if unthinkable. They learn from these templatized, memeplexes, absolutist thinking.
Again, we should learn to distance our definition of self from what we consume, or offered to consume.

To tackle this we have to fix our cultural gaps, to get our act together. Do everything we can to increase our cognitive diversity, to have more global voices, to look for bridge figures, to search for in-betweens be it geographical, ideological, cultural, or others.

We can build internet realities that take into consideration our complexity, our diversity, our various and myriads of ways of expression who and what we are, and how we want to live. To accept the multiple facets of human ingenuity and creativity.
We can deliberately dabble with this way of thinking, this “meaningness”. The acceptance of nebulosity and fluidity, to learn to find meaning in uncertainty, to learn inter-sectional, cross-cultural connectivity. We can find our gaps and fix our weaknesses within this space, in others, creating a collage of humanness.

However, this poses multiples forms of challenges to move out of our comfort zones and expand our world views. Societies move the burden on the individuals and ask them to provide the structure and system for themselves instead of delegating it to some absolute authority.

Yet as the meme trickled into other online spaces, the line “we live in a society” – originally intended to be an enlightened statement which denounced the many flaws and contradictions of society – instead turned into a piece of satire.

The online world and discussions that bridge cultures make issues more apparent. These controversial topics need to be addressed openly, not through mobs, but maturely. Each on their own.

Some authors and psychologists have written endlessly on the individual’s effort to make sense of experiences, the meaning-making process, metaperception, neo-perception, and self-transformation.
From the “Mental demands of modern society” from Kegan, to Piaget subject-object relations, to Erik Erikson “ego identities”, to Abraham Maslow “self-actualization”.

Learning to live in the informational society is being deliberately open and motivated to possibilities from the intersectionality of all the experiences and ways of seeing the world.
We’re learning to communicate on a global scale while coexisting with algorithms.

This concludes our review of how we can mature to better adapt to the internet medium. We’ve started by exploring educational curriculum such as web and internet literacy, so that we can be properly prepared. We’ve said this includes reading, writing, and participating. Next, we’ve said this curriculum should also take into consideration awareness to biases, knowledge being the best way to tackle them. Later, we’ve mentioned long-term adaptation, social adaptation to the internet, which include learning to manage our connections online by segregating them properly and not being put in templatized boxes. After that, we’ve examined completely disconnecting as a way to better reconnect. We’ve seen some studies about helping NGOs and how it reduced biases and our reaction to online outrage. Finally, we’ve dived into what it means to be a netizen of the information era and societies, to live in post-modernity reaching meta-modernity. We’ve discussed some of the artistic transitions and how they reflect our malaise and continuous improvement at including ourselves in this intersectional and cross-cultural world that has never been as connected as today.

Table Of Content


Attributions: Origin of the ‘Primum Mobile’, from: Robert Fludd, Philosophia Sacra, Frankfurt, 1626

April 26, 2021

Mark J. Nelson (mjn)

Deadlines for 2021 technical games research conferences April 26, 2021 12:00 PM

This is a bit late since some of the deadlines have already passed, but nonetheless, here are the deadlines I know of for 2021 editions of the conferences I track for my lists of institutions and researchers active in technical games research. Plus two journal special issues.

Deadlines still in the future (as of April 26):

DeadlineVenuePaper type
2021 May 28CoGShort/demo/other papers
2021 May 30ToGSpecial issue on Evolutionary Computation for Games
2021 Jun 07ICECAll papers
2021 Jun 07AIIDEResearch paper abstract registration
2021 Jun 14AIIDEAll papers
2021 Jun 25ICIDSAll papers
2021 Jul 16CHI PLAYShort/demo/other papers
2021 Jul 31ToGSpecial issue on User Experience of AI in Games
2021 Sep 06ACGResearch papers

Deadlines that have passed:

DeadlineVenuePaper type
2020 Dec 22I3DResearch papers
2021 Jan 25 Feb 01FDGResearch papers
2021 Feb 17CHI PLAYResearch papers
2021 Mar 16 Mar 30I3DPoster papers
2021 Apr 12 Apr 19CoGResearch papers
2021 Apr 12 Apr 20FDGShort/demo/other papers

I might make this a regularly maintained feature in the future, but this is a one-off list for now. Partly so I can stop having to repeatedly search for the deadlines.

There are some other deadlines omitted from the tables above, like workshops, tutorial proposals, doctoral consortia, etc. Click through for the complete CfPs and deadlines.

Ponylang (SeanTAllen)

Last Week in Pony - April 25, 2021 April 26, 2021 12:41 AM

Audio from the April 20, 2021 Pony development sync call is available.

April 25, 2021

Derek Jones (derek-jones)

Software engineering experiments: sell the idea, not the results April 25, 2021 09:31 PM

A new paper investigates “… the feasibility of stealthily introducing vulnerabilities in OSS via hypocrite commits (i.e., seemingly beneficial commits that in fact introduce other critical issues).” Their chosen Open source project was the Linux kernel, and they submitted three patches to the kernel review process.

This interesting idea blew up in their faces, when the kernel developers deduced that they were being experimented on (they obviously don’t have a friend on the inside). The authors have come out dodging and weaving.

What can be learned by reading the paper?

Firstly, three ‘hypocrite commits’ is not enough submissions to do any meaningful statistical analysis. I suspect it’s a convenience sample, a common occurrence in software engineering research. The authors sell three as a proof-of-concept.

How many of the submitted patches passed the kernel review process?

The paper does not say. The first eight pages provide an introduction to the Open source development model, the threat model for introducing vulnerabilities, and the characteristics of vulnerabilities that have been introduced (presumably by accident). This is followed by 2.5 pages of background and setup of the experiment (labelled as a proof-of-concept).

The paper then switches (section VII) to discussing a different, but related, topic: the lifetime of (unintended) vulnerabilities in patches that had been accepted (which I think should have been the topic of the paper. This interesting discussion is 1.5 pages; also see The life and death of statically detected vulnerabilities: An empirical study, covered in figure 6.9 in my book.

The last two pages discuss mitigation, related work, and conclusion (“…a proof-of-concept to safely demonstrate the practicality of hypocrite commits, and measured and quantified the risks.”; three submissions is not hard to measure and quantify, but the results are not to be found in the paper).

Having the paper provide the results (i.e., all three commits spotted, and a very negative response by those being experimented on) would have increased the chances of negative reviewer comments.

Over the past few years I have started noticing this kind of structure in software engineering papers, i.e., extended discussion of an interesting idea, setup of experiment, and cursory or no discussion of results. Many researchers are willing to spend lots of time discussing their ideas, but are unwilling to invest much time in the practicalities of testing them. Some reviewers (who decide whether a paper is accepted to publication) don’t see anything wrong with this approach, e.g., they accept these kinds of papers.

Software engineering research remains a culture of interesting ideas, with evidence being an optional add-on.

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — Adapting: Technical Solutions, Wars and Patches April 25, 2021 09:00 PM

I have robbed the golden vessels of the Egyptians to build a holiness for my God, far from the borers of Egypt

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 5 — Adapting
  • Section 3 — Technical Solutions, Wars and Patches
Table Of Content
  • Detection and Remedial, War of Algorithms
  • Changing the Way the Platform Work, Communication and Filters
  • Differential Privacy and Privacy First Internet
  • Defensive Tools
  • Attention Management
  • Decentralization, Transparency, and Free Software

When free market and regulations fail, when the laws of rights can’t properly protect anyone and trust has eroded, we’re left only with ourselves. In that scenario, tech is seen as the savior of the internet, the weapon and armor of choice for everyone, building and selecting software that resolve issues.

For social media platforms and other big entities such as governments, algorithms can be used for detection, categorization, and remedial. The first step of an issue is to know: without knowledge it’s hard to defend.

Platforms could employ algorithms that would automatically flag posts containing hate-speech or other sensitive content, and either remove them or let a person take further actions from there. There could also be automatic detection systems that would find bot accounts, satiric content (including deep fake), and state-linked accounts, then label them as such. As we said earlier, if there are laws to tag such accounts, it makes their activities transparent and reduces the chances of black propaganda.
Some even talk of contextualization engines that could bring up the context behind any piece of media posted.
Facebook has recently been experimenting with labeling bot accounts, adding an official stamp on state related and celebrity accounts, and labeling satiric and other types of posts as such. Other platforms are applying comparable labeling, at least for official accounts.

Algorithms could then be countered with algorithms. After detecting malicious intents some actors — be it the platforms to protect their own policies, or states entities — could launch offensive “pro-social” bots themselves, or other means of automatic counter-propaganda as a defense.

However, algorithmic solutions are problematic because they create an arms race. Moreover, the over-reliance on algorithms can have consequential results as they can be reflective of inner-biases. We’ve seen that before.

Another technical solution, in between internet platforms and their users, could be to change criteria that make the internet what it is: speed, exposure, and long-lasting.
There has been some arguable success with platforms that make messages ephemeral, for example. However, these are hard to control because these are innate attributes of the medium.

Rethinking the current way we communicate online and providing means to better communicate without toxicity, filter bubbles, addiction, and extremism is what a lot of people are thinking about today. It’s a question that remains unanswered.
Some have attempted to create platforms around, not only communicating, but also organizing ideas, increasing the efforts to partake in the activities, adding a positive creative gatekeeping.
Some have played with new recommendation algorithms that would increase cognitive diversity, pushing people out of filter bubbles by exposing them to different ideas, cultures, geographies, and anything unfamiliar.
Others are trying to build systems that would be able to put metrics on our biases, on our tendencies to associate with similar people, of clustering. Essentially, creating a self-monitoring to add serendipity, inclusiveness, and diversity in our own lives.
Some have tried a topic-based internet where people are randomly connected based on common interests, instead of vote metrics and shock-value.
Another way is to not put all the facets of our lives in the same place, to distinguish between different activities, hobbies, and interests, to keep them separate. This helps avoid social cooling and facades.
Others are trying to see if the micro-transactions and micro-payments we discussed in a previous section, would work to make an internet of the passion economy and drive away other incentives, leaving only pure human creativity and interests.

Technologically, internet giants are trying to win back the trust of the netizens by talking of differential privacy and privacy first tech.
Like we said before, this is similar to privacy as a product, privacy as market value. However, big techs are selling these words not only for the users but to protect themselves from the law.
Some of these companies are now disabling third-party cookies by default in their products, notifying users of tracking, providing privacy sandboxes, enabling end-to-end and other types of encryptions by default, using differential privacy or cohort-based analysis or context-based ads instead of micro-targeting.
Yet, these all make no sense without transparency and when these entities are for-profit.

Digital citizens still don’t trust them and would rather rely on defensive tools and products to feel safer online and avoid being tracked.
They use tools such as VPN, which we discussed earlier, proxies, and ad blockers. According to statistics 47% of USA internet users now utilize ad blocking software.

Additionally, many are now using attention management and informational management tools to get back the control over their attention. We’ve seen earlier how the internet can affect us long-term cognitively and how we are inundated with information.
Attention management tools are software that are used to warn people when they get inadvertently absorbed into activities or media. To be proactive instead of reactive.
Informational management tools are database systems used to organize in a clear and concise way the information we find important. They help to deliberately decide what should be in our memory extension, which we discussed when we saw the cognitive changes the internet brings.

One great thing about the internet, is that even though it’s convenient, people don’t have to use pre-existing platforms, they can be empowered to create their own.
The edification of standards that allow for decentralization and keeping platforms open are good technological ways to avoid bait-and-switch tactics, data leaks, privacy concerns, loss of access, being locked out of accounts, etc..
Avoiding monoliths is avoiding putting all eggs in one basket. Users can choose to spread their internet usage across different services and favor decentralized ones.

On top of this, if netizens have enough patience, expertise, and time, one of the best solution is to own the data and tools by hosting them, self-hosting. This is more than having backup copies of the digital assets, it’s also about regaining control, trust, and privacy.
This is what some popular YouTubers and others are doing by building their own site, to not be victims of the market and keep platforms decentralized and open.

Sometimes openness isn’t enough, we can host services ourselves but if the software is proprietary then we might still not trust it. What is needed in that case is transparency.

Transparency can be achieved in different ways, self-sovereign identity is one that we’ve seen in the previous section.
Another way is to use so-called “zero-data” applications, software that let us be in control of our data from the start, doesn’t access it, or doesn’t do any type of tracking.
Users can rely on feedbacks and recommendations from non-profit organizations that try to defend digital privacy such as the Electronic Frontier Foundation (EFF) to be up-to-date with the best practices and events in the online sphere. We’ll tackle the education part in the next section.

Yet, that can be limiting and not transparent enough. The most transparency we can have from software is when it is open source and when the licenses enforce the respect of freedom and liberty, what we call free software.

Certain non-profit organizations and projects have as mission to promote such kind of user freedom, namely the Free Software Foundation (FSF) and the GNU Project.
Technically savvy netizens could still rely on their own instinct and replace their tools and services with the open sources projects they deem deserve more trust.

Open source and free software licenses can enhance and create value for the public sector too. They can be used within the ICT framework in infrastructure and services offered by the institutions, all publicly funded developments.
Through the use of free software, the citizens and government will feel more in control over information technology. It would grant them digital independence, sovereignty, and security ensuring that citizen’s data is handled in a trustworthy manner.
The use of free formats and protocol will also influence the way development is done, increase trust and reduce distance between government software and the citizen involvement. Open source is collaborative and encourages collaboration by nature.
Having everything done in the open, open access (OA), would also reduce waste, avoiding non-replaceable software, and offering technological neutrality. This reminds us of the current trendy discussion rotating around the subject of electronic right to repair.

These could be applied to any government services, especially if they involve social media as a utility and digital identity. The possibilities are interesting.

Practically, this can be implemented at the state or institution level either through the legal system, regulations, policies, or encouragements or promotion (or non-encouragement).
Multiple nation-wide entities and bodies are transitioning to open source or free-software solutions through different measures. For example:

  • In 2002, the Peruvian government voted the adoption of open source across all its bodies to ensure the pillars of democracy were safeguarded.
  • In 2004, the Venezuelan government passed a law, decree 3390, that would also transition the public agencies to use open source software.
  • The National Resource Centre for Free and Open Source Software (NRCFOSS) in India since 2005 is promoting the use and development of free software.
  • The Malaysian Public Sector Open Source Software Program in Malaysia since 2008 is similarly discouraging the use of proprietary-software and encouraging free software.
  • In the same year, 2008, Ecuador passed a law, decree 1014, to migrate the public sector to Libre software.
  • New-Zealand, in 2010, through its open access initiative, NZGOAL, started promoting the use of free and open source licenses as a guidance for releasing publicly funded software.
  • In the same spirit, the UK Government Digital Service formed in 2011 after a proposal from 2010, included in its guideline the promotion of open source when it fits the government IT strategy.

…And countless other examples that show how government entities are promoting or using open source and free software to bring back trust and transparency. As we said earlier, this is a must during the trust and truth crises we are facing.

This concludes our review of technical and software solutions that can be used to avoid issues we’ve seen in this series such as filter bubbles, toxicity, truth and trust crises, attention and information management, and others. In a first time, we’ve seen how big entities can use algorithms to fight online content through detection and remedial, either removal or tagging/labeling of content. However, we’ve also seen that this would give rise to an arms race of bots. Then we’ve looked at ways internet platforms could change their approach to make communication less toxic and increase cognitive diversity. Next, we’ve mentioned how internet giants are patching themselves by introducing more privacy and security features in their products. After that, we’ve said netizens would still lack trust because of the lack of transparency, thus would rely on defensive privacy tools such as VPN, proxies, and ad blockers. Users could also use software tools to help them manage their attention and information. Lastly, we’ve explored finding transparency and trust by decentralizing services and the usage of free and open source software. This can also be applied at the national level to tackle the trust issue with governments.

Table Of Content


Attributions: J. Kepler, Mysterium Cosmographicum, 1660

April 23, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — Adapting: Legality, Transparency, Accountability, The Nations Intervene April 23, 2021 09:00 PM

I am the Black of the White and the Red of the White and the Yellow of the Red; I am the herald of the Truth and no liar

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 5 — Adapting
  • Section 2 — Legality, Transparency, Accountability, The Nations Intervene
Table Of Content
  • Laws and Governments as Moral Arbiters
  • Free Market Competition Is Not Enough, Privacy as a Human Right
  • Balance Between Sociality And Privacy
  • Consent, Ownership
  • Consent, Persuasive Tech
  • Consent, Right Of Non-Reception
  • Accountability, Securing Data
  • Accountability, Content + Gatekeeping
  • Transparency, Processing, Usage, Analytics, and Partnerships
  • Transparency, Disclosing Sponsors
  • Transparency, Bots
  • Example Laws
  • Drawback Of GDPR
  • Standardization Of Data
  • Privacy as a Utility, Government Monopoly on Identity
  • Social Media as a Utility
  • Digital Identity, Identity as a Public Infrastructure
  • Examples of Implementations

The market and corporate self-regulation have their limits. They cannot in themselves be sources of morals and ethics. This is the realm of laws, the legislations that governments make are the real arbiters of duties and rights.
The governments, as state entities, can impose the rules that should be followed to be able to act on their territories. However, laws are bound by geographical areas and as such cannot be international. They can only be inter-governmental if treaties and partnerships are in place. Companies can decide to comply to different regulations in different areas to be able to operate on these markets.

The best defensive tactic for nations is always legislations, instead of leaving it as a free-for-all medley. This is what we’ll dive into: when governments intervene on the internet.
The neoliberal mindset abhors this return to authorities because of the general lack of truth we’ve seen earlier. Yet, who do we trust more and want to grant more authority to: private for-profit entities on not-so-free markets or our governments and their legal systems?

In this section we’ll focus on rights and not regulations related to the private sector. Things related to transparency, accountability, consent, ownership, utility, and identity.
We’ve already seen market regulations for fair competition, conflicts of interests, collusions, antitrust, and others that are currently discussed or getting in place around the world.

Despite these new market regulations, many of the internet giants and their partners still stand tall and confident that their business model isn’t going to fall. Either because of nifty bypassing techniques, or because of lobbying. That’s why such regulations don’t coincide with what most of the netizens actually want.

For example, many recognize and want privacy to be a basic human right, part of human dignity. The “right to be let alone”.
Similarly, we’ve previously seen how researchers and human right activists are now thinking, when it comes to brain data privacy, of new rights such as: cognitive liberty, mental privacy, mental integrity, and psychological continuity.
On the surface, these will only be fancy words of wisdom and considerations if they aren’t written in laws, legally binding. Otherwise, companies could lie. Governments have to ensure the spirit of the laws are applied and not mere market regulations. It would be catastrophic if no such basic rights are in place and sensitive personal information such as brain data become common place on the internet.

The question today are related to who’s in charge of the privacy and what kind of balance we can have. Can our information be both a currency for some and protected as a right for others?

The actual implementation is a hard problem when it comes to privacy guidelines as there is a balance between sociality and privacy: sociality has always required the voluntary abandonment of privacy. In order to be social we must give up some of our private time and space to share it with others.
This takes new forms online where sociality is taken at the expense of privacy and social autonomy. Today, it rests mostly upon digital records and archives in the hands of private companies that track our social behavior. As we’ll see, it could be more attractive to have these records switch hands from the corporate world to the public sector.

The legal world is a tricky one, the laws need to be written so that they cannot be abused, covering as many aspects of an issue as possible to not leave room for ambiguity.

The first clarification that needs to be made is about defining what is meant by personal data and who is the actual owner of such data. We already gave our definition in the data and personal data section in part 1 of this series.

Personal data is any information that can be used to directly or indirectly identify a person, an attribute currently attached to them, a behavior, a preference or interest, personal history, anything.
These include things such as name, phone number, address, email, schoolbook, credit scores, geolocation, travel logs, interests, information about past purchases, health records, insurance records, online behavior, etc..
We call the person to which the personal data belongs a data subject.

Having this definition in a text law is the starting point.
Then, comes the issue of ownership of the data. As you might remember, we called the person which data was extracted from the data subjects. Hence, we need to specify what are the rights of ownership they have over their data, and in which cases they need to give consent to allow others to access it.

The consent itself needs to be delineated properly. We need to know what can be done with the data, for what purpose, for how long the retention can be, and most importantly, if consent is transferable. One company getting approval to access data could share this data with other entities, the transitive consent and transitive propagation of data.

Ownership also rhymes with control. Laws might want to spell out if a data subject should or should not be able to perform corrections, updates, or erasures of the data they have generated.

Yet, consent makes no sense if people can be coerced into giving it, if it isn’t informed consent. This is the case online with the use of persuasive technology, both to extract data and be subject to advertisements. They go hand in hand. Thus, the legal domain has to take this into consideration and posit whether such technology is allowed or not.
Additionally, it could make the parallel with gambling, alcohol, cigarettes, or other questionable products, basically limiting advertisements on mainstream channels.
Furthermore, when it comes to consent, the law has to deliberate if persuasive algorithms and designs should be allowed on children or not — age is often a factor when it comes to consent. Persuasive design and dark patterns could be made illegal altogether.

When it comes to advertisements, it has to be decided if consent needs to be given even before receiving them, or whether the person can at a later time refuse to receive them, retracting it.
The right of non-reception, is the right of someone not wanting to be the recipient of ads. If such right is in place, companies could be pursued in court and fined if they advertise to people who have opted-out of them, or to those who haven’t opted-in depending on how the law is phrased.
Such right is generally associated with online cookies, the right to not be traced online.
Offline, it is associated with not receiving ads in mailboxes, or at city-scale, to not have or limit billboards. This is also a method to avoid material waste.

Still, these can easily be bypassed with an ingenuous use of product placement and sponsoring instead of direct ads. Or even turning customers into brand evangelists, which is the norm with neoliberalism as we’ve seen earlier.

Once the personal data is stored, it could and is expected to be subject to access restriction and accountability. The companies and entities we have trusted with our data should be responsible for their safety so that nobody else retrieves them.
Thus, governments could create bodies and processes to verify the security, integrity, and confidentiality of the data storage in companies and entities that choose to host it. With the increase in quantity and types of consumer data, it is imperative to have such measures in place. We’ve heard too many data leaks stories in the news lately, either from hacking or rogue employees.
Arguably, even standards like the PCI-DSS, if applied to personal data, might not be enough without real-time checks.

Along with the responsibility of storing the data, laws could dive into the accountability related to the content of the data itself. Should platforms be accountable for the content it hosts, or should it be the data subjects themselves, or both?
This would in turn create new gatekeepers: a monitoring and reviewing of the content published and hosted. This could also include the algorithms that are used to promote content.

These algorithms have acted as amplifiers of extremism, enlarging dangerous fringe communities, and pushing biases (racial or others). Currently, the entities that are using them have been hiding behind the fact that these are algorithms, and thus “mathwashed” removing their responsibility because “they have no conscience of their own”. The myth of the impartial machine.

Many of the social network platforms, in the USA, have been avoiding monitoring the content posted on them by referring to the USA first amendment of free speech and the 1996 USA Communications Decency Act, shielding them from liability for content posted by their users.
However, this only applies to the USA and the different branches of the same platforms are able to monitor content in other countries.

To thwart this, governments need to either consider social media as news platforms, or find methods to strengthen the institutions that create and distribute reliable information: mainstream media, academia, nonpartisan government agencies, etc..
Once social media are part of the mainstream news system, the gatekeepers are back in place, they’ll have to uphold journalism standards for the content posted on them.
Governments can also enforce social networks to pay news that get published and distributed there, indirectly re-strengthening the local media publishers.

Apart from news, the platforms can be held liable for the undesirable content that is shared there.
This is most important when this content is shared in a public space that children can access. Laws can ensure children won’t be subject to neither surveillance, tracking, and be protected from predatory and dangerous content.
Contrary to popular belief, in the USA the COPPA (Children’s Online Privacy Protection Act) only takes the advertisement and tracking of children, but not whether the platforms are responsible for the content to which they are subject to. The Child Online Protection Act of 1998 (COPA, yes similar name) is the one that has the intent of preventing minors from accessing obscene material on commercial websites in the USA.
Most countries already protect their children against being subject to harmful material on public commercial channels.
India’s new intermediary liability and digital media regulations takes a step further and forces content to be traceable to its source, to be able to verify social media users, and rapidly take down non-consensual sexually explicit content and morphed/impersonated content within 24h of being notified.
With the same mindset, some countries have laws specifically to take down online hate speech and incitations of violence or terrorism.

If we can’t blame the hosts of the content, we might want to hold the people that generated it accountable. This, in turn, should make black propaganda ineffective. However, because most online-interactions are anonymous, it is very hard to do.
A solution to this would be to attach a real identity to whoever uses, posts, and creates online content, this is what the concept of digital identity is about.
Online platforms that allow bots could also be forced to tag them as such, making it clear to everyone that these aren’t real humans.
We’ll discuss this later, but as you can expect, while it does bring back gatekeepers, it also could create more social cooling.

Agreeing that the personal data should only be used for specific purposes is fine, but there needs to be some guarantee behind the agreement. For trust to set in, transparency is mandatory.

One type of transparency is about understanding why and how the recommendation engines, the curation systems of the internet, work. From advertisements, to news feeds and other algorithms. Having a clear picture of their inner-workings would help us make better decisions and responsibilities instead of overly relying and trusting them.

With the recommendation engines come the analytics, the categorizing, ranking, rating, and scoring of data: how data is processed internally. Transparency is about knowing where and how our data will be used, in what process.
It could be hard for companies to accept transparency at this level because that would mean opening their valuable extracted data storage to the world.

The transitive propagation of data needs also to be transparent if it happens. That includes disclosing who are the third parties that will access the data, for which reasons, what data points they used, the target audience, and who paid for the ads campaigns.
This type of financial accountability, full transparency regarding the amount spent by companies on ads, also applies to political campaigns sponsorship. Ad political campaigns, which on the internet were often overlooked, would now have their layer of anonymity removed. Yet again removing black propaganda and state-linked accounts from the equation and making microtargeting less obvious.

Many legislatures, regions, states, governments, authorities, and countries have erected or proposed laws to tackle the things we’ve seen, let’s mention some of them.

Related to the financial accountability, the Honest Ads Act, is a bill in the USA that was proposed such that online services would be required to reveal the description of the target audience of ads campaigns.
However, this bill wasn’t passed yet in favor of self-regulation. The USA, in general, is a laggard on the domain of privacy as it feels more threatened by the intervention of the state than the market.
Still, in 2018, the California Consumer Privacy Protection, a legislature passed in California (a USA state) would guarantee users in that region the right to know what data is being collected, and opting out of the sale of their data.
This opt-out approach isn’t the best but it’s a start.

Others might find that there needs to be a stronger data protection, that it’s part of human dignity and that it shouldn’t be threatened by any kind of intrusions. In that view, privacy and ownership over our private data is a default and not an opt-out.

The European Union was the first in 2016 to spear-head this change by rolling out the largest attempt at protecting user data with the General Data Protection Regulation.
Privacy laws are not new, for example France’s data protection law dates back to 1978, and the EU already had Data Protection Directive law about the protection of fundamental rights and freedoms in the processing of personal data, dating from 1995. However, the GDPR is an extension of all this with modernization, clarification, and making it actionable. It is a move that has spurred other states to enforce similar laws. It also gives them more confidence in intervening against the internet behemoths, which they wouldn’t dare attack before.

The French data protection law ensures the following rights:

Personal data must be collected and processed fairly and lawfully for specified, explicit, and legitimate purposes, and with the consent of the data subject. In addition to the right to consent, data subjects have been given the following rights: right to be informed, right to object, right of access, right to correct and delete information, and right to be forgotten.

These have been mapped unto GDPR to hold data processors accountable when manipulating data that isn’t mandatory for the functioning of the service. It defines clearly what is meant by personal data, what it means to process it, and what is required of organizations and entities that process it.
The organizations are expected to have a management procedure in place to keep the data safe and secure from breach. That means specific roles need to be in place, such as a Data Protection Officer, and the risk management assured. They are held accountable for the privacy and protection of the data, and data protection assessment can be performed to ensure this is properly applied.
The data subjects keep their rights over their data. They should be informed about the processing activities taking place, the data not being used for any non-legitimate purpose. Their explicit consent needs to be requested for its usage and collection. That means the data subjects have the right to update/correct and erase their data. Additionally, the data subject can ask to see which data is held about them and control whether its transfer can happen between third parties or not.

Most importantly, regulators can ask for demonstrations of accountability and impose fines for the entities that aren’t complying. This is the clause that makes the difference with any previous laws.

After it was passed, the internet has been shaken. Most websites have chosen to display annoying pop-ups/notification banners asking if users want to allow their information to be shared with third parties. It defaults to no but it is sometimes hard to manipulate and very intrusive.
Moreover, some companies are also allowing their users residing in the EU to download the personal data that has been gathered, and give them the possibility to delist it, essentially erasing it.

The way the GDPR has been applied by companies has given rise to more dark patterns, persuasive technologies trying to trick users into accepting to give their personal info.
Research has shown that only around 11.8% of consent forms in 2019 met the minimal requirements based on the European law.

Multiple countries have their own privacy laws and acts, and many today are modernizing and reforming them to add actionability like GDPR.
Some even include the right to data portability and the right to object to marketing, the right of non-reception which we mentioned earlier.

Let’s take some examples.
Canada has the Privacy Act from 1983, which it is modernizing to be similar to the GDPR in the Personal Information Protection and Electronic Data Act. China’s Personal Information Security Specification that took effect in 2018, and ratified in 2020 in the Personal Information Protection Law, has more strenuous and far-reaching requirements than GDPR applying to any personal data and not only sensitive personal data. The UK has its UK Data Protection Act from 1998, which has been updated in 2018 to follow and supplement the GDPR, and is being updated today to follow its own standards. The Irish Data Protection Commissioner (DPC) upholds the same fundamental privacy standards as the EU. Similarly, Australia has updated its 1988 Privacy Act and Australian Privacy Principles to be like the GDPR. Russia Federal Law on Personal Data of 2006 regulates data processors to protect the defined sensitive personal data against unlawful and accidental access. India’s PDP Bill of 2019 goes in the same direction as GDPR but is more integrated with Aadhaar portability which we’ll see in a bit. Argentina is going the same way, currently amending their laws to be like the GDPR. etc..

It’s apparent that most of the world is moving in the direction of edifying laws to protect the privacy of the citizen from the private corporate data brokers. Market regulations aren’t seen as enough.

Some nations take it a step further, considering the protection, privacy, standardization, digitalization, and usage of identity to be the role of the state and not corporations. The state should have a monopoly on identity, just like it should, in theory, have a monopoly on violence because it has the same destructive power.
As we said, sociality requires giving up a bit of privacy, and the digitalization of identity is a requirement for the information society. Some people are uncomfortable with the idea of making it the job of the state and would rather have centralized private entities do that. Yet we know that corporate incentives are only for-profit and not the shared good, which could hinder the transformation of the society into this new era.

Thus, some governments force the private sector to strictly follow their privacy rules, while the public sector uses identity as a utility, a part of public infrastructure for the public services.
This requires standardization, definition, transparency, and data portability of digital identity. We’ll come back to this in a bit but first let’s imagine social media as one of these services.

We already discussed social media as utilities in the social media section of part 1. We said back then that for it to be a utility, it should change from a want to a need, an essential infrastructure, a necessity like electricity.
Social media could be said to be an integral part of information society, in order to adequately take part in the 21st century as an individual, and thus would be important to have the government provide it to remove any profit incentives from it.

Some believe that current popular social media already act, feel, and are considered by people like utilities. So they push forward the idea of making them as such, just like the telecom sector or electricity sector: a public service that is regulated by the government.
It would make it easier to ensure the protection of the constitutional rights of users, such as freedom of speech in some countries. Additionally, the government could enforce search neutrality, modeled after net neutrality regulations — essentially assure equal access for everyone.

Yet, others argue that making social media similar to telecom regulations would be bad for the market, reduce innovation and growth because of the lack of competition. This is a market view of the subject.
Moreover, it can be said that social media haven’t gained the status of utility because countries don’t go haywire when they go down, that there is always an alternative, they are still new and replaced every couple of years.

Furthermore, from the neoliberal perspective, one that dreads authorities and prefers seeing things as a free flowing market, not catching nuances, this is the equivalent of the chilling rise of “authoritarianism” and “fascism”. Words that are used as emphasis to display the attack they feel towards their world view.
Still, there might indeed be a new kind of social cooling taking place when we become owners of our social media, instead of having them as private companies. State actors were already requesting personal data from these companies and using them for their gain, as we’ve seen before. Making them public utilities would instead force the general public to keep it in check, as it would be something they would be indirectly paying to keep running.

Another trend is the one of the digitalization of identity by governments. This is the standardization of identification and identity systems and management that become part of the public infrastructure.
In practice, this is more of a re-definition of national identity cards by making them digital, interoperable, and accessible by all citizens. Having it as an infrastructure means that all kinds of public and private services can rely on it to verify reliably the identity of people.

This comes with a lot of challenges for governments as they have to set in place the legal framework around the infrastructure, its usage, regulations, auditability, traceability, transparency, and obviously actually create a technology that is secure, confidential, keeps the integrity, is scalable, and respectful of all individuals.
The capability of the identity system needs to be well-defined, from authentication, authorization, roles, and delegations available. Some government bodies and international standards have been created for these identity management systems such as ISO, the EU eIDAS, European Self-Sovereign Identity Framework, European Blockchain Services Infrastructure (EBSI), and the GSMA Identity programme and Mobile Connect.

Like with social media as a utility, this comes with the same perception from those who’d rather be surveilled by private companies than their government. But with enough transparency and preparation, the most digitally-skeptic can get on board, especially if the solutions are free and open source as we’ll see in the next section.

So far, the implementations of digital identities by governments have only been applied when it comes to accessing public services and KYC processes when signing up for services such as telcos, energy, and banks.
Estonia, India, and China are countries that are leading the way. The upside is convenience and an economic boost. According to McKinsey Global Institute, countries that implement a digital ID scheme can boost their gross domestic product by 3 to 13 percent by 2030.

Estonia has been rolling its digital identity program for the past 20 years. Technically, it is a mandatory 11-digit unique identifier assigned to all citizens above the age of 15 that is stored along with a public-private key pair in a PKI (Public-Key Infrastructure).
The default format of this digital ID is a physical card, an i-card, similar to a credit card, passports, and sim cards: an embedded chip (UICC) protected by PIN codes. Internally these circuit cards use secure elements (SE) as a root of trust to store the sensitive information. We generally refer to this type of technology as hardware security modules (HSM). Other formats exist such as a mobile application that inspired China’s version of it, I am not currently aware if the Chinese solution additionally requires an UICC to be inserted in the phones.

The data is stored in a decentralized blockchain that is scalable and secure. Protected against cyber threats, which they’ve been subject to in the past, and are obviously continuously facing and tackling newer security challenges. Additionally, the data is also backed up in Estonia’s e-embassy in Luxembourg.

Practically, this allows someone to be verified during a KYC process (Know Your Customer), login to private or public services, to encrypt or decrypt files securely, and to legally digitally sign documents.
Note that e-signatures are allowed in the EU, through the eSignature Directive, to replace handwritten ones.

Users of such system are thus in the center, practicing self-sovereign identity (SSI), personal autonomy, in full control of who has access to their data. The system allows for traceability and auditability. Citizens can transparently see what data is collected about them, when, why, who has accessed to them, revoke or keep this information, and more. That is unless law enforcement requests access, but they’ll still be notified once the investigation wraps up.
This can be abused but the country has shown a strong legal stance whenever law enforcement or doctors took advantage of their positions. The traceability of each access also makes it harder for someone to touch the data without anyone else noticing — unlike traditional physical cabinet storages. This enables more privacy and empowerment that when a private company has control over personal data.

The advantages are obvious: designers of systems save time by relying on the government infrastructure, and citizens too by carrying their life activities online.
This includes banking, e-residency, accessing health records, medical prescriptions, voting, paying taxes, and more. One thing they are considering integrating is their own cryptocurrency.

Estonia also teaches cyber hygiene in elementary school. Education is an important factor that we’ll dive into next.

The EU is considering implementing a similar system, but leaving each national authority the choice. However, it is still being discussed and argued if it should be mandated across all nations. They have agreed on eSignature Directive, allowing electronic signatures to be the equivalent of handwritten ones, and took a few steps with encouraging national digital IDs in 2018 through the Electronic Identification And Trust Services (eIDAS) regulation. The implementation of digital ID in Europe would allow cross-border electronic transactions and electronic signatures, but only 14 out of 27 member countries have introduced online authentication systems, such as DigiD in the Netherlands, Belgium’s eID card, and Spain’s DNIe.

Another large implementation of digital identity is India’s Aadhaar. Technically, it is a non-compulsory 12-digit unique identifier, termed Aadhaar, that also comes with a public-private key infrastructure that stores biometric information.

Due to the non-mandatory nature, the government has to ensure that no service can be denied if the person hasn’t signed up for Aadhaar. However, in practice, many services are now using it for KYC as a replacement for paper proof of identification.
Moreover, because it isn’t mandatory, Aadhaar isn’t a proof of citizenship or residence: it doesn’t replace other IDs like passports or driver’s licenses.

Aadhar practical form is as a biometric ID system, allowing citizens to prove their identity by simply providing a fingerprint or iris scan for example. There is also an ID card that can be issued.
On top of Aadhar, the government has rolled out a Unified Payment Interface (UPI) allowing banks to interoperate with Aadhaar. UPI is designed to make person-to-person (p2p) and e-commerce transactions swifter and easier.

For India, this has been a game changer, enabling access to bank accounts and services that most couldn’t access before, making them more inclusive. Figures are not robust, but it can be assured that most (4 out of 5) Indian citizens have the ID card.
This saves a lot of overhead, someone can transfer money simply by showing their Aadhaar card, or paying with their fingerprint.

Unfortunately, there are a lot of criticism, calling it a “tech-solutionism” and saying the roll-out has been discriminatory.
Besides, there have been emerging reports regarding the security concerns of the personal data associated with Aadhaar ID being sold in alternate markets.

These flaws in privacy and security need to be addressed with accountability, traceability, and a strong legal framework, similar to what Estonia is doing. So far, Aadhaar doesn’t yet seem to put the person in control of their own information, self-sovereign identity. Plus, the document could be upgraded to be considered a true proof of citizenship.
Additionally, only basic biometrics is not enough, but adding a smart card with a pin to the mix was the way to go. It fulfills: something you know, something you have, something you are.
The development possibilities are there and the outcomes would be interesting.

China has slowly been pushing for its virtual/electronic ID card, a digital ID. Practically, it is a valid national ID that contains biometric information such as face and fingerprints.
The solution takes a mobile-first approach, as an app with QR codes, and would integrate well with all services, both online and offline. It is still in its pilot/trial phase, but would be a true replacement for official ID cards and could be used within popular applications such as WeChat and Alipay. For example, WeChat Payment is already extremely popular for making digital payments.

China’s implementation would be directly inspired by Estonia’s one, very similar. At China’s scale, like India, it would open a lot of future possibilities as 67 percent of Chinese residents do not currently have a credit record. The system would also be linked to a generic credit score database.

China is also eyeing entering the digitalization of the supply chain. Their Standardization Administration describes it neatly: “First-class companies do standards. Second-tier companies do technology. Third-tier companies do products.”

This means being able to associate identity and traceability of all the supply chain and trade processes across multiple geographies and organizations. A true verifiable life-cycle where you could identify compliance and transparency.
For that, they’ve partnered with international standard organizations such as ISO and ITU, and started designing and implementing the next wave of standardization in cyber-physical trade with its ambitious China 2035 Standards strategy. This initiative is also driven by other organizations such as UN/CEFACT, the ICC, World Customs Organization, the European Union Intellectual Property Office (EUIPO), and International Air Transport Association (IATA).

This type of digital meta-platform, this platform-of-plaforms, would empower individual actors by giving them the power of traceability and identity for supply chain & trade. This applies to both people within it and outside it.
For the consumer, that would mean being able to reliably see the origin of a product, including its materials, legal, geographical, intellectual-property, its origins, and more. This is essential for many of us, and would make it easy to see the quality of intangible goods.

Consumers, governments, and companies are demanding details about the systems, enterprises, and sources that delivered and transformed the goods along their value chain. They worry most about quality, safety, ethics, and environmental impact, to name just a few.

This would ensure the transparency when buying products from big platforms that contain sub-markets such as Alibaba, Amazon, and Wal-Mart. This adds a lot of trust and transparency with consumers, they can verify the digital twin of their physical good, along with its history.
Now, that is going digital!

A path towards societal digitalization through internet technologies.

Other countries such as Argentina, Afghanistan, Denmark, Pakistan, Kazakhstan, and more have or are introducing national ID cards with embedded chips used for services and are considering or in the progress of integrating with biometrics, payment, and the online world just like Estonia, India, and China are doing.
This seems like a global trend and is expected to increase with new types of root of trust that can be shared over the wire such as IoT SAFE.

This concludes our review of how governments can bring back order, morals, and ethics on the internet. In a first place, we’ve dabbled with how the market isn’t enough and how there’s a need for laws that would consider privacy as a human rights. Then we’ve seen the balance between sociality and privacy, to whom we’d prefer giving it. Next we went over three topics applied to legality: consent, accountability, and transparency. After that we’ve looked at some examples, how the world is moving in that direction. Finally, we’ve tackled identity and social media as utilities and infrastructures, and what that would imply, along with some of today’s examples.

Table Of Content


Attributions: Michael Maier, Atalanta fugiens, Oppenheim, 1618

April 21, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — Adapting: Free Market, Let It Solve Itself April 21, 2021 09:00 PM


  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 5 — Adapting
  • Section 1 — Free Market, Let It Solve Itself
Table Of Content
  • The Market Will Decide
  • Promoting Fair Competition
  • Privacy as a Feature
  • Changing The Economic Dynamics

In this last part of the series we’ll go over the adaptations we are undergoing to remove the frictions we have with the internet — anything to make it better suited for us. We’ll take multiple perspectives, primarily the ones of users, societies, and others that are encountering the issues we’ve seen related to this new communication channel.

Let’s start by taking the point of view of the market and economy as ways, in themselves, to provide solutions to problems we’ve had.
Keep in mind that these heavily depend on the locality, culture, and practices of the tech companies at the epicenter of the internet drastic changes that are taking over the world. Most of these giants embody a neoliberal mindset and originate from the USA.

A market solution is a solution based on trusting the free market before all else. It hinges on not having to intervene with regulations and laws but to believe the behavior of consumers, their choices and decisions, will eventually lead to an ethical and moral balance.
This laissez-faire attitude is anchored in what we’ve seen in the cultural ambiguity and truth and trust crises sections. They rely on the assumption that individuals understand the dynamics, are well-educated on the subject matter, and will always make the best (economic) decisions overall.
This line of thinking is contested, but let’s consider the adjustments and innovations that can be driven by users.

Market solutions are hardly international as they can only work by relying on fair competition and transparency between actors. Non-intuitively, at least in the eye of some, this can only be achieved through some regulations that keep the market in checks, and such regulations are geographically bound.

When it comes to the online world, because a lot of the money is made from ads and tracking, it means regulating the whole digital advertising tech supply chain, and data brokers along with their partnerships.

If this isn’t done properly, there could be conflicts of interests, some companies self-preferencing advertisements for the services they themselves provide. This is relevant because many of the ad-tech providers also offer other services, which they can promote through their ads to compete with other players.

This problem is exacerbated when only a few entities, which as we know is the case today, have the upper hand (hear by that almost monopoly) in the ad tech sector and data broker space. These companies can then thwart competitors by buying them or competing directly with their services.
The companies will then accrue larger datasets and it’ll be harder to compete with them. This is what we’ve seen in the new economies section. They can control the pricing and quality of services on this digital supply chain.

To promote fair competition, to make it a free market, many things can be done.
One of them is to break the large datasets into subsets to allow new smaller entities to compete instead of having systems of interlocking monopolies.
Another thing that can be done is to manage conflicts of interests, which is what some lawmakers are doing these days. This includes institutions such as the European commission, the Competition and Markets Authority in the UK, the Australian government, and the USA Congress that are now checking for risks of collusions, antitrust, and what can be done for fair competition. We’ll see more of the law and regulatory concerns next.

Additionally, one thing that can foster good competition is to have open standards describing what can be done with ad tech, the format of the data traveling on the pipeline. This includes implementing a common transaction and user ID for consumer and making them portable and interoperable across ad providers.
It would then be easier to port the data between rivals and might even allow consumers to regain control over their data, selecting with whom they’d like to share it. We’ll see more of that when discussing digital identity.
Apple uses something called IDFA, Google uses an analytics and advertising tracking ID, Facebook uses a Pixel ID and others.

Standardization also goes along openness such as the right to repair, to allow products to be repairable to last longer, and decentralization.

However, openness can also be abused for profit. We’ve seen countless times that standards that are initially open can be hijacked. The big players will in turn drive the standard towards more complexity and growth, consequentially kicking out any small player that can’t follow.
This is arguably what happened in the “browser war”, and is still happening today with web standards. The ones contributing the most will control and subdue them, while defending their actions on the basis of openness, improvement, and progress.

History also shows that the same companies that emerged thanks to openness tend to reject these principles when they undermine their strategic position. After gaining a substantial amount of users, after being rooted deep enough, they’ll use the network effect to justify changing their tactic and stance.
A too big to fail scenario.

Obviously, companies can also spend money to defend their interests, for example by lobbying the USA Congress, as we’ve seen in the new economies section.

There are many ways to make money online, on the internet market.
To make profit, as we’ve seen in the new economies section, the most popular models are the ones that go along with what users expect, namely zero-price models where data is exchanged instead of money. This goes under different names such as “free model”, “data-as-payment model”, “online advertisement-based business model”, “ad-supported model”, “freemium model”, etc..
There are other models that are possible, as we’ll see, such as a flat-rate model with unlimited or limited usage, a bundle-like model, and a per-usage or subscription based model where users are enrolled and charged recurrently.

The names for the free models actually have separate particular meanings that reflect how each view and make money from the data, personal data, or attention, which become the currencies of exchange. These models also provide insights into manners of approaching privacy.
The consumers are then free to select the model that they find the most suited to their needs, interests, and ethical concerns.

The ad-supported model, one in which companies make money from selling ads or from gathering data for ad-tech providers, poses the most privacy concerns. It is either targeted advertising, such as microtargeting, or context-based advertising.

A freemium model is one in which the product is offered free of charge and money, a premium, can be paid to access additional features or services.
The free of charge part can be combined as a hybrid with advertisement.

A data-as-payment model, aka PDE or Personal Data Economy, is a model in which individuals deliberately sell their personal data as an asset to pay for services and products. This model turns the surveillance economy upside down.
Similarly, a privacy-discount model is a model in which the users relinquish their privacy to obtain a discount on a product or service.
These two models are user-centric, letting the consumers take back a semblance of control over their data: with whom they share or disclose it, the price for it, and on which terms.

Each model has a different perception of what privacy is, either as a commodity/product/feature, as a luxury/premium, as a right, or as a utility.
In the market view it is primarily a feature or premium that can be transacted.

What we described as data-as-a-payment or privacy-discount are instances of perceiving privacy as a commodity. In these scenarios neither citizens nor governments care about privacy regulations but freely exchange and trade it. In this case, people are powerless about what information is being transferred, while tech companies can make huge profits by selling the data.

Privacy can also be seen as a luxury or something you get for a premium, aka PFP or pay-for-privacy, aka primium aka privacy-as-a-premium. In this scenario, consumers are encouraged to pay extra to gain back their privacy, or to acquire new products that will fill their desire for more privacy and security.
This is a case where companies profit from privacy by charging users for it. Transparency, awareness, and privacy have become competitive features delivered at a premium.
We’ll see that awareness and transparency are two things that can be enforced through laws in the next section.

On the other side, privacy can also be seen as a utility or universal right. Either something the governments regulate, control, and monitor, just like water, a tool — this is what we’ll see later with digital identity. Either as a universal right that needs to be protected by law, which is something we’ll see next too.

The debate over privacy can be confusing for consumers, they may feel torn between all these options. Some of them wanting to disengage with the virtual world to maintain their anonymity instead of putting themselves at risks. Without privacy, we are subject to social cooling, feeling tracked, caught in the polarization, as we’ve seen before.
This confusion is indeed what sparked the whole branch of privacy as a luxury, companies are capitalizing on it, for them it’s a new business opportunity.

Privacy is not only a feature in itself but also a stand, a brand, a show of force for publicity and marketing differentiation.
Many companies sole existence rely on this need: selling something at a higher price mark because of this or having products, devices, services, and tools, that will offer control back to users.
The market size of this type of business, according to Arthur D. Little, is estimated to be, as of 2020, around $5 to $6 billion.

VPN providers are one category of these flourishing businesses. They sell a service to avoid the continuous online tracking and data collection. A VPN being a proxy to internet requests, essentially hiding where they are originating from. Some commentators have suggested that all consumers should use a VPN.
In general, their model is based on offering a subscription, a pay-for-privacy product with recurrent payment. Though, ironically, some VPN companies also provide their services for free through the hybrid “ad-supported” freemium model.

Another company profiting from privacy is the device and OS manufacturer called Apple that sells products at a premium. It has recently (2021) stirred uproar from American companies that make money from the ad tech infrastructure and pipeline when it announced a recent update to their phone, tablet, and TV OS. Apple is a popular product in the USA among high earners, thus good ad targets.
The recent update would notify users when applications want to access location, track them across apps and websites, or others. The user would have the choice to allow or disallow such action.

The move is arguable, as Apple makes a lot of money from the ~30% cut (the Apple tax) it takes from in-app purchases, disallowing any other type of external purchases linked to apps. Such a change would force app developers relying on the ad model to switch to another model such as subscription or in-app purchases, which they’d take a percentage of.

As can be seen, the free market and laissez-faire approach brings with it a lot of conflicts. When big players can enforce business models on their platforms, after using the network-effect to acquire users, it enrages other players.
The dynamics at play are interesting.

When privacy is offered at a premium it raises concern about the inequality of acquiring it, a socioeconomic digital divide. “Advertising is a tax on the poor”, as Scott Galloway said. It creates incentives for the companies offering privacy and security-centered products to keep the status-quo, to make more money from it.
There’s also the lack of consumer knowledge and understanding on these issues and their implications. They could either be unaware that these are issues, or be misled by marketing schemes portraying products as privacy-aware while not providing any proof to back it up.
Privacy can be advertised on the front while in the back tracking is done. The tracking, the telemetric gathering, is not limited to advertisement purposes but could also be used for partnerships with other entities that would allow companies to gain an edge. For example, a partnership with state actors, internet providers, or mobile operators.

Let’s remember that companies are not moral agents, nor arbiters of truth, and that their decisions are both restricted by, on one side, the legislative geography they are operating on, and on the other side, the profit made from the appearance and satisfaction they offer customers. Ethics is only a cosmetic feature.
This can be hard to grasp for the neoliberal mind that tries to find their meaning through the purchase of products, as an extension of themselves. Especially when these products only provide that in return of profit.

One solution for consumers is to go toward models that remove data from the equation, thus changing the economic incentives for online businesses. These alternative models rely on putting real value on what we consume and use online, going back to transacting money for intangible assets.

While the free models were disruptive when they came out, it turns out, unsurprisingly, that consumers don’t like being harassed by constant ads.
Tech aware users will often install adblockers on their machines, rely on one of the above paid solutions to not be tracked, use browsers that block third-party cookies, and other mechanisms. We’ll dive more into the tech solutions later. Statistics show that in the USA 47% of users use adblockers.

This kills companies that rely mainly on the ad model. Either because they aren’t able to attract enough users to generate revenue, or because it damages the brand image.
Additionally, companies don’t want to be at the mercy of the ad tech monopoly, which puts them at risks when they decide on the budget and price they allocate for specific ads and on which platforms they’ll run them. They could demonitize them at any time or disallow them on the ad platform for arbitrary reasons.
Furthermore, even companies paying for ads are starting to see the limits of this model because of the lack of good insights, bots, and fake information that return to them.

For these reasons, we’re seeing the rise on the internet of the subscription economy, micro-transaction economy, passion economy, and the economy of patronage.

We already discussed the passion and patronage economy in part 2 section 2 so let’s focus on the others. All these are about putting the consumers in power, letting them make decisions through their wallet. In this case we’ll have a consumer-controlled world, thus a working free-market solution.

There are different types of subscription models, either based on usage volume, or based on time via a billing cycle. It stands in contrast with a pay-per-product model where users would pay only once, in subscription the payment is recurrent.
Consequentially, this works well for products that either perish quickly, need to be replaced, e-commerce loyalty programs, or intangible goods such as the ones provided by SaaS (Software as a Service) companies.
These include Netflix, Amazon Prime, Hulu, YouTube, Spotify, SirusXM, Pandora, VPN services, among others.

The subscription model has been estimated in 2017 to have 28k companies worldwide and had more than a 100% yearly growth. The largest retailers in the category generated more than $2.6 billion in sales in 2016, up from $57 million in 2011.

The model has the advantage that it is flexible and provides customer satisfaction. Buyers aren’t burdened by large fees and can stop the subscription at any time.
The companies providing the services are then forced to innovate, improve, and provide quality to be able to attract and keep customers. There are many instances that show that switching from an ad-based model to a subscription model has helped increase revenue.

Here the customer is at the center and their expectations are king. For this model to work the company has to build a long-term relationship with them, often focusing on the relationship more than the product or the transaction.

Nonetheless, it isn’t an easy task. A minimum number of subscribers is needed for the business to subsist, and it’s not easy to attract them when netizens are used to getting things for free.
This has especially been hard for newspapers, as we’ve said in the past. Statistics show that in the USA in 2017, only 11% of consumers were paying for online news. The rest of the world isn’t as generous either, the statistics have been replicated in other places with an average of 10% paying for news. Yet, recently we are now seeing an unexpected resurgence, a new growth. It has been reported that in 2017 the American newspapers The New York Times gained 130k subscribers and the WSJ gained 300%, the British newspaper the Guardian has also announced that it had seen profits for the first time after 20 years with more than 300k people making contributions.

The most successful stories of subscription based businesses are those that relied heavily on partnerships — between brands, consumers, affiliates, and others.
Despite all this, companies relying on the subscription model can still rely on product placement and advertisements, which puts us back at square one.

Another method to increase the number of customers, to persuade them of contributing, is to lower the barrier of entry. This could mean making it more affordable, or even to change the value of the product based on the purchasing power of the individuals or their locality.

The new disruptive approach is to make the payment infinitesimally small, a micro-payment, a micro-transaction, which can be fractions of cents, there’s almost no line of demarcation.
This can create a dynamic in which ads and trackers are unnecessary because users would pay directly with these monetary tokens the online content creators. The tokens essentially being the monetary form of the attention of users.

The word micro-transaction refers to purchasing virtual goods through micro-payments, however it also has a historical meaning attached to the gaming world.
Initially micro-payments acted as a sort of virtual currency that existed only within games.
Later on, some game developers used a model in which they provide virtual desirable assets, such as loot boxes, in exchange for micro-payments. Micro-transactions have thus gained a connotation with gambling and addiction, which brings some countries’ legal matters and regulations in the mix especially when it involves kids.

The main issue with micro-transactions is that they can only be possible if there is a monetary technology that allows real-time transfer, no fees, and privacy.
Cryptocurrencies on a blockchain seem to be the go-to solutions for this. They are far away from the traditional pipeline that is also linked to data brokers, as we’ve previously seen.

Unfortunately, some of the cryptocurrencies have issues regarding scalability and volatility, which makes them ill-suited for such endeavor. Cryptocurrencies are now more attached to investment and gambling than an internet free of ads.
Recently, payment gateways and banks have taken a particular interest in trying to gain power in this space. Namely, VISA and PayPal are now allowing certain cryptocurrencies to be used and converted on their processing platform. This would then bring information related to cryptocurrency to data brokers, removing the privacy aspect.

Still, the applications for micro-payments open avenues and unlock an entire economy between subscriptions and ads.
We’re slowly starting to see this appear in specialized browsers that allow to distribute attention tokens to websites and content creator, and newspaper that want to allow micro-transactions along with their other revenue streams.

This concludes our review of how the free market can be used as a method to adapt to the internet. First we’ve seen what this idea implies: well-informed customers making the right choices to balance the status and set things right. Then we’ve explored how this cannot be true if there aren’t regulations set in place for fair competition to happen between companies. Next, we’ve discussed some of the possible regulations and things that could be put in place to achieve this such as collusion checks and open standards. After that we’ve set ourselves on looking at different business practices, ways of making money online. Some of them are related to the ad business, others subscriptions. We’ve tackled how these models offer a glimpse into what privacy is considered: product, luxury, tool, universal right. When privacy is a product it’s traded freely. When privacy is offered at a premium, which we called primium, it’s done because of a need and not because of moral or ethical standards. We’ve listed some businesses that base their marketing or product around privacy. Finally, we’ve seen what removing data from the equation implies: having customers pay again to be able to choose, either through subscriptions or micro-payments.

Table Of Content


Attributions: S. Trismosin, Splendor solis, London, 16th century

April 20, 2021

Sevan Janiyan (sevan)

HFS “Incorrect number of thread records” error April 20, 2021 11:28 PM

Note: this post doesn’t provide a solution to address the “Incorrect number of thread records” file system issue but documents what I went through to see if I could when I faced it. I think your best bet would probably be to try ALSOFT’s Disk Warrior if you need to fix this issue. I was …

April 19, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Big Picture: Mass Hypnosis or Mass Psychosis April 19, 2021 09:00 PM

The creation of the world

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 4 — The Big Picture
  • Section 3 — Mass Hypnosis or Mass Psychosis
Table Of Content
  • Relations With Well-Known Dystopia
  • Attention, Awakening
  • Mind Control and Mass Hypnosis
  • Alienation and Mass Psychosis

The internet brings with it technological advancements that remind us of dystopia that sci-fi writers have talked about. So let’s go beyond what we’ve tackled so far and project ourselves in hypothetical futures to posit bigger consequences.

The two archetypical stories of dystopian futures are Brave New World by Aldous Huxley and 1984 by George Orwell.

In Brave New World, Huxley presents a hedonistic future that is governed by techno-science and eugenics. The people conform and adhere to the societies rules for the following reasons:

  • The future of people is predetermined based on their genetics, their place is chosen and what makes them happy too.
  • The people are conditioned in their sleep, while unaware, to find certain perspectives more enticing and to always be happy and satisfied of their position.
  • It is a future in which pleasure and happiness are everywhere, and it makes everyone prone to accept the status-quo. Who will rebel against pleasure?

We can see some parallel with neoliberalism, consumerism, and the passivity or paralysis we’ve seen earlier. However, we’ve also seen how people can’t deal with such world of templatized lives, even when pushed by the internet.

Mass production, homogeneity, predictability, and consumption of disposable consumer goods.

In 1984, Orwell presents a world of constant surveillance by Big Brother. This surveillance is extended through self-surveillance between individuals with a thought police. The world described is believed to be in perpetual war (forever war), historical facts and news are rewritten or negated to fit the accepted narrative, and language is used in a powerful way to accompany it.
The fictitious world employs many of the propaganda techniques and the eunemics we’ve encountered.

We can see some parallel with the previous sections related to the surveillance economy, the return of the collectivist narrative, and the internet wars of memeplexes that use censorship, call-out, canceling, and social cooling to keep everyone in check.

In one of these worlds facts are eliminated and in the other facts don’t matter. Today we see both of them, censorship and speech itself as a censorial weapon. Two techniques used to make people docile and well-behaved, two techniques that are omnipresent on any communication channel but even more on the internet. We’re in the center of an information warfare.

Beyond dystopian worlds, the attention economy on the internet carries with it questions related to the very nature of consciousness.

When we talk of attention we also indirectly refer to themes such as awareness and awakening. These are then tied to spiritual practices. For instance, how we pay attention is an inherent part of Buddhist practices.
Hence, attention can be said to be the outermost expression of our consciousness. Losing control of it would be the equivalent of losing oneself or being detached from reality and be brought into a fabricated universe.

In an attention economy, we would then live in memeplexes of ideas, packaged, wrapped, and opened effortlessly to let us access these collapsible concepts, these portable philosophical ideas, stories, emotions, narratives, and meanings.
We are what we focus on, we live what we are aware of. Our reality is limited by our perception of it.

We all think that the way we live is the best way to live. We all think that we see the world as it is, that we’ve come to the best possible conclusion about our direction in life. However, that’s just our definition of normality. And this normality isn’t even ours as most of it happens without us being there. We often can’t justify why we do what we do or why we like what we like.

When we spend so much time on the internet, and will possibly spend much more in the future, we’re indirectly unaware that it is defining us.
Looking around us, for some it might appear more explicit than others as they get caught in apparent internet cults or are memeoids combatant of their memeplex, but aren’t we all?
Isn’t most of the content that is fed back to us a reflection of who we already are? And as the internet takes a bigger part of our lives, will everyone get stuck in loops of paralysis like we’ve seen before?

One question we can ask is whether it’s possible to engineer these wants, to use mind control or hypnosis to make people more inclined to give their attention or perform specific actions.

A far-fetched, or not so far-fetched, concept from sci-fi is a psychotronic weapon or electronic harassment. This abstract weapon is a representation of the totality of all possible methods and means, be them suggestive, pharmacological, paranormal, technogenic, or others that influence the psyche of a person for the purpose of modifying their consciousness, behavior, or health.
While this idea is often associated with lunatic conspiracy theories, it’s interesting to tinker about the possibility that the internet could indirectly be used in the future, or now, for such ends.

One way to be able to achieve this would be to get a direct interface to the brain. The neurotechnology for brain scanning, which used to be limited to laboratories, is slowly getting more accessible, transportable, and less invasive. There are instances using electrical activity circuits, rhythmic sound or flashing light, ultrasonic or magnetic simulation, all to modify brain waves and patterns of electrical activities in the brain. These can work both as reading devices or output devices.

Today, these are still used mainly in labs to treat neurological and mental illness, and improve mood and cognition. The USA FDA has approved in 2008 transcranial magnetic stimulation to treat depression, pain, and migraine.

However, we can imagine that when this becomes more mainstream, when brain-related data starts to be common on the internet, that ad-tech companies will jump on the opportunity to access whatever comes out of these new body sensors.
There is nothing more private than thoughts. This can give the ability to know and predict what someone’s mental dispositions are, with whatever this implies: from knowing their cognition style, to identifying cognitive strengths and weaknesses, to perceiving personality traits, and to determine their aptitudes or likeliness for certain information.
When in the wrong hands this could be called a neuroweapon.

Having this type of information in the wild would be the most advanced form of social cooling, full-on paralysis, a forced “psychocivilized” society, not far from Huxley’s brainwashing dreams.
This is why researchers and human rights activists are starting to think about adding new rights: cognitive liberty, mental privacy, mental integrity, and psychological continuity.

Another method to alter minds that has been tried in the past is by using chemicals and pharmaceuticals drugs. The research has shown, at least as far as the public knows, that they aren’t very effective and their applicability comes with a lot of technical difficulties.

Some mind-altering and stimulating chemicals are already readily available on the market of most countries. Others such as hallucinogenic mushrooms, marijuana, heroin, LSD, and truth serums aren’t as available everywhere but the intelligence community showed that they weren’t effective when it comes to applying a particular military objective or operational deployment. This is because their effects isn’t reliable nor the information that people give when under them — people could become drowsy and spurt out fictitious stories.
Yet, we can imagine that these can be used without a particular goal but with a more generic approach. This is why some of these are now being introduced and tested as psychiatric drugs to treat anxiety, depression, or mania. They force people to enter new states of minds and emotions. Many of them work by returning overactive or underactive neural networks and signal to more “normal” levels — a reset button on the brain.
These drugs can also engender the opposite effect and actually increase confusion, anxiety, and depression. Some military have been using them for torture to force the person to cooperate or surrender.

An additional topic in relation to mind control is hypnosis.

Hypnosis is a state of awareness where someone detaches their of attention from their environment and get absorbed by their inner experience, their feelings, imagery, and cognition. These inner experiences and imageries are induced by a clinician that will guide the process.
The imagined feels real in that state, bridging the gap between the left and right hemispheres, the hypnotic reality. Similar to a trance state or meditative state, an obliteration of the ego.

Three things happen in the brain of the hypnotized person:

  • The activity in their salience network decreases, a place in the brain related to worry, so they get absorbed in the moment.
  • The brain-body connection increases.
  • There’s a reduction, even a disconnect, between the actions and the awareness of them.

As we said, this can be seen abstractly as a link — by using imagery — between the left-intentional brain (logical, rational, abstract, critical), and right-involuntary-state brain (emotions, feelings, instinctive, intuitive).

In practice hypnosis is studied and used along other forms of therapy such as CBT (Cognitive Behavioral Therapy) to fix misbehavior. It is used to increase the effectiveness of suggestion, give access to the mind-body link and unconscious processing. Alone it is not very useful, but it facilitates other kinds of therapies.

Hypnosis can be self-taught but it is most often administered by a clinician. It starts with a relaxing phase where the patient focuses their attention, giving it away, an induction step. This could be done in multiple ways, be it a visual focus or auditory one. The next step is a suggestion phase which consists of the clinician guiding the patient by helping them create a visualization of events and scenarios, be it verbally or using imagery, to help them address or counteract unhelpful behaviors and emotions.

In a sense, this is very like guided meditation where someone sits in a relaxing position, calmed by a mantra or sound, and is guided by a master that will help them address their inner woes.
“In hypnosis, you’re so absorbed that you’re not worrying about anything else.”

However, studies show that only a minority of the population is hypnotisable, about 10-20% are highly receptive to it. It has been shown that hypnotisability is a genetic trait and follows a Gaussian or bell-shaped distribution. So most research related to hypnosis focuses on this part of the population.

Now, related to our main topic, this doesn’t seem like mass hypnosis would be possible, nor very actionable either. Yet, we can imagine a world in which daily activities, wandering online, becomes a trance state and where imageries are transmitted using compacted packages such as memes.
Still, that is far from the truth as most studies show that it is almost impossible to make someone do something against their will. Moreover, only an extremely small part of the population would be both willing to do “evil” biddings and also be susceptible to hypnosis. But wouldn’t these fringe people be easily findable on the internet and band together anyway?

Homo homini lupus.
Men is a wolf to men.

When the psychology of a whole population changes it can create a deleterious unstoppable snow-ball effect.

Indeed, it is becoming ever more obvious that it is not famine, not earthquakes, not microbes, not cancer but man himself who is man’s greatest danger to man, for the simple reason that there is no adequate protection against psychic epidemics, which are infinitely more devastating than the worst of natural catastrophes. — Carl Jung

When a society loses the grip on reality, becomes driven by uncontrollable fear, when mental illness becomes the norm rather than the exception, social alienation takes over, a psychic epidemic.

All one’s neighbours are in the grip of some uncontrolled and uncontrollable fear… In lunatic asylums it is a well-known fact that patients are far more dangerous when suffering from fear than when moved by rage or hatred. — Psychology and Religion

Mass psychosis happens when a large portion of society descends into delusion, similar to what happened during the European witch hunt.
It is a paranoia taken to the extreme caused by a rise in anomie, societal stress, and a flood of negative emotions.
“A mysterious, contagious insanity”.

We can already see how this can be amplified on the internet where these types of emotions, ambiguity, confusion, fear, and doubts thrive. We’ve discussed amply the cultural insecurities and cognitive dissonance in the previous sections.

We can also wonder if these can deliberately be induced in another population through PSYOPs.
Don’t we also see the rise of this phenomenon through the memeoids and their memeplexes, taken over by ideas so powerful that they possess them, consume them, even destroy them? Mobs and self-justice carnage and destruction.
Moral absolutism, and categorical imperatives are cognitive distortions because most of reality and the living of everyday life takes place in the gray area, between the extremes.

Those who control the flow of information in a society, the ideas we accept as true or false exert a great power over the course of civilization.

This concludes our review of future societal meltdown, be them hypothetical sci-fi scenarios or realistic ones. We’ve started by taking a look at the relation with well-known dystopian stories such as Brave New World and 1984. We discussed the parallel with today’s internet, censorship and speech itself as a censorial weapon. Then we’ve talked about the deep inter-relation between attention, awareness, and consciousness. We’ve said that a control of attention is a control of reality but that online we’re often only given back what we already want. Next, we’ve wondered about different ways to engineer the “wants” of a person, from mind-control weapons, to getting access to data taken directly from the brain, to drugs and chemicals, to hypnosis. Finally, we’ve discusses mass-psychosis, the state of a society where mental illness becomes the norm and reality is distorted to fit an invented narrative.

Table Of Content


Attributions: Hieronymus bosch, outer wings of the Garden of Delights, c. 1510

Marc Brooker (mjb)

Tail Latency Might Matter More Than You Think April 19, 2021 12:00 AM

Tail Latency Might Matter More Than You Think

A frustratingly qualitative approach.

Tail latency, also known as high-percentile latency, refers to high latencies that clients see fairly infrequently. Things like: "my service mostly responds in around 10ms, but sometimes takes around 100ms". There are many causes of tail latency in the world, including contention, garbage collection, packet loss, host failure, and weird stuff operating systems do in the background. It's tempting to look at the 99.9th percentile, and feel that it doesn't matter. After all, 999 of 1000 calls are seeing lower latency than that.

Unfortunately, it's not that simple. One reason is that modern architectures (like microservices and SoA) tend to have a lot of components, so one user interaction can translate into many, many, service calls. A common pattern in these systems is that there's some frontend, which could be a service or some Javascript or an app, which calls a number of backend services to do what it needs to do. Those services then call other services, and so on. This forms two kinds of interactions: parallel fan-out, where the service calls many backends in parallel and waits for them all to complete, and serial chains where one service calls another, which calls another, and so on.

Service call graph showing fan-out and serial chains

These patterns make tail latency more important than you may think.

To understand why, let's do a simple numerical experiment. Let's simplify the world so that all services respond with the same latency, and that latency follows a very simple bimodal distribution: 99% of the time with a mean of 10ms (normally distributed with a standard deviation of 2ms), and 1% of the time with a mean of 100ms (and SD of 10ms). In the real world, service latencies are almost always multi-modal like this, but typically not just a sum of normal distributions (but that doesn't matter here).

Parallel Calls

First, let's consider parallel calls. The logic here is simple: we call N services in parallel, and wait for the slowest one. Applying our intuition suggests that as N increases, it becomes more and more likely that we'll wait for a ~100ms slow call. With N=1, that'll happen around 1% of the time. With N=10, around 10% of the time. In this simple model, that basic intuition is right. This is what it looks like:

The tail mode, which used to be quite rare, starts to dominate as N increases. What was a rare occurrence is now normal. Nearly everybody is having a bad time.

Serial Chains

Serial chains are a little bit more interesting. In this model, services call services, down a chain. The final latency is the sum of all of the service latencies down the chain, and so there are a lot more cases to think about: 1 slow service, 2 slow services, etc. That means that we can expect the overall shape of the distribution to change as N increases. Thanks to the central limit theorem we could work out what that looks like as N gets large, but the journey there is interesting too.

Here, we're simulating the effects of chain length on the latency of two different worlds. One Tail world which has the bimodal distribution we describe above, and one No Tail world which only has the primary distribution around 10ms.

Again, the tail latency becomes more prominent here. That relatively rare tail increases the variance of the distribution we're converging on by a factor of 25. That's a huge difference, caused by something that didn't seem too important to start with.

Choosing Summary Statistics One way that this should influence your thinking is in how you choose which latency statistics to monitor. The truth is that no summary statistic is going to give you the full picture. Looking at histograms is cool, but tends to miss the time component. You could look at some kind of windowed histogram heat map, but probably won't. Instead, make sure you're aware of the high percentiles of service latency, and consider monitoring common customer or client use-cases and monitoring their end-to-end latency experience.

Trimmed means, winsorized means, truncated means, interquartile ranges, and other statistics which trim off some of the tail of the distribution seem to be gaining in popularity. There's a lot to like about the trimmed mean and friends, but cutting off the right tail will cause you to miss effects where that tail is very important, and may become dominant depending on how clients call your service.

I continue to believe that if you're going to measure just one thing, make it the mean. However, you probably want to measure more than one thing.

April 18, 2021

Ponylang (SeanTAllen)

Last Week in Pony - April 18, 2021 April 18, 2021 11:07 PM

The supported version of FreeBSD has switched to 13.0 for ponyc and corral. The Roaring pony folks met again.

Derek Jones (derek-jones)

Another nail for the coffin of past effort estimation research April 18, 2021 09:32 PM

Programs are built from lines of code written by programmers. Lines of code played a starring role in many early effort estimation techniques (section 5.3.1 of my book). Why would anybody think that it was even possible to accurately estimate the number of lines of code needed to implement a library/program, let alone use it for estimating effort?

Until recently, say up to the early 1990s, there were lots of different computer systems, some with multiple (incompatible’ish) operating systems, almost non-existent selection of non-vendor supplied libraries/packages, and programs providing more-or-less the same functionality were written more-or-less from scratch by different people/teams. People knew people who had done it before, or even done it before themselves, so information on lines of code was available.

The numeric values for the parameters appearing in models were obtained by fitting data on recorded effort and lines needed to implement various programs (63 sets of values, one for each of the 63 programs in the case of COCOMO).

How accurate is estimated lines of code likely to be (this estimate will be plugged into a model fitted using actual lines of code)?

I’m not asking about the accuracy of effort estimates calculated using techniques based on lines of code; studies repeatedly show very poor accuracy.

There is data showing that different people implement the same functionality with programs containing a wide range of number of lines of code, e.g., the 3n+1 problem.

I recently discovered, tucked away in a dataset I had previously analyzed, developer estimates of the number of lines of code they expected to add/modify/delete to implement some functionality, along with the actuals.

The following plot shows estimated added+modified lines of code against actual, for 2,692 tasks. The fitted regression line, in red, is: Actual = 5.9Estimated^{0.72} (the standard error on the exponent is pm 0.02), the green line shows Actual==Estimated (code+data):

Estimated and actual lines of code added+modified to implement a task.

The fitted red line, for lines of code, shows the pattern commonly seen with effort estimation, i.e., underestimating small values and over estimating large values; but there is a much wider spread of actuals, and the cross-over point is much further up (if estimates below 50-lines are excluded, the exponent increases to 0.92, and the intercept decreases to 2, and the line shifts a bit.). The vertical river of actuals either side of the 10-LOC estimate looks very odd (estimating such small values happen when people estimate everything).

My article pointing out that software effort estimation is mostly fake research has been widely read (it appears in the first three results returned by a Google search on software fake research). The early researchers did some real research to build these models, but later researchers have been blindly following the early ‘prophets’ (i.e., later research is fake).

Lines of code probably does have an impact on effort, but estimating lines of code is a fool’s errand, and plugging estimates into models built from actuals is just crazy.

April 17, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Big Picture: Truth & Trust Crises April 17, 2021 09:00 PM

A. Kircher, Ars magna lucis, Amsterdam, 1671

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 4 — The Big Picture
  • Section 2 — Truth & Trust Crises
Table Of Content
  • Individualism, Neoliberalism, And Relativism
  • Lack Of Trust — Doubt As The New Normal
  • The Collapse Of journalism
  • Mobs & Self-Justice

Nobody likes to be put in a box, nobody likes to be paralyzed. In a world where the individual is put first, we want to feel in control. Yet, we are lost in a whirlpool of chaotic and clashing ideologies, memeplexes and absolutist templates. Who are we, how should we define ourselves? This makes us distrust any form of authority and renounce classical forms of media. As individuals, we’re looking for authenticity, whatever form it can take even if disingenuous, so that we can find our new roots, our new base. Then, there’s nobody better suited to affirm justice than internet mobs.
It feels like the internet is the new far-west, a lawless land of incessant doubt, lacking meaning and trust, where gurus, inspirational influencers, and vigilantes reign and data, personal data, and metrics of reputation are the currency held by the banks of social media.
Or it may also feel like eunemics in action — the deliberate improvement of the meme pool — and people wanting to take part in the cultural meme wars and genocides.

A lot of this can be attributed to the wide spread of neoliberalism values and mindset, which as we’ve seen in the previous section are the most prevalent online. Particularly, we’re faced with individualism, relativism, and inequalities all justified by the protection of personal rights.

If you see a world full of individuals then you’ll want a morality that protects those individuals and their individual rights.

You’ll emphasize concerns about harm and fairness.

But if you live in a society in which people are more likely to see relationships, context, groups, and institutions. Then you won’t be so focus on satisfying individuals, you’ll have a more sociocentric morality.

Which means that you place the needs of groups and institutions first, often ahead of the needs of individuals.

If you do that then a morality based on harm and fairness won’t be sufficient. You’ll have additional concerns. And you’ll need additional virtues to bind people together.

The Righteous Mind

The individual becomes reason in and of itself, and not only a figurant in their own life. We are expected, in the neoliberal theory, to develop our own opinions, outlook, stances through a considerable amount of solitary reflection. However, most important issues, such as the definition of our identity, happen over dialogue. We can’t define ourselves or recognize ourselves, in a void.
We’ve discovered how this can be beneficial in part 2 section 2, as networked individuals connected by interests. Interests become part of our identity and how we generate meaning.

On the internet, however, this leads to struggles as we’re incessantly exposed to a wide range of ideas, some of which may be breaking our local taboos. The Overton window keeps moving, for the better or the worst.
Isolated societies, communities, and ways of living may be forcedly assailed by ideas outside their comfort zone. We’ve touched this topic in the cultural malaise section.

I went a bit over this in my article entitled cross culture frustration.

The new mentality appears different and frightening — different acting, different social dynamics, different manner of building the “self”, different definition of who we are within the society. In addition, we hate being wrong, and we like confirmation. We create a cocoon and hide ourselves in it repeating the hokum over and over again. And we are afraid of the unknown — afraid that the stable self we’ve built to be part of a society is not a self that could partake and feel comfortable in the activities of another society.

In this mindset, the individual always comes before the large collective forces and needs to be protected against them. Personal life choices can’t be criticised as this would be tantamount to an assault on personal dignity. Inherently, this carries the acceptance of a soft moral relativism, any opinion should be respected and nobody should impede on others.
This translates on the social plane into political acts for individual rights, recognition, and equality. Anything else is understood as inflicting damage and being a form of oppression. We are more interested in dignity, universals, innate attributes, and equality.

Additionally, this requires the breaking of hierarchies and a loss of a grand sense of meaning. The perception is now centered on the self to find these, rejecting, and even erasing from awareness, issues and concerns that transcend the self.
We aren’t recognized through honors, distinctions, or any rare achievements, but by our self-generated worth. We’ve replaced these forms of recognition, the need for belonging and authenticity for example, by other things.

Personal and intimate relationships are now the definition and emphasis of life. We now expect to find the greater meaning, which was previously found in institutions and traditional structures, in these relationships. The domestic becomes the loci of “the good life”.
This is the core of what social media are about. They are filling a cultural gap as we said before.

Self-fulfillment and worth is also found through the instrumental reasoning of neoliberalism — actions driven by profit, optimization, market, and metrics. Meaning lies in efficiency, success is found in numbers, tangible assets, products, and money. In a way, it has replaced grand narratives and sense with short-term narcissistic, selfish, pleasures and pains, with little regard for others or society as a whole.
Similarly, the intimate relationships are also seen as tools for self-development and so are also serving an end.
It permeates every aspect of life. Even simple things such as food have become an individual matter, for personal consumption, rather than a convivial mean of regrouping and conversations.

Old social orders are shattered and replaced with resources for projects, economic growth, and prestige in technological progress.

No empathy. No loyalty. No forgiveness. Thanks to the market, the old-fashioned virtues have been rendered obsolete.

Common goods and social objectives are de-emphasized. The protection of the individual is only an extension of the protection of their private economic rights. The new morals are the morals of protecting the individual and their economic rights, the sanctity of the individual and the pursuit of self-interest as the highest moral ideal.

However, this system generates a lot of inequalities, there’s a wide distribution of wealth and income. It is hard to keep competition open and fair — a competitive attitude that also fosters more individualism. Corporations are free to self-regulate.
Comparativeness and cooperation are both part of human nature, but the one that is valued and promoted by the culture will thrive.

On the internet we’re presented with the myth of individual geniuses working in their garages without any investment, self-made success stories. Yet, this is only a myth.

And because worth is found in such success, it is then displayed in clear light for everyone to admire. Social media make this even more apparent.
These inequalities will undeniably boil into social unrest. Individual persons will demand their individual rights and feel entitled to do so.

Overall, there’s a sentiment of being stranded on an island, lost, afraid, and missing something vital. Many societies embracing or exposed directly or indirectly through the internet (or other political pressures) to neoliberalism have a hard time coping with this. The competitive and instrumental mindset encourages a fear of the other. The moral relativism destroys grand narratives, old societies heroes, and replaces them with ambiguity and an identity crisis. The cultural malaise, the feeling of anomie, we’ve discussed earlier.
We’re left with defining ourselves through interests, intimate relationships, metrics, economic activities, and a moral that only has one thing to say: “everyone is on their own, let them be”.

We are like islands in the sea, separate on the surface but connected in the deep — William James

More than anything, this damages and erodes our trust in any form of authority, this makes us doubt what we know and what others are telling us. We’ve seen how this is abused on social media in a previous section.

When reality doesn’t reflect our projections, it will backfire. Turning things upside down, changing love into hate, or any other ways of rebalancing our perceptions.
We project on people, ideologies, cultures, and politics, wanting to put our trust in something, to see them as extensions of us, to feel that our conception of reality is stable and secure.
When it doesn’t match the people revolt.

On social networks we are torn between different narratives that threaten each others. There’s no accountability or regulation for this user-generated content and it is often published under aliases that can’t be traced to any source.
Truth and trust are bent to their extreme. The digital revolution undermines mainly the control that governments used to have on them, replacing it with a technological alternative of wild-west, open, democratic, and sometimes manipulated discussions.
Infobesity and infoxication don’t help either, we’re inundated and can’t keep track of what is important, nor absorb it, nor understand it, nor know who said it. It makes it hard to effectively grasp issues and make decisions, as we’ve discussed in the paralysis section.
We’ve also previously seen how state entities are trying to get back the control by using speech itself as a censorial weapon.

The biggest consequence of this is that there has never been as much political discussion happening. It’s everywhere on the internet and it is what thrives.
However, the memetic material, the content, is always provocative, conflicting, shocking, and sensational because, as we keep repeating, this is exactly what thrives in the internet environment, namely: what touches our cultural sensitivities.
Consequentially, in relation to what we said before, this content will be about identity politics and identity in general, personal definitions. Discussions will rotate around recognitions of who some people are, the gaps they have in their cultures, how they patch them, the oppressions and damages they feel are being done to their previous or new-found identities.

Some discussions and content are more powerful and dangerous than others, activated ideologies that have real world consequences. This can lead to warranted or unwarranted protests, extreme ideologies, discriminations, assaults, suicides, acts of violence, and others.
In a previous section we’ve dabbled into how this can be used and initiated as part of PSYOPs.

This becomes alarming when doubt engenders the creation of an alternative reality that contests well-accepted scientific truth and facts, turning virtue into a vice. The lack of trust in authority being replaced by a trust in narratives that claim pseudo-authenticity and certainty. Sometimes coming from ill-informed people or from nations or entities that have an agenda — misinformation and disinformation.
This is what we’ve talked about when discussing memeoids, propaganda, and cults: a memetic equilibrium, a coherent world of cohesive meaning. The grand sense of meaning that was broken with neoliberalism but that couldn’t be recovered through anything else. The modern hermetism, or return of mysticism.
Doesn’t memetics itself reflect the concept of neoliberalism, all selfish memes fighting for their individual survival?

The word “truth” is often semantically associated with some religious or moral doctrines. This confusion also occurred in psychological literature.

In practice, this is embodied through conspiracy theories, built over time on the accumulation of confirmation biases and wishful thinking, and also through polarized and absolutist memetic wars that carry new narratives.

The side effect of these is the collapse of the classical form of journalism. Traditional media in its original shape is dead and along with it any form of “official narrative”.
There exists some media that are government owned, publicly funded, but the netizens do not trust them.

The online economy has ravage journalism. For example, in the USA, about a fifth of newspapers have closed and the survivors are shadows of what they were. The ones that are left rely on shock value to drive revenue from ads, the new business model, or have switched to subscriptions. Many geographical regions are now news deserts leaving people to rely on social media instead.
This results in the dramatic decline of professional reporting, yet a majority of people have not, and will not, notice any significant change.

The dominating ad networks, Google and Facebook, have their own alternative news channels and also decide on the internet dynamics: promoting or demoting a newspaper based on its popularity and how much revenue it attracts.

Citizen journalism is now king and the traditional media are following their footstep, redefining themselves as intermediates. Social media platforms have replaced the gatekeepers, as we’ve seen earlier, and the “democratic” curatorial systems decide which news are important and how they get framed.

While in the past the news were responsible for what they said, regulated, limited by their journalistic values, now it’s fully unregulated.

There are upsides and downsides to this new development. As we said, these are amateurs, people without journalism training, and so the quality of the reports and coverage can vary, is subjective, heavily opinionated, unregulated, and can be abused by malicious actors — such as spin doctors.
However, it can also promote human rights and democratic values to have the people join together and collaborate in this endeavor.

We’ve tackled before how the engines reinforce what people already believe, amplifying their narrative through their self-generated (often fabricated) news.
It’s even more distressing when heads of nations publicly attack the press, calling them “enemy of the people”, inflaming opinions, polarization, and partisanships. Politicians have learned how to play the game on the internet.

Along with the self-generated meaning and self-generated news and facts, we see the rise of self-justice, private justice.
Ironically, this is in stark contrast with what neoliberalism is about, it is the large collective that fights individuals, the return of the battle of the collectivist narratives.
The polarization with its absolute us vs them mentality. A return of a sort of absolute moral rigidity with categorical imperatives.

Laws should be there to protect minorities, not lead to mob violence, and vigilantism, the kind of fast justice, with total impunity, and auto-defence that is justified by militias around the world — the wild-west and absence of authority that favors the creation of these dynamics.
This lack of trust in institutions and truth makes us want to create our own. We substituted the old gatekeepers, so we can also redefine the social contract.

Corporations and multinationals have been doing this for some time now, pushing their own justice, influencing legislations, spying, and thwarting competitors using any means. This has definitely been facilitated by social media.

On the individual side, it’s understandable, we’re presented a world, through the lenses of the news and social media, of constant chaos, clashes, and protests everywhere. It seems to be the only logical conclusion, when doubting all authorities, that we should fight for our own rights and that no one else will do it.
The offline consequences of this are apparent so let’s talk about the online ones.

The collectivists narrative, one that sees each person as a member of group and not an individual, a kind of “social tribalism”, is the way people bring back order from the chaos of getting exposed to different ideas — a pre-neoliberal view within a neoliberal mindset.
It is associated with a make-belief of simple dichotomies, groups at war, enemies, activism and counter-activism. Societies then become a struggle for power, a zero-sum war of ideologies. These ideologies are, within neoliberalism, related to identities.
Some of these are hijacked or pushed by political actors for gain. This is by various kind of parties that adhere to different political ideologies on the political spectrum.

This forcing of homogeneity, is the same force that drives indoctrination, cults, and memeoids. On themselves, these distorted worldviews and conflicts are wars of memeplexes, each side carrying their package of culture.

The combatants fight to annihilate the other memeplexe, to achieve eunemics — essentially trying to deliberately control the meme pool.
The war of memeoids is fought by quarantining other memes, restricting their speech/thoughts through censorship, and also done by executing attacks that ridicule memes to detach them from the enemy’s memeplex.
“Tyranny is the deliberate removal of nuance.” – Albert Maysles
Only one reality must survive!

When it comes to these attacks, they are filled with derogatory terms, loaded words that carry with them cultural ideas, politicized insults: essentially memes in the form of words. As we said, ridicule and satire are the best tools to fight narratives and drive discussions.
Creating a new language is also creating new ideas. Satiric words are thus both attacking memes toward the enemy memeplex, and new memes to be incorporated in the memeplex of the generator. A memetic warfare. Benjamin Lee Whorf said: “Language shapes the way we think, and determines what we can think about.”

A lot of the popular words are taken from the USA politics, so we’ll extract some examples from there, any reader should be familiar with the terms considering their invasiveness in the online sphere.

One of these words is SJW, “Social Justice Warrior”, a pejorative term used to describe someone who promotes “socially progressive views” for personal validation rather than conviction. This word carries a political meaning in the USA politics, is a satire thus a meme.
The term itself is used by the opposite ideologists to refer to the internet combatants of the other groups. These SJWs rely on victimhood talk, a twist on the oppressor narrative along with emotions, to silence/censor their opponents. Absurdingly, the groups using the words are themselves portraying the world in a similar way, the back to the collectivist narrative.

The personal validation is described as “virtue signalling”, a way to show your virtues and allegiance to a group. Virtue signalling is the common phenomenon we’ve seen happening when people share memes on social media: informing and showing others that they belong, that they know what is right, following what they think is prestigious, along with a sense of superiority.
It is most often used to point out “fake outrage”, “political correctness”, “microaggression”, “microinvalidations”, and “self-righteousness” (also meme terms), but it can also be applied more generally to anything used to get social karma/points.
Indeed, this is related to the neoliberalism narcissism coming from finding meaning through metrics. Obviously, and contrary to popular beliefs, this noisemaking is done by anybody on the political spectrum and not only by a single political camp. Only the association and its cultural reference that packages it in the name makes it a novel meme.
“In a democracy, recognition matters. Everyone wants to be seen as they are. If they are not, then it’s impossible for them to enjoy the experience of being full citizens.” — Melissa Harris-Perry

The term is also associated with slacktivism, hashtag activism, and pathological altruism. These are ways of supporting causes on social media while not involved in the on-the-ground activities and the messiness. These are terms used, and mostly prevalent, by Western netizens to describe how they are distanced from reality and act from the clean pristine coccoons of their sofa.
They are pejorative because they imply that these actions have little effects, might even be ineffective, are low-cost efforts, and can’t substitute more substantive actions. However, as we’ve seen over the past sections, we know that online political talks do indeed have an effect on mindsets and allow for more discussions around a topic — temporarily getting attention, selecting what is important.

One technique that is extensively used today by all groups, political parties, or association as a defense mechanism is a sort of deliberate social cooling of others, a self-surveillance that goes by the name of call-out culture and cancel culture.
The names are normally used by a specific political side in USA politics as a derogatory term describing another side, but the techniques themselves are applied by everyone anywhere in the world. It’s about using the online-outrage machine we talked about earlier to attempt to defame and vilify characters that go against the memeplex of the group.
This public lynching type of censorship should be a reminder of the propaganda we’ve seen earlier. Speech as a censorial weapon.
A perfect way to achieve eunemics through the suffocation of memeplexes.

This concludes our review of the truth and trust crises accelerated by the internet. In a first part we’ve seen how neoliberalism and individualism have shaken our notions of stability and broken our vision of authorities and hierarchies. The individual is favored before the collective, and they now find meaning through metrics, economic activities, and moral as a protection of new personal identities. Next, we’ve looked at how this unbalance can backfire if projections don’t match reality, especially when presented with so many worldviews on social networks.
Then we went over how this generates an enormous amount of political discussions, especially related to identity, and how this generates alternative universes of meanings that live on social media, some of it conspiratorial or based on novel narratives. A kind of self-generated meaning.
After that we’ve observed the collapse of traditional journalism and all that it implies, especially that now people can generate their own news, their own facts.
Finally, we’ve concluded with a glanced at a self-generated justice, a private justice created through the return of a collectivist narrative, fights between groups. These groups being memeoids, that fight eunemics wars on the internet trying to anhilate other memeplexes using some techniques we’ve encountered in previous sections such as speech as a censorial weapon.

Table Of Content


Attributions: A. Kircher, Ars magna lucis, Amsterdam, 1671

April 15, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Big Picture: The Formatted Individual April 15, 2021 09:00 PM

The Parmasayika grid is a fundamental religious diagram which divides up the Hindu Pantheon according to the measure of the purusha of the cosmic primal man

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 4 — The Big Picture
  • Section 1 — The Formatted Individual
Table Of Content
  • Paralysis
  • Normalcy and Innovation
  • The Neoliberal Mindset

In this part of the series, we’ll explore the bigger picture and the generic issues and “ill effects” on societies that are brought by the emergence of the internet or accelerated by it. We’ll begin with a look at three inter-related subjects: a general social paralysis, an apparent sentiment of homogeneity, and the relation with the widespread neoliberal mindset.

Paralysis is experienced at many levels and in different ways. The first of these are cognitives, which we’ve brushed in part 2 section 2.

One of the cognitive effect of the internet, proven by studies, is how much it affects our attention on the short term and long term. It is more divided and we’re less able to sustain concentration, it is more shallow and guided by attraction.
Research has shown the effect is clear on children, and long term studies have also shown that, contrary to popular belief, constant multi-tasking actually impedes the very act of task switching and does not improve it. This last one is due to how frequent exposure to this environment increases our susceptibility to distraction from irrelevant environmental stimuli.
Overall, multi-tasking in an internet environment has been associated with significantly poorer overall cognitive performance, it trains us to pay attention to what is flashier. We then feel frustrated and hindered when we have to perform tasks that require concentration. We feel obstructed and paralyzed, as if we’ve been robbed of our concentration.

Another effect we’ve looked at before is how the internet changed the way we retrieve information and value knowledge, acting as a transactive memory — an externalized memory.
The result is that we remember more where to find the information rather than the information itself. We know less about factual information because we’ve offloaded it to this new medium.
Hence, when we have decisions that rely on facts it makes us dependent on the internet. This can be both beneficial and harmful, as we said, giving us the opportunity to focus on conclusions, emergent hypothesis, overall aspect rather than facts. However, it also makes us focus more on opinions, trust the first results to our query (usually confirming our thoughts), ask fewer questions related to facts but questions related to individuals and society, and put an emphasis on fads and trends.
Our decisions are thus slower, having to consult online resources to make them, and hampered by what someone else has written about the subject we’re looking for.

We’ve previously also seen that social media had the same neurocognitive effects as real life social interactions but with an instant feedback through clear metrics. Time is now objective, not subjective, anything to turn us into machines. This has a direct impact on the concept of the self and self-esteem, making us pay attention to social judgement and social comparison. We’ve talked about influencers, hyper-successful individuals, and the unrealistic expectations these can create.
These can make us feel inadequate, especially for people that can’t manage properly their emotions.

Another aspect that gives us cold feet is the over-reliability of the medium. We’re deeply aware that nothing disappears on the internet, that any faux pas will be remembered.
Yet, we’ve also never been as uncertain about anything. We’re constantly the spectators of theatrical clashes of opinions and it makes us doubt our own.
Moreover, we’re constantly overthinking, overanalysing things, we are submerged in information and can’t take decisions fearing we’ll make the wrong ones. An analysis paralysis due to the infobesity.

Joining these ideas related to judgement makes us stay on the bench. These are compounded in phenomena like social cooling, the 1% rule, and lurking.

Social cooling is the self-censure that people exert on themselves because of social pressure. The tension between always feeling watched by surveillance algorithms, knowing our personal data is gathered against our will, understanding that our weaknesses and flaws could be exposed and used against us, and fearing how we’ll be scored based on the whims of the internet crowd. We’re intimately aware that our digital reputation is now serious business and can impact our real life.
For these reasons, people change their behavior and adapt to whatever makes them look better on the platforms, have better scores or avoid doing anything that could add a negative side to their reputation. We hesitate before making a move. All over, it makes an environment socially rigid with risk-averse people that conform.
A chilling effect on society and a powerful form of control.

This is in direct relation with the notion of lurkers, a silent majority of passive users that is said to constitute 99% of the internet, according to the 1% rule, and that doesn’t participate in the creative aspect.
The vast majority of people only consumes and doesn’t contribute any content. This participation inequality has been researched, verified, and shown to be consistent on different online forums and wiki pages such as Wikipedia and others.

There are multiple reasons why most stay on the periphery as read-only users. Social cooling plays a big part in it and the studies make it clear.
When asked why people only observed and did not participate, the common reply is that they were gathering cultural capital: the knowledge that enables an individual to interpret various cultural codes. We take the time to evaluate the implicit and explicit norms of the group, to learn if we can fit, to understand the preferred method of interaction without making mistakes and avoid being rejected, to see the topics of conversations that are allowed, and to learn the conventions of the community. This is commonly done by finding the typical figures representing the community, the models to adopt, templates that exhibit the kind of dialogues we can have.
People also lurk because they fear being redundant or that their contribution would not be worthy. By lurking they can glimpse at what has already been said and done.
We want to copy, belong, but also be unique, that’s something we’ve already seen in the biases section. Maybe it’s because our human experience is so similar and yet distinct.

Lastly, people stay afar because they feel that browsing is enough for them. That could be simply because they are looking to find particular content from a place and nothing else.

On the psychological side, lurking is associated with loneliness and apathy, a sense of not belonging. Lurkers are found to be less satisfied and experience more distractions, they are distant.
In the case of social networking, they experience less intimacy and feeling of personal well-being, and an overwhelming loneliness as they watch others participate in the community.
Social media does indeed feel cognitively similar to amplified real life interaction.
This is akin to a TV audience but with the sense that what we see is real and not a fabricated reality.

What makes someone de-lurk is situational and depends on both the individual’s personality, the environment, and the community. What remains is that only a handful of superusers generate the vast majority of content.

Research shows that these contributors are highly motivated, feel a sense of duty, attachment, and belonging to their online community. Other research show that active participants seemed to be more extroverted and open, felt they were more empowered and in control of their environment, had confidence in their ability to influence, had higher self-efficacy, and a greater need for gratification.
That is in sync with what we’ve seen regarding frustrated individuals that had their cultural malaise and gaps filled by the internet. Participants need a place to express themselves and that is relevant to them.

Statistics show that the people contributing the most are out of the ordinary, they exhibit extraordinary behavior. That means that when we consume any content on the internet, we’re mostly consuming content created by people who, for some reason, spend most of their time and energy creating this content. These people clearly differ from the general population in important ways. The kind of people that move the Overton window, which we talked about.
This is worth keeping in mind when on the internet.

Knapp, had been submitting an average of 385 edits per day since signing up in 2005 as of 2012. Assuming he doesn’t sleep or eat or anything else (currently my favored prediction), that’s still one edit every four minutes. He hasn’t slowed down either; he hit his one millionth edit after seven years of editing and is nearing his two millionth now at 13 years. This man has been editing a Wikipedia article every four minutes for 13 years. He is insane, and he has had a huge impact on what you and I read every day when we need more information about literally anything. And there are more like him; there is one user with 2.7 million edits and many others with more than one million.

These days the 1% rule has been more or less debunked as the barrier to engagement has decreased. However, it hasn’t decrease enough, the number of lurkers is still relatively high and social cooling adds to the equation.

This paradoxical balance between lurkers, conformity, social cooling, and out-of-the-norm influencers and contributors in different communities, makes us feel a sense of normalcy and lack of innovation.

Normalcy is the consequence of adherence and conformity. We lurk because we want to adhere to the norm, not stand out, to fit in. We’re also interested in participating because of the homophily phenomenon we’ve discussed before, “birds of feather flock together”.
This all keeps the status quo and homogeneity in the community.

This creates a situation in which we’re offered templatized, cookie-cutter identities which we can choose from, and indirectly coerced to fit into them by the persuasive technology. It’s easier to market to persons that fit into well-separated categories, to “buyer personas”. One could even call these memeplexes and memeoids, honed through eunemics.
This is all decided by fads, opinions, and interests, a sort of “choose your character/future” mode of identity building. With these templates there’s an emphasis on attitude rather than facts, truth, or moral. Sometime called “Hypernormalization”.
“How can you have a personality if you have no knowledge to base it upon, and if you merely have opinions that have been given to you through slogans or clichés.”

It is hard to get out of a template, these mass one-sided personalities, because of all that we’ve seen before especially the social judgement, social cooling, the action-attention-reward addictive mechanism, and others. We need privacy as a right to not fit in, a right not to be perfectly molded, well-behaved, docile, and conformed humans.
We are guided by biased algorithms that are mathwashed, believing they are impartial.

This has detrimental effects on kids that are in the process of finding themselves, in the process of building their personality. They feel enormous social pressure to be perfect. Self-inflicted injuries, such as cuttings that are serious enough to require treatment in an emergency room, in the USA, have increased dramatically in 10- to 14-year-old girls, up 19% per year since 2009.
Less physically harmful, is the effect on young boys that are looking for achievements and get addicted to “competency porn”, any type of media that fulfills the need for recognition, achievement, and control over the environment — Be it video games or animated movies. The epic tales and escapist fantasies of today’s world.

It is astonishing because different persons are part of different internet realms, with different templates, and are given skewed pictures of reality — it’s as if everyone lived in alternate universes that sometimes, but rarely, cross.

Indeed, even though the UN considers the right to internet access a human right, the effect of the globalisation of the internet are unclear.
Already we’ve seen filter bubbles, homophily, and confirmation bias that keep people within the same realms. Additionally, we have to add to this the privileges associated with the internet. Not everyone can access the internet, not everyone can use it powerfully, not everyone can participate in community content creation, and not everyone has the language skills to participate in these various communities.
The content is still subject to being part of a hierarchy, still based on the same social and international hierarchies that exist today. Minority views in the real world are still minority views online, the same people facing discriminations offline also tend to face them on the internet.
And yet, it also fosters the democratization of speech if used properly, but the content we are given somehow reinforces the status quo.

It’s a global and local presence, we could say a glocalization. Everyone that has internet uses it in their own way, and the way they use it affects their perception. Some use the internet for research, some for hobbies, some to socialize, some to learn, some as a tool for work, etc.. Internet usage is shaped by the myriads of ways that we access it for our needs and values.
Yet, this exact habit of usage is shaped by the local culture and thus also creates a divide as the internet is inherently based on connecting networks together.

For example, even though we can use the internet for different things, the language we speak will only give us a narrow view of the whole spectrum. If we search using a language that is more “popular”, let’s say English, we’re prone to be given the templatized identities related to the most vociferous online English speakers (USA).
Language then is part of the dynamic of who defines and who is defined.
There are other imbalances that exist based on geolocal preferences. For instance, speakers of a certain language will prefer domestic issues.

Moreover, advertisers, the main source of revenue online, will prioritize specific consumers that fit a template from a certain culture and geographical area, and not others. This will incentivize content creation for that specific audience, as this is where more money is made. This in turn will widen the cultural gap, now a digital advertising gap, thus a media content quantity and quality gap.

There are many factors that make the internet both global and local at the same time. Again, it depends on our usage of it but when we default to the average person, there’s no room around being presented only narrow views.
The question remains whether internet truly makes people conform, what’s the state of creativity and counterculture, and how much propaganda comes into play.

The literature shows that a small subset of the population, around 2 to 3%, have international ties and act as bridge figures between cultures. These persons show a higher creativity than their peers because of their exposure to different views.
From this we can imagine that it’s the same thing on the internet.

When using a service you become part of its ecosystem, there are many platforms and the big ones, the FAANG, have a tech hegemony. Internet in itself didn’t kill counterculture, it is just not present on these platforms as they encompass a sort of new overarching powerful entity, sometimes more organized than governments.

This means that counterculture today isn’t any more about a war of ideologies but about going against the trends of the internet. The counterculture of the past, which rotated around personal expressions and identity are now used by platforms to drive engagement and profit, it’s part of their lucrative business. Social media in particular have been catering to the “demands” of a generation by filling the gap that the culture couldn’t — self-fulfillment as a product, as we’ve seen earlier.
This means that these types of old countercultures are still part of the system, but by embodying them we’re not in charge. We pull ourselves together by gathering pieces of templatized subcultures. Then, this collection becomes our niche personal branding, our personal expressions and ideologies.

As we went over in part 2 section 2, we are now seeking individuation, we are networked individuals connected by interests. This is in stark contrast with counterculture which requires a group to thrive, an “us against the world” mindset. The internet does allow regrouping but only on a shallow level, particularly for collective dissent and unsatisfaction, turning the groups to absurdities and cult-like behavior.
Directly opposing the system will only make the system stronger.

True counter-culture can either be found by embracing the algorithms, understanding their inner-working and warping them to our needs, or by signing-off and rejecting the whole system for an alternative one. Both of these are difficult in the era of like-and-share.

So we are bound to use the internet, and the internet to project our anthropological nature, its current state and flaws, be it conformity, homogeneity, or others.
We’ll be given a reflection according to the function it has for us, our lifestyle, current society, culture, and time. For a big part of the world that means the internet is becoming inherently neoliberal, used for neoliberal ends, and encourages neoliberal behaviors.

This isn’t surprising considering the origin of the internet, who are the biggest internet companies, and who are the most vocal online. The internet embodies the “American mindset”, a mindset of personal brands and marketing, the self-made man.
A considerable amount of people are driven by this, with different subset cultural bubbles living outside the hegemony of these giants because of different environmental constraints.

The internet is the perfect tool for the workings and reinforcement of neoliberalism values. It is a perfect display of how this way of viewing the world, which was heavily questioned during the Cold War, has taken over.

It focuses on a certain flavor of democracy, individualism, and a laissez-faire absolutism of the marketplace. The belief and undeserved faith in the efficiency of markets. It centers around deregulation, free trade, privatization, and puts the individual at the center of everything. Any other attitude, and especially government intervention, is frowned upon.
We can clearly see how this has unfolded on the internet, with Facebook’s many scandals clarifying the limits of corporate self-regulation.

There’s an over-emphasis on economic efficiency and growth at all costs. Political theorist Wendy Brown has gone even further and asserted that the overriding objective of neoliberalism is “the economization of all features of life”.

The self becomes a product, a personified capital that needs to accumulate symbols of status, a storyline, and advertise itself. With regards to the internet, and because the money is where advertisers and the audience put their interests, a personal-brand will only be profitable if it fits the mold of the popular culture.
We have to partner with the brands and products that make us shine. We own things that reflect our personal meaning. We calculate the benefits, efficiency, and productivity of each action. We had a look at this in the new economies section, this is in relation to this. The commodification of personal data and attention for profit.

This unfettered exploitation puts gains before anything and it doesn’t only apply to individuals but especially to corporations. That is why marketing is prioritized on the internet, an optimization of products and surveillance. We de-emphasize public goods to turn our eyes towards the market idolization, a corporatocracy.

Everything has a monetary incentive rather than a social objective. Anything needs a metric, an indicator, to be tracked and quantifiable. This focus on economic efficiency can compromise other, perhaps more important, factors, or promote exploitation and social injustice. We see this on the personal level on social media through likes, shares, and subscribers.

It’s arguable if neoliberalism, the advent and rise of this way of life, are bi-products of the internet or if it’s the opposite. It’s arguable if individuation is the product of neoliberalisation or vice-versa, and if they are accelerated by the internet.
It’s probably a mix of everything, societal changes with the private enterprises reflecting the will of individuation of private individuals, which are also projected on their internet — the internet driven by the big players.

We’ll see later how these societal change and acceleration bring social unrest.

The individuals are at the heart, all the responsibility and weight are on them, that’s why they focus so much on identity politics and individual gains, while dismissing broader identities and class consciousness.
A mix of looking for authenticity, as it seems to be the gap in cultures that have embraced neoliberalism, and an overwhelming majority of narcissistic individuals.
A mix of losing political freedom because of apathy, and looking for self-fulfillment as withdrawal from moral concerns. Are lurkers neoliberal? Are counter-culture and creativity now anti-neoliberalism?

Globalisation doesn’t necessary mean homogenisation, but it’s undeniable that the neoliberal mindset has spread further than any other and can’t be dismissed. Its acceptance, normalization, the incentives, indirect influences on the internet and its dynamics are apparent.
Isn’t this the best propaganda, one we don’t even notice unless we’re someone in the middle, someone cross-cultural?

Nonetheless, what happens online is only a function of the offline status quo, the offline paralysis. The internet does not exist in a vacuum. It is made with the purpose of letting people connect with their own networks, assuming that users are individuals, and inherently individualising them.
The online world recreates and accelerates — there is nothing inherent within the technology that makes it neoliberal, it’s only a tool. Each can have their own different internet experience depending on how they use it, as we kept saying.
We’re both formatted and free individuals — freedom as a menu.

This concludes our review of how the internet has consequences in the acceleration of cultural evolution, formatting individuals, paralyzing them, imposing an overwhelming homogenisation while letting fringe people contribute, and fostering and spreading the neoliberal mindset.
At first, we’ve examined the cognitive effects: attention, memory, and social cognition. Our attention is divided, our memory externalized, and our social fear amplified. Then we’ve seen how the internet makes us constantly anxious of making indelible mistakes, embodied in the concept of social cooling and self-censorship.
Next, we’ve observed the link with the notion of lurkers, read-only users, the 1% rule, and the out-of-the-ordinary users that actually contribute the content. These contributors do so because the internet has filled something they were missing, a cultural gap.
After this we’ve questioned conformity and homogeneity, how the internet offers template we select from, and puts pressure on individuals to fit perfectly. Yet, we’ve also observed that these templates, these experiences, and the internet function as a whole, differ and are divided per culture, environment, language, and others — the glocalization of the internet.
Later, we’ve added to this the fact that the internet is driven by monetary incentives and that it will create a dynamic in which more content is targeted at people that fit certain templates and others will be enticed to fill the templates of other cultures, an indirect cultural assimilation.
Following this we’ve discussed what it would mean to stand out of these templates, of all these memeplexes, to be more creative or counter-cultural and we’ve concluded it means either embracing the tech or going against it. Finally, we’ve linked the topic with neoliberalisation and how it exemplifies the phenomenon we’re seeing, both by emphasizing the market, efficiency, and the individual.

Table Of Content


Attributions: Parmasayika grid

April 14, 2021

Gustaf Erikson (gerikson)

The Pacific War Trilogy by Ian W. Toll April 14, 2021 01:23 PM

  • Pacific Crucible: War at Sea in the Pacific, 1941–1942
  • The Conquering Tide: War in the Pacific Islands, 1942–1944
  • Twilight of the Gods: War in the Western Pacific, 1944–1945

An excellent and readable account of the (US) war in the Pacific against Japan in World War 2. Highly recommended.

Marc Brooker (mjb)

Redundant against what? April 14, 2021 12:00 AM

Redundant against what?

Threat modeling thinking to distributed systems.

There's basically one fundamental reason that distributed systems can achieve better availability than single-box systems: redundancy. The software, state, and other things needed to run a system are present in multiple places. When one of those places fails, the others can take over. This applies to replicated databases, load-balanced stateless systems, serverless systems, and almost all other common distributed patterns.

One problem with redundancy is that it adds complexity, which may reduce availability. Another problem, and the one that people tend to miss the most, is that redundancy isn't one thing. Like security, redundancy is a single word that we mean that our architectures and systems are resistant to different kinds of failures. That can mean infrastructure failures, where redundancy could mean multiple machines, multiple racks, multiple datacenters or even multiple continents. It can mean software failures, where common techniques like canary deployments help systems to be redundant when one software version failures. I can also mean logical failures, where we recognize that state can affect the performance or availability of our system, and we try ensure that the same state doesn't go to every host. Sometimes that state is configuration, sometimes it's stored data or requests and responses.

An Example

Unfortunately, when we talk about system designs, we tend to forget these multiple definitions of redundancy and instead just focus on infrastructure. To show why this matters, let's explore an example.

Event logs are rightfully a popular way to build large-scale systems. In these kinds of systems there's an ordered log which all changes (writes) flows through, and the changes are then applied to some systems that hang off the log. That could be read copies of the data, workflow systems taking action on the changes, and so on. In the simple version of this pattern one thing is true: every host in the log, and every consumer, sees the same changes in the same order.

Event bus architecture, with three replicas hanging off the bus

One advantage of this architecture is that it can offer a lot of redundancy against infrastructure failures. Common event log systems (like Kafka) can easily handle the failure of a single host. Surviving the failure of a single replica is also easy, because the architecture makes it very easy to keep multiple replicas in sync.

Event bus architecture, with three replicas hanging off the bus, with host failures

Now, consider the case where one of the events that comes down the log is a poison pill. This simply means that the consumers don't know how to process it. Maybe it says something that's illegal ("I can't decrement this unsigned 0!"), or doesn't make sense ("what's this data in column X? I've never heard of column X!"). Maybe it says something that only makes sense in a future, or past, version of the software. When faced with a poison pill, replicas have basically two options: ignore it, or stop.

Event bus architecture, with three replicas hanging off the bus, with logical failures

Ignoring it could lead to data loss, and stopping leads to writes being unavailable. Nobody wins. The problem here is a lack of redundancy: running the same (deterministic) software on the same state is going to have the same bad outcome every time.

More Generally

This problem doesn't only apply to event log architectures. Replicated state machines, famously, suffer from the same problem. So does primary/backup replication. It's not a problem with one architecture, but a problem with distributed systems designs in general. As you design systems, it's worth asking the question about what you're getting from your redundancy, and what failures it protects you against. In some sense, this is the same kind of thinking that security folks use when they do threat modeling:

Threat modeling answers questions like “Where am I most vulnerable to attack?”, “What are the most relevant threats?”, and “What do I need to do to safeguard against these threats?”.

A few years ago, I experimented with building a threat modeling framework for distributed system designs, called CALISTO, but I never found something I loved. I do love the way of thinking, though. "What failures am I vulnerable to?", "Which are the most relevant failures?", "What do I need to do to safeguard against those failures?"

If your answer to "What failures am I vulnerable to?" doesn't include software bugs, you're more optimistic than me.

April 13, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — Biases & Self: Cultural Ambiguities & Insecurities April 13, 2021 09:00 PM

The Sufi-infuenced teachings were aimed at destroying man's illusory sel-image, and revealing him as being guided by mechanical reflexes

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 3 — Biases & Self
  • Section 2 — Cultural Ambiguities & Insecurities
Table Of Content
  • Constrained By Platforms Or Platforms Constrained
  • Beliefs and Values
  • Repressed Cultural Weaknesses
  • Polarization As A Natural Phenomenon
  • Primal Needs
  • True Believers

To be a netizen means to be part of the online social sphere. There’s no way around it, to have a voice and participate people have to join a platform, which comes with its own limitations.
The rules of the platforms are the rules of the information society but the platforms adapt more to fit us than we adapt to them. Anything happening on them is directly because of real people interacting together. People that have their own hopes, emotions, values, prejudices, and beliefs. Consequently, through our own cultural differences, ambiguities, and insecurities, we are indirectly manipulating ourselves.

We look for confirmations of our own experiences, as we said in the previous section, confirmation bias. We want to share and search for things that relate to our local lives, to make us look smart, empathic, cast us in a positive light, or that is useful in our day-to-day lives.
On the internet this is proved through many experiments. Emily Falk’s Lab, Director of the University of Pennsylvania’s Communication Neuroscience Lab, has demonstrated how the expectation of social confirmation and reward influence the likelihood of someone sharing a meme with someone else. We share for social comparison, to look favorable to other group members, for “virtue signalling”.
Similarly, according to research, on average people visit a maximum of 30 websites before making their mind about something. The bulk of these websites are part of the top results of the search engines of which 70% will always support the view portrayed by how the query was formulated. This is what we refer to as an echo chamber and filter bubble, it echoes back what we think.
We seek information sources that support and reinforce existing attitudes or beliefs, as well as process them. “All else [being] equal, people seem to prefer not changing their opinions to changing them” (Lebo & Cassino, 2007, p. 722).

We increasingly, inadvertently through selective exposure by our own judgement, are localized, not only geographically but also through our language, ideologies, time, and cultures. As we said earlier, the internet is made of networked separated individuals, linked by their interests and similarities.
This isn’t only about filter bubbles created through algorithms, but it’s our natural tendency to pay attention to those who have a lot in common with us. We lack cognitive diversity.
We call this phenomenon homophily, “birds of feather flock together”. We’re really good at it, we have this inner urge to find people similar to us, and this is confirmed by over 100 studies.
This isn’t necessarily negative, this allows us to create communities based on niches, on interests, be more democratic, etc.. All things we’ve seen in part 2 section 2.

We all think that the way we live is the best way to live. We all think that we see the world as it is, that we’ve come to the best possible conclusion about our direction in life. However, that’s just our definition of normality. And this normality isn’t even ours as most of it happens without us being there. We often can’t justify why we do what we do or why we like what we like.

This is as much as about externalities as internalities. We associate ourselves with others like us, as a way of expressing our own individuality by proxy.
We said people spent more time viewing and sharing memes that confirmed their views with others, not only because it confirms them, but also because it’s a way of expressing one-self and indirectly promoting who we are. We went over this in the meme section, memes are a representation of our mental models, how we comprehend and make sense of the world.

Ideologies are never force-fed, we are deliberately looking for them. Nobody can be persuaded if they don’t want to be persuaded.
That is why when we discussed propaganda we said that the best messages are those that resonate with the audience, that aren’t perceived as propaganda. Not imposed, but emerging from themselves and expressing the concerns, tensions, aspirations, and hopes of the persons.

We live through our media, they define us, we identify with them. That is why the interactions on the internet are a reflection of our different cultures.
Everyone has an opinion, however, what is worrying is when they give rise to consequential beliefs. When they manifest in the real world.

On social media, these opinions are now all expressed and openly visible. The pristine walls protecting us have fallen and we are faced with a messy world full of ambiguities.

This is increasingly true when netizens use persona, and even more when they use hide under an anonymous pseudonym. People are then free to express themselves without restraints, breaking accepted cultural codes, expectations, unspoken tensions, and preconceived notions. They aren’t burdened by what their peers think of them.

However, we are all biased, as we saw in the previous section, and this confluence of voices brings uncertainty to our own. Doubt sets in when many claim the contrary of something widely accepted. It’s also hard for us to accept coincidence as an explanation, which gives rise to some shouting their conspiratorial thinking. Trust is easy to lose and hard to gain.
A trust crisis is setting in.

On social media everyone has an equal voice and different views. We said this was moving us towards a low-context society or informational society. We’ve also just seen the tendency to regroup in bubbles with people and content that is in agreement with our current views.
Yet, research tells us that social media users have a more diverse news diet than non-users. It shows how much we are bound to encounter opinions and information that might upset us because they go in the opposite direction of our preconceived notions. The bubbles are bursting against each others, and this implicitly creates hostility.

As humans, we tend to connect ideas in a coherent way, to make sense of the world around us based on our culture and routine. We have expectations of what is true, know historical facts set by our institutions, build a narrative from the events we are told, hold public opinions, have pictures of what is normal or not, assumptions about how structures fall into place, and which established powers we think is better.
Yet, again, reality isn’t sterile and spotless, these preconceptions are but a thin layer holding a story, and it only covers an infinitely small portion of the existence on this planet. Cultures often have unspoken tensions, anxieties, weaknesses — no story is perfect. This is felt even more heavily when they clash on the internet.
A truth crisis setting in.

These contradictions need to be accepted and that might generate cognitive dissonance in some people. Cognitive dissonance is a word used to describe the effects someone feels when holding contradictory beliefs, ideas, or values and it’s denoted by psychological stress. The human-psyche wants to be consistent, otherwise it creates discomfort. We feel an urge to resolve these contradictions.

Then it’s no wonder that some content go viral, it’s because they’re unleashing a repressed cultural agenda. Online discourse promotes outrage because we feel the need to defend or attack our positions so that our world view doesn’t shatter — to keep our own consistency or to find it again.
If we didn’t have issues in our societies, these extreme forms of discussions wouldn’t materialize. Thus, it is reductive to point the finger at just the social media platforms, that’s why we’re pointing it at ourselves here.

When facing an unclear situation, a tension, a question, or decision, we instinctively take stances. Naturally we either are with or against. We put ourselves in camps. At least, the most prominent and outspoken people take sides, so on the internet these are the ones whose voice break through the noise.
That person will be exposed to a disproportionately large amount of info similar to their view and a small amount of info against their view. Naturally, they’ll want to react the contrarian viewpoints to defend what they think is right.

After that dance, opinions will be polarized and the diversity of point of views and nuances will disappear — the system will have reached its equilibrium.
Absolutist thinking is in human nature when there is an absence of certainty. We want to grasp for reassurance.

The researcher Cass Sunstein has shown that this is even more apparent when putting people that have similar mindsets in the same room. They divide themselves according to more polarized version of the same mindset.
Initially the group shows a lot of internal disagreement over issues but then, again, fall unto an equilibrium. People held more-extreme positions after speaking with like-minded individuals and were more homogeneous, squelching diversity. Either an all or nothing, a black and white vision, highly moralistic, with expectations on us and others.
Similar studies show the same results and adds that people have a less negative perception of opposite views if they get exposed to them more often, instead of mainly the ones confirming their own.

We’ve seen that people self-sort and seek confirmating views online, this widens the rifts between different points. The internet makes it exceedingly easy for people to move into extreme absolute positions and versions of whatever is discussed.

The internet, and particularly social media, also fills the voids that our societies cannot. Gaps in our primal needs: the need for certainty, the need for identity, the need for belonging, the need for accomplishment, competency, and approval, or higher spiritual and moral callings. All things that are missing in contemporary civilizations according to Charles Taylor.

Our lack of certainty about fundamental assumptions and loss of control inclines us to the middle but opinions on extremes look steady and we slowly move towards them — a need for closure, narrowing our perception.
We generally adopt the first belief that fills the gap, which is given to us by the flashy promotional culture or fringe ones.
These fillings do not have to be understood but believable, to look consistent and absolutely certain. The quality of the ideas themselves play a minor role but the meaning and confidence that they bring matters.

The persons the most sensitive are the ones who are bored, feel powerless, have insecurities in their culture, feel uneasy about the ambiguity of the world, and are frustrated. A feeling of anomie.
They are driven primarily by hope for a coherent future, their gaps filled by the ideology, being consoled and rewarded for taking part in it.
They go under the name of fanatics, highly motivated individuals, defenders of culture, true believers, memeoids, or hyper partisans.

The transformation or indoctrination can happen slowly over time, through repetition and a slippery slope, as we said, reaching a target level of saturation creating a full universe of meanings. Then stuck in a feedback-loop.

Undeniably, micro-targeting can be used to find these individuals which needs aren’t filled by society, to then turn them into true believers. Monitoring posts can give us insights on whether someone feels stressed, defeated, overwhelmed, anxious, nervous, stupid, useless, and a failure.
It’s not surprising that this is going to be used because hyper partisans are the most motivated, they are driven by their ideology and occupy a bigger space in the information market. This is a sort of tyranny of the minority, which drags people that are undecided.

Sarah Golding, the president of the Institute of Practitioners in Advertising, says “It has essentially weaponized ad technology designed for consumer products and services. There is a danger that every single person can get their own concerns played back to them”.

Cognitive dissonance and uncertainty open a window of opportunity to hack the mind and bring these individuals in.
The memeoids are known to self-isolate into fringe communities. Ironically, statistically these communities, with their extreme viewpoints, are the most influential on the internet. This is all unsurprising because they offer certainty and answers for the lost netizens.

Additionally, these communities always have gatekeepers and look enticing. They isolate the members and shun anyone that has a different opinion. The stories are reinforced through time. This reminds us that certain communities on the internet are no different from cults, and of the reasons why people join them.

This concludes our review of cultural angst and how they are reflected on the internet. First of all, we’ve talked about how the internet has its limitations but is more shaped by us than we are shaped by it. We’ve then dabbled with the concepts of beliefs and values which always get confirmed through the results given by recommendation/curation engines and other algorithms. Next, we’ve seen how when cultures clash they make us feel discomfort and bring to the surface cultural anxieties and weaknesses, thus we react. After that, we’ve talked about how polarization is natural, how we instinctively take sides to become more homogeneous. On the internet this creates rifts between ideas, people want to take sides. Later, we’ve covered how the internet can fill needs that aren’t filled by societies, be it a need for closure, accomplishment, belonging, or approval. Finally, we’ve looked at hyper partisanship, how people get stuck in these communities, and why they are the most vociferous online.

Table Of Content


Attributions: Alexander de Salzmann, cover-design for the programme of the “Institute for the Harmonic Development of Man”, Tiflis, 1919

April 11, 2021

Ponylang (SeanTAllen)

Last Week in Pony - April 11, 2021 April 11, 2021 11:47 PM

New versions of the ponylang crypto, http, http_server, and net_ssl packages are available.

Derek Jones (derek-jones)

Pricing by quantity of source code April 11, 2021 09:45 PM

Software tool vendors have traditionally licensed their software on a per-seat basis, e.g., the cost increases with the number of concurrent users. Per-seat licensing works well when there is substantial user interaction, because the usage time is long enough for concurrent usage to build up. When a tool can be run non-interactively in the cloud, its use is effectively instantaneous. For instance, a tool that checks source code for suspicious constructs. Charging by lines of code processed is a pricing model used by some tool vendors.

Charging by lines of code processed creates an incentive to reduce the number of lines. This incentive was once very common, when screens supporting 24 lines of 80 characters were considered a luxury, or the BASIC interpreter limited programs to 1023 lines, or a hobby computer used a TV for its screen (a ‘tiny’ CRT screen, not a big flat one).

It’s easy enough to splice adjacent lines together, and halve the cost. Well, ease of splicing depends on programming language; various edge cases have to be handled (somebody is bound to write a tool that does a good job).

How does the tool vendor respond to a (potential) halving of their revenue?

Blindly splicing pairs of lines creates some easily detectable patterns in the generated source. In fact, some of these patterns are likely to be flagged as suspicious, e.g., if (x) a=1;b=2; (did the developer forget to bracket the two statements with { }).

The plot below shows the number of lines in gcc 2.95 containing a given number of characters (left, including indentation), and the same count after even-numbered lines (with leading whitespace removed) have been appended to odd-numbered lines (code+data, this version of gcc was using in my C book):

North Star Horizon with cover removed.

The obvious change is the introduction of a third straight’ish line segment (the increase in the offset of the sharp decline might be explained away as a consequence of developers using wider windows). By only slicing the ‘right’ pairs of lines together, the obvious patterns won’t be present.

Using lines of codes for pricing has the advantage of being easy to explain to management, the people who sign off the expense, who might not know much about source code. There are other metrics that are much harder for developers to game. Counting tokens is the obvious one, but has developer perception issues: Brackets, both round and curly. In the grand scheme of things, the use/non-use of brackets where they are optional has a minor impact on the token count, but brackets have an oversized presence in developer’s psyche.

Counting identifiers avoids the brackets issue, along with other developer perceptions associated with punctuation tokens, e.g., a null statement in an else arm.

If the amount charged is low enough, social pressure comes into play. Would you want to work for a company that penny pinches to save such a small amount of money?

As a former tool vendor, I’m strongly in favour of tool vendors making a healthy profit.

Creating an effective static analysis requires paying lots of attention to lots of details, which is very time-consuming. There are lots of not particularly good Open source tools out there; the implementers did all the interesting stuff, and then moved on. I know of several groups who got together to build tools for Java when it started to take-off in the mid-90s. When they went to market, they quickly found out that Java developers expected their tools to be free, and would not pay for claimed better versions. By making good enough Java tools freely available, Sun killed the commercial market for sales of Java tools (some companies used their own tools as a unique component of their consulting or service offerings).

Could vendors charge by the number of problems found in the code? This would create an incentive for them to report trivial issues, or be overly pessimistic about flagging issues that could occur (rather than will occur).

Why try selling a tool, why not offer a service selling issues found in code?

Back in the day a living could be made by offering a go-faster service, i.e., turn up at a company and reduce the usage cost of a company’s applications, or reducing the turn-around time (e.g., getting the daily management numbers to appear in less than 24-hours). This was back when mainframes ruled the computing world, and usage costs could be eye-watering.

Some companies offer bug-bounties to the first person reporting a serious vulnerability. These public offers are only viable when the source is publicly available.

There are companies who offer a code review service. Having people review code is very expensive; tools are good at finding certain kinds of problem, and investing in tools makes sense for companies looking to reduce review turn-around time, along with checking for more issues.

April 10, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — Biases & Self: Cognitive Biases April 10, 2021 09:00 PM

The art of memory is like an inner writing. Those who know the letters of the alphabet can write down what is dictated to them and read out what they have written

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 3 — Biases & Self
  • Section 1 — Cognitive Biases
Table Of Content
  • Biases Aren’t Long Term
  • Who
  • Physical
  • Biases Related To Visibility
  • Biases Related To Information Availability
  • Biases Related To Social Pressure

Humans are prone to manipulations by their own experiences and biases. Let’s review what are some of the cognitive biases and cultural weaknesses that make us fall for all sort of things on the internet.
We’ve already looked at coercion, deception, and other types of persuasion principles in a previous section. In the following two we’ll emphasize on ourselves instead of external factors.

There are many cognitive biases that have been documented over the years, this is nothing new. However, it’s interesting to see them under the light of this medium, how they come into play on the internet.
I’ll try to list the ones that are the most prominent in my opinion.

In general, biases can affect us when it comes to beliefs, decision-making, reasoning, economic decisions, and behavior. Yet, biases don’t have a significant influence on the long run, at least the ones that aren’t inherently connected to our cultures. We’ll dive into culture in the next section.
Cognitive biases are temporary tricks, temporary stimulations that hack our brain. When someone is aware that another person has used such bias against them, they know they have been cheated. Regardless, studies show that even when trying to influence subliminally, without the audience realising, these types of stimulations only have negligible effects.

In most cases, a bias isn’t something that someone creates but something we all naturally have and that is exposed. As such, we can all have them, different persons having different ones under different circumstances. Kids in particular are more vulnerable to them.

Before diving into cognitive biases, let’s stop to mention some physical reactions.
In part 2, section 2, we’ve mentioned social cognition and how communicating on the internet had the same bodily-effects and cognitive responses as other types of social interactions. This implies, as research shows, the same status-to-hormone correlation: stress with cortisol and social confirmation/success with oxytocin, endorphin, and dopamine.
Practically, in the social media environment, which works with interfaces showing quick metrics of social success or failure, that means people’s Action-Attention-Reward system will lead them towards addiction. Social media acting like slot-machines — classic conditioning — and keep us hooked.

We all react similarly to these hormones; biased to associate pleasant and positive feelings with what triggers some, and associate negative feelings with others. Hence, we are incentivized to repeat the same behaviors that once got us a dopamine hit — be it finding a niche of expertise we cater to, or posting and finding something shocking.

A market in which everybody has the right to speak comes with the biases of everyone in it.

Some of the biases we can have on social media are in relation to the control over visibility. Biases such as attentional bias, salience bias, framing effect, mere exposure effect, anchoring effect, and others.
Attentional bias is the tendency to pay attention to what is more recent in memory. Similarly, salience bias is about paying attention to what is more remarkable and emotionally striking. The anchor bias is a similar bias, fixating on a single part of a story. In the attention economy we definitely pay attention to what is more extraordinary and recent, and we’ll be shown these stories more often.
Mere exposure effect is about being influenced to like something because we are familiar with them, just by being exposed more frequently. Framing is similar, it’s about extracting different conclusions from the same story depending on how it’s presented. What we are exposed to on the internet is selected by the best buyers, and by our personal interests and stubbornness.

These are compounded into something called continued influence effect, which is a bias related to how hard it is to correct misinformation in our memory after having seen it the first time.
Moreover, our visibility is also affected by biases that make us seek certain things more than others. The automation bias, on the internet, is a tendency to rely too much on the algorithms for recommendations. We know these algorithms are not impartial and can be biased.
While the well-known confirmation bias, illusion of validity, optimism and pessimism biases, are all related to how much we chase, interpret, remember, and focus on information that is consistent and related to what we already believe. We like to stay in our bubble, we are our own filters of visibility with our biases. We are especially interested in reading and sharing content that connects to our own experiences and senses. These are biases that play a major role with what we’ll see in the next section, namely cultural insecurities.

Other biases are in relation to the over-availability of information and how it relates to our memory.
One such bias is the information bias, which is a tendency to look and search for information even though this information is meaningless to us, and cannot affect our actions.
The quest for information is also related to something we’ve seen in a previous section regarding the extension of our memory. The Google effect is a bias in which we are more likely to forget information if we know that it is readily available online.
Additionally, another popular bias under the name of Dunning-Kruger effect is the tendency of unskilled people to overestimate their own skills. With the whole internet acting as an external memory, it seems like more people are having this bias.
A bias that goes in the opposite direction is the imposter syndrome in which someone with enough experience feels they aren’t adequately qualified and fraudulent. With the whole internet pushing forward images of successful and perfect individuals, it seems like some people are more prone to this bias.

The over-availability of information itself can also lead to biases that makes it bypass certain of our defenses. As far as this is concerned, internet memes are an interesting case to examine.

The humor effect is a tendency to more easily remember things that are funny because they both capture our attention, stand out in a flood of information, and make us feel good. Humorous content also takes less time to process cognitively.
The picture superiority effect is the notion that pictures are remembered more easily and frequently than any other format. We’ve seen that one in the meme section and also said that pictures are processed more rapidly, 60 thousand times faster than text.
There are two ways to process information, through fast thinking or slow thinking, and memes harness speed to skip any form of slow thinking.

These effects, along with content that somewhat goes along with our preconceived views of the world, mean that visual internet memes will bypass criticism, go straight through a peripheral route, and lodge themselves in the brain, to be memorable.

Let’s now go briefly over some social biases, which will be helpful for us when tackling the cultural perspective in the next section.
Some of them are related to conformity and others to individuality in a social context. Online, as we now know, social interactions are mostly the same as in the real world, plus instant feedback, big exposure, and a sprinkle of anonymity and fabricated opinions.

When it comes to individuality, the biases of people on the internet are due to the constant exposure they have. For example, the spotlight effect is the tendency to overestimate the amount that others will pay attention to us.
Another bias, in relation, is the reactance: the urge to do the opposite of what others would want us to do, feeling a need to stand out and to assert freedom. It’s very common to see this contrarian behavior on social media.
The social comparison bias, yet again related, is about favoring candidates for a position that don’t compete with our own. We always compete on the internet, social media profiles are like public CVs and so this bias comes into play.

Other social biases tend to be linked to conformity such as the authority bias, the bandwagon effect, groupthink, social comparison bias, and in-group and out-group biases.
Authority bias is the tendency to trust and be more influenced by something or someone based on their origin and perception. We’ve seen earlier how state actors can abuse this to trade-up-the-chain to reach trusted mainstream media.
The bandwagon effect, groupthink, herd behavior, and in-group biases are all linked to preferential treatment to our own group, and keeping harmony, consensus, and conformity in it.
On the other side, there’s out-group homogeneity bias where we see members of an outside group as relatively similar. Confusingly, this bias also implies that the members of our own group are dissimilar.

Some people are more inclined to social biases than others. Some will recall images better than others. Distinctive people develop or react to biases in their own ways, be it because of their gender, sex, context, environment, geographical location, professional experiences, life experiences, education, etc..

This concludes our review of cognitive biases and how they get reflected on the internet. We’ve first seen some of the physical aspects of online addiction. Then we’ve covered three broad categories of biases: some related to the visibility, others to the information availability, and finally some related to social pressure. The biases related to visibility are about our fixation on what we see more frequently. The biases related to information availability are about what information we seek and how it affects our memory or confidence. Finally, the social biases are about how we conform to a group while still seeking individuality. On the internet these can be used to frame our views, bought or selected by algorithms, decide how information is accessible, and how social interactions get mapped unto social media with an instant feedback.

Table Of Content


Attributions: R. Fludd, Utriusque Cosmi, Tractatus primi, Oppenheim, 1620

Pepijn de Vos (pepijndevos)

SimServer: Tight Integration with Decoupled Simulators April 10, 2021 12:00 AM

I am working on Mosaic, a modern, open source schematic entry and simulation program for IC design. With a strong focus on simulation, I want to offer deep integration with the simulator, but also be able to run it on a beefy server and shield my program from simulator crashes. To this end, I have developed an RPC abstraction for interfacing with simulators remotely.

Here is a demo of a short Python script that uses Pandas, Matplotlib, and Cap’n Proto to run a CMOS netlist on Ngspice and Xyce and a behavioural Verilog equivalent on CXXRTL, allowing for easy verification of the transistor implementation.

You can see that the behavioural simulation is nearly instantaneously, while the spice results stream in much slower because they are doing a fully analog transistor simulation. You can see there is a bit of overshoot at the edges, and zooming in on that, you can see minor differences between the analog simulators because Xyce is using an adaptive timestep.

close up of overshoot

Now let’s take a step back and take a look at the design and implementation of this system. There are several reasons why I chose for a simulation server.

  • Ease of installation. Xyce is notoriously hard to install and only works on Linux as far as I know. An RPC protocol allows Xyce to run in a Docker container.
  • Performance. My laptop might not be the best place to run the simulation. An RPC protol allows the simulator to run on a beefy server, while running my user interface locally for a snappy experience.
  • Integration. Running a simulator in batch mode provides no indication of progress and requires setting up and parsing output files. An RPC protocol allows for direct, streaming access to simulation results.
  • Stability. It would not be the first time I’ve seen Ngspice segfault, and I’d hate for it to take the user interface along with it. An RPC protocol allows the same tight integration as its C API without linking the simulator into the GUI.

For the RPC library I settled on Cap’n Proto, but the next question is, what does the actual API look like? Ngspice has quite an extensive API, but the same can’t be said for Xyce and CXXRTL. So I could offer the lowest common denominator API of “load files, run, read results”, but one of my main goals was deep integration, so this is unsatisfactory. What I ended up doing is define small interfaces that expose a single functionality, and use multiple inheritance to assemble simulator implementations.

So I currently have 3 implementations of the run interface, and on top of that Ngspice implements the tran, op, and ac interfaces, with more to follow. I hope that in the future JuliaSpice will be a simulator that provides even deeper integration.

Please check out the code, and let me know your thoughts: (How to expose simulator configuration and other functionality? Can we do remote cosimulation? Any other interesting usecases?)

Meanwhile, here is a demo of the example Python client running a transient and AC simulation on my VPS.

# on my VPS
docker pull pepijndevos/ngspicesimserver:latest
sudo docker run -d -p 5923:5923 pepijndevos/ngspicesimserver:latest
# in the examples folder
python ../ ngspice myvps:5923 rc.sp tran 1e-6 2e-3 0 ac 10 1 1e5

transient result AC result

April 09, 2021

Frederic Cambus (fcambus)

The state of toolchains in NetBSD April 09, 2021 10:42 PM

While FreeBSD and OpenBSD both switched to using LLVM/Clang as their base system compiler, NetBSD picked a different path and remained with GCC and binutils regardless of the license change to GPLv3. However, it doesn't mean that the NetBSD project endorses this license, and the NetBSD Foundation's has issued a statement about its position on the subject.

Realistically, NetBSD is more or less tied to GCC, as it supports more architectures than the other BSDs, some of which will likely never be supported in LLVM.

As of NetBSD 9.1, the latest released version, all supported platforms have recent versions of GCC (7.5.0) and binutils (2.31.1) in the base system. Newer (and older!) versions of GCC can be installed via Pkgsrc, and the following packages are available, going all the way back to GCC 3.3.6:

| Package | Version    |      Release date |
| gcc10   | GCC 10.2.0 |     July 23, 2020 |
| gcc9    | GCC  9.3.0 |    March 12, 2020 |
| gcc8    | GCC  8.4.0 |     March 4, 2020 |
| gcc7    | GCC  7.5.0 | November 14, 2019 |
| gcc6    | GCC  6.5.0 |  October 26, 2018 |
| gcc5    | GCC  5.5.0 |  October 10, 2017 |
| gcc49   | GCC  4.9.4 |    August 3, 2016 |
| gcc48   | GCC  4.8.5 |     June 23, 2015 |
| gcc3    | GCC  3.3.6 |       May 3, 2005 |

The focus on GCC doesn't mean that the GNU and LLVM toolchains cannot coexist within NetBSD, and work has in fact been done during the last decade to make it happen.

Despite currently not being built by default in official NetBSD releases, LLVM has been imported in the NetBSD source tree in 2013. Daily images are built from NetBSD-current for selected platforms (at least amd64, i386 and evbarm) with the MKLLVM and HAVE_LLVM build options enabled, and contain LLVM and Clang.

Moreover, NetBSD has invested a lot of work on LLVM during the past few years, including funding some developer contracts for Kamil Rytarowski (kamil@) and Michał Górny (mgorny@), which allowed them to work on various parts of the LLVM toolchain to add and enhance support for sanitizers, and to improve LLDB support.

They both published several dozen articles on the NetBSD blog along the way, retracing their journey. Kamil's final report about upstreaming support to LLVM sanitizers summarizes the work accomplished. Thanks to this work, sanitizer support on NetBSD is mature and mostly on par with Linux. As a result, because LLVM is upstream for GCC sanitizers, they are also available in GCC on NetBSD. Similarly, Michał's final report on his LLDB work details the achievements on the debuggers front.

As always, work continues towards keeping the toolchains up to date, and upstreaming local changes whenever possible.

April 06, 2021

Bogdan Popa (bogdan)

Screencast: Writing a Resource Pool Library for Racket April 06, 2021 04:42 AM

After hacking on redis-lib for a bit on Sunday, I decided to write a general-purpose resource pooling library that I can re-use between it and http-easy and I recorded the process. You can check it out on YouTube: You can find the library on GitHub. One particularly interesting bit about the library, that I did not to record, is that the tests are all property-based. I might do another screencast at some point to talk about how they work and the bugs they found in my original implementation (from the video).

April 05, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Actors and Incentives: State Actors: PSYOP, Narrative Warfare, And Weaponized Tech April 05, 2021 09:00 PM

Access to the mountain of the philosophers is blocked by a wall of false, sophistical doctrines.

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 2 — The Actors and Incentives
  • Section 3 — State Actors: PSYOP, Narrative Warfare, And Weaponized Tech
Table Of Content
  • Nothing New — Censorship and Propaganda
  • Psychology as a Weapon
  • Information Society = Information War & Weaponized Tech
  • Internal: Population Control, Data and Surveillance
  • Computational Propaganda
  • External: Cyber Wars
  • Social Media As Battleground — State Sponsored Trolling
  • Memes as Vectors of Propaganda

Since ancient times, nations have tried to write history from their own point of view. As they say, history is written by the victors. Today, the speed of the internet allows rewriting the narrative in real-time, and state actors will certainly take advantage of this. Namely, there are two ways, or generations, of state information-control practices: an information scarcity approach, aka censorship, and an embracing of information approach, speech itself as a censorial weapon. Both require monitoring of the population to properly adapt the tactic in use.

State surveillance is nothing new either, but the era of information abundance gives the capability of amassing data on a colossal scale.

Historically, nations have used propaganda and surveillance campaigns internally, to control their population, and externally, to sway the opinion of other countries, direct revolts, and other favorable incentives.
Today, these dynamics have moved to the cyberspace. The internet has been part of all contemporary conflicts as a force acting on the public opinion. Ultimately, the entity that is able to control the media, the discussion environment, the social networking sites, is in control of the narrative.

Before diving into the new era war, let’s review how psychology can be used as a weapon.

Psychology has always been used for military gain but got more traction with the concept of total war. This type of warfare includes civilian-associated resources and infrastructure as legitimate targets, mobilizing all resources of society to fight the war, prioritizing it over other needs — an unrestricted war. Thus, having a way to sway civilians in this type of war is a must-have in the fighting arsenal.
Although alone it isn’t sufficient and needs to be complemented with simultaneous diplomatic, military, and political tactics. Psychological war takes into consideration the full environmental scope.

A psychological war consists of psychological operations, also known in the contracted form as PSYOPs, which consist of planning psychological activities designed to influence the attitudes and behavior for a political or military objective — to evoke a planned reaction from the people.
This is a vague definition that includes anything we’ve seen from the previous influence, propaganda, and persuasion section. These tactics, when used as weapons, will be backed up by important state incentives making them more powerful, and arguably more successful, than when they are employed by other entities.

In a total war the target can be individuals, civilians, organizations, groups, or governments. The audience needs to be well delineated and studied so that the message is appropriately formed, as we’ve discussed beforehand. There are four audiences to a message:

  • The ultimate one, the real target.
  • The intermediate one, which are likely to receive the message and could be part of the target audience.
  • The apparent one, an audience that seemingly appears to be the target but isn’t the real intended one.
  • The unintended one, which the planner didn’t intend to reach but still received the message.

The receptivity, as we’ve kept saying, depends on many cultural and environment factors. However, in a military setting and with government sponsorship, vulnerabilities can be artificially created through means such as kinetic (bombs and guns), non-lethal biological weapons affecting human psyche and mood, and physical threats. The full environmental scope comes into play.
These operations will employ the schemes we’ve seen such as anchors, hooks, and imageries. They will be delivered through different methods, and will have a clear aim. The objective is either a positive one — to reinforce the behaviour and feelings of friendliness —, a negative one — to destroy morale, weaken the adversary, create dissonance and disaffection —, or to gain support from the uncommitted and undecided.

When faced with a new weapon, states have to develop defensive systems. These include doing PSYOPs on their own population, censorship, and counterpropaganda. Although some countries have laws that forbid applying psychological operations on domestic grounds and on their own citizens.

Nonetheless, most countries have integrated in their military structure distinctive branches that are specialized in the type of operations related to the information and psychological sector.
Here is a generic definition of the role of such unit:

The integrated employment of the core capabilities of electronic warfare, computer network operations, psychological operations, military deception, and operations security, in concert with specified supporting and related capabilities, to influence, disrupt, corrupt or usurp adversarial human and automated decision-making while protecting our own.

When deployed these well-trained units can apply these tactics at multiple levels.

  • Strategic: applies to anything outside the military area, even in peacetime, to prepare the global information environment.
  • Operational: applies to a joint military operation to help achieve consistency between the national objective and the objective of the coalition who they are partnering with.
  • Tactical: applies to on the ground operations, especially when facing an opposing force, to support the actions of a tactical commander.

What’s important for these units is to be able to measure the effectiveness of their operations (MOE). In military, if it can’t be measured then it’s no good. The internet offers clear metrics and is the perfect ground for that.

The ethics of these methods is arguable. Some say that it’s an arms race and that they are forced to apply them, that the development is unavoidable. Others say they offer a way to reduce the “significant loss of life”. As with anything, there’s a lot of subjectivity when it comes to these.

In an information society, information is the center of the new-generation war and thus these operational units are put to the forefront.
Information superiority is a must to win. Everyone fights for their narrative.

This new generation war is often referred to as the 6th one, where the aim is to destroy economic potential, not take part in physical battlespace, and be the dominant information holder.
Like the information society, this type of war is network-centric, taking into consideration the inter-relations between bubbles and how affecting one could lead to a quick victory. This requires information superiority to be able to plan in advance, conceal, and prepare the multiple aspects that will jointly create a favorable ground for the goal of the operation — to easily make a nation ply to your intentions.
From information, moral, psychological, ideological, diplomatic, economic, and so on, these are all required. Practically, this means using mass media, religious organizations, cultural institutions, NGOs, scholars, etc.. In today’s world this translates into using the right influential methods, and influencers on social media.

The data that is available on the internet, and particularly social media, can be gathered, stored, and linked, to finally become an armor or weapon in the hands of a state entity. It is a tool that can be used both defensively and offensively.

This intelligence gathering often take the form of social media surveillance. The information can be used to identify possible bad actors or individuals vulnerable for a PSYOPs campaign.

On the defensive side, a prominent explicit example of intel gathering is how the USA is now asking VISA applicants to submit five years of social media handles for some selected platforms.
However, the intel gathering probably happens most of the time unbeknownst to the data subjects. This is what has been revealed by internal whistle-blowers, and by more explicit state requirements like ISP keeping internet connection logs for at least 2 months in some countries.

Additionally, we need to remember that social media platforms are “independent” businesses. As such, they are bound by the legalities of the geographic areas in which they want to operate. Consequentially, governments can pressure them by taking the legal route and force them to act a certain way to operate on their legislature. Or they can simply buy access to that data from data brokers.
For example, it’s not uncommon that the platforms will have to obey when receiving an order to censure certain messages. Sometimes notifying users, sometimes not notifying them, according to the transparency rules of the platforms and the legislation in which it is operating. This is the simplest method to get rid of dissidents and keep order in a country. However, for the citizen of information societies, censure isn’t looked at too well. The people are craving honesty and individuality, while rejecting authority or association to greater organizations like nations and their wealth, as we’ve seen in the previous section.

On the offensive side, micro-targeting — that is using a conjunction of very specific attributes such as political leaning and interests to target individuals — can be used to amplify a message and have measurable metrics when performing a psychological operation. We’ll come back to this topic in a bit.

No need to use forceful actions when speech itself can be used as a weapon. This is what computational propaganda is about, using social media political discussions to sway opinions. Especially that this is the go-to place to discuss these things. This is known as platform weaponization. It can be used internally, to thwart opposition, or externally, to influence the decision-making of other states.
The act itself could either be state-executed, by the related military branch, state-coordinated, state-incited, or state-leveraged or endorsed.

Computational propaganda relies on the anonymity provided by the platforms which favors black propaganda and gives more credibility, making it seem more genuine. This means that states can deny involvement while inciting or leveraging activities happening on social media for their gain.
Without this knowledge, it is hard to know whether an opponent is attacking or not and how to defend against such attacks.

The attacks can be conducted with bots, automated scripts to scrap and spread data, with trolls, online accounts that deliberately target individuals to trigger and harass them, PR, public relation through news outlets and others, and with memes, which we’ve covered previously.

Bots, and in particular social-bots, give the ability to create fake online voices, and so to fabricate a public opinion. They are one of the weapon of choice in the cyber warfare of who can shout their narrative the loudest. Online, popularity is measured with “likes”, “votes”, “friends”, “followers”, and others which can all be faked.

The advantage that algorithmic campaigns provide is the ability to manipulate what people see, to temporarily boost the visibility of a message, to bypass the social media “democratic” process.

Social bots are accounts that impersonate people of a certain demographic, mimicking social interactions on the platforms. These can be used to create a sense of consensus where there is none, or generate doubt on a subject that is supposed to be widely accepted in the target audience.
With the visibility also comes awareness. This can be used by political parties to obtain votes, and to look more popular than others, to look more seductive and attractive.

Another use for such bot is something called astroturfing, fighting for a “turf”. This is about creating the impression of grassroots movements in favor or against something, be it an event, person, or idea.

This visibility can be used to choke off debates, smothering adversaries by making it seem like issues are one-sided, controlling the bulk of the conversation. A sort of artificially created social censorship. Trolls can be used for such effect too as we’ll see.

Alternatively, algorithmic amplification can be used to muddy political issues, create chaos and instability in the discourse (local or another country), to generate uncertainty, distrust, and divisions.
This is absolutely destructive if combined with on-the-ground actions when discord and confusion reigns in the population, and devastating when combined with economic pressure.

Not all types of content prosper on social media, we’ve seen that before. We’ve learned that outrage and politics resonate well, the things that clash with our cultural insecurities. That is why the bot-led political campaigns tend to come from the most radical parties, it is the most appropriate medium for them.
Apart from these, any type of negative content spreads like wild-fire. This is useful to amplify a sentiment of dissatisfaction and disagreement.

In retrospect, social media certainly has some part to play in the process of radicalization, however none of the messages there would resonate if they didn’t reflect some issues that were already present in us. As usual, it’s an exaggeration of our own ambiguous cultural code.

This is exactly what some state entities use to attempt to influence elections in other countries. Like bots generating discord and chaos regarding a certain topic, micro-targeting can be used to personalize messages towards hyper-partisans. In turn, these selected partisans will be pushed into their narrative, radicalized, polarized, and moving the Overton window. Furthermore, this radicalization gives them a bigger exposure and sets the ground for instability in a population.
This can be an attractive outcome in coordination with other types of actions.

These hyper-partisans can be converted into trolls, online accounts that deliberately target and harass particular individuals. They can either be paid/sponsored real people or bots, either aware or unaware of the tacit agenda, with or without explicit instructions, and either real partisans or fabricated ones.
This is frequently used along with black PR that are campaigns to deliberately engage in disinformation and harassment against a perceived opponent. It is used to intimidate and silence individuals.

There are countless examples of this tactic being used by states, either deliberately launched by them, or the states harnessing an already ongoing attack. These include, but are not limited to, making death and rape threats, amplifying vitriolic attacks, making accusations of treason or collusion, disseminating libelous disinformation, spreading doctored images and memes, and sowing acrimonious sexism.
The target of these attacks are usually journalists, activists, human rights defenders, and vocal members of an opposite ideology. These types of attacks are used everywhere in the world, some for capitalistic gains, and others for state gains.

Yet another efficient way for states to impose their ideas on the minds is through the news networks, all the headlines. They can employ the typical colored language, the emotive stories, imageries, hooks, and anchors we’ve seen before to shape the narrative.
Promotional culture is also an important aspect of pushing a narrative through the headlines.

Aside from these, nations can rely on the visibility principle, once again, to promote the stories they want. This could be by voting on social media with bots, or by infiltrating the news network and trading up the chain. News websites can be constructed from scratch and filled with bogus content, to then rely on the spreading mechanism and citizen journalism to gain credibility and attention. It might finally emerge on the mainstream news media consumed by the desired target audience.
Moreover, trading up the chain can be achieved by using fringe social media platforms which are prone to influence other online ecosystems. This works very well when the message is based on a kernel of truth but with a spin. Even when more reputable outlets are only debunking them, it’s giving the content a position in mainstream news.

All these are extremely hard to defend against, a David vs Goliath affair, and so states prefer to defend through a good offense. Counterpropaganda, part of counter-insurgency (COIN), is hard to practice on social media. However, one particular vector of information that can be used as COIN are memes, the new favorite shell for political messages.

Governments are getting more and more interested in memes, especially as a useful method to compact narrative and culture in a transportable container.
All modern, arguably post-modern, PSYOPs involve the propagation of memes on social media. Meme wars are an inherent part of political life. We’ve amply discussed them in a previous section.
They are the embodiment of the competition over narrative, creating coherent constellations of meanings and stories. We know it’s easy to overlook them because they are a common way of expression and often use humor, but that makes them the perfect craft for information warfare.

What appears like a prank or trope, soon turns into an influential narrative campaign spread by bots and trolls. Remember that memes aren’t limited to their internet format but that they go beyond, that this is only the appropriate envelope they take to move on that medium.
As such, like all that we’ve seen in this section, they can be used defensively, offensively, and to sway people on the fence, to recruit them or drive them away. They are used both for local and international sentiments.

Many countries employ, or discuss employing, memetics in their military sector. It consists of having separate units in a specialized center for these type of operations, which would be a sub-branch of the usual information and psychological operations but tailored to social media.
These meme warfare centers, or referred to as meme farms when described by opponents, would include interdisciplinary experts such as cultural anthropologists, computer scientists, economists, and linguists.

There are many open discussions about which countries actually employ these units, knowing their associations would reduce their power, but we find implicit indicators in the wild. Most countries and international associations have them in sight, take them seriously, or are actually armed with them already. Nowadays, any side of a conflict will use them, that includes all NATO nations, Russia, Hong Kong, China, Armenia, Azerbaijan, Palestine, Israel, India, Pakistan, etc..

However, it is an unconventional mean and so it is tricky to be put in place, or to let the citizens know it is in place. It is hard to include in a military structure. Memes are abstract concepts and they might sound unconvincing and bring obstacles related to finance — a lack of investments —, culture — the mindset to grasp the new generation warfare —, legalities — to encompass the ethics around using them —, and bureaucracies — who should be in command.
Additionally, there needs to be scientific proofs of their efficiency to justify the investment. Efficiency should be tested and measurable, and this isn’t straight forward when a change in attitude doesn’t necessarily correlate with a change in behavior.

If a nation lacks the framework, mindset, resources, knowledge, and tools, they’ll leave the advantage to others that can embrace this new paradigm. Nonetheless, it’s apparent that this is a threat but it wasn’t until 2017 that the EU and NATO established together a center to counter “hybrid threats”. Though its purpose is more analytical than proactive, for them, at least in the public eye, the best form of defence is awareness.

Theoretically, a meme, like a virus, can be categorized in infect, inoculate, and treat. To transmit, prevent or minimize, and contain a message.
A quarantine doesn’t really work in the virtual world though and combining this with the neutralization of the infected memeoids by killing them in the real world might have the opposite effect. History has shown that it is likely it will instead validate the ideology of the meme.

Detection plays an important role in the defence mechanism, to then be able to launch a counter-message. In practice, it has been attempted for counter-radicalization, a form of counter-insurgency, with mixed effects.
An invisible enemy is and invincible enemy, that is why identifying, cataloguing, and tracing should be done beforehand. Which can be almost impossible when the opponent, be it foreign or local, hides within the civilian population, netizens. Thus, the focus on “hybrid treats” center.

In all cases, displacing and overwriting dangerous pathogenic memes is normally done by replacing them with more benign ones. These reactionary memes have different degrees of applicability, actionability, operationalization, and success.
The fruitful reactionary memes have one thing in common: they use humor, irony, and sarcasm to deligitimize the message of an opponent by ridiculing and mocking it. This is similar to WWII propaganda techniques used in cartoons.
“No one is above satire.”

This concludes our review of how state actors are employing the internet narrative as part of their overall information and psychological operations. We’ve first seen how this is nothing new, how psychology has been used as a weapon for quite a while now, both on the defensive and offensive. Then we’ve looked at how data is at the center of wars, now that we’ve moved to an information society. Next, we’ve seen how this data can be part of the surveillance campaign of states. Later we’ve examined computational propaganda, how algorithm dictates our world and how consequently they can be used by nations. Finally, we’ve dived into social media, bots, trolls, news control, and memes, all communication vectors that rely on gaining visibility and credibility. Speech as a method of censorship, speech as a method of propaganda, speech as a weapon.

Table Of Content


Attributions: Geheime Figuren der Rosenkreuzer, Altona, 1785

Ponylang (SeanTAllen)

Last Week in Pony - April 4, 2021 April 05, 2021 12:30 AM

Ponyc 0.39.1 has been released! The Roaring Pony group has made progress, including an addition to the standard library math package. Sean T. Allen is looking for assistance with the ponydoc project.

April 04, 2021

Derek Jones (derek-jones)

The first computer I owned April 04, 2021 09:06 PM

The first computer I owned was a North Star Horizon. I bought it in kit form, which meant bags of capacitors, resistors, transistors, chips, printed circuit boards, etc, along with the circuit diagrams for each board. These all had to be soldered in the right holes, the chips socketed (no surface mount soldering for such a low volume system), and wires connected. I was amazed when the system booted the first time I powered it up; debugging with the very basic equipment I had would have been a nightmare. The only missing component was the power supply transformer, and a trip to the London-based supplier sorted that out. I saved a months’ salary by building the kit (which cost me 4-months salary, and I was one of the highest paid people in my circle).

North Star Horizon with cover removed.

The few individuals who bought a computer in the late 1970s bought either a Horizon or a Commodore Pet (which was more expensive, but came with an integrated monitor and keyboard). Computer ownership really started to take off when the BBC micro came along at the end of 1981, and could be bought for less than a months’ salary (at least for a white-collar worker).

My Horizon contained a Z80A clocking at 4MHz, 32K of RAM, and two 5 1/4-inch floppy drives (each holding 360K; the Wikipedia article says the drives held 90K, mine {according to the labels on the floppies, MD 525-10} are 40-track, 10-sector, double density). I later bought another 32K of memory; the system ROM was at 56K, and contained 4K of code, various tricks allowed the 4K above 60K to be used (the consistent quality of the soldering on one of the boards below identifies the non-hand built board).

North Star Horizon underside of boards.

The OS that came with the system was CP/M, renamed to CP/M-80 when the Intel 8086 came along, and will be familiar to anybody used to working with early versions of MS-DOS.

As a fan of Pascal, my development environment of choice was UCSD Pascal. The C compiler of choice was BDS C.

Horizon owners are total computer people :-) An emulator, running under Linux and capable of running Horizon disk images, is available for those wanting a taste of being a Horizon owner. I didn’t see any mention of audio emulation in the documentation; clicks and whirls from the floppy drive were a good way of monitoring compile progress without needing to look at the screen (not content with using our Horizon’s at home, another Horizon owner and I implemented a Horizon emulator in Fortran, running on the University’s Prime computers). I wonder how many Nobel-prize winners did their calculations on a Horizon?

The Horizon spec needs to be appreciated in the context of its time. When I worked in application support at the University of Surrey, users had a default file allocation of around 100K’ish (memory is foggy). So being able to store stuff on a 360K floppy, which could be purchased in boxes of 10, was a big deal. The mainframe/minicomputers of the day were available with single-digit megabytes, but many previous generation systems had under 100K of RAM. There were lots of programs out there still running in 64K. In terms of cpu power, nearly all existing systems were multi-user, and a less powerful, single-user, cpu beats sharing a more powerful cpu with 10-100 people.

In terms of sheer weight, visual appearance and electrical clout, the Horizon power supply far exceeds those seen in today’s computers, which look tame by comparison (two of those capacitors are 4-inches tall):

North Star Horizon power supply.

My Horizon has been sitting in the garage for 32-years, and tucked away in unused rooms for years before that. The main problem with finding out whether it still works is finding a device to connect to the 25-pin serial port. I have an old PC with a 9-pin serial port, but I have spent enough of my life fiddling around with serial-port cables and Kermit to be content trying a simpler approach. I connect the power supply and switched it on. There was a loud crack and a flash on the disk-controller board; probably a tantalum capacitor giving up the ghost (easy enough to replace). The primary floppy drive did spin up and shutdown after some seconds (as expected), but the internal floppy engagement arm (probably not its real name) does not swing free when I open the bay door (so I cannot insert a floppy).

I am hoping to find a home for it in a computer museum, and have emailed the two closest museums. If these museums are not interested, the first person to knock on my door can take it away, along with manuals and floppies.

Bogdan Popa (bogdan)

Screencast: Building a Redis Session Store for Koyo April 04, 2021 10:37 AM

I decided to write a library for storing koyo sessions in Redis today and I recorded the process. If that sounds appealing, you can check it out on YouTube:

April 02, 2021

Carlos Fenollosa (carlesfe)

Fed up with the Mac, I spent six months with a Linux laptop. The grass is not greener on the other side April 02, 2021 09:11 AM

This article is part of a series:

  1. Seven years later, I bought a new Macbook. For the first time, I don't love it
  2. How I moved my setup from a Mac to a Linux laptop
  3. This article
  4. (TBD) My review of the M1 Macbook Air

Due to very bad decisions by Apple's product marketing teams, Mac hardware and software had been in steady decline since 2016.

Therefore, there has been a trickle of articles on the Geekosphere about people switching from Macs to Linux or Windows.

This is the contrarian view. Don't do it.

The TL;DR is right there in the title: migrating to Linux is fine, but don't expect a better experience than the Mac.

My experience with the Dell XPS 13" Developer Edition was positive in general, including a self-hosted Cloud setup, but not good enough to convince me to stay with it.

We will cover:

  1. A comparison of generic productivity software: email, calendar, image manipulation, etc.
  2. Available power tools to customize your keyboard, trackpad, and more.
  3. A quick jab at decades-old issues which still haven't been solved.
  4. Misc stuff that Linux does better than the Mac.

I feel like I need to clarify that this is an article aimed at Mac users who are considering a migration to Linux in hope of a more polished system. As usual, personal experiences and requirements are subjective. I know that Ubuntu ≠ Gnome ≠ Linux. I also know that I'm not entitled to anything, everybody is welcome to send patches. Just let me say that if you try to cherry-pick any single issue, you're missing the forest for the trees.


Linux productivity software is fine, but there are rough edges for the power user

The typical disclaimer when recommending Linux to a Mac/Windows user is that some proprietary software may not be available, like MS Office, Photoshop, games, etc.

Nobody says, "the main problem you will find with Linux is that email and calendar clients fall apart when you scratch under the surface."

It is truly ironic because I ran MS Office with Wine and it worked well but I was unhappy with my email workflow.

Yes, the apps I missed the most from the Mac were,, and

I am an extreme power user, to the point that many of the keys on my keyboard don't do what the keycap says. I want my apps to let me do easy things fast while allowing me to do complex tasks with a bit of extra work.

I send and receive maybe 100 emails per day. Most of them are HTML, with attachments, video conference invitations, and such. I don't live in a vacuum. I can't ask my clients to send me plaintext email only. I need to send long emails with pictures, I want my zoom invites to appear automatically in my calendar.

For some reason gets a lot of criticism, but it does almost everything well. It has conversation view, search is fast and helpful, multiple accounts are combined seamlessly including autodetection of the "From" field based on the recipient, and smart folders (search folders) are updated when you need them.

On Linux, the only email client with a native "conversation view" is Geary, which is in early development and still very buggy. Evolution is fine and well-integrated with the rest of the desktop apps, but the lack of conversation view was a deal-breaker for me. Thunderbird is an excellent email client, but conversation view is provided by a plugin that is also buggy. Other options like Claws, Sylpheed, Kmail, and terminal clients are more limited in terms of features and don't work for me.

I ended up using Thunderbird, but I felt like I was doing my email with handcuffs. Suffice to say, I had both Thunderbird and Gmail open and used either one depending on the task I needed to complete.

The situation of calendar and contacts clients is similar, with the same contenders. I also ended up using Thunderbird along with Google Calendar.

About PDF and basic image management, anybody who has used will realize that it's much more than just a viewer. There is simply no replacement on Linux. You'll need to open either the Gimp or Xournal for any basic editing. I am an advanced Gimp user, but for most operations, is faster and more convenient.

Desktop notifications are something we don't think a lot about, but a bad system can be very annoying. Gnome has a system-wide framework, which is not well thought in terms of dealing with the actual notifications.

Most apps have their own notifications system which runs in parallel, especially Thunderbird and Evolution. You end up with different types of notifications on different parts of the screen, and a non-consistent UI to deal with them.

Finally, on the Mac, you can find an ecosystem of alternative paid PIM apps, like Spark, Fantastical, Things, and others. There is no equivalent ecosystem on Linux, probably because they would be difficult to monetize.

Power tools are more limited and more difficult to use

The previous section could be summarized as "Linux PIM software is fine in general, but gets in the way of power users."

That is counterintuitive, right? Linux is a much nerdier OS than the Mac and everything is customizable.

But when you jump from theory to practice, at some point you just want a tool to help you set up your config, without the need to edit your trackpad driver source file.

Any advanced Mac user knows about Karabiner, BetterTouchTool, Choosy, Alfred, Automator, and more.

With Linux, you can achieve almost the same feature set, but it is harder and more limited.

For example. To customize your keyboard, you will need a combination of xdotool, xbindkeys, xcape, xmodmap and setxkbmap to capture some event and then run a shell script. There is a Gnome Shell plugin that allows you to tweak your keyboard, but it's nowhere near Karabiner.

If you want to achieve some specific action you need to read four or five manpages, search online, and figure out how you are going to put the pieces together. That made me appreciate Karabiner and BTT much more.

Furthermore, I couldn't find a real alternative to BTT to customize trackpad multi-touch gestures. I tried a few approaches with libinput-gestures but none worked.

In the end, I was able to replicate most of my macOS power tools setup via input hooks and shell scripts, but it took much longer than it should have. I found it surprising that, given the number of nerds using Linux every day, there are no specific tools equivalent to those mentioned above.

"I Can't believe we're still protesting this crap"

Please allow me to make a bit of fun of issues that existed back in 1999 when I started using Linux and still exist today.

  • Screen tearing with the intel driver. Come on. This was solved on xorg and now with Wayland it's back. I fiddled multiple times with the settings but couldn't fix it. Even with OpenBSD it was easier to fix. The default settings should be better. I don't care if the video driver will use an extra buffer or whatever.
  • Resolving new hosts is slow, with a delay of about 2-3 seconds. I tried to disable IPv6 and other tricks from Stackoverflow threads, but none solved the issue completely. Again, an issue with the default settings. macOS does some DNS magic or something and the network feels much faster.
  • Resuming after suspend seems to work at first. As soon as you start to trust it and not save your work before closing the lid, it betrays you and you lose your work. Later, you upgrade the kernel and it works all the time for weeks until you upgrade the kernel again and it goes back to working 80% of the time. What a mess.

We've come a long way with Linux on the desktop but I think it's funny that some things never change.

Linux also hides some gems

I want to end this review on a positive note.

During those six months, I also took notes on apps and workflows that are still better on Linux.

  • Tracker/search is better and faster than Spotlight. It's a shame that not all apps take advantage of it, especially Thunderbird.
  • Firefox is amazing. On the Mac, Safari is a better choice, but I was very happy using Firefox full-time on Linux. I am going to miss some great plugins, like Multi-account containers, Instagram-guest, Reddit Enhancement Suite, and of course NoScript and uBlock Origin
  • Nautilus is better than the Finder. It's not even close.
  • The Gnome Shell Extension Gallery has many hidden gems, like Nothing to say which mutes the microphone system-wide with a shortcut, the Emoji selector, Caffeine to keep your computer awake, a Clipboard manager, and Unite to tweak basic UI settings. I am now using macOS equivalents to those, and I discovered their existence thanks to the Linux counterparts.
  • Insync for Linux is better than the official Google Drive File Stream for the Mac. In fact, I am now using the Mac version of Insync.
  • Gimp and Inkscape are excellent apps, and it's a pity that the macOS ports are mediocre. I'd rather use them than Pixelmator/Affinity Designer. Hopefully, someday either GTK or these apps will get decent macOS versions.
  • apt-get was a revolution when it was released in 1998 and it is still the best way to manage software today. brew is a mediocre replacement.
  • I paid for Crossover which allowed me to use MS Office and other Windows apps I needed. Kudos to the Wine developers for almost 30 years of continuous effort.
  • Xournal is an obscure app that allows you to annotate PDF documents as well as draw with a Wacom tablet. I used it constantly as a whiteboard for online presentations. The macOS port is very buggy, unfortunately, so I use OneNote which is not that good.

Hopefully, the success of paid tools like Insync or Crossover can encourage the developer ecosystem to continue developing quality apps, even if they are non-free, or supported by donations.

What's next?

On November 10th Apple showed us the future of the Mac and released again laptops worth buying. So I bought the 2020 M1 Macbook Air. You will read a review of it soon.

The hardware is much better than the Dell's and, I guess, every other PC laptop. The software ecosystem is a big improvement over my Linux setup, and Big Sur course corrects the absolute mess that Catalina was. With every passing year, the iCloud offering keeps getting better, especially if you have other Apple devices.

I am somewhat sad that I couldn't join the Linux Resistance. After all, I've been an annoying proselytizer heavy free software advocate in the past, and I still am, though I nowadays admit there are many nuances.

The experience of using Linux as a daily driver has been very positive for me, but I do need my productivity. I can work much faster with macOS and iCloud than I was with Linux and my self-hosted cloud setup.

If there ever was a period where the Mac experience was worse than Linux, it is now over. The Mac ecosystem wins again. Don't switch to Linux expecting it to have fewer papercuts than the Mac. It's quite the opposite.

There is definitely grass on the other side of the fence, but it is not greener.

Tags: apple, linux

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

April 01, 2021

Gergely Nagy (algernon)

The Logical Next Step April 01, 2021 12:30 PM

It's been a while I posted here, even though there were a lot of things happening around me. Alas, some of those led me onto a path that I am now ready to talk about: it's been brewing for a good while now, and I'm excited to announce it today!

I've spent the past two years working on Kaleidoscope and Chrysalis, and enjoyed every moment of it, keyboards, ergonomic keyboards, are my passion. Today, I'm taking that a step further, and am joining a major keyboard manufacturer to work on their input devices from now on. Not only will I be working with an incredible team developing the next generation of smart, cloud-connected input devices, but we will be bringing my own devices to the market aswell!

I was originally approached by a different team from the company, due to having a background in IoT and security, but the talks quickly turned to ergonomics and input devices (my greatest fault is that I can't stop talking about things I'm passionate about), and after a bit of back and forth, we ended up deciding that we can combine all our experiences, all our knowledge, and build products where all of them are put to good use. This is an incredible opportunity to put everything I've learned so far while working on Kaleidoscope, Chrysalis, and the many different projects of the past, towards the goal of building products that will revolutionize what we think about input devices.

I can't wait to share the crazy new ideas we've come up with during the past year of negotiation!

Gustaf Erikson (gerikson)

March April 01, 2021 07:35 AM

Derek Jones (derek-jones)

Linux has a sleeper agent working as a core developer April 01, 2021 12:36 AM

The latest news from Wikileaks, that GCHQ, the UK’s signal intelligence agency, has a sleeper agent working as a trusted member on the Linux kernel core development team should not come as a surprise to anybody.

The Linux kernel is embedded as a core component inside many critical systems; the kind of systems that intelligence agencies and other organizations would like full access.

The open nature of Linux kernel development makes it very difficult to surreptitiously introduce a hidden vulnerability. A friendly gatekeeper on the core developer team is needed.

In the Open source world, trust is built up through years of dedicated work. Funding the right developer to spend many years doing solid work on the Linux kernel is a worthwhile investment. Such a person eventually reaches a position where the updates they claim to have scrutinized are accepted into the codebase without a second look.

The need for the agent to maintain plausible deniability requires an arm’s length approach, and the GCHQ team made a wise choice in targeting device drivers as cost-effective propagators of hidden weaknesses.

Writing a device driver requires the kinds of specific know-how that is not widely available. A device driver written by somebody new to the kernel world is not suspicious. The sleeper agent has deniability in that they did not write the code, they simply ‘failed’ to spot a well hidden vulnerability.

Lack of know-how means that the software for a new device is often created by cutting-and-pasting code from an existing driver for a similar chip set, i.e., once a vulnerability has been inserted it is likely to propagate.

Perhaps it’s my lack of knowledge of clandestine control of third-party computers, but the leak reveals the GCHQ team having an obsession with state machines controlled by pseudo random inputs.

With their background in code breaking I appreciate that GCHQ have lots of expertise to throw at doing clever things with pseudo random numbers (other than introducing subtle flaws in public key encryption).

What about the possibility of introducing non-random patterns in randomised storage layout algorithms (he says waving his clueless arms around)?

Which of the core developers is most likely to be the sleeper agent? His codename, Basil Brush, suggests somebody from the boomer generation, or perhaps reflects some personal characteristic; it might also be intended to distract.

What steps need to be taken to prevent more sleeper agents joining the Linux kernel development team?

Requiring developers to provide a record of their financial history (say, 10-years worth), before being accepted as a core developer, will rule out many capable people. Also, this approach does not filter out ideologically motivated developers.

The world may have to accept that intelligence agencies are the future of major funding for widely used Open source projects.

March 31, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Actors and Incentives: Internet, In Between The Corporate, Private, And Public Spheres March 31, 2021 09:00 PM

For all are Men in Eternity, Rivers, Mountains, Cities, Villages. In your imagination, of which this World of Mortality is but a shadow. For it includes all things, and everything that exists, only exists through it

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 2 — The Actors and Incentives
  • Section 2 — Internet, In Between The Corporate, Private, And Public Spheres
Table Of Content
  • Availability of Information
  • Networks And Cultures
  • Individuation
  • Self-Generated Societies
  • Making Up Your Own Mind + Learning & Creativity
  • Citizen Journalist
  • Cognitive effects
  • Private Life Becoming Public Life
  • Passion Economy
  • Extreme Noeliberalisation

The internet is a new resource and when introduced in our social structures it has fueled the construction of utilities around it. Like any tool it has no effects on its own but only through its usages. In particular, it has altered our capacity of communication making it interactive, multimodal, asynchronous or synchronous, global or local, many-to-many, one-to-many, one-to-object, object-to-object.
In this section we’ll go over some of the things the internet allows to do through the platforms and services hosted on it. We can’t go over everything because that would be equivalent to describing what modern life is about. Instead, we’ll paint a rough picture of the internet in between the corporate, private, and public sphere.

We can describe ourselves as an information society, an internet-everywhere society. Calling ourselves digital natives, digital citizens, digital cosmopolitans, and netizens.
Everything that can be digitized gets digitized. According to a study published by Martin Hilbert in Science in 2011, 95% of all information in existence on the planet is already digitized and accessible through the internet.
This has shrunk the globe, brought knowledge to everyone, altered the way we communicate and connect, changed the way we make money, and changed the way we think.

The internet reflects us, it can convey the best and worst of humankind. It is an open-ended system, like our societies, with the inter-relations mapped in the digital space, with all the same logics that exist between cultures, organizations, and technologies.
Similarly, the internet is networked and so are our societies: constructed around personal and organizational networks, without clear boundaries. Thus, the same paradigms and dynamics that apply in our real world are echoed online. Studying the internet is studying anthropology.

Previously we’ve seen how this bridging of cultures online makes us shift from high-context cultures to low-context ones, or informational ones. Because of this, we are forced to learn to communicate in a different way, with an emphasis on collaboration, learning to negotiate, to accept disagreement as normal.
Paradoxically, this has made us more socially conscious, and social in general, on the internet than in the physical world. There is less distance, it feels like the globe has shrunk, and that has intensified social interactions, for better or worse.
Some studies have observed that overall, there is a correlation between sociability and internet usage. This applies to civic engagements and other types of relationships. This is a new kind of sociability, one that is driven by permanent connectivity, especially on social media.

When bridging cultures a common language has to be chosen, and on the internet it is often English that dominates. It composes around 30% of the internet according to estimates, but isn’t the native tongue of most users. This can limit the spread of information from one cluster to another, create boundaries for communities, an inequality of information, an inequality of representation, and shape the digital experience and views of people on the internet. The language, and indirectly culture, barriers can direct the flow of information, tending in the direction where it is sometimes more domestic. We’ll get back to this in the consequences part of the series.

The apparent reduction in distance also creates networking opportunities. In the past interactions were limited to the personal networks; people being favored for whether they went to a prestigious school, university, or worked at a well-known company. These days we can be exposed to a diverse range of networks that are much more accessible, increasing the employment prospect.
This applies to creativity too, being exposed to so many diverse ideas, different ways to see a problem. Creativity is an import-export business. Taking things that are ordinary from multiple places and putting them in totally different settings. What is called cognitive diversity.
Additionally, widespread connectivity brings the possibility of working from almost anywhere in the world, at any time of the day, if the job doesn’t require to be on-place — that is if it is an information related job. This unlocks tremendous freedom and autonomy.

The concept of autonomy is an important one, as it relates to another one: individuation. When moving from high-context society to low-context information society, the process of individuation takes place. Some ironically calls it the “ME-centered” society, or the “selfie generation”. This isn’t something new, it’s been happening for a couple of years, but it is more widely spread and the internet is a catalyst.
Concretely, this translates into the decline of the definition of the self based on organizations and organizational roles, such as work, nation, and family. The individual is now in the center and doesn’t find meaning in these classical system, sometimes leading to a cross culture frustration.

Nevertheless, this is not the end of the concept of community but the end of having it based on places of interactions. Instead, it has shifted toward the self-authorship of the community, one based on personal ties, social relationships, individual interests, values, and projects. As a matter of fact, the individual is not isolated, nor less sociable, but redefined as a networked individual with a quest for like-minded people.
The information society is a network of self-authored individuals.

The transformation and evolution in culture bring with them a reorganization of economic, communication, and political activities. According to some research, it is correlated with a sense of empowerment, increased feelings of security, personal freedom, and influence — all in relation with overall happiness and well-being.

Whole communities join together and participate in their own identity development, as a choice. In this model, people replace blind reverence for authority with authenticity.
In the past information were shaped and based on hype, now people crave what is genuine and transparent. This is a change in perception with a focus on raw emotions and connections.

This authenticity is an essential aspect for identity development, it is how we identify ourselves and create our stories. Our narratives are based on our interactions in this participatory network. We are the creators, refining our persona with the help of the individuals that surround us, to whom we tell our story.
McAdams (1993) asserts, “Once an individual realizes that he or she is responsible for defining the self, the issue of self definition remains a preoccupation through most of the adult years”.

Being the subject, the individual, in the process of creation is what is meant by autonomy. This is accompanied by an urge to not submit to institutions, especially for minorities. This switch in mindset is highly correlated with social autonomy, independent-minded citizen able to participate in self-generated networks.
This autonomy materializes itself in society through entrepreneurship, creative works, proactive consumers, self-informed critical thinkers, pedagogical platforms that allow self-learning, e-governments, grassroots movements, etc..
The paper entitled “The Social Media Mindset” lists six major types of autonomy:

  • Professional development
  • Communicative autonomy
  • Entrepreneurship
  • Autonomy of the body
  • Sociopolitical participation
  • Personal, individual autonomy

More than anything, to be autonomous means mastering our tools, grasping how to make algorithms work for us. They are ubiquitous, we are surrounded by them, the internet is mediated by them. Hence, we need to muster their strength.
To be internet literate implies understanding the technology that drives it. The customization and recommendation engines can be used to facilitate our daily routine, letting the users create their own personal optimized environments.

Social networks are used to create our own constructed world, which includes everything from e-commerce, education, entertainment, to sociopolitical activism. The users group together according to their interests and tailor it to their needs. A self-constructed society with people yearning for honesty, genuineness, transparency, common interests, and connections.
Many of the real world has migrated to the internet, from web marketers, work organizations, service agencies, governments, and civil societies.
Technology, being material, is based on products made by people based on their ideas, values, and knowledge. We adapt the technology to fit us, rather than adopting it. It is built into our society and thus it has to follow its constraints. Obviously, social media platforms are trying to monetize the process, turning it into a business.

Collectively, this has brought us to an era of creativity. Where everything is a remix and shared. We aren’t bound by the mainstream media gatekeepers like TV, radio, newspapers, and academia. Information is now freely available on the internet, accessible by anyone, from the worst atrocities to the most awe-inspiring instances of human kindness.
In the internet society we are forced to broaden our collective perspective, a perspective that is more global.

The informational freedom carries with it transformations in the sociopolitical realm. There’s a widespread availability of tools for creating and distributing content across the globe. Most often, there is no need for training nor expertise to participate, everyone is empowered to mass media — A direct access to million of other people in an expanded community.
We bypass traditional media, letting us challenge the assertions of powerful corporations, government officials, news reports, and television advertisements. We do it through blogs, Twitter, Facebook, and others.

Everyone is an author and can express their thoughts to the world and this undoubtedly means that some people are going to spread disinformation, misinformation, and other types of erroneous information.
Our new distrust in authority and increasing doubts, can be abused through our cognitive biases. The attention economy, with the never ending competition between all actors, is filled with manipulation.
This is something the netizens are slowly learning. We’ll come back to biases later.

The constant connectivity has given rise to a more democratic and participatory form of journalism: citizen journalists. The audience employs the press tools to inform one another. The public now plays the role of collecting, reporting, analyzing, and disseminating news and information.

There are upsides and downsides to this new development. As we said, these are amateurs, people without journalism training, and so the quality of the reports and coverage can vary, is subjective, heavily opinionated, unregulated, and can be abused by malicious actors — such as spin doctors.
However, it can also promote human rights and democratic values to have the people join together and collaborate in this endeavor.

Social network websites, accessible to everyone, have provided an alternative for activists to report on their activities. Social media also allow real time participation and messages can be sent quickly in reaction to events.
This is particularly helpful during an earthquake, street movements, and protests.

Lots of things are now shaped and decided on the internet, it is tightly linked to political changes in the process. Citizen journalists now surround all the protests and movements these days. These movements, rejecting authority, are often without leaders, more spontaneous, and full of emotional outbursts.

The media of choice to express feelings about a situation are internet memes. They act like the modern era version of caricatures found in traditional newspapers.

Some memes have been put forward in democratic campaigns such as sharing messages about climate change. Though arguably, the research hasn’t found their impact in altering mindsets that impressive. In that study only 5% of participants changed their minds.
Memes, like caricatures, can also act as a power and counterpower, a battle of communication. Humor has always been a tool for people to express feelings and thoughts that were repressed, to fight against oppression.
This is the case in countries where people have to bypass censorship and rhetoric by reusing the semantic meaning of the slogans, ideas, and terminology. They reuse the language, twist it, satire it, to reinterpret it in their own way. They use the official language in reverse by applying it with humor.

Still, we have to remember that citizen reporting is full of biases, doubts, lies, polarization, conspiracy theories, and subjectivities. These can also obviously be manipulated by other actors, as we said.

Similar to citizen journalists we are seeing the rise of citizen science, which we won’t dive into here.

Let’s now jump a bit into some of the effects that being constantly online can have on our cognitive abilities. In a place of never-ending multitasking, attention economy, infobesity, and new social concepts.
This will be reviewed again in the consequences part of the series.

Our attention is now always divided between multiple media sources at the expense of sustained concentration. Everything tries to attract our attention in this hyperlinked addictive environment and we got used to it.
Education providers can already see the effects this has on children attention span: their attention is more shallow.
Studies show a significant effect on cognitive performance, where even short term engagement reduces attention scope for a sustained duration after coming offline.

Memory-wise, the vast array of information has changed the way we retrieve, store, and even value knowledge. The internet is now acting like a transactive memory, externalizing our thoughts — a helper, a memo, a memento.
We have most of the factual information in existence at our fingertips, introducing the possibility that the internet could ultimately negate or replace certain parts of our memory system, especially the semantic memory — memory of facts.
Multiple research show that people searching online were more likely to remember where they found the facts rather than the facts themselves. We are more reliant on the internet for information retrieval.
I’ve dived into this topic previously here and here.
At the group level this can be efficient but at the individual level it can hinder how we recall the actual information. The internet sometimes even bypasses the ability to retain where we found the information when it gives us the opportunity to search for it. We don’t have to remember the exact information anymore, nor where its located, just that it can be searched and that it’ll be available.
This cognitive offloading using an external long term storage can be beneficial to focus, not on facts, but on aspects that are not retrievable like meta information, conclusions, and emergent hypothesis. It frees some space to think about other things than plain facts.
Though other research show that analytical thinkers are less likely to rely on such means in their day-to-day than other people.

Cognitively, the internet also creates new ways to learn. Any new skills and information are available in this space. People now have more breadth of knowledge instead of depth of knowledge.

When it comes to social cognition, there are a lot of debates regarding whether the internet provokes the same effects as real interactions would. So far, the studies and evidences show that it does indeed reflect real things like self-concepts and self-esteem. We clearly see this through the neurocognitive responses to online social interactions.
These online social relationships, like any relationships, are connected to feelings of happiness, and mental and physical well-being. Thus being accepted or rejected online feels the same way in the brain, an interpretation, which we’d like to remember, in the offline world is often ambiguous and left to self-interpretation.
However, on social media, unlike the real world, we are presented with clear metrics which give us an indication of whether we fit in or not. They take the form of “followers”, “shares”, “likes”, “friends”. The potential can be painful, or addictive because of the immediacy. There is a direct feedback on our self-esteem.
Additionally, the public life on these social media also means that upward social comparisons are omnipresent. People take part in artificial environments manufactured to picture hyper-successful persona. Obviously, this is what the platforms capitalize on. You can go back to the previous section to get an idea.

On social media we aren’t anonymous but often create persona, a self-presentation of a real person. One that we mold to be perfect. Everyone has the ability to become an orator, a super-star in front of a gigantic crowd. An online life is like being followed by a million paparazzi.

We know that the medium is quick, indelible, and that a misstep can make us victim of the internet outrage machine. Be it positive, or not, because of private justice or vigilantism, or any other reasons. We are intimately aware of how extreme views proliferate and how ideas are hard to separate — always coming as blocks, hooks, and anchors.

With social media our personal life and work life are intertwined, they come as a package that can’t be separated. Our views in one field can propagate and affect other aspects of our lives.
Personal life is now public life, and not only for celebrities.

This is compounded with the feeling of always being scored and tracked. According to the Pew Research Center, 72% of people feel that almost all of what they do online is being tracked by advertisers, technology firms or other companies.

Having private personal details online and our digital reputation can have repercussions in the real world affecting our relationships, job opportunities, bank loans, visa applications, and more.
It is hard to erase something once it is on the internet, and so we apply self-surveillance between ourselves. Indeed, we know our weaknesses are mapped and become increasingly transparent and this leads to self-censorship, conformity, risk-aversion, and social rigidity.
This is a phenomenon called “social cooling”. “You feel you are being watched, you change your behavior.” Isn’t this a form of censorship? Privacy is the right to be imperfect and not judged. We’ll come back to this in the last section of the article when discussing some of the solutions.

I was told that I couldn’t be trusted since people can’t check online what I’m doing when I’m not around.

On the good side, the ability to easily get exposure, the appetite for authenticity, and the itch for autonomy have given birth to the passion economy.
The passion economy consists of taking the cliché of “following your passion” to the extreme. While it didn’t work properly in the past, it is now increasingly easier to achieve, either through an online patronage system or simply because of the new exposure.
The individual is empowered and liberated from previous notions of fixed career path.

The passion economy is centered around purposeful creators that are motivated and driven by their passion, displaying their expertise and experiences with the world.

The audience is as passionate and niche as the creator, influenced and transported through the craft. This is in stark contrast with the apathetic consumers of previous years that made price-focused decision about trinkets of mass productions.
People want to be involved in the making process, they want the whole experience, to engage with the services and ideas they like, to associate with their stories, to build a relationship with the authors. They want to find individuality in their products.

Along with an always public life comes the idolization of hyper-successful individuals. There is a rise in unrealistic expectations of oneself.
Individuation puts the weight on people to find who they are, and it is easy to go back to role models for escapism. These people might in turn use their influence to create a monetizable audience.
The gold rush for scores and fame, the new cult of personality, the new promotional culture going viral.
This is the rise of the “influencers” who’ll sell their authenticity for paid product placements and advertisements.

Lastly, these coalesce — individuation, influencers, passion economy, private life becoming public life, and social media — into neoliberalization.
We’ll see more of this later, as it also depends on culture and how people perceive and use the internet in different ways.

This is the ailment of modern society in general, and the internet makes society more “modern”. A shift in motivations to be goal-oriented and efficient, instead of being based on the more traditional values.
The individuals, with the help of the internet, are now accustomed to marketing themselves as products. This is the modern rule of the game: corporate life merging with private life.

Many products are sold to fill this “need”. I’ll digress, as I have discussed this in other articles, and here. We’ll come back to this later.

This concludes our review of the person’s life on the internet in between the corporate, private, and public spheres. We started by seeing how the availability of information is now everywhere and shrinks distances. Then we’ve taken a look at the relationship between networks in societies being mapped online and the effects on cultures. Next we’ve discussed the switch to the information society which creates a need for individuation, networks of people grouping based on their interests. Then we’ve seen how this allows self-generated societies and making our own mind about things. Later we’ve discussed the empowerment of the average person through citizen journalism. After that we’ve glanced at some of the cognitive effects in the realm of attention, memory, and social cognition. Following this we’ve concluded with three subjects: the private life becoming public, the passion economy, and a new extreme form of neoliberalisation that is normalized.

Table Of Content


Attributions: W. Blake, The Sun at its Eastern Gate, c 1815

March 29, 2021

Ponylang (SeanTAllen)

Last Week in Pony - March 29, 2021 March 29, 2021 04:39 PM

The ‘Force declaration of FFI functions’ RFC has entered final comment period. OpenSSL and LibreSSL builders have been updated. There are also some releases from last week.

Kevin Burke (kb)

Preferred Stock and Common Stock Aren’t The Same March 29, 2021 04:00 PM

When you get an offer from a tech company it will usually be some combination of cash and stock. Small companies give out stock because it's hard to compete, cash-wise, with bigger companies, and a grant of stock or options offers the chance at a large payday down the road.

Valuing the cash part of an offer is pretty simple. How should you value the stock? Well, one answer is whatever someone's willing to pay you for it.

To that end recruiters will sometimes give you information like, "our last round closed at $4.80 per share, so, if you get X shares per year, your stock compensation is worth $4.80 * X." Sometimes recruiters will send a fancy tables showing the value if the company doubles or 10X's its value.

This is a classic bait and switch. When a company raises a round from, say, Sequoia, Sequoia wires money to the company and gets preferred shares. When you are granted stock options as an employee, you are getting common shares. The differences will vary from company to company, but commonly:

  • Holders of preferred shares get paid out before common shareholders (ie you). Bankruptcy is not intuitive. If you get in a traffic accident, the insurers will usually say something like, the other party is X% at fault, and you were Y% at fault, so this is what you owe. Bankruptcy is not like traffic court. All the rights holders ahead of you will get paid out 100% before you see a cent. If the company is sold or liquidated, the preferred shareholders will likely be paid in full before any holder of common stock sees a dollar. Because of this, preferred shares are more valuable.

  • Preferred shares have different voting rights than common shares. A preferred share might get five or ten (or zero) votes, to one for a common share.

  • Preferred shares may have other downside protection in the event an IPO or sale does not reach a target price.

So preferred shares are worth more than common shares. It is a mistake to look at the preferred share price and multiply by the number of shares and use that as the current value of your common shares. Your shares are worth much less than that.

One reason this happens is that preferred shares are easier to value, because there are regular funding rounds, insider sales. Common stock doesn't trade hands that often before an IPO because stock sales often require board approval. But that doesn't excuse anyone from giving you common shares and pretending they are worth as much as preferred shares.

The recruiters and VC's next trick is to pretend that you shouldn't be concerned about the difference between common and preferred stock, because in an IPO preferred stock is usually converted to common stock. That's true. But this is sort of like saying a home without fire insurance and a home with fire insurance are worth the same as long as they don't burn down. If you have common stock, you don't have fire insurance. And a startup is significantly more likely to fail than a home is to burn down.

If anyone tries to convince you that the difference doesn't matter, ask them if they'd like to swap their preferred shares one-for-one with your common shares.

If you are being recruited and someone tries this on you, please point out the difference and explain that you don't appreciate being condescended to. You should also think less of the company. Every one of your coworkers went through the same process of being lied to about their potential share value.

If you are an employer and you want to do this honestly, quote the most recent share price you have, and then explain that that's the preferred share price, but you are not giving out preferred shares. Explain that recruits should value their shares lower than the preferred share price you quoted - exactly how much is difficult to say, but the preferred share number should be an upper bound on that value. If your common stock is traded, or any of your shareholders are forced to mark their shares to market (Fidelity holds them in a mutual fund, for example), you should disclose that.

(You could also let your employees sell their equity more often, maybe to insiders.)

March 28, 2021

Derek Jones (derek-jones)

The aura of software quality March 28, 2021 09:45 PM

Bad money drives out good money, is a financial adage. The corresponding research adage might be “research hyperbole incentivizes more hyperbole”.

Software quality appears to be the most commonly studied problem in software engineering. The reason for this is that use of the term software quality imbues what is said with an aura of relevance; all that is needed is a willingness to assert that some measured attribute is a metric for software quality.

Using the term “software quality” to appear relevant is not limited to researchers; consultants, tool vendors and marketers are equally willing to attach “software quality” to whatever they are selling.

When reading a research paper, I usually hit the delete button as soon as the authors start talking about software quality. I get very irritated when what looks like an interesting paper starts spewing “software quality” nonsense.

The paper: A Family of Experiments on Test-Driven Development commits the ‘crime’ of framing what looks like an interesting experiment in terms of software quality. Because it looked interesting, and the data was available, I endured 12 pages of software quality marketing nonsense to find out how the authors had defined this term (the percentage of tests passed), and get to the point where I could start learning about the experiments.

While the experiments were interesting, a multi-site effort and just the kind of thing others should be doing, the results were hardly earth-shattering (the experimental setup was dictated by the practicalities of obtaining the data). I understand why the authors felt the need for some hyperbole (but 12-pages). I hope they continue with this work (with less hyperbole).

Anybody skimming the software engineering research literature will be dazed by the number and range of factors appearing to play a major role in software quality. Once they realize that “software quality” is actually a meaningless marketing term, they are back to knowing nothing. Every paper has to be read to figure out what definition is being used for “software quality”; reading a paper’s abstract does not provide the needed information. This is a nightmare for anybody seeking some understanding of what is known about software engineering.

When writing my evidence-based software engineering book I was very careful to stay away from the term “software quality” (one paper on perceptions of software product quality is discussed, and there are around 35 occurrences of the word “quality”).

People in industry are very interested in software quality, and sometimes they have the confusing experience of talking to me about it. My first response, on being asked about software quality, is to ask what the questioner means by software quality. After letting them fumble around for 10 seconds or so, trying to articulate an answer, I offer several possibilities (which they are often not happy with). Then I explain how “software quality” is a meaningless marketing term. This leaves them confused and unhappy. People have a yearning for software quality which makes them easy prey for the snake-oil salesmen.

Gustaf Erikson (gerikson)

The Eighteenth Brumaire of Louis Bonaparte by Karl Marx March 28, 2021 10:37 AM

Available online here.

Come for the class analysis, stay for the bon mots.

It’s probably fitting that the only way obscure French politicians are remembered today is through their skewering in this piece.

Hegel remarks somewhere that all great world-historic facts and personages appear, so to speak, twice. He forgot to add: the first time as tragedy, the second time as farce. Caussidière for Danton, Louis Blanc for Robespierre, the Montagne of 1848 to 1851 for the Montagne of 1793 to 1795, the nephew for the uncle. And the same caricature occurs in the circumstances of the second edition of the Eighteenth Brumaire.

The period that we have before us comprises the most motley mixture of crying contradictions: constitutionalists who conspire openly against the constitution; revolutionists who are confessedly constitutional; a National Assembly that wants to be omnipotent and always remains parliamentary; a Montagne that finds its vocation in patience and counters its present defeats by prophesying future victories; royalists who form the patres conscripti of the republic and are forced by the situation to keep the hostile royal houses they adhere to abroad, and the republic, which they hate, in France; an executive power that finds its strength in its very weakness and its respectability in the contempt that it calls forth; a republic that is nothing but the combined infamy of two monarchies, the Restoration and the July Monarchy, with an imperial label – alliances whose first proviso is separation; struggles whose first law is indecision; wild, inane agitation in the name of tranquillity, most solemn preaching of tranquillity in the name of revolution – passions without truth, truths without passion; heroes without heroic deeds, history without events; development, whose sole driving force seems to be the calendar, wearying with constant repetition of the same tensions and relaxations; antagonisms that periodically seem to work themselves up to a climax only to lose their sharpness and fall away without being able to resolve themselves; pretentiously paraded exertions and philistine terror at the danger of the world’s coming to an end, and at the same time the pettiest intrigues and court comedies played by the world redeemers, who in their laisser aller remind us less of the Day of Judgment than of the times of the Fronde – the official collective genius of France brought to naught by the artful stupidity of a single individual; the collective will of the nation, as often as it speaks through universal suffrage, seeking its appropriate expression through the inveterate enemies of the interests of the masses, until at length it finds it in the self-will of a filibuster. If any section of history has been painted gray on gray, it is this. Men and events appear as reverse Schlemihls, as shadows that have lost their bodies. The revolution itself paralyzes its own bearers and endows only its adversaries with passionate forcefulness. When the “red specter,” continually conjured up and exercised by the counterrevolutionaries finally appears, it appears not with the Phrygian cap of anarchy on its head, but in the uniform of order, in red breeches.

The coup d’etat was ever the fixed idea of Bonaparte. With this idea he had again set foot on French soil. He was so obsessed by it that he continually betrayed it and blurted it out. He was so weak that, just as continually, he gave it up again.

The army itself is no longer the flower of the peasant youth; it is the swamp flower of the peasant lumpen proletariat. It consists largely of replacements, of substitutes, just as the second Bonaparte is himself only a replacement, the substitute for Napoleon. It now performs its deeds of valor by hounding the peasants in masses like chamois, by doing gendarme duty; and if the natural contradictions of his system chase the Chief of the Society of December 10 across the French border, his army, after some acts of brigandage, will reap, not laurels, but thrashings.

Confessions of a Long-Distance Sailor by Paul Lutus March 28, 2021 08:17 AM

A self-published book available online recounting the author’s solo round the world sail.

A worthy entry in the long roster of such accounts.

Libra Shrugged: How Facebook’s dream of controlling the world’s money crashed and burned by David Gerard March 28, 2021 08:09 AM

A short account of how Bitcoiners tried to create a Facebook currency and how the rest of the world reacted.

March 27, 2021

Pierre Chapuis (catwell)

Booting GDM on my XPS with kernel 5.11 March 27, 2021 08:15 PM

When I updated my Linux kernel to 5.11 I had the bad surprise to end up with a blinking underscore on reboot. It had been many years since an update had broken my system like that. I fixed it rather easily by booting in rescue mode and downgrading the kernel. I had no time to investigate so I just added linux to IgnorePkg at the time, But I don't use Arch to run old kernels so today I took the time to fix it "properly".

To do so, I reproduced the issue, then downgraded again and looked at the logs with journalctl -b --boot=-1. It quickly let me understand that it was GDM that was failing due to something wrong with graphics initialization.

To keep things short, let me skip to the conclusion: if you run into this issue on an old-ish Dell XPS with an Intel Iris Plus 640 graphics card like mine with GDM, Arch and Wayland (or something similar), try enabling early KMS by adding i915 to the MODULES array in mkinitcpio.conf and rebuilding the initramfs, that fixed it for me.

Andreas Zwinkau (qznc)

How you can handle The Diamond with CMake March 27, 2021 12:00 AM

CMake requires old-school include-guards and prefix at scale

Read full article!

March 25, 2021

Marc Brooker (mjb)

What You Can Learn From Old Hard Drive Adverts March 25, 2021 12:00 AM

What You Can Learn From Old Hard Drive Adverts

The single most important trend in systems.

Adverts for old computer hardware, especially hard drives, are a fun staple of computer forums and the nerdier side of the internet1. For example, a couple days ago, Glenn Lockwood tweeted out this old ad:

Apparently from the early '80s, these drives offered seek times of 70ms, access speeds of about 900kB/s, and capacities up to 10MB. Laughable, right? But these same ads hide a really important trend that's informed system design more than any other. To understand what's going on, let's compare this creaky old 10MB drive to a modern competitor. Most consumers don't buy magnetic drives anymore, so we'll throw in an SSD for good measure.

XCOMP 10MB    Modern HDD   Change Modern SSD  Change
Capacity 10MB 18TiB 1.8 million times   2 TiB 200,000x
Latency 70ms 5ms 14x 50μs 1400x
Throughput 900kB/s 220MB/s 250x 3000MB/s 3300x
IOPS/GiB (QD1) 1400 0.01 0.00007x 10 0.007x

Or there abouts2. Starting with the magnetic disk, we've made HUGE gains in storage size, big gains in throughput, modest gains in latency, and a seen a massive drop in random IO per unit of storage. What may be surprising to you is that SSDs, despite being much faster in every department, have seen pretty much the same overall trend.

This is not, by any stretch, a new observation. 15 years ago the great Jim Gray said "Disk is Tape". David Patterson (you know, Turing award winner, RISC co-inventor, etc) wrote a great paper back in 2004 titled Latency Lags Bandwidth that made the same observation. He wrote:

I am struck by a consistent theme across many technologies: bandwidth improves much more quickly than latency.


In the time that bandwidth doubles, latency improves by no more than a factor of 1.2 to 1.4.

That may not sound like a huge amount, but remember that we're talking about exponential growth here, and exponential growth is a wicked thing that breaks our minds. Multiplying Patterson's trend out, by the time bandwidth improves 1000x, latency improves only 6-30x. That's about what we're seeing on the table above: a 250x improvement in bandwidth, and a 14x improvement in latency. Latency lags bandwidth. Bandwidth lags capacity.

One way to look at this is how long it would take to read the whole drive with a serial stream of 4kB random reads. The 1980s drive would take about 3 minutes. The SSD would take around 8 hours. The modern hard drive would take about 10 months. It's not a surprise to anybody that small random IOs are slow, but maybe not how slow. It's a problem that's getting exponentially worse.

So what?

Every stateful system we build brings with it some tradeoff between latency, bandwidth, and storage costs. For example, RAID5-style 4+1 erasure coding allows a system to survive the loss of one disk. 2-replication can do the same thing, with 1.6x the storage cost and 2/5ths the IOPS cost. Log-structured databases, filesytems and file formats all make bets about storage cost, bandwidth cost, and random access cost. The changing ratio between the hardware capabilities require that systems are re-designed over time to meet the capabilities of new hardware: yesterday's software and approaches just aren't efficient on today's systems.

The other important thing is parallelism. I pulled a bit of a slight-of-hand up there by using QD1. That's a queue depth of one. Send an IO, wait for it to complete, send the next one. Real storage devices can do better when you give them multiple IOs at a time. Hard drives do better with scheduling trickery, handling "nearby" IOs first. Operating systems have done IO scheduling for this purpose forever, and for the last couple decades drives have been smart enough to do it themselves. SSDs, on the other hand, have real internal parallelism because they aren't constrained by the bounds of physical heads. Offering lots of IOs to an SSD at once can improve performance by as much as 50x. Back in the 80's, IO parallelism didn't matter. It's a huge deal now.

There are two conclusions here for the working systems designer. First, pay attention to hardware trends. Stay curious, and update your internal constants from time to time. Exponential growth may mean that your mental model of hardware performance is completely wrong, even if it's only a couple years out of date. Second, system designs rot. The real-world tradeoffs change, for this reasons as well as many others. The data structures and storage strategies in your favorite textbook likely haven't stood the test of time. The POSIX IO API definitely hasn't.


  1. See, for example, this Reddit thread, unraid forums, this site and so on. They're everywhere.
  2. I extracted these numbers from my head, but I think they're more-or-less representative of modern mainstream NVMe and enterprise magnetic drives.

March 24, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Actors And Incentives: New Economies March 24, 2021 10:00 PM

The spectacular widespread magnetic cures of doctor Franz Anton Mesmer. Work based on the magnetic bipolarity of the human body and the influence of an all pervasive vital fluid.

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 2 — The Actors and Incentives
  • Section 1 — New Economies: Information Economy, Attention Economy, Data Economy, Surveillance Economy
Table Of Content
  • Data And Attention As New Assets
  • How Much Value
  • The Actors (Social Media, Ads Networks)
  • What They’d Do To Get It
  • What They’d Do To Protect It

The artifacts we’ve previously seen have given rise to new types of economies, tools, and arsenals that can be used by different actors for different incentives.
Let’s start by focusing on the economic aspects by looking at actors such as social media platforms, newspapers, and advertisement companies.

Life in an information society is about moving data, the new raw material, around our manufacturing pipelines. This data is then consumed by either paying with attention or money.
Data and attention are the two assets of the digital economies that emerged.

Whoever holds the means of productions, by centralizing and monopolizing them, has a tremendous advantage in any economy. In a data economy these are the data brokers who figuratively data mine the raw material, store it, and keep it well guarded — as is required in any intangible economy to succeed (See Capitalism Without Capital: The Rise of the Intangible Economy).

The other new asset is attention. In an environment where it is at the same time so hard to reach an audience and, once reaching a threshold, so easy to spread virally, having the skills to capture attention is valuable.
This is due to multiple factors such as the rise of social media, the average users becoming generator of content, and the ever-growing infobesity.

Davenport & Beck (2001) define attention as “focused mental engagement on a particular item of information. Items come into our awareness, we attend to a particular item, and then we decide whether to act.
Its scarcity turns it into a prized resource, constructing an attention economy. Attention is exchanged as a commodity, either by giving it or taking it — an intangible good on one side, money and physical goods on the other.
There is a fight for this resource, the corporate entities need it because the digital economy is in sync with advertising. The drivers and generators of money are buyers of attention and data, namely marketers and advertisers. Just like traditional media advertising, the online world follows a model that uses a linear process called AIDA: Attention, Interest, Desire, and Action.
Hence, social media platforms, online newspapers, and others make money by selling ad spaces, selling targeting information using the data gathered, and giving access to or selling the data itself.

Marketers have two choices when it comes to social media: either passive marketing — gathering data from their consumers posts and public information — or active — engaging with users, creating polls, paying influencers to advertise a product, etc..
Both parts are facilitated by platforms that allow tracking and analyzing the results of their campaigns. Advertisement is only a sub-section of marketing.

As for online newspapers, and many others, they have chosen to either sell ad space, thus attention, or switched to a subscription model in which the customer pays to get access.
Some argue that the centralization of ad services forces companies to offer paid services, leading to an overall more expensive online experience. There are big questions regarding the value of the content we consume online and how much should be paid for it. The concept of micro-transactions and micro-payments comes to mind, we’ll come back to them in the solutions part.

The internet conundrum is that users have developed a habit of expecting online services to be free. So, to make the services offered profitable, the owners have to extract valuable things from their users, namely their attention and generated data.

The digital culture expert Kevin Kelly describes the expected attributes of this economic space as:

  • Immediacy - priority access, immediate delivery
  • Personalization - tailored just for you
  • Interpretation - support and guidance
  • Authenticity - how can you be sure it is the real thing?
  • Accessibility - wherever, whenever
  • Embodiment - books, live music
  • Patronage - “paying simply because it feels good”,
  • Findability - “When there are millions of books, millions of songs, millions of films, millions of applications, millions of everything requesting our attention — and most of it free — being found is valuable.”

In direct relation with the data economy, a sub-type of economy has appeared that specializes in a particularly profitable data type: personal data. The surveillance economy, as it came to be called, centers itself around making profit from gathering personal information, commodifying it.
This data is obviously more valuable and can be sold for reasons other than precise targeted marketing, such as creating a voters profiles database to be sold to state actors. This was the case of Cambridge Analytica, who used Facebook as a tool to gather information. Allowing them to join up to 7 thousand data points on 240M people. We’ll see more of that later.

This grants companies access to information that can be used to tailor their products to their customers’ needs, and customize their marketing campaigns to specific categories of people, to what we call “buyer personas”. This is good for passive marketing.
For example, a job ad can be sent precisely to men between 25 and 36 that are single and have interests in cars.
When so many data points are involved in identifying an audience, this is referred to as microtargeting.

In the past advertising was used with the goal to inform consumer choices, today it is predominantly used to persuade them, going back to our section on persuasion and propaganda. It steers consumers toward the brands which invest more in advertising, rather than those that are sustainable and responsible but less competitive. Thus, driving global consumption artificially.

Active marketing, engaging with users, works well in a surveillance economy too. Companies can perform direct continual experiments on their customers without them being aware of it nor giving consent.
An A/B or Wizard of Oz testing phase that would’ve taken months to perform can be done in a matter of hours on social media, an almost instantaneous feedback.

In the past marketing used to be more of a one-to-many type of communication, with the internet it now appears more intimate, a one-to-one customized experience with the brand. The company can build a relationship with their customers for long-term success.
This relationship can then work in their favor through the viral nature of social media. The consumers who find what they are looking for will share their positive experience, while those who didn’t will also communicate it. The opinions will proliferate and increase in number.

From the data gathering side of the equation, big tech companies are then justified in amassing data at all costs, tracking and analyzing. This affects privacy and control over our lives, we’ll see some of the deleterious consequences later. The data includes location history, preferences, age, name, and other things we’ve seen in the previous data and metadata section.

These three or four types of economies, digital, data, attention, and surveillance, clash together. The first one generates a mountain of advertisements and content that floods people, making them more picky about what they pay attention to. Accordingly, the value of attention rises, it being more scarce, creating additional incentives to invest into specific targeted advertisement, hence more surveillance.
A true arms race driven by a contamination of the supply that is often filled with irrelevant, redundant, unsolicited, low-value information. An adverse effect of this information revolution.

One of the many consequences, which we’ll see later, is something called continuous partial attention, a phenomenon where someone’s attention is constantly divided. It can lead to increase stress and an inability to reflect or make thoughtful decisions, sometimes related to analysis paralysis.
This consequence is convenient to some actors, especially in relation to what we’ve seen in the propaganda and persuasion section.

There’s a lot of money involved and many reasons to secure a position in these new economies. As with any new commodity that spawns, is lucrative, and fast growing, it attracts regulators and policymakers that want to restrain its control and flow.

Every year, companies worldwide spend more than 1.3 trillion dollars on commercial communication and 600 billion dollars on advertising. The titans of the internet, Alphabet, Amazon, Apple, Facebook and Microsoft look unstoppable. They are part of the most valuable listed firms in the world. Their profits keep surging: they collectively racked up over 25 billion dollars in net profit in the first quarter of 2017.

This is compounded with the feedback effect that more data encourages more data. This confers an enormous advantage to already large incumbents, the ad technology suppliers that have amassed a wall of well guarded data.
In this economy it is quantity over quality.

The giants span different fields, Google can see what people search for, Facebook what they share, Amazon what they buy. There’s no room for small players such as online newspapers, or any other SMEs, they can’t compete. This means the biggest players have a strong grip on the supply chain, both the data, attention, and the ad tech.

Size isn’t the issue here but the danger it can have on the consumers in these economies. It impacts everyone along the supply chain, from publishers, to advertisers buying ads, and the consumers.
This can create conflicts of interests, giving a self-preferenciation to specific suppliers of ad technology. These later can then control the price of access to it, its quality of service, and the transparency going along with it.
With such control, these companies can thwart competitors, either by hoarding more data and making it impossible for others to rival them, or by acquiring or destroying contenders.
For instance, many think that Facebook’s acquisition of WhatsApp for $22 billion falls in that category, namely creating a barrier of entry and stifling competition.

Unquestionably, these data brokers are nothing without their network of buyers, the advertising companies and other entities interested in getting access to mined data and attention. While there are many small players, the bigger ones will often sign partnership deals with the data brokers, so-called third-party partnership (or sub-contractors), to exchange and interoperate data laterally.
There are countless examples of these generally non-transparent deals, at least non-transparent with regard to the data subjects which have their information exchanged without their consent.

For example, AT&T may use third-party data to categorize subscribers without using their real names to show them ads. In practice, that means AT&T can find out which subscribers have which health issues — or which ones bought over-the-counter drugs to treat it. The companies extracted this kind of information from the databases accessed through third parties which contained subscribers’ spending.

An investigation by the New York Times from 2018 detailed how Facebook had made special agreements to share/give access to personal user information and private chats to 150 tech firms, including Amazon, Apple, Microsoft, Netflix, Spotify, and Yandex. The list also included online retailers, car-makers, banks, phone manufacturers, and media organizations.

Facebook distinguished between two types of relationships: integration and instant personalization partnerships.
The first one is about offering Facebook’s features outside its own app or website, the users having to sign in to their account to allow it.
The second one, “instant personalization”, is about allowing other apps to interact with Facebook private messages to customize them.
Facebook noted that it ended “nearly all” of the integration partnership and most of its personalization partnerships.

Facebook is a gargantuan player, but only one among many like Google and Twitter. They all have such deals and act as data brokers.
The most interesting type of partnership relying on these gigantic databases of personal information are the ones with state actors. For now, let’s only note that the USA government requires would-be immigrants and visa applicants to submit five years of social media handles for specific platforms identified by them, mainly Facebook. We’ll see how lucrative social media are for states later.

With lots of money and people involved, many aggressive tactics are used to get the attention and personal data of people, any legal means are allowed. That is why they rely on persuasive technology, a term used to refer to any technology designed to change attitudes or behaviors through persuasion and social influence. In our case, they are used to persuade to act based on ads, pay attention, and give personal data.

The use of persuasive design is custom-fit to the receiver, for example it can be based on age-appropriate desires.
The preliminary information can be gathered from all the sources we mentioned in our data and metadata section such as smart utilities, and our daily ubiquitous online activities. This is all possible because the user inadvertently agreed to a hidden privacy policy, in most cases without reading it. In so far as this is the argument used by these platforms to allow this, while making it as cumbersome as possible to understand what the policies imply.

Then, once the target is set, the software platform algorithms will explicitly or implicitly attempt to win the attention of the users through interface and psychological tricks based on the interests and personalizations from the data extracted earlier.
This includes mundane things such as catchy click-bait titles of news articles, and visual tricks such as using shocking pictures, impactful fonts, and attractive colors. The usual in marketing and other types of media (TV, newspapers, radio).
The new nefarious techniques rely on dark patterns, user interfaces that are designed to mislead and direct users into acting a certain way. This includes hiding parts of the interface that are supposed to be there for legal reasons, encouraging users to sign up or buy services or items they didn’t intend to, and more

Additionally, the platforms will create an environment that nurtures attention, shaping it using the data acquired. This ranges from benign things such as showing users what they are looking for, to turning the digital environment into a casino-like environment.
Social media are especially known to employ this stratagem, relying on the craving for social feedback that people want and mixing it with a reward system, be it “like”, “views”, or “shares”. Overall, this is a vicious cycle made up of a classical conditioning scheme that creates a drug-like addiction for the users participating on social media.

These techniques work well on adults but even better on kids. Perniciously, the different platforms involved have used them to target younger and younger people. The typical age when kids get their first smartphone has fallen to 10yo.
There’s more profit to be made out of a blank mind than an already filled one. This manipulation for profit is a reminder of our section on propaganda and how having full control of an environment allows for indoctrination.

The children are then caught in a virtual environment replacing their real-world needs and basic human drives: to be social and obtain goals.
To reiterate clichés, for teenage girls it could be wanting to be socially successful, and for teenage boys to want to earn competences. Meanwhile, they can easily fall victim to the internet’s attention and surveillance economies — an endless pursuit of stimulating virtual points which they believe will make them happy and successful.
I digress, we’ll have a full part with consequences.

Strikingly, the tracking and fight for attention starts earlier than childhood, the digitalization of our lives starts before birth. This is why data on pregnant women is the most valuable. We are the very first generation to be datafied before birth, this is a historical transformation.

On social networks, people most of the time want to entertain themselves, and entertainment is what makes them stay. One type of very popular entertainment is political infotainment, a sort of caricature of politics. A large portion of end users visit social media only for politics, and so the platforms will be incentivized in promoting this type of content. They give what people are looking for.

Many of these digital spaces rely on something called a “feed” or “home” page that will be generated and presented by default. This page will be optimized and curated by algorithms for the goal of the platforms. It is highly individualized, the news and information are selected by engagement value, intentionally designed to isolate end users from information that will make them leave, and creating a self-reinforcing feedback loop of confirmation, thus satisfaction.

From recommendation engines, search results, trends, autocomplete forms, it all happens through the mediation of algorithms. Whatever content is able to satisfy the criteria of the algorithms will be promoted, be it through clickbait titles, paying for fake views, fake reviews of products, bots interaction, and outrage-related and emotion inducing content.
The objective is to predict what we are more likely to interact with: click, view, engage.

The algorithms themselves are impartial, they do not know what the actual content is, and cannot discern between what is true and what isn’t. They are simply good at what they were programmed to do.
The algorithms are trade secrets of the platforms, opaque tools that influences millions of people each day. The data brokers and networks use this as a justification of non-accountability for the consequences, something we call mathwashing. Hiding behind the algorithm and pleading free speech.

Algorithm amplification is the phenomenon we are experiencing, the automatic radicalization of views and disinformation as an instrument for more views and spiraling out of control. It is a business choice made in pursuit of profit.
Social media will reward, through the point systems, extreme stances on subjects, these serve as the lubricant for these businesses, making them more profitable and influential. Studies have shown how users feel satisfied when finding a niche to cater to that generates a lot of these points (likes/shares).

This is an effect that is comparable with TV long-term media effects theory. It states that the more an individual watches television, the more they believe social reality matches what they see on it.

These have ramification in the real world, the consequences can be dire, as we’ll see later. For example, hate speech amplification can thrive and fuel genocide. Internet platforms play a dominant role in our conversation, extremism cultivated online has real world ramifications.

Evidently, they are now under public scrutiny and lawmakers want to intervene but social network, ad tech service providers, and others in the loop will fight to defend their castle of data.

The most common argument is that users have consented to what is happening by agreeing to the privacy policy and EULA they were presented with when signing up to the service. It is ok because it is the inevitable conclusion of our online activity. In other words, asking companies whose business models revolve around exploiting personal data for consumer-influence techniques to explain their privacy policies is like asking someone to shoot themselves in the foot.

The companies are now tweaking their tactics, using a patch-like approach rather than tackling the real issue, which is the core of their business. We’ll see in the last part some real solutions.

For example, AT&T recently said they would stop sharing users’ location details with data brokers, Facebook said it stopped allowing advertisers to use sensitive categories for targeting, Google said it’s now allowing users to download masses of their data.
Recently Google is also going to remove third-party cookies, to then allow third-parties to have access to the data only through them and getting information about groups and not individuals. They also get information directly from other sources such as their analytics and AMP service. A sort of differential privacy. Arguably, these give Google even more power over this type of data and drives away competitors.

In some countries, like the USA, the companies hide behind laws related to free speech to justify their policies. They are claiming that they do not want to be arbiters of truth.
That helps in taking the blame away from them, leaving them unaccountable as no USA critics want any platform to act as a censor.

As a last resort, like in any industry big enough, these companies don’t shy away from using their money to convince the lawmakers anywhere in the world.
Between 2005 and 2018, the big five tech in the USA have spent more than half a billion dollars lobbying the USA Congress, and donating to parties. Literally purchasing power.
Moreover, other databrokers also spend as much, companies such as CoreLogic spent as much as $215,000 on lobbying in 2020; and Acxiom, which spent $360,000 on lobbying in 2020 for issues related to data security and privacy.

Amazon, Apple, Facebook, Google and Microsoft have collectively thrown $582 million at legislators in an attempt to influence the lawmaking process and make sure their voices (and interests) are taken into account.

All of the firms have taken a position on privacy policy, with the issue coming up 3,240 times in all reports files—by far the most talked about topic. Last year alone, $66.4 million was spent on privacy issues by the tech giants. That includes $16.6 million from Google, $15.1 million from Amazon and $13 million from Facebook.

Whatever legality they deal with, like anything legal, depends on the country where the law is applied. Thus, any solutions will be localized, as we’ll see later.

This concludes our review of the emergence of new economies in the digital space with regards to social networks and ad tech providers. We’ve first seen the two new assets: data and attention. Then we’ve looked at their worth and which actors had incentives to capture them. Next, we’ve seen how a few are owning the pipeline in this new infrastructure and how they have agreements with third parties. After that we’ve dived into some of the manipulation tactics used to gain such assets. Finally, we’ve listed a few of the defenses some of these companies have used against policymakers.

Table Of Content


Attributions: E. Sibley, A Key to Magic & the Occult Sciences, c. 1800

March 21, 2021

Phil Hagelberg (technomancy)

in which there is no such thing as a functional programming language March 21, 2021 11:35 PM

There is no such thing as a functional programming language.

Ahem. Is this thing on? Let me try again.

There is no such thing as a functional programming language.

All right, now that I've got your attention, let me explain. Functional programming does not have a single definition that's easy to agree upon. Some people use it to mean any kind of programming that centers around first-class functions passed around as arguments to other functions. Other people use it in a way that centers on the mathematical definition of a function such as ƒ(x) = x + 2; that is, a pure transformation of argument values to return values. I believe it's more helpful to think of it as a "spectrum of functionalness" rather than criteria for making a binary "functional or not" judgment.

oregon coast

So functional programming is an action; it describes something you do, or maybe you could say that it describes a way that you can program. Functional programming results in functional programs. Any given program exists somewhere on the spectrum between "not functional at all" to "purely functional". So the quality of "functionalness" is a property that you apply to programs.

Obviously "functional programming language" is a term in widespread use that people do use to describe a certain kind of language. But what does it really mean? I would argue that a language cannot be functional; only a program can be more or less functional. When people say "functional programming language" what they mean is a language that encourages or allows programs to be written in a functional way.

Except for very rare cases, the language itself does not force the programs written in it to be more or less functional. All the language can do is make it more or less difficult/awkward to write functional programs in. Ruby is rarely called a functional programming language. But it's possible (and often wise) to write functional programs in Ruby. Haskell is basically the textbook example of a functional programming language, but imperative Haskell programs exist. So calling a programming language functional (when taken literally) is a bit of a category error. But "a language that encourages programming in a functional way" is an awkward phrase, so it gets shortened to "functional programming language".

Incidentally the exact same argument about "functional programming language" can be applied to the term "fast programming language". There is no such thing as a language that is fast. Only programs can be fast[1]. The language affects speed by determining the effort/speed trade-off, and by setting an upper bound to the speed it's possible to achieve while preserving the semantics of the language[2]. But it does not on its own determine the speed.

Please don't misunderstand me—I don't say this in order to be pedantic and shout down people who use the term "functional programming language". I think it's actually pretty clear what people mean when they use the term, and it doesn't really bother me when people use it. I just want to offer an alternate way of thinking about it; a new perspective that makes you re-evaluate some of your assumptions to see things in a different light.

[1] If you want to be even more pedantic, only individual executions of a program can be fast or slow. There is no inherent speed to the program that exists in a meaningful way without tying it to specific measurable runs of the program.

[2] For instance, the Scheme programming language has scores of different implementations. The same program run with the Chez Scheme compiler will often run several times faster than when it's run with TinyScheme. So saying "Scheme is fast" is a category error; Scheme is not fast or slow. The same is true of Lua; you will usually get much faster measurements when you run a Lua program with LuaJIT vs the reference implementation.

Ponylang (SeanTAllen)

Last Week in Pony - March 21, 2021 March 21, 2021 11:19 PM

Version 0.6.3 of ponylang/ponyup has been released. The first recording is available for implementing RoaringBitmap in Pony.

Derek Jones (derek-jones)

Software engineering research problems having worthwhile benefits March 21, 2021 11:06 PM

Which software engineering research problems are likely to yield good-enough solutions that provide worthwhile benefits to professional software developers?

I can think of two (hopefully there are more):

  • what is the lifecycle of software? For instance, the expected time-span of the active use of its various components, and the evolution of its dependency ecosystem,
  • a model of the main processes involved in a software development project.

Solving problems requires data, and I think it is practical to collect the data needed to solve these two problems; here is some: application lifetime data, and detailed project data (a lot more is needed).

Once a good-enough solution is available, its practical application needs to provide a worthwhile benefit to the customer (when I was in the optimizing compiler business, I found that many customers were not interested in more compact code unless the executable was at least a 10% smaller; this was the era of computer memory often measured in kilobytes).

Investment decisions require information about what is likely to happen in the future, and an understanding of common software lifecycles is needed. The fact that most source code has a brief existence (a few years) and is rarely modified by somebody other than the original author, has obvious implications for investment decisions intended to reduce future maintenance costs.

Running a software development project requires an understanding of the processes involved. This knowledge is currently acquired by working on projects managed by people who have successfully done it before. A good-enough model is not going to replace the need for previous experience, some amount of experience is always going to be needed, but it will provide an effective way of understanding what is going on. There are probably lots of different good-enough ways of running a project, and I’m not expecting there to be a one-true-way of optimally running a project.

Perhaps the defining characteristic of the solution to both of these problems is lots of replication data.

Applications are developed in many ecosystems, and there is likely to be variations between the lifecycles that occur in different ecosystems. Researchers tend to focus on Github because it is easily accessible, which is no good when replications from many ecosystems are needed (an analysis of Github source lifetime has been done).

Projects come in various shapes and sizes, and a good-enough model needs to handle all the combinations that regularly occur. Project level data is not really present on Github, so researchers need to get out from behind their computers and visit real companies.

Given the payback time-frame for software engineering research, there are problems which are not cost-effective to attempt to answer. Suggestions for other software engineering problems likely to be worthwhile trying to solve welcome.

Gokberk Yaltirakli (gkbrk)

Constructing a sponge function from MD5 March 21, 2021 09:00 PM

.iocell { display: flex; justify-content: space-between; align-items: center; } @media (max-width: 540px) { .iocell { flex-direction: column; } .iocell * { width: 100%; } }

While doing some research, I came across the term “Sponge function”. After playing around with them and implementing one inside my kernel, I decided to write this blog post about how to write a simplified version. To keep low-level cryptographic code to a minimum, we will be relying on the hash function MD5. Buckle in, this is going to be a long one.

This article will start from the simple concept of an MD5 hash, and incrementally build on it until we implement a lot of common functionality that seems like black boxes. Every step should be small enough to be digested individually, while still contributing to the overall understanding of the topic. If anything is unclear, feel free to discuss it in the comments. The post is designed so that you can pause and play around with the concepts yourself any time you want, or speed through it if that is what you prefer.

Since we are going to base our project on the MD5 hash function, let’s set aside a small section to go through what it is. We will treat MD5 as a black box and ignore any complicated details for the sake of brevity.

MD5 - Background

MD5 is a cryptographic hash function that maps an arbitrary amount of data into 16 bytes (or 128 bits). In its heyday, MD5 was the go-to choice for hashing passwords, checking files for corruption, and tagging data against tampering. Those days are long gone. It has been considered broken for some time, and it is not recommended using it for anything security-related. But it is well-known and has been implemented for practically any computing device ever created. Luckily for us, Python comes bundled with a collection of hash functions in the hashlib module. Let’s quickly see how it works.

In [3]:
md5(b"Test 123").hex()
Out [3]:
Out [3]:

As we can see, inputs of different lengths map to fixed-size outputs, and small changes in the input lead to completely different output values. Basically what we would expect from a hash function. In this article, we will be (ab)using the MD5 hash function to create a sponge function. But before we can do that, we need to know a sponge function is.

Sponge functions

A sponge function is a cryptographic function that can “absorb” any number of bits and “squeeze” any number of bits, like a sponge. This is different from what we observed about MD5; while MD5 will only produce fixed-size outputs of 16 bytes, a sponge function can output 1 byte, 26 bytes, 5000 bytes or any number that you like. This sounds fun and could be useful for a lot of different tasks, so let’s do some unholy programming and turn MD5 into a sponge function.

Sponge functions are fascinating. You can use a sponge function as a hash function, random number generator, Message Authentication Code or for data encryption. It would be apt to describe one as a cryptographic Swiss army knife.


In order to create a sponge function, we need an internal state (which is just a buffer), and function to pseudorandomly transform one state into another. We will take advantage of the two properties of the MD5 hash, our state buffer will be the 16 bytes of MD5 output and our transform function will be MD5 itself.

Sponge functions keep most of their internal state hidden. Both the bits absorbed and the bits squeezed only touch a small portion of it, so the output never reveals the full state of the function.

  • The first step is to initialize the state, either to zero or to a sensible default value.
  • For each byte of the input
    • The first byte of the state is XOR’ed with the input byte.
    • State is replaced by MD5(State).

This process absorbs all the input data into the state. After we have absorbed our input, we can squeeze out as many output bytes as we want by following a very similar process. For each byte that we want to produce:

  • Output the first byte of the state.
  • Transform the state using MD5.

Warning! You probably don’t want to use this for anything too sensitive. This is a proof-of-concept implementation using the broken MD5 function as a base. At least pick something better like ChaCha20 or SHA-512. In general, we want a large state and a transform function that mixes the state really well.


Now that we have briefly gone over the theory, let’s get to the implementation. We will take this step by step and implement each operation we mentioned above. Our first step is the transform function.

Transform function

According to the theory we outlined above, we need a transform function that will take our state and pseudorandomly map it to another state. In our case, the MD5 hash function will be doing the heavy lifting for us. And by heavy lifting I mean MD5 will be doing pretty much the whole job.

We can transform the current state by passing it through the MD5 function. Here’s a small demonstration.

In [5]:
# Initial state

# Transform once

# Transform again

# And so on...
Out [5]:
Out [5]:
Out [5]:

Looks like everything is working. Let’s encapsulate this in a method of the Sponge class. Every time we absorb or squeeze a byte, we will mix the state using this method.

In [6]:
class Sponge(Sponge):
    def transform(self):
        self.state = md5(self.state)


As we mentioned before, the state needs to be initialized before we start absorbing and squeezing any bits. Since we are using MD5, we want our state to be 16 bytes. Fortunately, MD5 makes sure that no matter the value we provide, the state will end up being 16 bytes. So we can pick any value, including an empty string. Let’s go with that option.

In [7]:
class Sponge(Sponge):
    def __init__(self):
        self.state = b""

Let’s see if everything is working. After creating a Sponge instance, we should be greeted with the MD5 of the empty string, d41d8cd98f00b204e9800998ecf8427e.

In [8]:
s = Sponge()
Out [8]:

Absorbing a byte

Using the logic from the theory section, we can write the code for absorbing a single byte easily. We will replace the first byte of the state with input XOR first byte, and then transform the state.

In [9]:
class Sponge(Sponge):
    def absorb_byte(self, byte):
        self.state[0] = byte ^ self.state[0]

We can quickly test that this results in different states after we absorb different data. Let’s try to absorb [1,2] and [2,1] and observe the difference in the states.

In [10]:
s = Sponge()

Out [10]:
In [11]:
s = Sponge()

Out [11]:

Absorbing a buffer

Generalizing this to buffers of arbitrary sizes is trivial. Just iterate over a buffer and absorb the bytes one-by-one. This is a useful abstraction because in real code we will commonly work with buffers instead of individual bytes.

In [12]:
class Sponge(Sponge):
    def absorb(self, buffer):
        for byte in buffer:

Here’s a quick sanity check: Our state should be different from the empty state after absorbing bytes. Let’s quickly verify this before moving on.

In [13]:
s = Sponge()
Out [13]:

Squeezing a byte

Since we don’t need to do any input-mixing, our squeeze logic will be simpler than our absorb logic. Following the theory part, we will output the first byte and transform the state again in order to produce one byte.

In [14]:
class Sponge(Sponge):
    def squeeze_byte(self):
        byte = self.state[0]
        return byte

Let’s produce some bytes and see if it’s working.

In [15]:
s = Sponge()
[s.squeeze_byte() for _ in range(5)]
Out [15]:
[40, 243, 39, 189, 220]

Squeezing a buffer

Going from extracting single bytes to exctracting buffers is not too difficult. We can use a list comprehension to write this in a concise way.

In [16]:
class Sponge(Sponge):
    def squeeze(self, size):
        buf = [self.squeeze_byte() for _ in range(size)]
        return bytes(buf)
In [17]:
s = Sponge()
Out [17]:

It might seem like a very small amount of code, but this is all we need. It might be useful to add some convenience funtions later, but for 99% of the use cases these methods will be sufficient. Now we can start playing around with our sponge function.

Use cases

In the beginning, we mentioned that sponge functions have a wide range of cryptographic use cases. In this section I will implement them in simple ways to provide some examples on how useful sponge functions can be.

Hash function

Hashing is the easiest thing to implement with a sponge function. In fact, we already saw a demonstration of this when testing the squeeze function above. To clarify; we can produce a hash by absorbing all the input and squeezing a fixed number of bytes.

In [18]:
def sponge_hash(data):
    s = Sponge()
    return s.squeeze(10).hex()

sponge_hash(b"Test 123")
sponge_hash(b"Test 113")
Out [18]:
Out [18]:
Out [18]:

This seems to fit our criteria of a hash function; inputs of different sizes map to fixed-size outputs, and small changes in the output result in completely different hashes. You can substitute 10 with any other length in order to change the output size of your hash. In general, longer hashes are less likely to have collisions but take up more space. You can play around and pick the sweet spot for your use case.

Random number generator

Random number generation is also something that can be done with sponge functions. The basic idea is to absorb the RNG seed and then squeeze out bytes for as many random numbers as you need. In the following example, I am using a fixed seed in order to generate 10 unsigned 16-bit integers.

In [19]:
import struct

s = Sponge()
s.absorb(b"Seeding the RNG")

def rng():
    buf = s.squeeze(2)
    return struct.unpack('H', buf)[0]

[rng() for _ in range(10)]
Out [19]:
[29342, 19407, 47040, 9984, 55893, 40500, 56312, 36293, 58610, 10880]

If we use the same seed, we will always get the same output. This might sound counterintuitive for the goal of generating “random” numbers, but it is commonly required to be able to replicate random results. If this is not something you need, you can seed from an actually random source or something that changes regularly like the current time. This depends on what qualities you expect from your random numbers. Below is a demonstration of how to read a random seed from /dev/urandom.

In [20]:
s = Sponge()

with open("/dev/urandom", "rb") as urandom:

[rng() for _ in range(10)]
Out [20]:
[56437, 39690, 47308, 16515, 29378, 11318, 32523, 18419, 47972, 4874]

Idea! You can absorb values while generating them as well, this allows you to periodically reseed your RNG using external sources.

Message Authentication Code

We can use a sponge function in order to produce a mechanism that can produce and verify signatures using a secret key. This is a very common technique, especially in mobile and web applications where it is used to store the session on the client without letting them tamper with it. If you want to read more about this use-case, check out JSON Web Tokens.

In order to produce a signature; we will absorb the data, and the secret key. After this, we can squeeze any number of bits that can be used as the signature.

In [21]:
def sign(data, key):
    s = Sponge()
    return s.squeeze(5)

data = b"Hello world!"
key  = b"password123"
signature = sign(data, key)

Out [21]:

Verification of a signature can be done by generating the signature yourself and comparing the signature you received with the signature you generated. If they match up, the data and the signature have not been tampered with.

In [22]:
def verify(data, sig, key):
    correct = sign(data, key)
    return sig == correct

verify(data, signature, key)
Out [22]:

As expected, the signature can be verified successfully. Let’s try to modify the data a little and switch two characters around.

In [23]:
data = b"Hello wordl!"

verify(data, signature, key)
Out [23]:

Similarly, we can have the correct data and tamper with the signature instead. The verification fails, showing that both the signature and the data are protected against corruption and tampering.

In [24]:
data = b"Hello world!"
signature = bytes.fromhex("481e4c2b9d")

verify(data, signature, key)
Out [24]:

Stream cipher

A stream cipher allows us to encrypt and decrypt a stream of bytes using a single secret key. This can be used to make sure only you, or anyone you entrust with the secret key can decrypt the data.

In [25]:
def stream_cipher(data, key):
    s = Sponge()
    output = bytearray(data)
    for i in range(len(data)):
        key = s.squeeze_byte()
        output[i] ^= key
    return output

data = b"Hello, world!"
encrypted = stream_cipher(data, b"password123")

Out [25]:

Decoding a stream cipher is very simple, in fact it takes no code at all. Simply encrypting the already encrypted value with the correct key will end up decrypt your data. Let’s try to decode our data with the correct and incorrect passwords.

In [26]:
stream_cipher(encrypted, b"password123")
stream_cipher(encrypted, b"password132")
Out [26]:
bytearray(b'Hello, world!')
Out [26]:

Idea! You can combine the Message Authentication Code and the Stream Cipher in order make a chunk of data that is encrypted and resitant against tampering. This is called Authenticated encryption commonly done in real protocols. Try to implement the AE and AEAD variants.

Warning! It is recommended to also include an IV / nonce with your key in order to make sure the same plaintext encrypts to different ciphertexts.

Time-based one-time password

You might have noticed that a lot of services these days ask you for one-time tokens when trying to authenticate. These tokens are usually displayed as 6 digits and expire in ~30 seconds. Using a sponge function, we can implement our own version pretty easily. Here’s how these one-time tokens work.

  1. The server and the client have a pre-agreed secret key.
  2. When authenticating, the server asks the client to produce a token.
  3. The client absorbs the current time and the secret key in order to produce a token, and sends it to the server.
  4. The server independently produces a token using the same key and following the same rules.
  5. If the tokens match up, the client is granted access.
In [27]:
import time

key = b"Secret key 123"

def get_otp(key, period=10):
    t = time.time()
    value = int(t / period)
    time_left = period - (t % period)
    s = Sponge()
    otp = [s.squeeze(1).hex() for _ in range(3)]
    otp = ' '.join(otp)
    return otp, int(time_left)

otp, time_left = get_otp(key)

f"OTP is '{otp}'."
f"Valid for {time_left} more seconds."
Out [27]:
"OTP is '7c 0b c8'."
Out [27]:
'Valid for 7 more seconds.'

If the code is still valid, meaning that time_left is still not zero, the OTP will be considered valid.

In [28]:
otp == get_otp(key)[0]
Out [28]:

If we wait until the timer runs out, our OTP will no longer validate.

In [29]:
time.sleep(time_left + 1)

otp == get_otp(key)[0]
Out [29]:

Idea! It is recommended to also accept codes that should have been generated before or after the current time in order to account for clock skew. After all, the current time is the input that determines what the code will be, so authentication won’t be possible if the clocks don’t match up.

Block cipher

Stream ciphers that use cryptographic hashes have a risk of running into cycles. This is when calling the transform function on the state will eventually go back to a previous one. In order to mitigate this, we can use a block cipher instead.

The difference of a block cipher is; instead of constructing the sponge once and squeezing bytes from it for the whole stream, we instead absorb a counter into the sponge along with the key and nonce in order to generate a fixed block of bytes. This is where we get the name “block cipher”.

In [30]:

def get_block(key, counter):
    s = Sponge()
    return bytearray(s.squeeze(BLOCKSIZE))

def block_encrypt(data, key):
    size = len(data)
    result = b""
    counter = 0
    while data:
        # Chop off BLOCKSIZE bytes from the data
        data_block = data[:BLOCKSIZE]
        data = data[BLOCKSIZE:]
        # Generate a block cipher block
        block = get_block(key, counter)
        for i, byte in enumerate(data_block):
            block[i] ^= byte
        result += block
        counter += 1
    return result[:size]

data = b"Hello, world! Don't forget to stay hydrated."
encrypted = block_encrypt(data, b"test")
Out [30]:

As we did with the stream cipher, let’s try to decrypt our data with correct and incorrect keys.

In [31]:
block_encrypt(encrypted, b"test")
block_encrypt(encrypted, b"TEST")
Out [31]:
b"Hello, world! Don't forget to stay hydrated."
Out [31]:

Closing words

If you made it this far, thank you for reading my article. I’d appreciate any emails or comments. You now have an understanding of how to implement some commonly used cryptographic technologies from scratch, please let me know what projects you end up doing with sponge functions.

March 20, 2021

Gonçalo Valério (dethos)

And… the blog is back March 20, 2021 10:30 PM

You might have noticed that the website has been unavailable during the last week (or a bit longer than that), well, the reason is quite simple:

OVH Strasbourg datacenter burning OVH Strasbourg datacenter burning (10-03-2021)

It took sometime but the blog was finally put online again, new content should be flowing in soon.

And kids, don’t forget about the backups, because the good old Murphy’s law never disappoints:

Anything that can go wrong will go wrong


March 19, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Artifacts And Spaces: Memes & Cults March 19, 2021 10:00 PM

Metamorphosis from a frog's head to Apollo. Repelled bestiality

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 1 — The Artifacts and Spaces
  • Section 4 — Memes & Cults
Table Of Content
  • Defining Memes
  • Internet Memes As A new Mode Of Communication
  • Memeplex — The Meme Ecosystems
  • Effective Memes
  • Memeoids
  • Relation To Cults

In this last section of our first part about artifacts and places, we’ll explore memes, internet memes, and cults, their definitions, how they have transformed and crisscrossed, their relationship, their role as communication vectors, and the extreme forms that they can take.

Meme is a word that comes up in many conversations. It can refer to two related definitions, the classic one and the internet-related one.

Let’s start with the classic definition.
The Oxford English Dictionary defines a meme as “an element of culture that may be considered to be passed on by non-genetic means, especially imitation”.
Keith Henson defines memes as “replicating information patterns: ways to do things, learned elements of culture, beliefs or ideas.”

A meme is a unit of culture, it is copied from other individuals, and propagates by being actively shared. It encompasses constituents of culture within itself. A mix of narrative, ideas, behaviors, and tropes.
The meme information has not only to propagate but also to have an impact and persist.

The memotype, the actual information-content of a meme, its semantic core, can have direct or indirect meaning. It can use figurative language that resonates within a culture such as commonly recurring images, rhetorics, motifs, and clichés.
In its physical form, its appearance, a meme can be contained in any media shell: verbal, visual, textual, auditory, gustatory, sensory, or anything the senses can allow.

These definitions are enticing but don’t include the essential novelty that the concept of meme has brought: a parallel with genes. The name comes from a mix of gene and mimetes, from the Greek mimetes, the imitator or pretender. It was initially supposed to be called Mimeme but got abbreviated to meme.
Thus, memes also add the notion that the transmitted information is granular and can be studied from an evolutionary standpoint. Memes are transported in their media package, the copies are subject to variation, and they compete for space in our memories and for the chance to be copied again. Only successful variants can survive.

Susan Blackmore re-stated the definition of memes as: “whatever is copied from one person to another person, whether habits, skills, songs, stories, or any other kind of information. Memes, like genes, are replicators in the sense as defined by Dawkins.”

Memetics is the study of the analogy between the elements moving in an “information society” and Darwinian evolution. This is done in multiple ways, one of them is to apply models to visualize the transfer of information, to see the criteria that makes them stick, to analyze what makes them successful and reproduce. We’ve done a great overview in the previous sections on big data and models of information flow.

There are two movements that argue different definitions of what memes are.

  1. A unit of cultural transmission (that can be copied, located in the brain). That can be described as a memory item, a portion of neurally-stored information that is instantiated/caused by interacting with another person’s nervous system. (I know this is heavy).
  2. Observable cultural artifacts and behavior.

Some people think memes have a physical nature in the brain and others think that they only exist within society and not as observable entities in the brain. Essentially, one group defines memes as internal and another defines them as external to oneself.
The first group argues that in the future we could obtain technologies that would allow us to find such observable meme.
Due to these separate definitions, it’s better to differentiate and study memes using two words.

  • i-meme, for the internal cognitive phenomena like neuronal behavior, studied via fMRI, neuroimaging techniques, genetic profiling, hormonal tests, and neurochemical reactions.
  • e-meme, for the external phenomena like culture and behavior studied by simulated propagation on social networks, game theory, attention and awareness studies, motivations, emotions, etc..

Like any scientific endeavor, it’s important that things be quantifiable and well-defined, otherwise it’s a show-stopper. Research on memes always precisely do that, for example let’s quote a paper: “A meme is information (using Shannon’s definition as that which reduces uncertainty) transmitted by one or more primary sources to recipients who, as secondary sources, retransmit the information to at least an order of magnitude more recipients than primary sources, where propagation or the information persist at least ten hours and the information has observable impact in addition to its transmission.”
Most research focus on e-memes.

It’s also good to be aware that there are skeptics about the terminology used when comparing memes to genes as they don’t equate to one another. The critics call memetics a pseudoscience because memes do not follow the same selection criteria as genes. Namely, the definition of a meme is still vague, there is no scientific demonstration of the replication happening, the encoding of a meme isn’t known, and the mechanisms of replication aren’t stable.
Memes can thus be placed in between the science fiction ideas of brainwashing, viral marketing, and Darwinian evolution.
Regardless, playing with the notion that cultural information moves like genes is interesting in and of itself as a tool for thoughts.

In common parlance when someone says meme they can mean “internet meme,” usually a photograph with a clever caption that is shared around the Web. Often created anonymously, remixed endlessly, and shared constantly, the most viral memes seem to materialize out of nowhere.
The term has seen a rebirth with the internet, even the Merriam-Webster notes that a meme is now popularly defined as “an amusing or interesting item or genre of items that is spread widely online especially through social media.”

Limor Shifman characterizes these internet memes as:

  • A group of digital items sharing common characteristics of content, form, and/or stance, which
  • were created with awareness of each other, and
  • were circulated, imitated, and/or transformed via the Internet by many users.

The author of the original meme idea, Dawkins, noted in 2013 that it “has itself mutated and evolved in a new direction. An internet meme is a hijacking of the original idea. Instead of mutating by random chance, before spreading by a form of Darwinian selection, internet memes are altered deliberately by human creativity. In the hijacked version, mutations are designed — not random — with the full knowledge of the person doing the mutating”

However, even though internet memes are associated with creative medium like funny image macros, this perception is limiting. Internet memes are “culturally resonant items easily shared and spread online”, and as with anything on this communication channel they have to follow its rules to succeed.
As a consequence, the sharing and virality aspects are self-explanatory, these criteria are required of social media messages for them to pierce through bubbles. In that case, it isn’t surprising that the concept of memes, when applied to the internet, will have to be deliberately designed using attention grabbers such as visual content. In that sense, internet memes are only instances of memes and not a brand-new definition.

However, unlike generic memes, the appearance of internet memes is limited to what can be transported on the internet. This can be textual, audio, or visual.

  • Verbal media memes: slogans, words, text, keywords, hashtags, etc..
  • Audio media memes: songs, melodies, etc..
  • Visual media memes: images, posters, “Photoshoped” pictures, caricatures, etc..
  • Hybrid/audiovisual media memes: video, vines, etc..

Internet memes are culturally relevant, broadly resonant, organically developed, and voluntarily spread.
On the internet they have become a new shared cultural language, a way to convey meaning. It’s a new type of vehicle for communication, packing ideas into transportable containers that can evoke meaning, emotions, memories and more.
The sentence “a picture is worth a thousand words” applies well to internet memes.

Memes are convenient and accessible mental model creators. They offer a quick way to construct a mental representation of a situation, event, or object and allow to process, organize, comprehend, explain, judge, and formulate predictions and inferences about these former.
They provide a narrative in a condensed package, and so are tools for analysis. They are media rhetoric, deconstructing complex ideas using compressed symbolic images/text/audio.
As you can see, it makes memes excellent persuasion and propaganda anchors and hooks, you can revisit the section on propaganda and persuasion for more info.

Even if some memes weren’t created with a purpose, they will sooner or later be appropriated by political or business elites for leverage. Many memes, because of their symbiosis with social media, have to frequently show strong emotions that is with or against ideologies, show humor, parody, be snarky, silly, witty, angry, and most predominantly be poignant.
As was said in the social media section, it is our own confused cultural codes that are abused to make messages spread. That is why successful memes take stances, either as meme-aggressor — discrediting an idea, political program, politics, or media person or authority — or as meme-protector — a message to keep the status quo, enhance a person’s profile or authority, or protect a state ideology.
Memes frame the attention on a single aspect of an issue and cannot confer enough nuances due to their condensed form. They are filtered views of the world.

However, in the eye of the average person, internet memes are looked at as entertainment content, more precisely infotainment — like anything that is on social media. It is easy to overlook their power as they do not appear to have substantive content and can take the form of hastily constructed cartoons. Nevertheless, after the 2016 USA elections, the media outlets of the world started crowning memes as “the new king of political communications”.

Unlike TV, printed media, and most other forms of communication, internet memes are built for speedy consumption on smartphones and are prioritized because of the visual aspect of social networking. They provide a quick emotional hit in comparison with long-winded articles, and thus an instant payoff.
Studies have shown that people perceive internet memes via a peripheral route in their brain, according to Richard Petty’s elaboration likelihood model of persuasion (ELM). This means that they rely on a general impression, the mood, and early part of the message contained in the meme and not on any significant cognitive efforts.
This goes together with the notion that internet memes are perceived as infotainment.
This peripheral route isn’t effective when it comes to long-term changes in attitudes but can have some influence on the short-term. However, as you can imagine, this can be countered with repetition, more units of the same message.

More than anything, memes are a way for individuals to express themselves, to show who they are and what they believe in. Consuming and sharing memes is an act of identification.
It has been found that there is a close relation between belonging to a sub-culture or having affinity to a group and the propagation of related internet memes. They are used as a form of commentary, or even rebellion against or for official discourse.
People might even seek out memes as a type of emotional release.

This creates a constellation of memes, a meme pool. In it, there are organizations of large groups that are copied and passed together, co-adapted for their survival. We call them meme complexes, or memeplexes.
The memes are aware of each others, like cross-breeding, sometimes cross-referencing the format, appearance, message, or semantic core of other memes.
This symbiosis reinforces individual memes.

The memeplex, like an ecosystem, balances itself with different memes, creating a memetic equilibrium. Similar to how propaganda sells a coherent world of cohesive meaning. Some people like to talk of memes as the new urban legends medium, the modern hermetism or mysticism, others as an ecological forest.
Experts in the domain of semiotic are getting interested in memes for this reason. Semiotic being the study of signs and symbols with an emphasis on their interpretation, context, and meaning. It covers many of the things we’ve seen and we’ll see, such as: ideologies, persistent beliefs, commonly held views, cultural anxieties, dominant mindsets, received wisdoms, expectations, preconceived notions, assumptions, widespread preoccupations, historical facts, principles, ideals, supposed eternal truths, unspoken tensions, shared hopes, master narratives, public opinions, prevalent attitudes, considerations of what is normal, outlook, established power structures, and more.

Through different means, such as the gathering of big data, it is possible to track the propagation of the memes across multiple communities and measure their influence, discover who influences others the most and which variants are best.

A research analyzing memes generation on online platforms has showed that fringe communities such as 4chan and reddit r/The_Donald (defunct) were big generators of memes, which then were influential on more frequented social networks such as Twitter. Their messages were mainly political ones, imbued with emotional and hateful content.
Nothing surprising here either.

Even though we can track the spread of memes, the real origin of them is often unknown. Some have come to call this phenomenon crowd-sourced democratic expression, and others an intellectual apocalypse. That reminds us of what social media are about.
This means that there is no accountability for the creation of the message. It makes it harder for political groups to blame or retaliate against the memotype, or even criticise the creator, as they could with mainstream media reporters or journalists.
Indirectly, that’s also an advantage for the propagandist, they can use memes as tools of black propaganda, hiding their identity.

We’ve seen that emotions and shock-value make a meme spread. What other criteria make a meme successful?

Let’s state something from the get go: Memes do not have to be true to replicate. Memes aren’t about their creation but about their spreading. The constant production of new content leads to a fight for the survival of the fittest.

Like genes, meme survival is measured along 3 axis fecundity, fidelity, and longevity. They also spread like viruses. There are studies quantifying the analogy with evolutionary pressure, showing the selection process and the movement between different communities, moving from fringe ones to more mainstream ones. Prosperous memes have to bridge cultures to live on. Being restricted to silos is a disadvantage and can lead to extinction.
Other studies show how the SIR (Susceptible - Infected - Resistant) model of disease applies perfectly to memes. Susceptible people being the ones who hadn’t seen the meme, infected people those who were actively interested in its content and to spread it further, and the recovered were those who had seen the meme and lost interest.
These research and models confirm that memes can be studied like infections. Consequently, they can be treated or have their propagation hindered by quarantine for example.

This is a topic that the author of the concept of memes, Richard Dawkins, directly tackles in the essay entitled “Viruses of the Mind”. Though the essay dives into the parallel with religious doctrines, it still puts down criteria that viruses have to fulfill to survive:

  • Spread silently, be difficult to detect (by its host)
  • Not spread if the virus is already present in the host (double infection)
  • Not kill its host, or infer too much damage, at least not rapidly enough to stop the spread

The essay then extracts and applies this to memes with two conditions:

  • A readiness to replicate information accurately, perhaps with some mistakes that are subsequently reproduced accurately.
  • A readiness to obey instructions encoded in the information, so replicated.

When it comes to the shell of the message, the visual nature of internet memes adds to their potential persuasive power on social media, making them more attractive. They are comparable to editorial cartoons in newspapers.
An image has the advantage of stickiness and brevity, communicating concepts quickly. This is something that has been studied. Our brain interprets images faster than text, thus reducing the likelihood of thinking deeply about the content, as we said earlier. 70% of our neurons are in our eyes, and we process images 60 thousand times faster than any other media type.
Rational discussion is less effective than a visual campaign. Images are more emotionally evocative, and emotional cues have a preferential path in the brain. We do not even need to be aware that these emotions are elicited, they could be subliminal.
As far as metrics go, research in the field of marketing clearly show that visual content is more shared than anything else, help retain information, and make following instructions easier.

When it comes to the memotype, the semantic core of the message, we’ve already mentioned that successful memes are reflections of us. It is worth reiterating.
A meme is an expression of our culture, our beliefs, behaviors, and values. The ones that go viral are unleashing our repressed cultural agenda, they don’t use novelty but use a potential that was already there. They exploit our latent, yet intrinsic, gaps in our cultural code, using our cultural vulnerabilities, the ones we aren’t immune to. Therefore, memes don’t compete for dominance by appealing to our intellect, our compassion, or anything related, but they compete on triggering our automatic impulses.
Logic and truth have nothing to do with memes, but they rely on our flight-or-flight reactions, the fastest reactions. From that perspective, memes aren’t really pro-social but anti-social.
Like viruses, memes reveal less about themselves than about their hosts. We wouldn’t be sick if we had an immune system capable of recognizing the shell and then neutralizing the code it contains.

Thus, memetic material that is provocative, sensational, making news headlines, and shocking will be more successful. Studies show that people are more willing to share memes that evoke stronger disgusts (Heath et al.).
Another study conducted in 2017 related to an analysis of the keywords used in memes shows that the most popular memes were politically related. From USA MAGA movement, to the USA libertarian, to the president of the USA at the time (This study was USA-centric).
That means the content often acts like a propaganda anchor or hook, and is receptive to the current paradigms and belief systems.

Then this all manifests through real consequences such as media confusions, lack of trusts, protests on the streets. It’s all viral memetics in action. We’ll dive into consequences in another part of this series.

Another criteria that helps survival is whether a meme can create a symbiotic relationship with other memes, what we referred to as memeplex. The meme could already fit properly with others, cooperating, or it could develop progressively a real or artificial compatibility.
Ganging up helps the meme flourish, and the longer they are attached together, the more they’ll merge into a single package, a co-evolution.
Additionally, memes can have defensive systems, reject other memes that would hurt it. Consequently, creating an intolerance within the meme complex.

The content and the medium aren’t sufficient alone, the context is extremely important too. We need to take into account the external factors such as:

  • The timing to coincide with major events
  • The architecture and dynamics of the network
  • The media literacy and facility of the target population

Let’s now review some of the results of research related to meme success.

The University of Memphis research shows that for an image macro to be successful the predictor variables were:

  • A shorter memes (fewer than four words)
  • The use of concrete terms (more memorable than abstract language)
  • Avoiding swear words as they hamper virality (negatively correlated)

A research on military memetics by Dr.Robert Finkelstein digs into the measurement of the fitness of memes. It calculates the memetic fitness unit as the number of person-hours transformed. The most fit memes correlated with:

  • Propagation (.428): number of recipients, type of recipients, dispersion of recipients
  • Persistence (.284): duration of transmission, duration of memory
  • Entropy (.087): small, medium, large (At this time, we consider smaller memes to be more important than larger memes. Larger memes may all be memeplexes (e.g., the typical human brain cannot hold more than 10 “chunks” of information in short-term memory))
  • Impact (.20): Individual consequence, societal consequence

That means, according to this research, that the most important factor to fitness are propagation and persistence, and impact of a meme, but mostly propagation.

The research also summarizes some criteria that should be applied to the information content of the meme, such as:

  • accuracy
  • relevance
  • timeliness
  • usability
  • completeness
  • brevity
  • security

The computer scientists at the Hebrew University tried to use an algorithm to predict the spread of certain hashtags on Twitter using a dataset of more than 400 million Tweets. Their findings were that successful memes factors were based on:

  • The meme’s content
  • The meme’s context
  • The social graph

Another paper on the military application of memes proposes the following:

  • Meme conciseness (1-10 words)
  • Choosing information which propagates, has impact and persists (Info-PIP), and be able to monitor them via metrics

Some papers even go as far as to say that it’s not really about the meme criteria but the network criteria. That’s what the Indiana University’s Center for Complex Networks and Systems Research found: memes that went viral were no different from those that didn’t, the success was due to the structure of the social network.

These research all go in a direction that echoes what we said earlier: memes are a reflection of us. These memes do not necessarily have to be malicious, they could have beneficial purposes. Today memes have surpassed genes as the dominant driver in human behavior and we have to understand their importance. For that we have to develop a healthy cultural immune response to hostile ones. We should learn to quarantine, distance ourselves, and apply immune attacks to neutralize them when necessary.
These are requirements in an information society.

Two concepts come to mind in relation to our society: memocides — the deliberate eradication of a memeplex from a population, by either killing the host or by censorship — and eunemics — inspired by eugenics but applied to memes, deliberately improving the quality of the meme pool by selective breeding. Eunemics is done through memetic engineering, using engineering principles to mold them.
This means memes can also be informational weapons, propaganda tools.

When meme and memeplex are brought to their extreme, a person is overtaken by them, letting go of their own survival instinct and favoring the meme instead. In the symbiotic relationship, the meme has won and consumed its host. The people affected by this phenomenon are called memeoid.
Examples of this include kamikazes, suicide bombers, cult members, and others. Keeping in mind that the previous criteria for reproduction still applies, namely that the meme should keep their host alive long enough to spread properly.

This is a trading between genetic reproductivity and memetic reproductivity, which might not coincide. Promoting the memeplex might not be correlated with promoting a gene pools. Replacing the purpose of one’s existence with the spread of the meme.

Richard Dawkins cites the following when it comes to memeplex consuming individuals, it becomes a sort of narrative.

  • Impelled by deep inner conviction that something is true/right/virtuous, against all reasons
  • Make positive virtue out of the strong stance on the meme
  • Conviction that mystery is a good thing
  • intolerant behaviour towards perceived rivals of the meme
  • If follow a different meme from parent it comes from a charismatic individual
  • The internal sensations of the meme holder are similar to ones associated with sexual love

Strong memeplex are thus similar to cults, people being mind controlled by ideologies and its members turning into puppets of the memes.
Singer’s six conditions of mind control reverberate this idea:

  • Keep the person unaware of what is going on, and what changes in themselves/their behavior is happening.
  • Control the person’s social and/or physical environment, especially the person’s time. Remind them of the narrative all the time.
  • Systematically create a sense of powerlessness in the person. Keep them away from their former social support group or identity.
  • Manipulate a system of rewards, punishments, and experiences to inhibit the former identity. Accomplished through various methods of trance induction, speaking patterns, guided imagery, etc..
  • Manipulate a system of rewards, punishments, and experiences to promote the group’s ideology and behaviors. Compliance are rewarded while questioning or doubts are met with redressed and rejection.
  • Put forth a closed system of logic and an authoritarian structure that doesn’t permit feedback and refuses to be modified except by leadership.

These are all things that we’ve seen in the mix of social media space and the usage of memes in their memeplex.
This is especially true of the attention-reward system of social media. As social primates, attention is a measure of status. When we get attention our brain releases chemicals such as dopamine and endorphins, creating an addiction (AAR: Action - Attention - Reward). This system can be shortcut with memes. The memeplex, and the ones carrying it, requires intense social interactions and behaviors such as sharing the meme or reenacting its content.

The members then create their own isolated bubbles of coherent meaning. Some of them opposed, clashing, and creating the rise of many of the things we’ve seen so far.

Clearly, as with propaganda, children are more vulnerable to subversion and are easy pray for memeplex. Children are pre-programmed to absorb useful information at thigh rate. It is harder as adults to remove the pernicious and damaging information that have been introduced early.

Deprogramming studies, used to get people out of cults, can be applied to memeoids too. They rely on invoking new capture-bonding social reorientation mechanisms.

Similar to our conclusion on propaganda, we have to say that some memeplex can be considered pathogenic but others can be beneficial or relatively harmless. The judgement is subjective.
In a strict sense, everyone, including you and I, are memeoids, carriers of memeplexes. We are all hosts, one way or another, but not everyone reaches a degenerative state, even though we have the potential.

A skill that should be developed is metamemetic thinking: being able to recognize and acknowledge the content of memeplexes, the illogicality in them, and anything related to their attributes.
The metamemetic can also be used to understand the people associated with them. Coming back to our data and metadata section, memes are information and thus can be used to characterize, analyze, and classify people.

This concludes our review of what memes are and their relation with the internet. We’ve first seen their definitions, one is based on a comparison with genes applied to cultural information, and the other to internet image macros and tropes. We’ve then made the parallel to how these two aren’t dissimilar and how internet memes are only instances of memes adapted to social media. We’ve taken a look at how people perceive these memes, how they are used as a way to communicate and express one’s identity. Then we’ve dived into the memeplex, how memes associate themselves to create a world, to better survive. Next, we’ve inspected what makes some memeplex successful by summarizing some research. Finally, we’ve investigated the close relation between the extreme form that memeplex can take, turning people into memoids, and cults.

Table Of Content


Attributions: J.C. Lavater, Physiognomik, Vienna, 1829

March 17, 2021

Gustaf Erikson (gerikson)

One year since WFH started March 17, 2021 05:55 AM

One year since WHO declared a pandemic March 17, 2021 05:54 AM

March 15, 2021

Ponylang (SeanTAllen)

Last Week in Pony - March 14, 2021 March 15, 2021 01:58 AM

We’re migrating the Pony Tutorial and Pony Patterns to MkDocs-Material. There’s a new tool for interactively exploring the Pony AST from a debugger. ponylang/http has been updated.

March 14, 2021

Derek Jones (derek-jones)

The impact of believability on reasoning performance March 14, 2021 10:38 PM

What are the processes involved in reasoning? While philosophers have been thinking about this question for several thousand years, psychologists have been running human reasoning experiments for less than a hundred years (things took off in the late 1960s with the Wason selection task).

Reasoning is a crucial ability for software developers, and I thought that there would be lots to learn from the cognitive psychologists research into reasoning. After buying all the books, and reading lots of papers, I realised that the subject was mostly convoluted rabbit holes individually constructed by tiny groups of researchers. The field of decision-making is where those psychologists interested in reasoning, and a connection to reality, hang-out.

Is there anything that can be learned from research into human reasoning (other than that different people appear to use different techniques, and some problems are more likely to involve particular techniques)?

A consistent result from experiments involving syllogistic reasoning is that subjects are more likely to agree that a conclusion they find believable follows from the premise (and are more likely to disagree with a conclusion they find unbelievable). The following is perhaps the most famous syllogism (the first two lines are known as the premise, and the last line is the conclusion):

    All men are mortal.
    Socrates is a man.
    Therefore, Socrates is mortal. 

Would anybody other than a classically trained scholar consider that a form of logic invented by Aristotle provides a reasonable basis for evaluating reasoning performance?

Given the importance of reasoning ability in software development, there ought to be some selection pressure on those who regularly write software, e.g., software developers ought to give a higher percentage of correct answers to reasoning problems than the general population. If the selection pressure for reasoning ability is not that great, at least software developers have had a lot more experience solving this kind of problem, and practice should improve performance.

The subjects in most psychology experiments are psychology undergraduates studying in the department of the researcher running the experiment, i.e., not the general population. Psychology is a numerate discipline, or at least the components I have read up on have a numeric orientation, and I have met a fair few psychology researchers who are decent programmers. Psychology undergraduates must have an above general-population performance on syllogism problems, but better than professional developers? I don’t think so, but then I may be biased.

A study by Winiger, Singmann, and Kellen asked subjects to specify whether the conclusion of a syllogism was valid/invalid/don’t know. The syllogisms used were some combination of valid/invalid and believable/unbelievable; examples below:

        Believable                  Unbelievable
        No oaks are jubs.           No trees are punds.
        Some trees are jubs.        Some Oaks are punds.
        Therefore, some trees       Therefore, some oaks
                   are not oaks.               are not trees.
        No tree are brops.          No oaks are foins.
        Some oaks are brops.        Some trees are foins.
        Therefore, some trees       Therefore, some oaks
                   are not oaks.               are not trees.

The experiment was run using an online crowdsource site, and 354 data sets were obtained.

The plot below shows the impact of conclusion believability (red)/unbelievability (blue/green) on subject performance, when deciding whether a syllogism was valid (left) or invalid (right), (code+data):

Benchmark runtime at various array sizes, for each algorithm using a 32-bit datatype.

The believability of the conclusion biases the responses away/towards the correct answer (the error bars are tiny, and have not been plotted). Building a regression model puts numbers to the difference, and information on the kind of premise can also be included in the model.

Do professional developers exhibit such a large response bias (I would expect their average performance to be better)?

People tend to write fewer negative tests, than positive tests. Is this behavior related to the believability that certain negative events can occur?

Believability is an underappreciated coding issue.

Hopefully people will start doing experiments to investigate this issue :-)

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Artifacts And Spaces: Data & Metadata: Personal Data & Models March 14, 2021 10:00 PM

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 1 — The Artifacts and Spaces
  • Section 3 — Data & Metadata: Personal Data & Models
Table Of Content
  • What Is Personal Data
  • How Is It Extracted, What’s Its Value
  • Information Flow Analysis
  • The Power of Models

The internet is littered with nuggets of information, some made of gold and some worthless. From this mine we can extract meta-information, inferences that can lead to more lucrative valleys.

Two of the most talked about are personal data and models, also known as statistical trends visualizations, big data analysis, and predictive studies. Let’s take a look at what is personal data, where and why people share them, why they’re valuable, and how they can be used with different types of models.
In another part of the series we’ll focus on the actors and consequences.

Personal data is any information that can be used to directly or indirectly identify a person, an attribute currently attached to them, a behavior, a preference or interest, personal history, anything.
These include things such as name, phone number, address, email, schoolbook, credit scores, geolocation, travel logs, interests, information about past purchases, health records, insurance records, online behavior, etc..
We call the person to which the personal data belongs a data subject.

Information gathering is omnipresent, we are deliberately, or not, leaving trails in the digital space. On a daily basis, in modern society, through our regular actions, it is inevitable that our activities will generate data that is collected, which may or may not require our permission or knowledge.
As we’ll see later, the online services and their sub-contractors benefit from this exchange process, relying on the brokerage of big data about their consumers.

The digital world is interwoven with our physical world, we are incentivized to enter information to access services and utilities. Everything is being datafied, anything that can be turned digital will be, including us from birth till death. Are we an information society?
Data collection has been normalized, and we will see the general effects of this in another section. This is justified by the advantages this can convey such as having an optimized service.

There are five places where data gathering happens: casual online browsing and searches, social media, third party marketing, smart widgets, and traditional data repositories.

The biggest collectors are internet-based navigation sites like Google, Yahoo, and Bing. Apart from the data source coming from their search engine services, these companies have a broader business model where they offer tools such as emails, document editors, file storage facilities, and others in which they gather information about users.
These are used for targeted ads. For example, Gmail, a Google owned email service, will direct their marketing campaigns based on the type of mails a user has exchanged.

Likewise, massive amount of data is collected via social networking websites and mobile applications like Facebook, Twitter, LinkedIn, etc..
Users on these platforms either voluntarily communicate information with a public audience by messaging them, or indirectly share their behavioral patterns and preferences by filling a persona — The persona having meta-information stored and used by the platforms and advertising partners.
The data subjects might be lead to believe they own the data they generate through their activities on the platforms, however depending on the legal policy of the services and the legislation in which they reside, it is often not the case.

Third-party marketing providers are good source of consumer data. This includes companies such as KBM, Acxiom, and Equifax which have built consumer databases with information including wage data, occupations, past purchases, transactions, etc.. It is frequent that these third-parties partner or sell their information database to other entities.
In this category we also find credit card companies that might share processing data.

Yet another source of data are all the connected widgets. That include devices such as smart watches, smart doorbells, facial recognition cameras, fingerprint scanners, toys, and others. These could connect to smartphone applications, or directly to the internet, and the data might be forwarded to a third-party by the parent company.

Finally, the classical places where we find data collections are the traditional repositories. These are the ones that have been used since forever by insurance companies and that contain information such as credit scores, vehicle registration records, medical records, and other official and non-official knowledge.

Each piece of information in itself might not be valuable but the aggregation and processing makes it so. It’s quantity over quality, and the more data the more the approximate improvement.

This data confers a huge advantage to companies that use it compared to competitors that don’t. Like a GPS using multiple points to find a spot, the data allows for finely grained personalization of services and data-driven decisions. Companies can understand customers needs and wants, and have better marketing, products, and services.
Some argue that the companies might even know more about you than you know yourself.

Most companies and industries already used similar knowledge base in the past for marketing, as a result they only supplement their existing internal source with the external ones they can buy.
This gives rise to a business model, a new type of economy, in which personal data is commodified, gathered at all costs to be resold later to these companies. We’ll get back to this in the next part.
This type of business is lucrative, there is a lot of money involved. Simply taking a look at names such as Alphabet, Amazon, Apple, Facebook, and Microsoft that have insane profits racking up over $25B each year should give an idea. Or even looking at commercial consumer database owners such as Acxiom that makes sales of $1.13 billions and have customers such as big banks, investment services, department stores, and automakers.

This data is pieced together, shared, aggregated, and monetized, fueling a $227 billion-a-year industry. This occurs every day, as people go about their daily lives, often without their knowledge or permission.

There is so much money involved because the return-on-investment for firms that have embraced it shows that it works. In an ever-moving world, it is always a leverage to know about your consumer target and predict trends.
The data isn’t available to everyone though, and the companies owning them aren’t keen on letting go of their business. Moreover, there are moral, ethical, and legal regulatory concerns that are starting to develop around the topic.

Practically, this gathered data needs to be processed to be used by companies or other entities. That amounts to collecting, structuring, organizing, storing, sharing, and modeling the information.

The digitalization allows not only to deduce direct information from the data (such as personal interests in a topic), but also indirect ones like the flow of information. For example, it can allow knowing which persons are more prone to interact together. How data moves is information in itself.

This can help to know via clear metrics if a message was received by the community or group it was intended to target. The message could then be honed or sent through different more adapted means for the audience. Essentially letting the message sender influence a group without interacting directly with them, while still receiving feedback response.

Seeing how a message propagates can also be used to identify which hubs or communities are more influential than others, if information is more likely to originates from certain bubbles.
We can examine what are the criteria of these messages originating from the influential communities. For example, some studies have noticed that more fringe and extreme communities can influence other ecosystems on the internet.
The flow of ideas between these interconnected networks is likely why the mainstream media now gets some of their news from social media.

Indirectly, the collection of information and the visualization of silos can be used to categorized people belonging to them. We could attribute certain characteristics and qualities to persons frequenting a group.

Epidemiological model such as the SIR spread model can be used to see the propagation of messages through time and space. Similarly, percolation models can be used to see which variety of information can reach a tipping point and spread virally: its qualitative attributes or the ones of the network it’s spreading on.

Another information we can get is whether applying different reproductive strategies for messages work better. Answering questions such as: Does quality or quantity matters? Which works best an R vs K selection strategies? What makes a message good, is it the high-volume, frequent repetition or high quality and low-volume?
Some studies have shown that on the internet it’s a game of volume and brute force and not specifically intent and design to be able to reach a large audience.

Theoretically, all this data can be used to see trends, the spirit of the time, the zeitgeist. Having insights into the architecture and dynamics of the networks and the information that lives on them open many possibilities. The related field of big data, consisting of extracting value out of a humongous amount of data, is booming.

Unfortunately, as with anything on the internet, a considerable quantity of the data generated doesn’t come from real humans but from algorithms. Studies approximate that 40% of the internet consists of non-human users. Consequently, the visualization of flow, the insights, the models, and the refining of messages are partly based on information given by bots.
Thus, a company’s decisions based on such data might be indirectly based on the will of algorithms, hence leading them to deceitful conclusions.

Furthermore, the knowledge and visualization of the flow of information can be abused by actors wanting to spread a narrative.
For instance, they might rely on side channels such as citizen journalists and fake news portals to then have their message be regurgitated by mainstream media. Effectively, they would have traded up the chain by using their insight of the connection between different networks.

Lastly, there are still questions regarding personal data and data gathering in general. Who is responsible when there is a breach? What type of processing is allowed on the data? Is the data subject allowed to update or correct their data? Who is the actual owner of the data? Can the data be transferred freely without authorization from the data subject?
These are all important questions we’ll tackle in the last part of this series when we’ll discuss the solutions to the issues brought by the internet.

This concludes our review of what personal data is and how data on the internet is collected to be used in models. We’ve first seen what personal data consists of. Then we’ve looked at all the places personal data is generated and collected. We’ve seen the normalization of this process in societies. Later, we’ve examined how this data is useful for marketers, and how it creates a lucrative business to gather the data to resell it to them. Next we’ve pondered about different types of models and ways to use the data apart from direct marketing. And lastly, we’ve considered how this model could lead to misleading insights if it is based on algorithms and not human behaviors.

Table Of Content


Attributions: W. Blake, Death Door, from: Gates of Paradise 1793

Gokberk Yaltirakli (gkbrk)

Status update, March 2021 March 14, 2021 09:00 PM

It’s been a while since the last Status Update on November 2020 so I thought it would be good to have a catch-up post. Be prepared, this is going to be a long one. After all, I haven’t done one of these in 120 days.

Development environment

There has been one small change to my development environment. I’ve gotten a CLion license. I’m planning to play around with it for a while to see if it improves my C++ workflow.

DSP / Radio

There has been some good progress on signal processing and radio-related projects since the last status update.

Frequency-hopping spread spectrum (FHSS)

I made a modulator for a frequency-hopping signal, and wrote receiver plugins for SDR Sharp and GQRX. The modulator changes the frequency of transmission based on the hash of the current time, and hops frequencies 10 times a second. It is able to be received and demodulated without problems.

It is possible to share the same frequencies between multiple transmitters, as their collisions are not very common. This modulation is good at avoiding narrowband noise, but it provides no processing gain to the signal as there is no correlation done. All I’ve done is transmit a narrowband FM signal with a hopping frequency.

APT image encoder fixes

There has been some improvements to my APT image modulator apt-encoder. Previously, the images created by GIMP needed to be manually edited to get rid of comments. Now the program knows how to ignore comments in image files.

Spectrogram drawing

I wrote a modulator that can transmit image files using the HackRF One. It can draw simple pictures on the spectrogram. The height and width of the transmitted image, which translate to the transmission speed and the bandwidth, can be configured.

I have written an article about it here.

Signal correlation

I started learning how to do signal correlation in order to get processing gain from spread-spectrum signals. I haven’t written a full modulator/demodulator yet, but I have tested locally with Python and things are looking promising.

Kernel / OSDev

There has been some kernel development in February. Below are the main points. As always, the kernel source code can be found in gkbrk/kernel.

Threading improvements

From the first days of multi-processing in the kernel, it was not possible to pass any data when spawning threads. You could pass a function pointer and communicate with the thread using shared memory. Thanks to some changes in the Task class, it is now possible to push and pop arbitrary objects to the stack of the process.

The first thing I did after implementing this is to make the music player fork to another process so it can play in the background while you are dropped back in the command prompt.

Improved PS/2 driver

One of the first drivers we implemented was a polling keyboard reader. We now replaced this with a proper PS/2 driver that handles keyboard and mouse events. Yes, we also support PS/2 mouse now.

A driver for the “mouse integration” that is common in VM emulation software was also implemented, so we can seamlessly use the mouse in the kernel now.

RNG and entropy pool

The only method of generating random numbers in the kernel was a simple LCG PRNG. This made the random number quality bad, and (most importantly) not suitable for anything that needs to be secure.

We now have a system that can collect and extract entropy from kernel events and CPU interrupts. This means anything that depends on external input, such as keyboard and mouse events, and hard drive access patterns, now contribute to a “system entropy pool”.

For now, the only output is the occasional line on the debug console with a random byte. In the future, I am planning to add a sponge function that will collect entropy and produce arbitrary amounts of randomness.

Build system

We were using Ninja as the build system for some time, but recently I switched the ./configure script to emit Makefile syntax again. This was mostly done to integrate better with tooling and to get rid of a build dependency.

We also hash the output file names and put them all in a flat build directory. This makes it easier to clean the project, and allows us to reuse file names without having a nested build folder.

When using the ./mach command to build, all the subcommands still work the same so there are no changes.

Blog comments

I used to have a comment system that used CGI scripts to collect, process, filter and display comments. It required JavaScript for fetching and displaying the comments. At some point, I migrated servers and never bothered to set up comments again.

I decided to fix that, so now we have comments again. And guess what, they even work without JavaScript now. I am using Paul Graham’s spam filtering algorithm, which seems to be doing a great job so far.

Android application template

I wrote a small template for an Android application that displays a web page in a webview while exposing some useful APIs over the JavaScript bridge. It can be compiled and packaged entirely from the command-line, avoiding the Android Studio bloat.

Small utilities


I wrote a small utility called bencode2json to convert bencoded data to JSON. This is useful to me because I end up using bencode a lot when prototyping network protocols, but I don’t feel like rewriting jq for bencode.

Since I wanted to use this utility for external data from the network, I wanted to make sure it wouldn’t be able to do anything bad to my system. To this end, I learned about how to use seccomp to sandbox a Linux process. For better security and for a small challenge, I used the strict mode which only lets the process call read, write, exit and sigreturn. This pretty much eliminates almost all attack vectors.

LZ77 Compression / Decompression utility

In order to understand compression algorithms better, I decided to write a worse version of a common compression algorithm called LZ77.

The version I wrote keeps a 255-byte window in order to use only a single byte for the length values. Due to the very small window size, it is only able to achieve a 90% compression ratio on plaintext.

I am planning to play around with different algorithms and larger window sizes in the future.

Container runtime script

In order have more reproducible build without messing around with build environments and configurations, I wrote a shell script that uses systemd-nspawn to start and interact with temporary Linux containers. I decided to call this script, you can find it on my github.

Caius Durling (caius)

Fix Edgerouter DHCP ? entries March 14, 2021 03:55 PM

Occasionally I end up with devices on the local network that don’t emit their hostname over DHCP, so when listing the current leases on the EdgeRouter’s cli, they just appear as “?".1 These usually just irritate me, but occasionally when I’m looking for a machine on the network it means I can’t find it and end up poking the different “?” IPs using nmap or ssh to discover which machines they are.

$ show dhcp leases
IP address Hardware Address Lease expiration Pool Client Name
---------- ---------------- ---------------- ---- ----------- a8:1d:16:74:xx:yy 2021/03/14 16:38:27 trusted ? b8:27:eb:c5:xx:yy 2021/03/14 16:36:13 trusted picontrol1 a8:1d:16:75:xx:yy 2021/03/14 16:37:48 trusted ? 14:f6:d8:53:xx:yy 2021/03/14 16:40:29 trusted ?

The EdgeRouter lets me assign static entries in the DHCP subnet, which solves the problem of knowing which hostnames they are, but also pins those devices to (effectively) static IPs within the subnet which leads to me having to know which IPs are free when I assign them, etc. Avoiding that is why I have DHCP on the local network.2

Provided the EdgeRouter is configured to use dnsmasq to provide DHCP services3, you can lean on the dhcp-host option in the dnsmasq configuration to assign a hostname based on MAC address, without prescribing a specific IP address for the machine. This solves the issue of “?” devices showing up in show dhcp leases, whilst also allowing dynamic IP assignment.

You’ll need to know the MAC address in question, and pick a hostname to be assigned to the machine. You’ll then want to inject these through dnsmasq’s configuration file, which set service dns forwarding options xxx nicely injects into on the EdgeRouter.

$ configure
set service dns forwarding options "dhcp-host=14:f6:d8:53:xx:yy,cb1"
set service dns forwarding options "dhcp-host=a8:1d:16:75:xx:yy,cb3"
set service dns forwarding options "dhcp-host=a8:1d:16:74:xx:yy,cb2"

Then follow the usual compare, commit, verify your DNS/DHCP still works, save dance to apply & persist the changes.

Now when you login to the router and list the current DHCP leases, you’ll see the hostnames available - and you can now lookup the machines in local DNS via their hostname too. 🎉

$ show dhcp leases
IP address Hardware Address Lease expiration Pool Client Name
---------- ---------------- ---------------- ---- ----------- a8:1d:16:74:xx:yy 2021/03/14 16:38:27 trusted cb2 b8:27:eb:c5:xx:yy 2021/03/14 16:36:13 trusted picontrol1 a8:1d:16:75:xx:yy 2021/03/14 16:37:48 trusted cb3 14:f6:d8:53:xx:yy 2021/03/14 16:40:29 trusted cb1

  1. On my network currently these are Chromebooks, and Sonos speakers. I’ve also observed native SmartOS Zones behaving like this previously (I think they might have fixed this now.) I believe the device fails to send the current hostname (option 12) in either the DHCPDISCOVER or DHCPREQUEST packets. ↩︎

  2. Also, if I assign static host mappings to a device they vanish entirely from show dhcp leases, which stops me being lazy and checking one place to figure out where a device is. ↩︎

  3. To find out if you’re using dnsmasq for DHCP, check show service dhcp-server use-dnsmasq returns “enable” ↩︎

Andrew Owen (yumaikas)

Learning without Burnout March 14, 2021 12:45 AM

This article is a response to the excellent question by @Nellies.

This lifelong learning thing, is there any talk on how to deal with the infinite amount of stuff there is to learn?

I do ok, but sometimes the todo list does get cumbersome.

Is that part of the whole burnout in tech? No worries, I have broad shoulders, just curious.

Initial caveats

Most of my examples are going to biased towards full stack or frontend web development, since that’s where I have the most familiarity. This is also written to people trying to break into software development, or just want to understand what it’s like to manage this aspect of a software development career. I also gloss over a lot of things, especially cultural factors that make software development harder for those who can’t pass as white males.

Types of knowledge in software development

As mentioned by Nellie, one of the aspects of software development is that software developers are expected to constantly be learning new things. Or, at least, that’s the impression that happens from the outside. There are some mitigating factors, however.

The first is that once you learn the fundamentals for a particular technology stack, moving between similar flavors of that stack is made much easier. Once you understand the fundamentals of HTTP, HTML, CSS and JS, picking up Rails, Django or ASP.NET becomes much easier.

The second is varying types of knowledge have different shelf-lives. The cool new shiny JS framework being promoted by its creator? That could be relevant for only 6 months, or it could be the next React. It can be hard to tell up front, especially when you’re new. Thankfully, most jobs also lag the hotness by a certain amount, not everything asks for the hottest new tech.

But, if you’re learning HTTP status codes, or headers and auth works? That knowledge stays relevant a lot longer, and is part of all web apps. As a bonus, learning more fundamental knowledge gives you an edge when it comes to debugging problems, which is one thing that can help you stand out in your early years. Also, different types of knowledge have different acquisition costs. Basic JS or HTML is relatively easy compared to writing a full stack web app, which is easier than building a reasonably full scripting language, which, in turn, is easier than building a production grade data store like Postgresql or MongoDB.

Another factor is that it’s possible to specialize inside a given stack. So, for instance, with Rails, if you hit an area where your co-workers don’t seem to know a lot, that’s a good place to start a deep dive into how it works. Because then you’ll be able to build a reputation as the person who can figure out tricky things. You don’t have to know the whole stack at that same level of detail. If you’re known as “The ActiveRecord Bender” or “The Database Index Guru” or “The CSS Animations Witch”, people will give you more credit when you ask for help in their area of expertise. This also synergizes with learning fundamentals, and is where fundamentals can help you out.

Tradeoffs of what to learn

There are some trade-offs in play here. The first is that most foundational knowledge doesn’t show up as keywords on a resume, and most companies are poor at assessing it (to my knowledge). Either they assume that you have it if you have relevant keywords, or that you can pick up (or have) foundational as you go, or they pattern match you during the interview against what they expect people to know for the job. And nobody has time to learn everything all at once, no, not even that super impressive open source developer or Indie game development.

Learning on the Job

So, don’t learn everything all at once. You will have to look a lot of things up during your day to day work, even after a decade in the field. But, especially in your early years, do make time to learn things. Jobs for junior candidates will expect you spend a lot of time learning things. If they don’t they shouldn’t have hired you for that level. Any programming job involves learning the companies code base for at least the first 2-6 months you’re on a project, unless they follow a very conventional structure. So take advantage of the junior position and learn your job’s tech stack as effectively as you can.

Be sure to take deep dives into the tech involved at your first position, trying to understand one layer deeper than the one you’re working at. This doesn’t have to happen every ticket, but it should be happening at least once every paycheck or two. Often the lower layers are simpler than you might think, or make understanding the higher levels a lot easier. Understanding the lower levels also gives you a much better nose for when code is fluff vs substance. This guides your debugging, grows your fundamentals, and speeds up future work.

Initial learning time

When you are in your early years, you will need to front-load learning the basics. Unless you have some other skill set that can get you into the tech industry (there are a few, Quality Assurance/Testing comes to mind, as does being an engineer in a more continuous field, like aerospace, materials design, or the like), this is going to have to come out of your own time. Bootcamps are good for getting resume keywords under your belt, though they typically don’t spend much time on fundamentals. Colleges focus a lot more on fundamentals, but might use tech 5-20 year out of date. Either one will require you to figure out how to fill the gaps left by it.

For me, I had the luck to be able to carve out school time in high school (I was home schooled) on Fridays for morning each week to work through Head First C#. At the same time, I had an aspirational project to build a Scientific Calculator GUI app, which was solidly outside what I could pull off with what I started out knowing. For the first year or two, especially before you get a job, I’d recommend setting aside a time box in a similar fashion. If life doesn’t afford that timebox, then you’ll have to be a lot more proactive about making time.

Also, don’t try to code in a void, sit down with a game plan of what you’re going to try to do or learn. If you get to your time, and you don’t have that, take a walk and think about it, or do a chore with the plan in the back of your head. Often problems are solved away from the screen, or in conversations with ducks or teddy bears. Tunnel vision is a dangerous trap in software development

Maintaining a learning pace over time

Once you have your first job (or two), and have established your base knowledge, then the question comes up of how to maintain the learning required to keep up with tech. And this ultimately depends on what you plan do to with your career. If software is a stepping stone that takes you on to the next thing, then you can afford to burn a lot more of your motivation and time, and focus where the market is hot, and try to get 3-5 jobs out of that, setting yourself up as much for the next step as you can.

If you’re wanting to stay in software for the long haul, then I’d recommend a different approach from chasing the hype. Try to find the fun in software development. And, as you learn how to find the fun in building and learning things, be sure to pace yourself. I’ve had 6-week stretches of working hard at learning a new language, ecosystem, or side-project, followed by months of focusing on other things.

You don’t have to learn at 100% every week once you feel comfortable in your ability to build projects of a small-medium size. For web development, this would probably be about the point you’re able to write your own blogging software, for whatever functionality you’re looking for. You should be learning new things on a semi-regular basis. But keep it sustainable. One thing that really helps here is to have a process for documenting what you’ve learned, in some fashion. I’ve biased towards publishing code in the past, because that’s what’s been easiest for me.

Don’t use side projects as your only tool for learning software. I used to think this was the best way to learn, because it was how I had learned a lot of things. It has disadvantages, however. For one, side projects that are actually useful are more time-consuming than something you can throw away, and the kernel of what you’re learning might be discoverable in the shorter exercise. Noah Gibbs has a lot more to say here, and I owe a lot to his writing and companionship for changing how I approach learning in the last year.

But, whether through side projects, blogging, or building a portfolio, I do think that finding a way to publish your learnings is valuable. The nice thing about publishing and shipping smaller things on the side is that you do end up building a signal to employers that you have “passion”. This also allows you to ratchet your perceived learnings more sustainably than always trying to be at 100% learning mode.

High Output times

The main time I think that it’s worth going a highly regular output (like, once a week or more often) on the side is if you are aiming to become a proper software development celebrity. This is not for everyone, and likely doesn’t pay as well as you’d think, unless you spend a lot of time figuring out how to get it to do so, and you’re magically successful with it. There’s also a whole host of Not Writing Code that goes into making money off of it.

Doing this for a season, akin to an art challenge, is a great way to learn a lot in a short time, if you have the time and energy. If you’re looking to make a lot of money on the side, however, freelancing is likely a better bet, once you have the skillset to pull that off. As much as selling your hours for money isn’t the path to higher wealth bands, it’s a much less risky tradeoff compared to product development or trying to build a personal brand.

Final thoughts

Try to find the fun, as well as a sustainable long-term motivation. Look for trade-offs rather than dogma. Use binary search debugging. Listen to your body, and be honest with yourself.

March 12, 2021

Jeremy Morgan (JeremyMorgan)

Should Developers Join Clubhouse? March 12, 2021 12:18 AM

Clubhouse has been all the rage lately online, and people are flocking to it. If you’re a developer or tech person, is it worth signing up for? A total waste of time? Let’s take a look at it. Already on Clubhouse? Join me tomorrow for a chat! My Experience With Clubhouse I was hesitant to join Clubhouse. The premise seems silly. My first response: “So we’re going to volunteer to go on conference calls now?

March 11, 2021

Gustaf Erikson (gerikson)

13,000 dead in Sweden March 11, 2021 02:09 PM

March 09, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Artifacts And Spaces: Social Media And The Democratization Of Speech March 09, 2021 10:00 PM

It is clad in seven robes or skins (tunicas), to which the seven planetary spheres correspond in the macrocosm. The Cabalists connected this ten part structure of the eye with the Sephiroth. The blind spot was a term applied to the high Sephira

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 1 — The Artifacts and Spaces
  • Section 2 — Social Media And the Democratization Of Speech
Table Of Content
  • Efficiency
  • Democratization of Speech and Removal of Gatekeepers
  • Intersection of Social and Media Space
  • Infotainment And Overton Window
  • Anonymity, Automation, And The Role Of Platforms
  • It’s About Us

With the advent of smartphones, social media platforms are on the rise. Let’s step back and try to understand what social media are, not by citing events and instances happening on the different platforms, nor the countless consequences but by describing the characteristics of this new informational channel. We’ll go over the rest later but let’s take some distance for now.

A media is an outlet or tool used to deliver information. Before social media, this term most often referred to mass media communication such as newspaper, cinema, advertising, political speech, printing press, etc..
In contrast, social media are platforms hosted on the internet that facilitate the creation, engagement, and sharing of information between the average person and virtual communities. It relies on simulating aspects of social interactions.
Social media is the mix of media space and social space.

As a media, it stands out for its efficiency. This is characterized by numerous people on the platforms, high spreading speed, massive amount of information available, and the long-lasting lifetime of this information.

According to a Facebook’s statistics, there are 2.8 billion monthly active users on the platform. Pew research also shows that, in 2019 in the USA, Facebook was used by 69% of adults, while YouTube was used by 73% of them. Additionally, Worldwide statistics of the same year (2019) show that people spend on average 2 and a half hours a day on social media. These massive penetration rates are clear indicators of how social media have taken part in the lives of many.
Many of the platforms rely on a network effect to attract and create social hubs — The more the people on a platform, the more others will want to join it. This is the inherent social aspect, that’s why we also call them “social networks”.
Consequently, any information circulating on the popular social media could benefit from exposure that no other medium has.

The exposure is complemented with an instantaneous spread, a fast pass-it on effect. The engagement of users allowing the fast sharing of information. Quickly creating an amplification of the message, letting it travel through social circles that would’ve never been reached before — a wider transmission that breaks distances.

Another fitness criteria is how enduring and suitable the storage media is. The internet has a long-lasting memory, by the time the message is out it will have been copied and stored in multiple places. It will be almost impossible to erase.

These three criteria: exposure, speed, and memory, break previous media standards. However, to exploit this new medium successfully the information sent through it needs to adapt symbiotically to the ecosystem and niche in which it exists. If it is hostile, the information won’t be transmitted.
For example, the message will need to take in consideration the augmented Chinese Whisper effect taking place due to the speed and social aspects.

Another component that needs to be considered in the social media space, is the infobesity, also known as the infoxication phenomenon. The information is long-lasting, as we said, and accumulates over time. The users of the social media are overloaded with them.
The senders of mass messages will have to keep this in mind and craft their messages appropriately to penetrate social media. Indeed, the people on the platforms will never be able to consume all that is presented to them, and have to be picky with what they choose to process.

These are all indicators of the scale that the internet platforms, particularly social platforms, have and their potential use.
Most importantly, they have dethroned traditional mainstream media gatekeepers and allowed the common people to reach a wide audience, bypassing filters and safeguards.

In the past, there were two main ways to get information: through peers or through traditional media.
With peers, we’d learn about the world events through their stories, rumors, and anecdotes.
Traditional media, as the only other source, were subject to different types of manipulation. To protect the public and stand as gatekeepers of truth they erected journalism standards, free press, fact checking, and other types of regulation present in multiple countries.
At least that’s the theory. It remains ambiguous if traditional media are still influenced, controlled, and driven by ideological stances or corporate incentives.

Nowadays, we are presented with an alternative: the ordinary person as a media producer and filter. On social media, we aren’t subject to the standards and safeguards that mainstream media have.
People democratically filter and promote the news, ideas, and information that they consider important. The majority decides what is relevant and what isn’t. The phenomenon is similar to conversations with friends being converted from one-to-one into one-to-many, an amplification of decentralized voices. These voices are driven by an appeal to emotions and popularity, and not by reason, deliberations, qualifications, backgrounds, incentives, history, or others.
These are the messages that pierce through.

It is now frequent that mainstream media accentuate the legitimacy and credibility of social media by regurgitating their news from the different platforms. This can be interpreted as a confirmation and encouragement for participating on them.

The generation and distribution of media has been made effortless. Anyone can easily take part in it. This is the will of the actors owning the platforms, which we’ll see in another section.
The availability of digital equipment makes it a breeze to edit, write, capture, record, and film content.
The public is socially rewarded for their engagement, be it sharing, commenting, producing, or distributing. The usual social rules are in places such as wishing for fame or conforming and belonging to a community. We’ll also discuss in depth the psychological aspects in another section.
In sum, the public discourse on social media is a new place where the creation of narratives about events happens, and where they are discovered.

The digital platforms are also social places used for cultural, personal, and intimate relationships. This combines senses of trust and familiarity along with individualization and personality.

People maintain profiles on social media, facades of their personality that they mold with the aim of displaying it to an audience. The platforms normally allow for the customization and individualization of what can be interacted with, is recommended, and the style and types of messages received. It goes both ways: the person forms the virtual persona they want, and the digital platform assists in surrounding that persona with an environment that will foster it within boundaries.
Practically, that means the social media user will have personalized news and media, and interact with others, according to how they portray themselves.

Furthermore, the individualization, along with the fact that other users on the platform might already have social bonds with the person outside the digital space, induce feelings of trust and familiarity. However, this is complemented with an inability to establish new organic social bonds through social media — we aren’t able to connect deeply. We feel closer, yet further.
Familiarity is also felt through the repetition of the customized messages targeted for the particular persona.

As with any type of social place where communication happens, conversations will touch topics that conflict across cultures.
Just like mainstream media newspapers could be leaning toward different points of view, now everyone on social media reiterate their ideologies. An outer expression and clash of the inner confused and unaddressed stances of our societies and cultures.

Unquestionably, the most favored topic, the one driving the most engagement, is politics. Social media are the hubs for political discussions. According to Pew Research Center, 39% of adults in the USA used social media solely for political purposes. Interacting with media that has political meaning, which translates into “liking”, sharing, reposting, discussing, creating political posts.

Interestingly, social media are positioned between socialization, entertainment, and information consumption. For a vast majority of users political interaction has turned into infotainment, information sharing as entertainment. A caricaturization of politics, reducing political acts to quick and easy bona fide participation on social platforms — What is called hashtag activism or slacktivism.

Along with the overload of information and the democratic filter, users will indirectly encounter all types of conflicting ideas while browsing to entertain themselves. These having spread exactly because they interpolated the engagement of so many others.
The types of information that usually stand out are the ones that challenge our ideologies and culture, as we said. These types of messages can pierce through.

Some theory implies that because divergent views exists at the same time on social media, this latter is akin to low-context societies.
High-context mode are the communications that happen with an inner-group, including the subtle codes, traditions, and meanings. While low-context mode are the communications that are straight forward, reducing ambiguity, flat, and global.

This can be what some call an information society, a society in which creating, manipulating, and sharing of information is central. The people being part of this society would then be digital citizens.

We are having an incredibly difficult time adapting to this new medium, now that everyone has an equal voice.
We still refuge ourselves in different cocoons that collide in parody-like confrontations with one another. Again, a reflection of our own cultural weaknesses, our cultural insecurities.

These bubbles then can only be pierced through by more extreme content and ideas that will travel further. That means we are moving the Overton Window — the ideas tolerated in public discourse. Fringe ideas on social media are the ones that are heard the most and might be accepted as normal. The medium having voted democratically that this was what the majority voted as relevant.

Obviously, the votes are not all done by real human beings, the platforms have their share of responsibilities and reasons to promote certain types of discourse that drive engagement, as we’ll see soon.
This reminds us of the nudge theory and persuasive tech we’ve learned about in the previous section and how it could only be achieved when an actor has full control of the environment; This is the case with most social media. They can, and do, deliberately create digital environments that users feel fulfill their basic human drives.

Hiding the actors affect the messages greatly, this is done in two ways: anonymity and automation.
The use of persona that aren’t attached to real individuals let people express themselves without restraints. This attracts bitter views and clashes, as people can now break the accepted cultural codes.
These anonymized profiles can then be automated by algorithms, what we call bots, and exchange with other social media users to promote a message. Consequentially, cheating the democratization of speech and artificially creating a public opinion by giving importance to selected ideas.
We referred to these autonomous agents as computational propaganda in the previous section. Statistics show that around 40% of the traffic on social media is composed of these autonomous agents. We’ll explore later the incentives of the actors.

Let’s finally mention that there are different types of social media platforms. Some have specific rules, some are only used to share a single type of media, some are centralized, some are decentralized, some are anonymous, some require identification, some are more fringe than others, etc..
Social media have all the essential components we’ve described, but they are mainly guided by the platforms themselves. They are new instruments, new tools in our hands.

That is why some persons start to see social media as public utilities, even though they are far from it. A public utility being “an infrastructural necessity for the general public where the supply conditions are such that the public may not be provided with a reasonable service at reasonable prices because of monopoly in the area.” For that, a governing body will need to be erected and have monopoly over the social media in place and justify them being a necessity.

Overall, the medium reveals more about the people than about itself. It is how we use it that matters and the confusion we feel could be a manifestation of our own confusion. Or is it?

This concludes our review of what social media are as new communication channels. We’ve first seen its efficiency characteristics: huge reach, high speed, reliability. Then we’ve contemplated how it has broken the walls of the mainstream media gatekeepers and putting the keys of the castle in the hands of the common person. Later we’ve seen how it intertwines a social space with a media space, including al our cultural issues and how we want constant entertainment. Next, we’ve described how hard it is for the information to pierce through and how that increases extremeness in the messages to actually reach us. And finally, we’ve discussed anonymity and automation as ways to cheat the pseudo-democratic process that is supposed to happen on social media to elect what is important and what isn’t.

Table Of Content


Attributions: Gregor Reisch, Pretiosa Margarita, Freiburg, 1503; Basle, 1508

March 08, 2021

eta (eta)

Why asynchronous Rust doesn't work March 08, 2021 12:00 AM

In 2017, I said that “asynchronous Rust programming is a disaster and a mess”. In 2021 a lot more of the Rust ecosystem has become asynchronous – such that it might be appropriate to just say that Rust programming is now a disaster and a mess. As someone who used to really love Rust, this makes me quite sad.

I’ve had a think about this, and I’m going to attempt to explain how we got here. Many people have explained the problems with asynchronous programming – the famous what colour is your function essay, for example.1 However, I think there are a number of things specific to the design of Rust that make asynchronous Rust particularly messy, on top of the problems inherent to doing any sort of asynchronous programming.

In particular, I actually think the design of Rust is almost fundamentally incompatible with a lot of asynchronous paradigms. It’s not that the people designing async were incompetent or bad at their jobs – they actually did a surprisingly good job given the circumstances!2 I just don’t think it was ever going to work out cleanly – and to see why, you’re going to have to read a somewhat long blog post!

A study in async

I’d like to make a simple function that does some work in the background, and lets us know when it’s done by running another function with the results of said background work.

use std::thread;

/// Does some strenuous work "asynchronously", and calls `func` with the
/// result of the work when done.
fn do_work_and_then(func: fn(i32)) {
    thread::spawn(move || {
        // Figuring out the meaning of life...
        thread::sleep_ms(1000); // gee, this takes time to do...
        // ah, that's it!
        let result: i32 = 42;
        // let's call the `func` and tell it the good news...

There’s this idea called “first-class functions” which says you can pass around functions as if they were objects. This would be great to have in Rust, right?

See that func: fn(i32)? fn(i32) is the type of a function that takes in one singular i32 and returns nothing. Thanks to first-class functions, I can pass a function to do_work_and_then specifying what should happen next after I’m done with my work – like this:

fn main() {
    do_work_and_then(|meaning_of_life| {
        println!("oh man, I found it: {}", meaning_of_life);
    // do other stuff

Because do_work_and_then is asynchronous, it returns immediately and does its thing in the background, so the control flow of main isn’t disrupted. I could do some other form of work, which would be nice (but here I just wait for 2 seconds, because there’s nothing better to do). Meanwhile, when we do figure out the meaning of life, it gets printed out. Indeed, if you run this program, you get:

oh man, I found it: 42

This is really exciting; we could build whole web servers and network stuff and whatever out of this! Let’s try a more advanced example: I have a database I want to store the meaning of life in when I find it, and then I can run a web server in the foreground that enables people to get it once I’m done (and returns some error if I’m not done yet).

struct Database {
    data: Vec<i32>
impl Database {
    fn store(&mut self, data: i32) {;

fn main() {
    let mut db = Database { data: vec![] };
    do_work_and_then(|meaning_of_life| {
        println!("oh man, I found it: {}", meaning_of_life);;
    // I'd read from `db` here if I really were making a web server.
    // But that's beside the point, so I'm not going to.
    // (also `db` would have to be wrapped in an `Arc<Mutex<T>>`)

Let’s run this…oh.

error[E0308]: mismatched types
  --> src/
27 |       do_work_and_then(|meaning_of_life| {
   |  ______________________^
28 | |         println!("oh man, I found it: {}", meaning_of_life);
29 | |;
30 | |     });
   | |_____^ expected fn pointer, found closure
   = note: expected fn pointer `fn(i32)`
                 found closure `[closure@src/ 30:6]`

I see.

Hang on a minute…

So, this is actually quite complicated. Before, the function we passed to do_work_and_then was pure: it didn’t have any associated data, so you could just pass it around as a function pointer (fn(i32)) and all was grand. However, this new function in that last example is a closure: a function object with a bit of data (a &mut Database) tacked onto it.

Closures are kind of magic. We can’t actually name their type – as seen above, the Rust compiler called it a [closure@src/ 30:6], but we can’t actually write that in valid Rust code. If you were to write it out explicitly, a closure would look something like this:

struct Closure<'a> {
    data: &'a mut Database,
    func: fn(i32, &mut Database)

impl<'a> Closure<'a> {
    fn call(&mut self, arg: i32) {

There are a number of things to unpack here.

An aside on naming types

Being able to name types in Rust is quite important. With a regular old type, like u8, life is easy. I can write a function fn add_one(in: u8) -> u8 that takes one and returns one without any hassle.

If you can’t actually name a type, working with it becomes somewhat cumbersome. What you end up having to do instead is refer to it using generics – for example, closures’ types can’t be named directly, but since they implement one of the Fn family of traits, I can write functions like:

fn closure_accepting_function<F>(func: F)
    F: Fn(i32), // <-- look!
    /* do stuff */

If I want to store them in a struct or something, I’ll also need to do this dance with the where clause every time they’re used. This is annoying and makes things harder for me, but it’s still vaguely workable. For now.

msql-srv example code

[image: from the msql-srv crate, showing an example of many where clauses as a result of using closures]

An aside on ‘radioactive’ types

The way Rust is designed tends to encourage certain patterns while discouraging others. Because of ownership and lifetimes, having pieces of data that hold references to other pieces of data becomes a bit of a problem. If my type has a & or a &mut reference to something, Rust makes me ensure that:

  • the something in question outlives my type; you can’t go and drop the thing I’m referring to if I still have a reference to it, otherwise my reference will become invalid
  • the something in question doesn’t move while I have the reference
  • my reference to the something doesn’t conflict with other references to the something (e.g. I can’t have my & reference if something else has a &mut reference)

So types with references in them are almost ‘radioactive’; you can keep them around for a bit (e.g. inside one particular function), but attempting to make them long-lived is usually a bit of an issue (requiring advanced tricks such as the Pin<T> type which didn’t even exist until a few Rust versions ago). Generally Rust doesn’t really like it when you use radioactive types for too long – they make the borrow checker uneasy, because you’re borrowing something for an extended period of time.

borrowing semantics visualization

[image: from the RustViz paper, showing borrowing semantics]

Closures can be pretty radioactive. Look at the thing we just wrote out: it has a &'a mut Database reference in it! That means while we’re passing our Closure object around, we have to be mindful of the three rules (outlives, doesn’t move, no conflicting) – which makes things pretty hard. I can’t just hand off the Closure to another function (for example, the do_work_and_then function), because then I have to make all of those rules work, and that’s not necessarily easy all the time.

(Not all closures are radioactive: if you make them move closures, they’ll take everything by value instead, and create closure objects that own data instead of having radioactive references to data. Slightly more of a pain to deal with, but you lose the blue radiation glow the things give out when you look at them.)

Also, remember what I said about being able to name types? We’re not actually dealing with a nice, written-out Closure object here; we’re dealing with something the compiler generated for us that we can’t name, which is annoying. I also lied when I said that it was as simple as making all of your functions take F, where F: Fn(i32) or something – there are actually three different Fn-style traits, Fn, FnMut, and FnOnce. Do you know the difference between them?

So. A closure is this magical, un-nameable type that the compiler makes for us whenever we use || {...} syntax, which implements one of three traits (and it’s not immediately obvious which), and it also might be radioactive. Try and use one of these, and the Rust compiler is probably going to be watching you very carefully.

The thing I really want to try and get across here is that Rust is not a language where first-class functions are ergonomic. It’s a lot easier to make some data (a struct) with some functions attached (methods) than it is to make some functions with some data attached (closures).

Trying to use ordinary structs is downright easy:

  • they’re explicitly written out by the programmer with no funky business
  • you choose what traits and methods to implement on them and how to set them out / implement them
  • the struct can actually be referred to by other parts of the code by its type

Trying to use closures is hard:

  • the compiler does some magic to make a closure type for you
  • it implements some obscure Fn trait (and it’s not immediately obvious which)
  • it might be radioactive (or force you to use move and maybe insert a bunch of clone() calls)3
  • you can’t actually name their type anywhere or do things like return them from a function

Importantly, the restrictions applied to using closures infect types that contain them – if you’re writing a type that contains a closure, you’ll have to make it generic over some Fn-trait-implementing type parameter, and it’s going to be impossible for people to name your type as a result.

(Other languages, like Haskell, flip this upside down: functions are everywhere, you can pass them around with reckless abandon, etc. Of course, these other languages usually have garbage collection to make it all work…)

Bearing this in mind, it is really quite hard to make a lot of asynchronous paradigms (like async/await) work well in Rust. As the what colour is your function post says, async/await (as well as things like promises, callbacks, and futures) are really a big abstraction over continuation-passing style – an idea closely related to the Scheme programming language. Basically, the idea is you take your normal, garden-variety function and smear it out into a bunch of closures. (Well, not quite. You can read the blue links for more; I’m not going to explain CPS here for the sake of brevity.)

Hopefully by now you can see that making a bunch of closures is really not going to be a good idea (!)

*wibbly wobbly scene transition*

And then fast forward a few years and you have an entire language ecosystem built on top of the idea of making these Future objects that actually have a load of closures inside4, and all of the problems listed above (hard to name, can contain references which make them radioactive, usually require using a generic where clause, etc) apply to them because of how “infectious” closures are.

The language people have actually been hard at work to solve some (some!) of these problems by introducing features like impl Trait and async fn that make dealing with these not immediately totally terrible, but trying to use other language features (like traits) soon makes it clear that the problems aren’t really gone; just hidden.

Oh, and all the problems from what colour is your function are still there too, by the way – on top of the Rust-specific ones.

Beginner (and experienced) Rust programmers look at the state of the world as it is and try and build things on top of these shaky abstractions, and end up running into obscure compiler errors, and using hacks like the async_trait crate to glue things together, and end up with projects that depend on like 3 different versions of tokio and futures (perhaps some async-std in there if you’re feeling spicy) because people have differing opinions on how to try and avoid the fundamentally unavoidable problems, and it’s all a bit frustrating, and ultimately, all a bit sad.

Did it really have to end this way? Was spinning up a bunch of OS threads not an acceptable solution for the majority of situations? Could we have explored solutions more like Go, where a language-provided runtime makes blocking more of an acceptable thing to do?

Maybe we could just have kept Rust as it was circa 2016, and let the crazy non-blocking folks5 write hand-crafted epoll() loops like they do in C++. I honestly don’t know, and think it’s a difficult problem to solve.

But as far as my money goes, I’m finding it difficult to justify starting new projects in Rust when the ecosystem is like this. And, as I said at the start, that makes me kinda sad, because I do actually like Rust.

(Common Lisp is pretty nice, though. We have crazy macros and parentheses and a language ecosystem that is older than I am and isn’t showing any signs of changing…)

  1. This is really recommended reading if you aren’t already familiar with it (as you’ll soon see) 

  2. Seriously – when I put out the last blog post, the actual async core team members commented saying how much they appreciated the feedback, and then they actually went and made futures 1.0 better as a result. Kudos! 

  3. You might be thinking “well, why don’t you just only use move closures then?” – but that’s beside the point; it’s often a lot harder to do so, because now you might have to wrap your data in an Arc or something, by which point the ergonomic gains of using the closure are outweighed by the borrow checker-induced pain. 

  4. You can actually manually implement Future on a regular old struct. If you do this, things suddenly become a lot simpler, but also you can’t easily perform more async operations inside that struct’s methods. 

  5. (sorry, I mean, the esteemed companies that deign to use Rust for their low-latency production services) 

March 07, 2021

Ponylang (SeanTAllen)

Last Week in Pony - March 7, 2021 March 07, 2021 11:49 PM

We have new releases of Ponyc and Corral.

Derek Jones (derek-jones)

Code bureaucracy can reduce the demand for cognitive resources March 07, 2021 09:46 PM

A few weeks ago I discussed why I thought that research code was likely to remain a tangled mess of spaghetti code.

Everybody’s writing, independent of work-place, starts out as a tangled mess of spaghetti code; some people learn to write code in a less cognitively demanding style, and others stick with stream-of-conscious writing.

Why is writing a tangled mess of spaghetti code (sometimes) not cost-effective, and what are the benefits in making a personal investment in learning to write code in another style?

Perhaps the defining characteristic of a tangled mess of spaghetti code is that everything appears to depend on everything else, consequently: working out the impact of a change to some sequence of code requires an understanding of all the other code (to find out what really does depend on what).

When first starting to learn to program, the people who can hold the necessary information on increasing amounts of code in their head are the ones who manage to create running (of sorts) programs; they have the ‘knack’.

The limiting factor for an individual’s software development is the amount of code they can fit in their head, while going about their daily activities. The metric ‘code that can be fitted in a person’s head’ is an easy concept to grasp, but its definition in terms of the cognitive capacity to store, combine and analyse information in long term memory and the episodic memory of earlier work is difficult to pin down. The reason people live a monks existence when single-handedly writing 30-100 KLOC spaghetti programs (the C preprocessor Richard Stallman wrote for gcc is a good example), is that they have to shut out all other calls on their cognitive resources.

Given time, and the opportunity for some trial and error, a newbie programmer who does not shut their non-coding life down can create, say, a 1,000+ LOC program. Things work well enough, what is the problem?

The problems start when the author stops working on the code for long enough for them to forget important dependencies; making changes to the code now causes things to mysteriously stop working. Our not so newbie programmer now has to go through the frustrating and ego-denting experience of reacquainting themselves with how the code fits together.

There are ways of organizing code such that less cognitive resources are needed to work on it, compared to a tangled mess of spaghetti code. Every professional developer has a view on how best to organize code, what they all have in common is a lack of evidence for their performance relative to other possibilities.

Code bureaucracy does not sound like something that anybody would want to add to their program, but it succinctly describes the underlying principle of all the effective organizational techniques for code.

Bureaucracy compartmentalizes code and arranges the compartments into some form of hierarchy. The hoped-for benefit of this bureaucracy is a reduction in the cognitive resources needed to work on the code. Compartmentalization can significantly reduce the amount of a program’s code that a developer needs to keep in their head, when working on some functionality. It is possible for code to be compartmentalized in a way that requires even more cognitive resources to implement some functionality than without the bureaucracy. Figuring out the appropriate bureaucracy is a skill that comes with practice and knowledge of the application domain.

Once a newbie programmer is up and running (i.e., creating programs that work well enough), they often view the code bureaucracy approach as something that does not apply to them (and if they rarely write code, it might not apply to them). Stream of conscious coding works for them, why change?

I have seen people switch to using code bureaucracy for two reasons:

  • peer pressure. They join a group of developers who develop using some form of code bureaucracy, and their boss tells them that this is the way they have to work. In this case there is the added benefit of being able to discuss things with others,
  • multiple experiences of the costs of failure. The costs may come from the failure to scale a program beyond some amount of code, or having to keep investing in learning how previously written programs work.

Code bureaucracy has many layers. At the bottom there is splitting code up into functions/methods, then at the next layer related functions are collected together into files/classes, then the layers become less generally agreed upon (different directories are often involved).

One of the benefits of bureaucracy, from the management perspective, is interchangeability of people. Why would somebody make an investment in code bureaucracy if they were not the one likely to reap the benefit?

A claimed benefit of code bureaucracy is ease of wholesale replacement of one compartment by a new one. My experience, along with the little data I have seen, suggests that major replacement is rare, i.e., this is not a commonly accrued benefit.

Another claimed benefit of code bureaucracy is that it makes programs easier to test. What does ‘easier to test’ mean? I have seen reliable programs built from spaghetti code, and unreliable programs packed with code bureaucracy. A more accurate claim is that it can be unexpectedly costly to test programs built from spaghetti code after they have been changed (because of the greater likelihood of the changes having unexpected consequences). A surprising number of programs built from spaghetti code continue to be used in unmodified form for years, because nobody dare risk the cost of checking that they continue to work as expected after a modification

March 06, 2021

eta (eta)

Getting PIV-based SSH working on a YubiKey March 06, 2021 12:00 AM

I bought a YubiKey 5C Nano recently. These devices are great – I’ve built a lot of my (metaphorical) empire on top of them, seeing as they’re capable of acting as an SSH agent (store your SSH keys on them, securely!), an OpenPGP smartcard (do encryption and decryption on the key!), FIDO U2F ‘security keys’ (use them as a 2-factor authentication method!), and probably more.

Getting the thing to work as an SSH agent was, however, not the easiest thing I’ve ever done. There are multiple options here – you can use the OpenPGP applet and then configure GnuPG to work as an SSH agent, but that’s a brittle solution in my experience (gpg-agent is quite flaky, and often requires restarting when it forgets about the YubiKey). Instead, I wanted to see whether I could use the YubiKey’s PIV (Personal Identity Verification) applet to get this working.


For the SSH agent part, we’re going to use Filippo Valsorda’s yubikey-agent, so you’ll want to have that installed. In my testing, yubikey-agent’s built-in yubikey-agent -setup command errored out, so we’ll configure the PIV applet of the YubiKey manually. (This’ll also leave the door open for you to do other things with the PIV applet later if you like.)

You’ll also need Yubico’s own yubikey-manager (the ykman cli tool) installed.

Okay, here goes:

  1. Ensure pcscd (the PC/SC smartcard daemon) is installed and running. This might already have been done for you by your Linux distribution but, if not:
    • $ sudo systemctl enable --now pcscd
    • (This is safe to run if it’s already been set up; the command will just do nothing).
  2. This exercise assumes a fresh YubiKey (i.e. one where you haven’t touched the PIV applet yet). If that’s not the case, and you want to erase all of the PIV data and start afresh (losing all data encrypted with the PIV keys!), use the ykman piv reset command.
  3. First, change the PIN from the default one (123456).
    • $ ykman piv change-pin
    • As the command’s help says: The PIN must be between 6 and 8 characters long, and supports any type of alphanumeric characters. For cross-platform compatibility, numeric digits are recommended.
  4. Then, change the PUK (‘personal unblocking key’). This is used to reset the PIN if you ever forget it.
    • $ ykman piv change-puk
    • The PUK has the same entry requirements as the PIN (i.e. also 6-8 ASCII characters).
    • It might be prudent to generate a random PUK and keep it safe (e.g. by writing it down and locking the paper away). If you lose both the PIN and PUK, you will need to reset the PIV applet, losing all data encrypted with the PIV keys (and SSH access to hosts you don’t otherwise have access to).
  5. The YubiKey PIV applet by default has a well-known management key used to make changes to the PIV keys (etc.). It’s best practice to change this to something else. We’ll use the option to generate a random one, store it in the YubiKey, and secure it with the PIN.
    • $ ykman piv change-management-key -pt
    • -p: Store new management key on your YubiKey, protected by PIN. A random key will be used if no key is provided.
    • -t: Require touch on YubiKey when prompted for management key.
    • If you’re more paranoid than me, you can use the -g option instead, which will generate a key for you to note down and give back later. However, the extent to which you can cause damage with a management key is limited (you can delete and regenerate keys, but not decrypt data or anything), so this is arguably not worth the hassle – especially since forgetting the management key means you’d have to reset the PIV applet to make changes.
  6. Generate a public/private keypair on the key, using the 9a (“PIV Authentication”) key slot. We’ll use ECC (elliptic-curve cryptography) because it’s fast, secure, and has vastly smaller key sizes, and configure the key to always require touches and require the PIN on first use. You might want to use different settings here.
    • $ ykman piv generate-key -a ECCP256 --touch-policy ALWAYS --pin-policy ONCE 9a ./yubikey-public.pem
    • Note: Some outdated SSH servers/implementations only support RSA keys. If this applies to you, leave off the -a option to use RSA 2048 instead.
    • See ykman piv generate-key -h for a full description of all available options.
  7. Use the newly generated key to make a self-signed PKCS#11 certificate to act as our SSH identity.
    • $ ykman piv generate-certificate -s 'my-yubikey-ssh' -d 365 9a ./yubikey-public.pem
    • Modify the -s parameter to include a human-readable description of the key or the machine the key is installed in.
    • Modify the -d parameter to set how many days the key will be valid for. The value above specifices 1 year.
    • You can get rid of yubikey-public.pem after this step.
  8. Enable yubikey-agent.
    • $ systemctl --user enable --now yubikey-agent
    • Note: You might need to kill any running instances of gpg-agent if you had that running (and it decided to try and use your YubiKey), and potentially restart pcscd after doing so.
    • Note: Check that yubikey-agent started properly with systemctl --user status yubikey-agent.
      • On my machine, it failed to start because of namespacing issues.
      • If this happens, edit the unit file with systemctl --user edit --full yubikey-agent, and remove all lines under [Service] apart from ExecStart and ExecReload.
  9. Update SSH_AUTH_SOCK to point to the running instance of yubikey-agent.
    • $ export SSH_AUTH_SOCK=/run/user/1000/yubikey-agent/yubikey-agent.sock
    • The above command assumes bash. Other shells may vary.
    • If your user ID isn’t 1000, you’ll need to change the above command (you can find the right path by ps aux | grep yubikey-agent).
    • Note: You’ll want to put the above command in .bashrc or similar.
  10. Get the key fingerprint for your newly generated SSH identity.
    • $ ssh-add -L
  11. Copy the key fingerprint to your remote host(s), and put it in ~/.ssh/authorized_keys.
  12. You should now be able to SSH using the YubiKey!
    • $ ssh <host>
    • The first time you try this, it should pop up a window asking for the PIN, after which you’ll need to touch the flashing YubiKey.
    • Subsequent attempts will only require a touch.
    • If it didn’t work, check the notes under step 8. In particular, using OpenPGP and SSH simultaneously with the YubiKey usually requires you to kill either pcscd or gpg-agent, depending on what you want to do.
  13. Ensure you have a backup plan in place for when this YubiKey fails, or is lost.
    • The key we generated is local to the YubiKey, and cannot be exported. It can very easily be wiped by anyone running ykman piv reset (which requires no authentication!), or you could lose the key.
    • Make sure you have another way of getting into all hosts you use this YubiKey with (encrypted regular SSH key, out-of-band console, etc.).
  14. You’re done!

Hopefully that worked. If it didn’t, well, there’s a comments thing below if you can be bothered, I guess?

March 05, 2021

Sevan Janiyan (sevan)

LFS, round #5 March 05, 2021 05:35 PM

Up to this point I’ve been working with a chroot to build OS images from loop back mounted flat file which is then converted to the vmdk format for testing with virtualbox. I created packages for bpftrace and BCC, BCC was fairly trivial and the availability of a single archive which includes submodules as bcc-src-with-submodule.tar.gz …

March 04, 2021

Frederic Cambus (fcambus)

OpenBSD/loongson on the Lemote Fuloong March 04, 2021 11:10 PM

In my article about running OpenBSD/loongson on the Lemote Yeeloong back in 2016, I mentioned looking for a Fuloong. All hope seemed lost until the Summer of 2017, when a fellow OpenBSD developer was contacted by a generous user (Thanks again, Lars!) offering to donate two Lemote Fuloong machines, and I was lucky enough to get one of those units.

This machine uses the same CPU as the Yeeloong, a Loongson 2F which is a single-core MIPS-III 64-bit processor running at 800/900 MHz.

As hinted in my previous article, contrarily to the Yeeloong, the Fuloong is less strict with the type of RAM it accepts, and my device is happily running with a Kingston 2GB DDR2 SO-DIMM module (ASU256X64D2S800C6), replacing the original 512MB module.

Here is the result of a quick md5 -t benchmark:

MD5 time trial.  Processing 10000 10000-byte blocks...
Digest = 52e5f9c9e6f656f3e1800dfa5579d089
Time   = 1.726563 seconds
Speed  = 57918535.263411 bytes/second

For the record, LibreSSL speed benchmark results are available here.

System message buffer (dmesg output):

Copyright (c) 1982, 1986, 1989, 1991, 1993
	The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2021 OpenBSD. All rights reserved.

OpenBSD 6.9-beta (GENERIC) #74: Fri Feb 26 08:02:25 MST 2021
real mem = 2147483648 (2048MB)
avail mem = 2116452352 (2018MB)
random: boothowto does not indicate good seed
mainbus0 at root: Lemote Fuloong
cpu0 at mainbus0: STC Loongson2F CPU 797 MHz, STC Loongson2F FPU
cpu0: cache L1-I 64KB D 64KB 4 way, L2 512KB 4 way
bonito0 at mainbus0: memory and PCI-X controller, rev 1
pci0 at bonito0 bus 0
re0 at pci0 dev 6 function 0 "Realtek 8169" rev 0x10: RTL8169/8110SCd (0x1800), irq 4, address 00:23:9e:00:0f:71
rgephy0 at re0 phy 7: RTL8169S/8110S/8211 PHY, rev. 2
sisfb0 at pci0 dev 8 function 0 "SiS 315 Pro VGA" rev 0x00: 640x400, 8bpp
wsdisplay0 at sisfb0 mux 1: console (std, vt100 emulation)
glxpcib0 at pci0 dev 14 function 0 "AMD CS5536 ISA" rev 0x03: rev 3, 32-bit 3579545Hz timer, watchdog, gpio, i2c
isa0 at glxpcib0
com0 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com1 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
mcclock0 at isa0 port 0x70/2: mc146818 or compatible
gpio1 at glxpcib0: 32 pins
iic at glxpcib0 not configured
glxclk0 at glxpcib0: clock, prof
pciide0 at pci0 dev 14 function 2 "AMD CS5536 IDE" rev 0x01: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <WDC WD1600BEVS-00VAT0>
wd0: 16-sector PIO, LBA48, 152627MB, 312581808 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
auglx0 at pci0 dev 14 function 3 "AMD CS5536 Audio" rev 0x01: isa irq 9, CS5536 AC97
ac97: codec id 0x414c4760 (Avance Logic ALC655 rev 0)
audio0 at auglx0
ohci0 at pci0 dev 14 function 4 "AMD CS5536 USB" rev 0x02: isa irq 11, version 1.0, legacy support
ehci0 at pci0 dev 14 function 5 "AMD CS5536 USB" rev 0x02: isa irq 11
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00 addr 1
apm0 at mainbus0
vscsi0 at root
scsibus0 at vscsi0: 256 targets
softraid0 at root
scsibus1 at softraid0: 256 targets
pmon bootpath: bootduid=53153d3cd8ddc482
root on wd0a (53153d3cd8ddc482.a) swap on wd0b dump on wd0b

PCI device data:

# pcidump
Domain /dev/pci0:
 0:6:0: Realtek 8169
 0:8:0: SiS 315 Pro VGA
 0:14:0: AMD CS5536 ISA
 0:14:2: AMD CS5536 IDE
 0:14:3: AMD CS5536 Audio
 0:14:4: AMD CS5536 USB
 0:14:5: AMD CS5536 USB

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Artifacts And Spaces: Defining and Understanding Propaganda, Influence, And Persuasion March 04, 2021 10:00 PM

Descartes compared the creation of pictures of memory in the brain with the traces left by needles in fabric

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 1 — The Artifacts and Spaces
  • Section 1 — Defining and Understanding Propaganda, Influence, And Persuasion
Table Of Content
  • Communication
  • Influence
  • Persuasion
  • Propaganda
  • Types Of Propaganda
  • Administering
  • Reinforcement, Anchor, Hook, Imagery
  • Ethics And Issues Of Identification

Our journey begins with three terms: propaganda, influence, and persuasion. To most, they sound like synonyms but each convey a different concept. Like any form of communication the internet can be used as a channel for them. What does communication consist of?

The very broad domain of communication studies — with multiple sub-disciplines such as journalism, film critic, public relation, and political science — often use a simple reductionist model called the Shannon–Weaver model of communication.

Shannon-Weaver's Model Of Communication Diagram

The model has 7 different concepts: Information source, Encoder, Channel, Decoder, Destination, and Noise. In our case, the internet is used as the pipe that conducts the signal, the transmitter and receiver are the chosen form of media (video, text, audio), and the information source and destination are the people and algorithms at both ends.
This model is not complete as it does not take into consideration the personalities of the persons involved, nor context, history, time, or any other clues that make communication complex. Moreover, it is limited to one-to-one scenarios and doesn’t cover one-to-many.
Regardless of these flaws, the model gives a generic picture of what communication is: It is about sharing meaning — a message — through the exchange of information via symbols that have perceived meanings on both ends.
The Shannon-Weaver model is a great starting point to see what needs to be done so that the conduct becomes more reliable, reducing uncertainty. That is, we need to keep in mind the qualities of context, sender, intent, message, channel, audience, and response. Concretely, the more the exchange the less the ambiguity, both parties converging towards a common point, common interest, or focus.

All this to say that the internet is a medium for transporting information. Just like the printing press lead to novelties, change in thoughts, and brought doctrines to the public like never before, the internet has its own characteristics but can only be defined by what we use it for.
For now, let’s pause and go back to what propaganda, influence, and persuasion are.

Influence is a generic word to convey that someone’s decisions have been affected by something. Influence is passive, the person makes the decision themselves unconsciously. It can happen with or without any direct communication (verbal or nonverbal). It is a spontaneous mix of inner motivations and environment. Consequently, it is rarely, if ever, controlled by another entity’s will unless this entity has full control over the environment.
The nudge theory argues that an entity can influence people through positive or negative reinforcement, indirect suggestions. However, the long term effects of these suggestions are highly contested as they get mixed with other environmental factors like peer pressure and social norms.
An example of influence is product placement, where a product is desired based on its indirect association with a portrayed lifestyle that the viewer would, in theory, like to obtain.

Persuasion, on the other hand, is based on a two-way interaction. It is often used in negotiation when one party or many parties want to take over the other, imposing their point. The difference with negotiation is that the core of the message never changes, it is not about resolving a common conflict but gradually convincing someone.
Persuasion is thus transactional and voluntary. The party that wants their message to come across has to bridge through the individual psychological process, to exchange ideas freely and openly so that the other will be convinced. In persuasion there is a back and forth between persuader and persuadee, the persuader adapting, shaping, reinforcing, changing the message at each iteration to make it more appealing depending on the persuadee’s responses.
If the persuasion succeeds, it should result in a reaction such as “I never saw it that way before”.

Persuasion can be verbal or nonverbal. It can either look like a consensual dialogue to reach an understanding, or look like strategic information or awareness campaigns that appear one-way but are still using a feedback mechanism to hone the message. This former is called Organized Persuasive Communication (OPC), and refers to all organized persuasion activities (advertising, marketing, public relations, organizational communication), but the term can also refer to more manipulative means.
In both these cases there is a free flow of information and the persuadee is fully aware of what is happening.

People are reluctant to change and so to convince them the persuader has to shape the message in a relatable way to something the persuadee already believes, or use lies and tactics morphing the facts.
This means persuasion can take a manipulative form, which may or may not be consensual. For example when someone is persuaded under false pretences, incentivized via promises, provided extra benefits, or coerced through threats or actual infliction of costs (including withdrawal of benefits).

Propaganda goes in that manipulative direction. It is exclusively a one-to-many massive persuasive strategy that is deliberately trying to shape perceptions, cognitions, and the direct behavior to achieve the desired intent of the propagandist.

Historically, propaganda didn’t have the same meaning it has today. It meant “to propagate” or “to disseminate”, but the term has changed over time as it got attached with the dissemination of information by governments for particular purposes.
Purpose is what differentiate propaganda. It is always meant to be targeted at a specific group, category of people, or society with the goal to push them in the desired direction. This objective is often linked to institutional ideologies, that is why we call propaganda activated ideology.
This means it is inherently advantageous to the propagandists, and their ideological fellows, but not necessarily the recipients. In some cases, it is arguable, and depends on the viewer, if the message is propaganda or education.

Propaganda doesn’t shy away from using whatever means available to inculcate the message. It isn’t based on mutual understanding, omits or hides information, uses deception, uses manipulation, frames issues, coerces or deceptively coerces, distorts, lies, overemphasizes, and misdirects. Anything goes, as long as the goal is reached.
The message is crafted in a deliberate way to support a viewpoint, a narrative, a goal of an ideology.
This is why propagandist consultants are often called “spin doctors” or “spinmeisters”, reinterpreting the events under a particular perspective.

The corollary of propaganda is censorship. The entity that wants to control the perception of its ideology silences any message that doesn’t fit. It achieves the same purpose by, not filing people’s mind with approved information, but preventing them from being confronted with opposing points of view.

That is why propagandists require control of information flow so that they can manufacture it. This allows them to select the current agenda — telling the public what is important and what isn’t. Or to frame problems, influencing how the public pictures a situation, creating the images and opinions that go along the ideological goal. The propagandists try to control the media as a source of information distribution and to present distorted information from what appears to be credible sources.
Indeed, it is also imperative that the message should be packaged in ways that conceal its manipulative persuasive purpose, as we’ll see later.

Another interesting typical aspect of propaganda as communication is its heavy use of symbols and association. That is in direct relation to its institutional and ideological root. For instance: justice, liberty, devotion to a country, etc..
Symbolism is also heavily employed in advertising propaganda to sell products as ideologies and ways of living.

Wherever you look and ask, all domains of research and work that touch propaganda dismiss their association with it. Public relation dismisses it as consensual and not manipulative, and the same in the advertising industry. The studies of propaganda are filled with euphemism, diverting attention, using the techniques they are themselves describing.

The historian Zbyněk Zeman categorizes propaganda as either white, grey or black, but there are other categorizations too.

White propaganda is when the message is sent from an official source and the intent is more or less clear. For example, a message on TV by the president of a country.

Black propaganda is when the message conceals the source and intent by making it seem as if it emerged from the same people, group, or organization it is targeting. Obscuring the identity of the originator is done to make the message more credible, giving the impression it comes from the people it’s trying to discredit or manipulate. It is also used for diplomatic reasons to hide government involvement.
For the propagandist, this is the type that is safer and has greater persuasiveness.

Grey propaganda lives in between, it is a message coming from an ambiguous or non-disclosed source, and the intent isn’t known. This could be a rumor or a message based on unnamed insider’s info.

Counter-propaganda is the reaction to propaganda. It differs from it in how it is defensive, based on facts and truth, clear (understood by everyone), calling out logical fallacies. It is most often administered as a white propaganda.
Propaganda can only be countered effectively if the counter-propaganda acts quickly to overcome its damage. Propaganda uses an anchor, as we’ll see, and it needs to be dismantled as soon as possible. That means the older the propaganda the more it will be deeply rooted in people’s mind.

Self-propaganda is a type of propaganda where someone deliberately tries to convince themselves of something, even if irrational. This is akin to a Stockholm syndrome in some cases.

Sub-propaganda refers to anything that isn’t direct propaganda but to ways to keep the door open for future opportunities of propaganda.
Within this category we find promotional culture that is used in marketing and other domains. It consists of promoting commodities, celebrities, politics, civil society, social cultural traditions, economic ways, etc.. It is the salesmanship of a culture and thus cobbled with direct and indirect marketing means.

Computational propaganda is, according to Phil Howard, “the assemblage of social media platforms, autonomous agents, and big data tasked with the manipulation of public opinion.”
Computational propaganda can be directed by agendas or not. It could passively happen because of the incentive of the algorithms on the platforms — usually social media and thus views and clicks as we’ll see in other sections.
Propaganda, like any other type of communication, depends heavily on the environment and context in which it is moving.

Another related concept is persuasive technology, whenever a digital platform has full control of the environment of a user and attempts to change their attitudes through social influence and persuasion techniques. This consists in creating an environment that guides the person to perform certain actions and not others, inciting them through different means like social feedback, extrinsic or intrinsic motivations, and positive and negative reinforcement..

When it comes to administering persuasion and propaganda, the process is intuitive: Choose the target group and message to transmit. Choose the media form, the media shell, depending on how easily it can spread. Pack the message material in the shell, and send it towards either the mass/group, or towards specific individuals that could spread it further. If an opinion leader spread it then their credibility, expertise, trustworthiness, and attractiveness with respect to the target group should be analyzed to match the message.
Then ensues a series of iterations to polish the message through reinforcement, repetition, targeting opinion leaders through indoctrination, and the addition of new psychological techniques, manipulation of the message, emotions, and logical fallacies appropriate to the target audience.

If the propagandists’ or persuaders’ goal go to the contrary of the habits of the audience, it will be difficult to achieve and will require awareness and information related to behavioral change and predictors of behavior of the target audience.
That is, the message, to have its effect, will need to be adapted and employ tactics to resonate with the audience.

The message shouldn’t be imposed, instead the recipient should feel it flow effortlessly within themselves. Any propaganda shouldn’t look like propaganda to the target audience. “Propaganda is most effective when it is least noticeable”. The change will need to be self-imposed. This is similar to the nudge theory of reinforcement, it should appear as influence and not propaganda.
For that, the message should seem to give expression to the recipient’s own concerns, tensions, aspirations, and hopes. It must identify with them, this is what we call the anchor.

This anchor activates the ideology of the propagandist by tying down new attitudes or behaviors along with existing ones. It is the starting point for change because it represents something already widely accepted by potential persuadees — it is the path of least resistance.

These anchors can be categorized as either belief, value, attitude, behavior, and group norms.
An anchor of belief is one of the best anchor, and the stronger the belief of a receiver, the more likely they will attach to the propaganda goal.
A value is a kind of long-term belief that is not likely to change. It is hard to use as anchor because these are unwavering and so might conflict with the propagandist goal.
An attitude is the readiness to respond to an idea, object, or course of action. It is a predisposition that already resides within the receiver. The propagandist can activate the predisposition to further achieve the goal.
A behavior or behavioral pattern is a clear indication of how receivers are currently acting and often predicts future behaviors.
Group norms are beliefs, values, attitudes, and behaviors derived from the membership of the receiver in a group. The propagandist can use this tendency to conformity to make the message organically move laterally within the group.

A concept similar to the anchor is called the hook. It differs from the anchor in that it is based on a unique event or idea that is constantly repeated to amplify its effect.
The hook is generally a story which central element makes it newsworthy and evokes strong emotional responses, making it stick to memory. It could be fabricated, misleading and creating doubts, or real, but most importantly it should spread quickly and inundate all conversations on a topic. The hook is then attached with whatever piece of information it was packed with.

Hooks and anchors regularly use typical imagery to reach the ideological goal. For example, portraying the enemy as a baby killer.
The compaction of big ideas into small packages that can be distributed effortlessly is a must. This can be done through emotional wording, hook-like stories, symbolic images, and impactful videos. This is a form of portable philosophy created through repetition, emotions, and stories that lead to new meanings.
Notably, when using symbols, words, and images, that already live in the collective imagination, they will bring with them a baggage of feelings, concepts, and ideas. Think of all the recently rehashed tropes in media like movies, songs, images, and news. The use of strong wording will color the message, creating an association in the collective memory: genocide, patriotism, invader, victim, democracy, anarchy, enemy, confrontation, massacre, coup, radicalization, etc..

To spread further and faster, the anchors, hooks, and imageries must all interpolate and abuse our own confused cultural code. They work by exploiting issues we haven’t adequately addressed as a society. That is why people show strong responses to messages that are linked to injustice, outrage, partisanship à la we-vs-them, and controversies. We are more likely to react to things we don’t agree with.
From economic inequalities, race, nationalism, gender roles, sexual norms, etc.. It doesn’t matter which side of the issue people are on, the message will be spread and will provoke a reaction — if this is the goal of the propagandist. It could also be fabricated controversies, seeding doubt on purpose, making it seem like we’re still on the fence, ignorant (the domain of agnotology). The propagandist expertly triggers hope, satisfaction, pride, enthusiasm, but also fear, anxiety, anger, outrage, and disgust.

Let’s take a look at how Robert Cialdini details 6 principles of persuasion:

  • Reciprocity: People feel the need to give back to someone who provided a product, service, or information.
  • Scarcity: People want items that they believe are in short supply.
  • Authority: People are swayed by a credible expert on a particular topic.
  • Consistency: People strive to be consistent in their beliefs and behaviors.
  • Likability: People are influenced by those who are similar, complimentary, and cooperative.
  • Consensus: People tend to make choices that seem popular among others.

These go hand in hand with what we described so far and also reminds us of the conveyor of the packed message. We should pay attention to the credibility of the source, if the source is revealed.
Yet, a trick in propaganda is to keep confirming what you said, to never argue and assume the truth. Our mind is wired to accept and believe what others say as true, especially if not contradicting with any prior belief or value.

The rule of thumb is that the message shouldn’t create any cognitive dissonance but instead build a world that feels real to the recipient. To create a cohesive cosmology of meanings, to sell a coherent memorable story.
However, propaganda still denies the distance between the source and audience because it puts the ideology first.

We’ll dive in the psychological aspects in another part. For now, this should have created a strong foundation to understand influence, persuasion, and propaganda.

The exact identification of what can and should be considered propaganda is debatable and extremely hard to do. Some researchers have attempted to create a framework to discern manipulative persuasion from consensual informational and dialogical ones, but this is hardly applicable to the real world. And, as we said, propaganda could be considered education in the eye of some people.

When it comes to ethics, the subject of children indoctrination comes to mind. Children are the most vulnerable targets of propaganda as they haven’t built strong defenses against them. They are the least prepared for critical reasoning and contextual comprehension and thus are impressionable. Their mental world is in the process of being built and is easier to mold. There is no strong forces to oppose the propagandist goal.

On that note, it is subjective whether certain types of persuasions and propagandas are ethical or not. Propagandists are driven by ideologies, and consequently strongly believe that all means are necessary. As with anything related to ethics and moral, it differs based on personal judgement of the actions. For some people, it is ethically wrong to employ any subterfuge or tactics to reach a goal. For others, it is not about the tactics but whether the goal itself is right or wrong.
There’s a lot to be said on the black and white division of ethics, but we’ll leave it at that because the world is inherently nebulous.

This concludes our review of propaganda, influence, and persuasion. We’ve taken a look at the basic idea of communication, the packing of a message. We’ve then defined influence, persuasion, and propaganda, with propaganda being a type of persuasion that is one-to-many, manipulative, and driven by ideology. We’ve seen different types of propaganda and their definitions such as black, white, grey, self-, and others. Then we’ve seen how this package can be administered effectively through the person or channel carrying it and the actual shell of the message. This message will be successful if it doesn’t conflict with the receiver, if it is reinforced, uses an anchor, a hook, or strong imagery. The message will spread further and faster if it triggers strong emotional responses, especially something people feel they have to stand against. Finally, we’ve glanced at the issue of identification and the ethics concerning propaganda and the most vulnerable.

Table Of Content


Attributions: René Descartes, Traité de l’homme

March 03, 2021

Jeremy Morgan (JeremyMorgan)

What Do You Do for a Living Anyway? March 03, 2021 05:36 PM

I get asked this question a lot. My friends and family barely understand what I do, and my title Technical Marketing Manager doesn’t help at all. So I’ll try to explain it. TL;DR: My job is awesome. I build cool stuff, talk with amazing people, and have fun. If you want to join me, click here and let’s talk In 2002, I went from being a Network Administrator/IT person to a developer.

Robin Schroer (sulami)

The Shape of Tests March 03, 2021 12:00 AM

Many tests for an operation iterate over a mapping of different inputs to expected outcomes. By looking at the tests for a single operation as the same test with different inputs and output expectations, we can start to question how we should model those tests.

Tests as Matrices

By simply enumerating every possible combination of input values, we can construct a matrix with as many dimensions as inputs. We can then define the expected result for each set of inputs, and write a generalised test function:

∀ input∈{(a, b, …, n) | a∈A, b∈B, …, n∈N} f(input)

The number of possible test cases is thus:

|inputs| = |A × B × ⋯ × N|

As soon as our operation accepts an input that has more than a few possible values, that is any kind of number, string, or complex data structure, enumerating every possible input combination becomes impractical. Instead we can resort to groupings of values via properties.

This is a test matrix for division which uses properties instead of values, with the rows being dividends, and the columns divisors:

÷ Positive Zero Negative
Positive Positive undefined Negative
Zero Zero undefined Zero
Negative Negative undefined Positive

Matrices like this are necessarily exhaustive, and force us to think about the result for every possible combination of the input values we have included.

This is an implementation of the same property matrix in Clojure:

(ns division-matrix-test
  (:require [clojure.test :refer [deftest is testing]]
            [clojure.spec.alpha :as s]
            [clojure.spec.gen.alpha :as gen]))

(defn safe-divide
  "2-arity `/`, but returns `nil` on division by zero."
  [dividend divisor]
    (/ dividend divisor)
    (catch ArithmeticException _

(defmacro test-matrix
  "Generates tests for a two-dimensional test matrix."
  [test-fn matrix]
  (let [columns (rest (first matrix))
        rows (map first (rest matrix))
        combinations (for [[row idy] (map #(vector %1 %2) rows (range))
                           [col idx] (map #(vector %1 %2) columns (range))]
                       [row col (-> matrix
                                    (nth (inc idy))
                                    (nth (inc idx)))])]
    `(doseq [combination# [~@combinations]]
       (apply ~test-fn combination#))))

(deftest safe-division-test
  (let [gen-input
        (fn [kind]
          (case kind
            :pos (gen/generate (s/gen pos-int?))
            :neg (gen/generate (s/gen neg-int?))
            :zero 0))]


     (fn [x y result-pred]
       (let [dividend (gen-input x)
             divisor (gen-input y)]
         (is (result-pred (safe-divide dividend divisor))
             (format "Failed with: %s / %s" dividend divisor))))

     [[nil    :pos   :zero  :neg ]
      [:pos   pos?   nil?   neg? ]
      [:zero  zero?  nil?   zero?]
      [:neg   neg?   nil?   pos? ]])))

In this case we are testing a safe variant of the division function /, which returns nil if the divisor is zero. This simplifies the testing process, because we do not have to include any exception catching logic in our test function, or invent a notation to mean this set of inputs should result in a thrown exception.

It is worth noting that such a direct interpretation of a matrix is only possible in a language as malleable as Clojure. In other languages, we might have to resort to enumerating a set of (dividend divisor result) tuples, losing the guarantee of covering all possible combinations.

But even in Clojure, more than two dimensions in this matrix will quickly become unwieldy and hard to follow, and a tuple-based approach would scale better to larger numbers of input parameters.

Tests as Trees

Another way we could structure our tests is as a tree. A tree does not have to be exhaustive the same way a matrix has to be. We can omit certain combinations of inputs by pruning their branches. In this way we are implying that if a single input has a given value, it defines the result regardless of the other inputs’ values.

In the division example all branches with a divisor of zero could be collapsed into a single case, as the dividend does not matter in this case. This only works if the first level of branching describes the divisor, and the dividends are on the second level.

(ns division-tree-test
  (:require [clojure.test :refer [are deftest testing]]
            [clojure.spec.alpha :as s]
            [clojure.spec.gen.alpha :as gen]))

(deftest safe-division-test

  (testing "with a positive divisor"
    (let [divisor (gen/generate (s/gen pos-int?))]

      (testing "and a positive dividend"
        (let [dividend (gen/generate (s/gen pos-int?))]
          (is (pos? (safe-divide dividend divisor)))))

      (testing "and a zero dividend"
        (let [dividend 0]
          (is (zero? (safe-divide dividend divisor)))))

      (testing "and a negative dividend"
        (let [dividend (gen/generate (s/gen neg-int?))]
          (is (neg? (safe-divide dividend divisor)))))))

  (testing "with a divisor of zero"
    (let [dividend (gen/generate (s/gen int?))]
      (is (nil? (safe-divide dividend 0)))))

  (testing "with a negative divisor"
    (let [divisor (gen/generate (s/gen neg-int?))]

      (testing "and a positive dividend"
        (let [dividend (gen/generate (s/gen pos-int?))]
          (is (neg? (safe-divide dividend divisor)))))

      (testing "and a zero dividend"
        (let [dividend 0]
          (is (zero? (safe-divide dividend divisor)))))

      (testing "and a negative dividend"
        (let [dividend (gen/generate (s/gen neg-int?))]
          (is (pos? (safe-divide dividend divisor))))))))

This might look more verbose, but in exchange we get a unique label for every tree branch, which can improve readability. The nesting also naturally lends itself to lexical scoping, so we only have the values in scope which apply on a given branch.

A key advantage of the tree structure is flexibility. If one of the branches requires special code, we can confine it to that branch, avoiding complicating the remaining branches more than necessary.

Trees also scale better with larger numbers of inputs or options for inputs. A tree might grow overly wide or deep, but we can split it if that becomes a problem.

There is a downside to omitting branches though. If we change our safe-divide function to return different results depending on the dividend when the divisor is zero, our tests might still pass, depending on the specific inputs used, but we will lose test coverage for certain code paths. We have chosen to not test certain input combinations, and we need to be aware of this omission when we are changing the code under test.

Tests as Definitions

Considering the formula describing the generalised test function above, we could also consider translating this directly into code. This can work, but only if we can test results without re-implementing large parts of the code under test, otherwise we are overly coupling the tests to the code. In the division case, we can decide the sign of the result based on the signs of the inputs.

(ns division-spec-test
  (:require [clojure.test :refer [deftest is]]
            [clojure.spec.alpha :as s]
            [clojure.spec.test.alpha :as stest]))

(defn- check-safe-divide-result [{{:keys [dividend divisor]} :args
                                  ret :ret}]
    (zero? divisor) (nil? ret)

    (zero? dividend) (zero? ret)

    (or (and (pos? dividend) (pos? divisor))
        (and (neg? dividend) (neg? divisor)))
    (pos? ret)

    :else (neg? ret)))

(s/fdef safe-divide
  :args (s/cat :dividend number?
               :divisor number?)
  :ret (s/nilable number?)
  :fn check-safe-divide-result)

(deftest safe-divide-spec-test
  (let [check-result (stest/check `safe-divide)]
    (is (not check-result)
        (format "Failed with: %s"
                (-> check-result

This solution is specific to Clojure, though many other languages have property based testing tools that work similarly.

By adding a spec to our function, we can run a large number of different inputs against our function, and assert a property about the result based on the inputs. It will even shrink the inputs to find the simplest set of inputs to trigger a spec failure.

This means we do not have a programmer writing a matrix or a tree by hand anymore, which has some advantages. The main one being that a programmer might not consider all possible inputs.

Fail in safe-divide-spec-test
Failed with: {:args {:dividend ##NaN, :divisor 0}, :ret ##NaN}

Fail in safe-divide-spec-test
Failed with: {:args {:dividend 1, :divisor ##Inf}, :ret 0.0}

Fail in safe-divide-spec-test
Failed with: {:args {:dividend 6.812735744013041E-108, :divisor 2.7578261315509936E216}, :ret 0.0}


The optimal shape of a test depends mainly on the structure of the inputs to the operation we are testing, as well as its nature.

For pure functions which we expect to use widely and change rarely, property-based testing can be desirable to avoid unintended consequences. There is also a certain speed requirement for test shrinking to work effectively.

Operations with a small number of possible inputs can also be tested via test matrices, which have fewer limitations, but do not guarantee correctness, as only the programmer can assert the completeness of the matrix. They are easy to extend with additional values for parameters, but harder to extend with additional values. Their declarative nature can be useful for documentation purposes.

At the other end of the spectrum, tree-shaped tests are the most flexible, and scale best for larger operations with many inputs. If different branches require fundamentally different setup, test trees can isolate that complexity to where it is required. They also require the most care to keep tidy, and have a tendency to sprawl if unsupervised.

March 02, 2021

Gokberk Yaltirakli (gkbrk)

Emacs idle config March 02, 2021 09:00 PM

Due to some bad computer habits, I end up opening and closing editor instances a lot instead of starting my editor and using it for the duration of the project. In particularly bad cases, I end up closing my editor for every code change to run a compiler or check some files in the same terminal.

The proper solution to this problem is to get rid of bad habits and start using multiple windows like a modern human in 2021. But the easier solution is to make my editor startup time faster in order to mitigate some of the pain in using it.

This is not the fault of Emacs of course, as Emacs is expected to be running for at least the whole programming session. This doesn’t change the fact that I need to speed it up to make my life better though.


The first step to making a program faster is to run a profiler to see which parts are slowing it down. Since running emacs -Q (which makes Emacs start without loading the user config) is instant, this means my customization is causing Emacs to start up slower.

I used two methods for profiling. The first method is to run ProfileDotEmacs, which is a small Emacs Lisp script that prepares a report on how long each section of your .emacs file takes to execute. If all your configurations are done in your .emacs file, in a mostly top-level fashion, ProfileDotEmacs will probably be sufficient to know what is causing the slowdown.

The second method, which actually provided more useful information for me, is to wrap your config in profiler statements. This can be done like this.

(profiler-start 'cpu)

;; Your existing config


Now you will be able to run M-x profiler-report whenever you want in order to a nested profiler report that you can interactively drill down to.

Speeding everything up

The title is a little dishonest, as we won’t actually be improving the run-time of any code. What we will be doing is to take the blocking code that runs on startup, and make it run once the editor is idle for a few seconds. The most important thing is that the editor starts up quickly, displays the contents of the file instantly and allows me to move around. Less crucial settings and comfort plugins can be loaded once I stop pressing the keys to think for a few seconds.

The way to schedule a command to run when the editor is idle is to call run-with-idle-timer. You can choose how many idle seconds you want to wait before executing your callback and what arguments to pass to you callback. I ended up writing these two helper functions for loading a file after the editor becomes idle.

(defun leo/load-file (file)
  "Load an elisp file from .emacs.d"
  (load (concat "~/.emacs.d/" file)))

(defun leo/load-idle (sec file)
  "Load a file after N idle seconds"
  (run-with-idle-timer sec nil 'leo/load-file file))

Delaying everything by one second was enough for my needs, but you can choose a tiered approach where the essentials are loaded immediately and heavier, less-important plugins get loaded over time.

March 01, 2021

Gustaf Erikson (gerikson)

On the dates March 01, 2021 06:13 PM

When I started recording the dates when Sweden’s death toll from COVID-19 exceeded round thousands, I did not foresee the project continuing into the next year. But here we are.

I used to set the dates when I noticed Swedish media report them, but I’ve now gone to FHM’s stats page and got them from there.

This has led to some reshuffling - especially on Jan 6 2021 which now has its own tally.

This table has also been updated.

12,000 dead in Sweden March 01, 2021 12:28 PM

Mark J. Nelson (mjn)

Converting arbitrary base^exponent to scientific notation March 01, 2021 12:00 PM

I had the following question: Given a number in $base^{exponent}$ format, like $3^{500}$ or $7^{300}$, is there a way to convert it to scientific notation without explicitly constructing the (possibly very large) number as an intermediate step? We will assume here that both $base$ and $exponent$ are positive. For example, $7^{300}$ should be converted to $3.384\ldots \cdot 10^{253}$, also written 3.384...e+253.

This can be useful when looking at exponential growth with different bases. I happened to come across this particular question, right now, with game tree branching factors. If one game has an average branching factor of 3 and lasts 500 turns on average, is that more or fewer total states than one with an average branching factor of 7, which lasts 300 turns on average? We can eyeball the answer easily by converting to scientific notation, with its more familiar base 10: $3^{500} \approx 3.636 \cdot 10^{238}$, while $7^{300} \approx 3.384 \cdot 10^{253}$, so the latter is larger by 15 orders of magnitude.

Beyond games, exponential growth is recently in the news in epidemiology of course. Given an effective reproduction number $R$ and current cases $i$, there will be an expected $i \cdot R^t$ cases after $t$ additional time units pass. But how to easily compare different $R$ and $t$ combinations? Converting to scientific notation is one way.

I spent more time googling unsuccessfully for this answer than it ultimately took to just derive it from equations solvable using high school mathematics. Maybe I am bad at Google. But nonetheless, I thought I'd summarize the result.

There are two steps. First, change bases from $base$ to $10$. The resulting exponent is likely not to be an integer. Secondly, take this $10^x$ and convert it to scientific notation, meaning an expression of the form $m \cdot 10^e$, for some mantissa $m$, $0 m 10$ and integral exponent $e$.

Step 1. Changing bases

We want to turn $base^{exponent}$ into $10^x$. That means solving for $x$ in the equation $10^x = base^{exponent}$. For example, $10^x = 7^{300}$.

First, take the base-10 logarithm of both sides: $$\log_{10} 10^x = log_{10} base^{exponent}$$ Applying the definition of a logarithm as the inverse of exponentiation, this is equivalent to: $$x = \log_{10} base^{exponent}$$ This gives us a closed form for $x$, but still requires evaluating $base^{exponent}$, which is what we were hoping to avoid. But we can apply the logarithmic identity that $\log a^b = b \cdot \log a$, resulting in: $$x = exponent \cdot \log_{10} base$$

Now we have something that can be computed efficiently, at least in approximate terms. For example, we can convert $7^{300}$ to $10^x$ by computing $x = 300 \cdot \log_{10} 7$, yielding $10^{253.529\ldots}$.

This is almost scientific notation, except that scientific notation has integral exponents, not exponents like $253.529\ldots$.

Step 2. Round down the exponent and attach a mantissa

We now have a $10^x$ with probably non-integral $x$, such as the running example of $7^{300} \approx 10^{253.529}$.

The exponent for the purposes of scientific notation will simply be this $x$ rounded down to the nearest integer, denoted $\lfloor x \rfloor$. That will naturally result in a smaller number. So to approximate the original number, we need to multiply it by a mantissa $m$. This is the basic idea of scientific notation: a number between 0 and 10 times an integral power of 10. Put in the form of an equation, we want to solve for $m$ in: $$m \cdot 10^{\lfloor x \rfloor} = 10^x$$ This can be solved with high school mathematics actually a little more elementary than Step 1, though for some reason it took me longer to recall that fact: $$m = \frac{10^x}{10^{\lfloor x \rfloor}}$$ $$m = 10^{x - \lfloor x \rfloor}$$ Now we again have a closed-form equation that can be computed reasonably efficiently with floating-point arithmetic (not requiring a gigantic intermediate result).

To return again to our running example of $7^{300}$. In step 1 we determined this was equal to $10^{253.529\ldots}$. Now we turn it into scientific notation of the form $m \cdot 10^e$, where:

  • $e = \lfloor 253.529\ldots \rfloor = 253$
  • $m = 10^{253.529\ldots - e} = 10^{0.529\ldots} \approx 3.384$
Yielding $7^{300} \approx 3.384 \cdot 10^{253}$ or 3.384e253, without explicitly constructing the large intermediate number.

* * *

Translating the above to Python:
  import math

  base = 7
  exponent = 300

  raw_exponent = exponent * math.log10(base)

  sci_notation_exponent = math.floor(raw_exponent)
  sci_notation_mantissa = 10 ** (raw_exponent - sci_notation_exponent)

Gustaf Erikson (gerikson)

The Anarchy: The East India Company, Corporate Violence, and the Pillage of an Empire by William Dalrymple March 01, 2021 09:36 AM

A good history of the EIC. Dalrymple gives equal space to the “opposing” viewpoints, sidestepping the historiographical triumphalism of most English-language histories.

February March 01, 2021 09:01 AM


Last image(?) from this area of Stockholm from me, as we’ve moved our office.

Feb 2020 | Feb 2019 | Feb 2018 | Feb 2017 | Feb 2016 | Feb 2015 | Feb 2014 | Feb 2013 | Feb 2012 | Feb 2011

February 28, 2021

Derek Jones (derek-jones)

Fitting discontinuous data from disparate sources February 28, 2021 11:16 PM

Sorting and searching are probably the most widely performed operations in computing; they are extensively covered in volume 3 of The Art of Computer Programming. Algorithm performance is influence by the characteristics of the processor on which it runs, and the size of the processor cache(s) has a significant impact on performance.

A study by Khuong and Morin investigated the performance of various search algorithms on 46 different processors. Khuong The two authors kindly sent me a copy of the raw data; the study webpage includes lots of plots.

The performance comparison involved 46 processors (mostly Intel x86 compatible cpus, plus a few ARM cpus) times 3 array datatypes times 81 array sizes times 28 search algorithms. First a 32/64/128-bit array of unsigned integers containing N elements was initialized with known values. The benchmark iterated 2-million times around randomly selecting one of the known values, and then searching for it using the algorithm under test. The time taken to iterate 2-million times was recorded. This was repeated for the 81 values of N, up to 63,095,734, on each of the 46 processors.

The plot below shows the results of running each algorithm benchmarked (colored lines) on an Intel Atom D2700 @ 2.13GHz, for 32-bit array elements; the kink in the lines occur roughly at the point where the size of the array exceeds the cache size (all code+data):

Benchmark runtime at various array sizes, for each algorithm using a 32-bit datatype.

What is the most effective way of analyzing the measurements to produce consistent results?

One approach is to build two regression models, one for the measurements before the cache ‘kink’ and one for the measurements after this kink. By adding in a dummy variable at the kink-point, it is possible to merge these two models into one model. The problem with this approach is that the kink-point has to be chosen in advance. The plot shows that the performance kink occurs before the array size exceeds the cache size; other variables are using up some of the cache storage.

This approach requires fitting 46*3=138 models (I think the algorithm used can be integrated into the model).

If data from lots of processors is to be fitted, or the three datatypes handled, an automatic way of picking where the first regression model should end, and where the second regression model should start is needed.

Regression discontinuity design looks like it might be applicable; treating the point where the array size exceeds the cache size as the discontinuity. Traditionally discontinuity designs assume a sharp discontinuity, which is not the case for these benchmarks (R’s rdd package worked for one algorithm, one datatype running on one processor); the more recent continuity-based approach supports a transition interval before/after the discontinuity. The R package rdrobust supports a continued-based approach, but seems to expect the discontinuity to be a change of intercept, rather than a change of slope (or rather, I could not figure out how to get it to model a just change of slope; suggestions welcome).

Another approach is to use segmented regression, i.e., one of more distinct lines. The package segmented supports fitting this kind of model, and does estimate what they call the breakpoint (the user has to provide a first estimate).

I managed to fit a segmented model that included all the algorithms for 32-bit data, running on one processor (code+data). Looking at the fitted model I am not hopeful that adding data from more than one processor would produce something that contained useful information. I suspect that there are enough irregular behaviors in the benchmark runs to throw off fitting quality.

I’m always asking for more data, and now I have more data than I know how to analyze in a way that does not require me to build 100+ models :-(

Suggestions welcome.

February 27, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — Introduction February 27, 2021 10:00 PM

Astronomicum Cæsareum,

  • Internet: Medium For Communication, Medium For Narrative Control
  • Introduction

To no one’s surprise, the internet has permeated all aspects of our lives. All other means of communication have dwindled in comparison, even though the technological behemoth is relatively young (around 50 years old as of 2021).
Worldwide, according to statistics from 2019, people spent an average of 2 and a half hours a day on social media. The top place goes to The Philippines with 3h53min per day.

This raises an iceberg of loaded questions.
At the top: How controlling is the internet today?
Or, asking in reverse: How does internet, as a new vector of communication, let different actors control us? How is the internet changing the way communication works and, indirectly, how we think?

These are broad questions and there are plenty of sub-questions underneath. Indeed, we keep hearing in the news about social media, extremism on the rise, and a salad of convoluted keywords thrown in articles trying to rationalize all this.
Is there really an information war?
What are the elements of it?
Who are the actors?
What’s the relation with social media?
Is it related to all the conspiracy theories we keep hearing about?
Is journalism dying?
What is the attention and data economy?
Are we all part of a giant hypnosis experiment?
More importantly, can we, and should we, do something about this?

Like many of you, I’ve asked myself these questions countless times, got buried in a mountain of headlines and news articles selling shock-value pseudo-deciphering. I temporarily felt clear-headed but quickly got back to a muddy state of comprehension.
Thus, I’ve set myself to consume all the literature I could find, peel it, parse it, organize it, categorize it, hone it, filter it, etc.. So that I could, at least partly, remove the haze surrounding the topic.

This series of articles is going to be my attempt at answering how the internet changes communication regarding narrative control. My own mini internet studies.

Here is the rough plan of our exploration.

Table Of Content

Get ready because this is going to be a long ride!


  • Peter Apian, Astronomicum Caesareum (Ingoldstadt, 1540)

February 26, 2021

Gokberk Yaltirakli (gkbrk)

Giving search engines a fair access to data February 26, 2021 09:00 PM

Search engines are difficult to create. They are even harder to improve to a point where you get good-enough results to keep regular users. This is why it’s so rare to see decent search engines that aren’t front-ends to the Bing or Google APIs.

This doesn’t mean there are none though. There are a small number of search engines with their own crawlers and search logic. More and more of them appear over time, but most of them cannot improve to the point of catching on. This is because of a common resource they lack: Data.

I am not talking about the slimy, personal kind of data that Google and friends like so much. What are people searching for right now? How many of those do I have good results for? How do people form their queries? Those are all difficult to answer and improve if people aren’t using you search engine. But no one will use your search engine unless you improve those. Great then, we are in a chicken-and-egg situation with no escape in sight.

The data problem

Before tackling the problem, let’s explore what the problem is in the first place. The first problem is the number of humans testing the result quality. In almost all cases, the creator(s) will be testing the results. Friends and family will try it a few times before going back to their default search engine. Social media and Hacker News will provide a swarm of clicks that only last for a few hours. This is not data, at least not enough data.

The second problem is a little trickier. Most people from our already small set of users will not provide data that is too valuable. Let’s break down our users into two segments, the creators and the people testing it out.

The creators are programmers who research very specific pieces of information all day. While this makes them very good at using search engines, it makes them very bad at testing the results. A programmer knows the exact query that will bring them results before typing it. This query is usually so good that even a bad algorithm will find the results they are looking for.

The people testing it out have a different problem. When put on the spot for testing a search engine, it is not easy to come up with queries for difficult questions. But those are the exact situations that need the help of a good search engine. You will only see these queries once people see you as reliable and pick you as their default engine.

The current situation

We can separate the current search ecosystem into three distinct groups.

Google gets almost all the search traffic. They have more than enough data, both personal and aggregated, to serve all their needs. Their monopoly on search, and their hostility for the open web makes this undesirable. A good solution will decrease the amount of data and they get, or give more data to their competitors.

DuckDuckGo and other API front-ends get a small chunk of search traffic. They are a good compromise between keeping the web open, and having a good search experience as a user. Most of these engines stay as API wrappers forever, so the data they get doesn’t improve them much.

Independent search engines have to make do with scraps. This makes it hard for them to become popular or earn money to support themselves.

How to improve the situation

In this post; I will propose different ways to improve this situation. Each have different trade-offs in user convenience and their usefulness to search engines. The best option would be to use and promote independant search engines. But for a lot of people, it is hard to commit to a sub-par experience even if it is the better long-term option. One can look at how people handle environmental issues to see a prime example of this effect.

Feeding data to search engines

With the first option, you keep using your favourite search engine.

An automated process will send a sample of the queries to different search engines. This way, the engines can get their hands on organic usage data before finding full-time users.

This approach makes automated requests without user interaction. Depending on their systems, this might mess up their data collection or make it difficult to serve real users. To be considerate to the service operators, we should make our requests with a User-Agent header that explains what is happening. This header will allow them to log our requests, handle them in a cheaper way, and to filter them out of the data for their real users.

Redirecting to different search engines

Another approach is to have each search go to a random search engine. Compared to the previous approach, this one is more beneficial to search engines and more incovenient for the user. The user won’t be able to reproduce searches as the same query will end up going to different search providers. Similarly, a smaller search engine might give unsatisfactory results to the user, forcing them to perform the same query multiple times.

This approach can be combined with the previous one as well. By putting a few “good” engines on the random redirect list and feeding data automatically to the rest of them, the downsides could be improved.

Embedding the results of multiple engines

There are already meta-search engines, like Searx, that satisfy some of these requirements. The problem with them though is, each data source they add clutters the main results and slows down search. I think if Searx adds the option of sending data to small search engines in the background without slowing down the main UI, it will be a really good solution to this.

One could use iframes to do this as well, but browsers not being “User Agents” any more, they allow the websites to control their embeddability.

Centralized vs. shared

Another trade-off to consider is where the automated query submission should happen. If you choose a centralized approach, you end up trusting a third-party with your search queries. If you instead choose to handle this yourself without a centralized third-party, you are now sending all your queries to all the other engines in an identifiable way.

There are a few ways to work around this. One of them is to have small public instances like the Fediverse. Everyone would pick who to trust with their queries, and even on small instances the queries would be mixed enough to protect identities. Another approach would be to keep the queries saved locally, and submit them using random proxies.


If there are solutions satifying this need in the future, I am planning to implement this. I just wanted to write this and put it on the internet in case other people are planning similar things. I already have the random search engine redirect working, but in my opinion the most important piece is the automatic data feeding.

The way I will most likely implement this is either a web endpoint that can be added to browsers as a search engine, which can be hosted locally or on a server, or a browser extension.

Gonçalo Valério (dethos)

Django Friday Tips: Subresource Integrity February 26, 2021 06:26 PM

As you might have guessed from the title, today’s tip is about how to add “Subresource integrity” (SRI) checks to your website’s static assets.

First lets see what SRI is. According to the Mozilla’s Developers Network:

Subresource Integrity (SRI) is a security feature that enables browsers to verify that resources they fetch (for example, from a CDN) are delivered without unexpected manipulation. It works by allowing you to provide a cryptographic hash that a fetched resource must match.

Source: MDN

So basically, if you don’t serve all your static assets and rely on any sort of external provider, you can force the browser to check that the delivered contents are exactly the ones you expect.

To trigger that behavior you just need to add the hash of the content to the integrity attribute of the <script> and/or <link> elements in question.

Something like this:

<script src="" integrity="sha256-KSlsysqp7TXtFo/FHjb1T9b425x3hrvzjMWaJyKbpcI=" crossorigin="anonymous"></script>

Using SRI in a Django project

This is all very nice but adding this info manually isn’t that fun or even practical, when your resources might change frequently or are built dynamically on each deployment.

To help with this task I recently found a little tool called django-sri that automates these steps for you (and is compatible with whitenoise if you happen to use it).

After the install, you just need to replace the {% static ... %} tags in your templates with the new one provided by this package ({% sri_static .. %}) and the integrity attribute will be automatically added.

February 22, 2021

Ponylang (SeanTAllen)

Last Week in Pony - February 22, 2021 February 22, 2021 03:11 PM

Ponycheck has become an official Ponylang project. @ergl has opened a new RFC related to FFI declarations. We also have notes from Sean T. Allen and Theo Butler on how to start contributing to Pony.

Marc Brooker (mjb)

Incident Response Isn't Enough February 22, 2021 12:00 AM

Incident Response Isn't Enough

Single points of failure become invisible.

Postmortems, COEs, incident reports. Whatever your organization calls them, when done right they are a popular and effective way of formalizing the process of digging into system failures, and driving change. The success of this approach has lead some to believe that postmortems are the best, or even only, way to improve the long-term availability of systems. Unfortunately, that isn't true. A good availability program requires deep insight into the design of the system.

To understand why, let's build a house, then a small community.

A house, with four things it needs to be a working home

Our house has four walls, a roof, and a few things it needs to be a habitable home. We've got a well for water, a field of corn for food, a wood pile for heat, and a septic tank. If any one of these things is not working, let's say that the house is unavailable. Our goal is to build many houses, and make sure they are unavailable for as little of the time as possible.

When we want to build a second house, we're faced with a choice. The simple approach is just to stamp out a second copy of the entire house, with it's own field, wood, well, and tank. That approach is great: the failure of the two houses is completely independent, and availability is very easy to reason about.

Two houses, with full redundancy

As we scale this approach up, however, we're met with the economic pressure to share components. This makes a lot of sense: wells are expensive to drill, and don't break down often, so sharing one between many houses could save the home owners a lot of money. Not only does sharing a well reduce construction costs, but thanks to the averaging effect of adding the demand of multiple houses together, reduces the peak-to-average ratio of water demand. That improves ongoing economics, too.

Five houses, sharing a well

In exchange for the improved economics, we've bought ourselves a potential problem. The failure of the well will cause all the houses in our community to become unavailable. The well has high blast radius. Mitigating that is well-trodden technical ground, but there's a second-order organizational and cultural effect worth paying attention to.

Every week, our community's maintenance folks get together and talk about problems that occurred during the week. Dead corn, full tanks, empty woodpiles, etc. They're great people with good intentions, so for each of these issues they carefully draw up plans to prevent recurrence of the issue, and invest the right amount in following up on those issues. They invest in the most urgent issues, and talk a lot about the most common issues. The community grows, and the number of issues grows. The system of reacting to them scales nicely.

Everything is great until the well breaks. The community is without water, and everybody is mad at the maintenance staff. They'd hardly done any maintenance on the well all year! It wasn't being improved! They spent all their attention elsewhere! Why?

The problem here is simple. With 100 houses in the community, there were 100 fields, 100 tanks, 100 piles, and one well. The well was only responsible for 1 in every 301 issues, just 0.33%. So, naturally, the frequency-based maintenance plan spent just 0.33% of the maintenance effort on it. Over time, with so little maintenance, it got a little creaky, but was still only a tiny part of the overall set of problems.

Plot showing how the percentage of action items related to the well drops with scale

This is one major problem with driving any availability program only from postmortems. It feels like a data-driven approach, but tends to be biased in exactly the ways we don't want a data-driven approach to be biased. As a start, the frequency measurement needs to be weighted based on impact. That doesn't solve the problem. The people making decisions are human, and humans are bad at making decisions. One way we're bad at decisions is called the Availability Heuristic: We tend to place more importance on things we can remember easily. Like those empty wood piles we talk about every week, and not the well issue from two years ago. Fixing this requires that an availability program takes risk into account, not only in how we measure, but also in how often we talk about issues.

It's very easy to forget about your single point of failure. After all, there's just one.

February 21, 2021

Derek Jones (derek-jones)

Research software code is likely to remain a tangled mess February 21, 2021 11:32 PM

Research software (i.e., software written to support research in engineering or the sciences) is usually a tangled mess of spaghetti code that only the author knows how to use. Very occasionally I encounter well organized research software that can be used without having an email conversation with the author (who has invariably spent years iterating through many versions).

Spaghetti code is not unique to academia, there is plenty to be found in industry.

Structural differences between academia and industry make it likely that research software will always be a tangled mess, only usable by the person who wrote it. These structural differences include:

  • writing software is a low status academic activity; it is a low status activity in some companies, but those involved don’t commonly have other higher status tasks available to work on. Why would a researcher want to invest in becoming proficient in a low status activity? Why would the principal investigator spend lots of their grant money hiring a proficient developer to work on a low status activity?

    I think the lack of status is rooted in researchers’ lack of appreciation of the effort and skill needed to become a proficient developer of software. Software differs from that other essential tool, mathematics, in that most researchers have spent many years studying mathematics and understand that effort/skill is needed to be able to use it.

    Academic performance is often measured using citations, and there is a growing move towards citing software,

  • many of those writing software know very little about how to do it, and don’t have daily contact with people who do. Recent graduates are the pool from which many new researchers are drawn. People in industry are intimately familiar with the software development skills of recent graduates, i.e., the majority are essentially beginners; most developers in industry were once recent graduates, and the stream of new employees reminds them of the skill level of such people. Academics see a constant stream of people new to software development, this group forms the norm they have to work within, and many don’t appreciate the skill gulf that exists between a recent graduate and an experienced software developer,
  • paid a lot less. The handful of very competent software developers I know working in engineering/scientific research are doing it for their love of the engineering/scientific field in which they are active. Take this love away, and they will find that not only does industry pay better, but it also provides lots of interesting projects for them to work on (academics often have the idea that all work in industry is dull).

    I have met people who have taken jobs writing research software to learn about software development, to make themselves more employable outside academia.

Does it matter that the source code of research software is a tangled mess?

The author of a published paper is supposed to provide enough information to enable their work to be reproduced. It is very unlikely that I would be able to reproduce the results in a chemistry or genetics paper, because I don’t know enough about the subject, i.e., I am not skilled in the art. Given a tangled mess of source code, I think I could reproduce the results in the associated paper (assuming the author was shipping the code associated with the paper; I have encountered cases where this was not true). If the code failed to build correctly, I could figure out (eventually) what needed to be fixed. I think people have an unrealistic expectation that research code should just build out of the box. It takes a lot of work by a skilled person to create to build portable software that just builds.

Is it really cost-effective to insist on even a medium-degree of buildability for research software?

I suspect that the lifetime of source code used in research is just as short and lonely as it is in other domains. One study of 214 packages associated with papers published between 2001-2015 found that 73% had not been updated since publication.

I would argue that a more useful investment would be in testing that the software behaves as expected. Many researchers I have spoken to have not appreciated the importance of testing. A common misconception is that because the mathematics is correct, the software must be correct (completely ignoring the possibility of silly coding mistakes, which everybody makes). Commercial software has the benefit of user feedback, for detecting some incorrect failures. Research software may only ever have one user.

Research software engineer is the fancy title now being applied to people who write the software used in research. Originally this struck me as an example of what companies do when they cannot pay people more, they give them a fancy title. Recently the Society of Research Software Engineering was setup. This society could certainly help with training, but I don’t see it making much difference with regard status and salary.

Carlos Fenollosa (carlesfe)

Whatever Clubhouse is, they are onto something February 21, 2021 11:35 AM

I've been following Clubhouse for a few weeks. As a podcaster, it piqued my interest. So it's like podcasts, but live?

The official slogan is drop-in audio chat. But that's not good. It only makes sense once you've used the app, and it doesn't describe the whole thing.

For me, the perfect definition is: it's Twitch for audio. But then, you need to know what Twitch is.

Yesterday I received an invitation and finally got to try it first hand. And I think that Clubhouse is onto something.

Radio vs Podcasts

Everybody knows radio. Even during this Internet revolution, it still has survived. Why? Because it's convenient. You tune in to some station and listen to music or people talking. It requires zero effort.

Radio has two problems: the fact that it's live, and the selection of topics.

Nowadays it's easy to download aired shows, so if you really like some program but you missed it when it was live, just go to their website and download the mp3 file.

However, the selection of topics still is an issue. Due to the fact that a station is a business, and that its model is airing ads, it requires volume. Therefore most radio stations produce mainstram content.

With the coming of the internet, a few nerds started using a new technology called Podcasts. You could record any audio content with a 1€ microphone and publish it on the internet.

Even though podcasts are naturally asynchronous, many shows air live too. Some listeners can listen to the stream, but most of them just download the audio file later.

Publicly searchable podcast directories aggregate both amateur and professional audios. Thanks to that, we have reached this point where anybody in the world has access to an ocean of audio content about any topic, either mainstream or niche.

Enter Clubhouse

What Twitch did to Youtube, Clubhouse has done to podcasts. For the sake of this explanation, let's ignore that podcasts are an open ecosystem and Youtube is proprietary.

Youtube is a video discovery platform. It has some tools to livestream, but it's not their main focus. Twitch has a much better product (and ToS) for livestreamers and their audience.

Want to watch somebody playing Minecraft? Open Twitch, search for Minecraft, and boom! hundreds of streams right there. Join one, chat with the community, and if you're lucky the streamer may shout out to you.

You can't do that with podcasts.

First of all, there can be some interactivity by combining an Icecast stream with an IRC channel, but it is not a good system.

Second, live podcasts are not aggregated anywhere. It is just impossible to search for "strategies to control your stress during covid-19" and find live shows.

So, if only as a directory of live audio content, Clubhouse has future.

But it is not only that. The product is very well thought and lets the audience participate, with audio.

A naive approach would have been to include a text chat on top of the audio stream. That would replicate the current solution on an integrated app. Okay, not bad.

However, the Clubhouse team spent some time thinking about the use case for audio streaming, which is not the same as for video streaming, nor public chat rooms.

Most of us listen to audio while we are doing other tasks and most of the times our hands are busy. This is why people jokingly call it the Airpods social network. You can participate while being away from a phone or computer.

In Clubhouse, you can tap a button to "raise your hand", and the moderators may "unmute" you. Then you can talk to the rest of the audience. Of course, not all show formats allow for that, but the option is there.

Being able to talk to your idols or even talk to the community of fans is very powerful. My first experience with Clubhouse was moving. I was listening to a concert and after the show all the listeners gathered up to talk about their experience and to have a chat with the band. Everybody agreed that with Clubhouse you can feel that there's people at the other end. Not only the speakers, but also the audience.

You don't get that with podcasts, even with live ones with a chat room.

A new category

Clubhouse has definitely invented a new category which combines the best of radio and the best of podcasts.

The product implements a selection of novel features which, when brought together, create an exciting and very addictive experience:

  • Directory of live audio streams ("rooms") about any imaginable topic
  • You can quickly drop in any room, listen for a few minutes, and jump to another one
  • The audience can participate via audio, which creates a great sense of community
  • Basic tools to follow people and interests, and get notified when they live stream
  • Of course, streamers may record the audio and publish it afterwards, so it's trivial to use Clubhouse in combination with the current podcasting ecosystem.

If you're in the podcasting community you should try to find an invitation. It is the real deal.

Tags: internet, podcasting

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

Pepijn de Vos (pepijndevos)

Switching Continuously Variable Transmission February 21, 2021 12:00 AM

What if you took a boost converter and converted it to the rotational mechanical domain? Switching CVT!

At the University of Twente, they teach Bond Graphs, a modelling system for multi-domain systems that is just perfect for this job. Unlike domaing-specific systems or block diagrams, Bond Graphs model a system as abstract connections of power. Power is what you get when you multiply an effort with a flow. The two examples we’re interested is voltage × current and force × velocity, or to be exact, angular momentum × angular velocity.

Here is a schematic of a boost converter (source). It goes from a high voltage (effort, force) to a low voltage, but from a high current (flow, velocity) to a low current. It works by charging the inductor by shorting it to ground, and then discharging it via the diode into the capacitor.

boost converter

The classic example of model equivalence is that an electrical inductor-capacitor-resistor system behaves equivalent to a mechanical mass-spring-damper system. In the rotational domain, the equivalent of a switch is a clutch, and the equivalent of a diode is a ratchet. So we have all we need to convert the system! Step one is making the bond graph from the electrical system.

boost converter bond graph

Quick Bond Graph primer if you’re too lazy to read the Wikipedia page. Se is a source of effort. R, I, and C are generalized resistance, inertance, and compliance. mR is a modulated resistance I used for the switch/clutch. D is a diode/ratchet that I just made up. 0 junctions are sum of flows, equal effort. 1 junctions are sum of effort, equal flow. An ideal electrical net has equal voltage (effort), and a sum of currents, but a mechanical joint has an equal velocity (flow), but a sum of forces. With that in mind, we can convert the bond graph to the mechanical system.

mechanical boost converter

I’m not sure if those are even remotely sane mechanical symbols, so I added labels just in case. The motor spins up a flywheel, and then when the clutch engages it winds up the spring. Then when the clutch is disengaged, the ratched keeps the spring wound up, driving the output while the motor can once more spin up the flywheel.

It works exactly analogous to the boost converter, and also suffers from the same problems. Most ciritically, switching/clutching losses. I imagine applying PWM to your clutch will at best wear it down quickly, and maybe just make it go up in smoke. Like with a MOSFET, the transition period where there is a nonzero effort and flow on the clutch, there is power loss and heat.

Anyway, I decided to build it in LEGO to see if it’d work. I used a high-speed ungeared motor that can drive absolutely no load at all, and connected it with a 1:1 gear ratio to the wheels with only a flywheel, clutch, ratchet, and spring inbetween. This proves that there is actually power conversion going on!

lego mechanical boost converter

If you get rich making cars with this CVT system, please spare me a few coins. If you burn out your clutch… I told you so ;)

February 18, 2021

Pete Corey (petecorey)

Genuary 2021 February 18, 2021 12:00 AM

I didn’t participate in Genuary this year, but I was inspired by a few of the submissions I saw to hack together my own sketches. Here’s what I came up with.

I was originally inspired by this Reddit post on expanding circles, so I recreated it and added some extra layers of depth. My kingdom for a plotter and a mechanical pencil:

From there, I thickened the stroke width of each circle, and colored each pixel based on the number of circle intersections there (using 2D SDF to check for intersections, and the cubehelix algorithm for coloring). There’s some really cool kaleidoscope vibes in some of these variations:

The SDF technique caught my imagination, so I spent some more time playing with using SDF and cubehelix to render simple circles:

This post inspired me to play with turmites a bit. The &lbrace&lbrace&lbrace1, 2, 1}, &lbrace1, 8, 1}}, &lbrace&lbrace1, 2, 1}, &lbrace0, 2, 0}}} turmite is especially cool. Coloring it based on the number of visits to each cell, and removing the “state lines” shows some interesting structures:

While I didn’t officially participate, I had a lot of fun with Genuary this year.

February 15, 2021

Ponylang (SeanTAllen)

Last Week in Pony - February 14, 2021 February 15, 2021 02:47 AM

The supported version of FreeBSD is moving from 12.1 to 12.2. The Apple M1 support team has an initial report. The documentation site website,, is being shut down. The default branch renaming is underway. Interested in contributing to Corral or other Pony tools?

Andreas Zwinkau (qznc)

Software Architecture: How You Make Them Care February 15, 2021 12:00 AM

Software architects must code, talk business, and tell stories.

Read full article!