TL;DR: the Gemini protocol removes too much functionality for it to interesting to me.

What is Gemini?

Gemini is a simple web publishing protocol. It can be seen as a descendant, successor of Gopher. Gemini primarily emphasizes developer simplicity, and secondarily user privacy.

This comes with significant trade-offs for the author, however. Compared to standard vanilla HTML4, there are no inline links, no provision for media other than text on a page, and the styling of the content is left to the client.

My background in web publishing - I’ve been fascinated by publishing since I was a kid and I’ve been involved in printing zines and in student newspapers etc through the years. The idea that I can publish what I want, when I want, at whatever lengths I want, for effectively free, is still mind-blowing to me, almost 30 years since I copied some HTML code and made it mine.

Here’s where Gemini falls down for me.

First, there’s no official client. The fact that it’s so easy to implement a client means there’s a Cambrian explosion going on, and the filtering die-back has not yet occurred. This might change in the medium future.

Second, the styling limitations are crippling. I can probably survive without having images etc. on the same page, but the lack of inline links (each link has to be on its own line) leads to stilted, quasi-academic jargony text like this:

Check out my cool blog[1]! It’s full of cats!

[1] https://gerikson.com/blog

I’m not going to abandon three decades of hypertext authoring habits to make a developer’s life slightly easier.

Third, Gemini puts the cart before the horse when it comes to privacy. The solution to widespread tracking and user surveillance isn’t a bespoke hairshirt protocol that no-one is going to use. The solution is widespread legislation that makes using people’s personal data for targeted advertising illegal or very expensive. (This is not limited to Gemini. A great many influential Internet people are convinced politics is utterly broken, so “technical solutions” are all that’s left).

Gemini, to me, is part of the nostalgia for a past that never really was — the halcyon days of the Internet, before the Eternal September. But time is the great filter. What has survived from that era is not the spam, the pointless Usenet arguments, the shitposting, but finely polished nuggets. If you weren’t there, it might have seemed a paradise, but I was, and it wasn’t. It was today’s internet, but text-only and with proportionally even more white dudes.

Update

This entry was submitted to Lobste.rs and spawned an interesting discussion. Among others, this piece was posted (comments), and this post was also mentioned.

I suggest those interested in Gemini and what it means to people to read the linked items and discussions.

I’d also like to clarify that I wrote this rant basically to have a single place to point Gemini proselytizers to. I need to emphasize that if you personally use Gemini, or find Gemini useful or fun, more power to you! I am not suggesting you stop or that your efforts are in vain.

Sadly, there are some Gemini enthusiasts who are the equivalent of people handing out flyers for some obscure club, and who get shirty when you try to politely explain that you’re not interested in attending. Those people are annoying, and they give Gemini a bad name.

I have found that the secret for discovering data treasure troves is persistently following any leads that appear. For instance, if a researcher publishes a data driven paper, then check all their other papers. The paper: Composing Effective Software Security Assurance Workflows contains a lot of graphs and tables, but no links to data, however, one of the authors (William R. Nichols) published The Cost and Benefits of Static Analysis During Development which links to an amazing treasure trove of project data.

My first encounter with this data was this time last year, as I was focusing on completing my Evidence-based software engineering book. Apart from a few brief exchanges with Bill Nichols the technical lead member of the team who obtained and originally analysed the data, I did not have time for any detailed analysis. Bill was also busy, and we agreed to wait until the end of the year. Bill’s and my paper: The CESAW dataset: a conversation is now out, and focuses on an analysis of the 61,817 task and 203,621 time facts recorded for the 45 projects in the CESAW dataset.

Our paper is really an introduction to the CESAW dataset; I’m sure there is a lot more to be discovered. Some of the interesting characteristics of the CESAW dataset include:

• it is the largest publicly available project dataset currently available, with six times as many tasks as the next largest, the SiP dataset. The CESAW dataset involves the kind of data that is usually encountered, i.e., one off project data. The SiP dataset involves the long term evolution of one company’s 20 projects over 10-years,
• it includes a lot of information I have not seen elsewhere, such as: task interruption time and task stop/start {date/time}s (e.g., waiting on some dependency to become available)
• four of the largest projects involve safety critical software, for a total of 28,899 tasks (this probably more than two orders of magnitude more than what currently exists). Given all the claims made about the development about safety critical software being different from other kinds of development, here is a resource for checking some of the claims,
• the tasks to be done, to implement a project, are organized using a work-breakdown structure. WBS is not software specific, and the US Department of Defense require it to be used across all projects; see MIL-STD-881. I will probably annoy those in software management by suggesting the one line definition of WBS as: Agile+structure (WBS supports iteration). This was my first time analyzing WBS project data, and never having used it myself, I was not really sure how to approach the analysis. Hopefully somebody familiar with WBS will extract useful patterns from the data,
• while software inspections are frequently talked about, public data involving them is rarely available. The WBS process has inspections coming out of its ears, and for some projects inspections of one kind or another represent the majority of tasks,
• data on the kinds of tasks that are rarely seen in public data, e.g., testing, documentation, and design,
• the 1,324 defect-facts include information on: the phase where the mistake was made, the phase where it was discovered, and the time taken to fix.

As you can see, there is lots of interesting project data, and I look forward to reading about what people do with it.

Once you have downloaded the data, there are two other sources of information about its structure and contents: the code+data used to produce the plots in the paper (plus my fishing expedition code), and a CESAW channel on the Evidence-based software engineering Slack channel (no guarantees about response time).

Borja (@ergl) released pony-protobuf, an experimental Pony protocol buffers library and compiler.

I see a shiny grassy green diamond, some guys in uniform. One guy on a patch of dirt in the middle throws a ball, and another guy tries to hit it with a stick. If he hits it he wins, if he doesn’t: the other guy wins. It looks like most sports.
On second glance however, I realize things are not as clear. The guy with the stick — I guess it’s called a bat — seems to be alone against a team. The rest of his team is looking on. Is that fair? Even a soccer penalty — a mostly rare event — it’s still one against one. And how often can he try to hit the ball and how often can this other guy throw it? This is not clear. And what happens after he hits it? And also, what’s an inning? And how many are there? And are there any other sports where they wear hats?

And it’s not just the hats though. Belts too. Which athletes wear belts? Really.
Both teams wear collarless uniforms that look like they haven’t changed in a 100 years, which must be intentional. Some wear long pants, others wear knee socks. Either way, the uniforms are buttoned up and either striped or clear. But always neatly tucked in. Except for the guy trying to catch the ball — behind the guy with the bat — he has his own thing going on.

And then there is this oversized strange looking leather glove. It makes the players look like crabs who lost a claw. But every player wears one like it’s perfectly normal. The glove is mostly used to catch balls, but it also doubles as an accessory to share secrets with teammates that you want to keep from the other team, whose players must all be prodigious lip readers.

So everyone wears a glove. Except the guy with a bat. A bat that’s either made of wood or metal. Okay, so let’s see, if you have the bat you get three attempts to try and hit the ball? If you hit it correctly, you have to run to a first designated stop — or a base — and if you make it to this first base without your ball being caught or you ball arriving there before you — because someone picked it up and threw it there — you’re safe. Three more bases for a point. If you hit it really hard you might try and run to two, or even three bases in one go. But this is hard. Only if you hit it out of the park, you get to go past all bases and collect a point. That’s called a homerun. But mostly you try to make it to first base, and you wait for your teammate to try and hit the ball. If he does, you get a chance to run to second base (or more). Unless of course a ball is caught or hitters haven’t hit the ball for a combined number of three times, then teams switch sides.

Pitchers — as the guys that throw the ball are called — seem to center themselves by straightening their hat — a lot — so that their cap is perfectly perpendicular to their faces. And they like to fondle the ball between their fingers while either specifically looking at it or distinctively not. Then they do a little bit more hat straightening before curling up by pulling their knee to their chest, channeling all their power into their throw to make the ball fly around 90 mph between a small rectangle box that is only visible on TV and not to the players. And they do this again and again and again. He has to throw the ball three times correctly between the rectangle — without being it being hit — for the hitter to be out. If he throws it four times outside the box, the hitter is free to walk to first base.

Hitters also seem to try to get in a meditative state of mind. There are quite a few mannerisms and straight out ticks they perform while standing at the plate. They touch the middle of the plate, swing the bat an exact numbers of times, pull the fingers of their gloves — different gloves — an even number of times. Adjusting, centering, focussing. If you look closely some guys have stains on their right collarbone or shoulder, it’s where the bat touches as they twist and twirl it while waiting for the ball to arrive. It seems that wooden bats are tarred up, for grip, maybe? It’s not as much a physical as it is game about focus and wits. Nonetheless you have to throw the ball really hard, and hit it really hard. It is a multidimensional game.

Players all look very serious and focused. There is no flopping or cheating, everybody knows the rules and umpires are on top of things. And maybe it’s this aspect that I like most. It’s a fair game.

It quickly becomes clear that this a game of variables. Lots and lots of variables. And watching a baseball game must be a supremely different experience from playing the game. Watching the game I am constantly assessing stats, making small calculations, holding variables in my head. There are so many intangibles to keep track of, your head needs to clear out the rest of whatever it is working on. Watching baseball is almost meditative. For the guys on the field however, the only focus can be the next pitch.

I learn that an inning has two parts — a top and a bottom — and a pitcher has to try and strike out hitters three times for a team to change sides, this constitutes an inning. As I learn this the numbers on the screen start to make a little bit more sense, it’s as if I am slowly let in on a secret. A very old and unchanged secret.

A secret that can be endlessly explored and excavated further. There is no shortage of variables to do so. So an RBI is a Round Batted In, that seems important. And I am glad the announcers explain what the 7th inning stretch is. This sport is drenched in tradition and statistics, and if you want you can drown in it. Not only are there 120 years of history to dive in to, even with such a long history the sport is still evolving and historic events occur in this day and age.

Baseball is played all over the US: little league, high school, college, minor league and more. But Major League Baseball is of course where the best of the best compete. There are 30 teams in the Major League Baseball. Which doesn’t seem excessive for a country as big as the US. But teams play 162 games in a regular season. Please read the last sentence again, this is not a typo. This adds up to 2430 baseball games in one MLB season! You might call that excessive and very on-brand for the US. So the number of games and stats to obsess over are dazzling. And maybe it’s because of these numbers, but there is something about this sport. Because special things tend happen or are — statistically speaking — bound to happen. And they do. All of the time.

And so much so that you might even call this sport romantic. This makes complete sense.

The post A European watches Major League Baseball appeared first on Jan van den Berg.

I've been wanting to learn more about compilers and toolchains in general for a while now. In June 2016, I asked about recommended readings on lexers and parsers on Twitter. However, I have to confess that I didn't go forward with reading the Dragon Book.

Instead, I got involved as a developer in the OpenBSD and NetBSD projects, and witnessing the evolution of toolchains within those systems played a big role in maintaining my interest and fascination in the topic. In retrospective, it now becomes apparent that the work I did on porting and packaging software for those systems really helped to put in perspective how the different parts of the toolchains interact together to produce binaries.

Approximately one year ago, I asked again on Twitter whether I knew anyone having worked on compilers and toolchains professionally to get real world advice on how to gain expertise in the field. I got several interesting answers and started to collect and read more resources on the topic. Some of the links I collected ended up on toolchains.net, a collection of toolchain resources which I put online in February.

But the answer that resonate the most with me was Howard's advice to learn by doing. Because I seem to be the kind of person which need to see some concrete results in order to keep motivated, that's exactly what I decided to do.

I started by doing some cleanups in the binutils package in NetBSD's pkgsrc, which resulted in a series of commits:

• 2020-12-20 - ca38479 - Remove now unneeded OpenBSD specific checks in gold
• 2020-12-15 - 7263eee - Add missing TEST_DEPENDS on devel/dejagnu
• 2020-12-14 - b1637da - Don't use hard-coded -ldl in the gold test suite.
• 2020-12-13 - 146def2 - Remove apparently unneeded patch for libiberty
• 2020-12-12 - 6b347a9 - Remove CFLAGS.OpenBSD+= -Wno-bounded directive
• 2020-12-11 - f53b2d8 - Remove now unneeded patch dropping hidden symbols warning
• 2020-12-10 - b037380 - Enable building gold on Linux
• 2020-12-03 - 75d00bc - Remove now unneeded workaround for binutils 2.24
• 2020-12-03 - adfee30 - Drop all Bitrig related patches

Meanwhile, I also got the opportunity to update our package and apply security fixes:

• 2021-02-11 - 761e000 - Update to binutils 2.36.1
• 2021-01-27 - ba983e5 - Update to binutils 2.36
• 2021-01-07 - 7aef5c0 - Add upstream fixes for CVE-2020-35448
• 2020-12-06 - 99fdf39 - Update to binutils 2.35.1

I eventually took maintainership of binutils in Pkgsrc.

Building it repeatedly with different compilers exposed different warnings, and I've also run builds through Clang's static analyzer.

All of this resulted in the opportunity to contribute to binutils itself:

• 2021-04-14 - 5f47741 - Remove unneeded tests for definitions of NT_NETBSDCORE values
• 2021-04-12 - 0fa29e2 - Remove now unneeded #ifdef check for NT_NETBSD_PAX
• 2021-03-12 - be3b926 - Add values for NetBSD .note.netbsd.ident notes (PaX)
• 2021-01-26 - e37709f - Fix a double free in objcopy's memory freeing code

Most recently, I also wrote a couple of blog posts on the topic:

• The state of toolchains in NetBSD
• Speedbuilding LLVM/Clang in 5 minutes
• Speedbuilding LLVM/Clang in 2 minutes on ARM
• The state of toolchains in OpenBSD
• Playing with DJGPP and GCC 10 on DOS

And the journey continues. I'm following a different path from traditional compiler courses starting with lexers and parsers, and doing the opposite curriculum somehow, starting from binaries instead. I will be focusing on the final stages of the pipeline for now: compiling assembly to machine code and producing binaries.

My next steps are to read the full ELF specification, followed by the Linkers and Loader book, and then refresh my ASM skills. My favorite course at university was the computer architecture one and especially its MIPS assembly part, so I'm looking to revisit the subject but with ARM64 assembly this time.

This is Gibson’s worst novel. Not recommended.

It’s in the same (multi)verse as The Peripheral.

While that novel had engaging characters, this one doesn’t. The Jackpot protagonist, Wilf, shows us the horrifying prospect of the repressed Englishman surviving global collapse and an 80% die-off of humanity. The present-day character has no inner life to speak of. I have no clue how she managed to get in a relationship with her world’s Elon Musk analog. We’re supposed to believe that Eunice, the AI that the Jackpot side uses to try to avert nuclear war, is this fantastically engaging personality everyone loves, but in the end she’s basically sassy Magic Negro. The novel has entire chapters describing drives through the Bay Area.

In the end it reads as therapy for Gibson to cope with the Trump years.

When creating a variable name, to what extent are developers influenced by their native human language?

There is lots of evidence that variable names are either English words, abbreviations of English words, or some combination of these two. Source code containing a large percentage of identifiers using words from other languages does exist, but it requires effort to find; there is a widely expressed view that source should be English based (based on my experience of talking to non-native English speakers, and even the odd paper discussing the issue, e.g., Language matters).

Given that variable names can prove information that reduces the effort needed to understand code, and that most code is only ever read by the person who wrote it, developers should make the most of their expertise in using their native language.

To what extent do non-native English-speaking developers make use of their non-English native language?

I have found it very difficult to even have a discussion around this question. When I broach the subject with non-native English speakers, the response is often along the lines of “our develo0pers speak good English.” I am careful to set the scene by telling them of my interest in naming, and that I think there are benefits for developers to make use of their native language. The use of non-English languages in software development is not yet a subject that is open for discussion.

I knew that sooner or later somebody would run an experiment…

How Developers Choose Names is another interesting experiment involving Dror Feitelson (the paper rather confusingly refers to it as a survey, a post on an earlier experiment).

What makes this experiment interesting is that bilingual subjects (English and Hebrew) were used, and the questions were in English or Hebrew. The 230 subjects (some professional, some student) were given a short description and asked to provide an appropriate variable/function/data-structure name; English was used for 26 of the question, and Hebrew for the other 21 questions, and subjects answered a random subset.

What patterns of Hebrew usage are present in the variable names?

Out of 2017 answers, 14 contained Hebrew characters, i.e., not enough for statistical analysis. This does not mean that all the other variable names were only derived from English words, in some cases Hebrew words appeared via transcription using the 26 English letters. For instance, using “pinuk” for the Hebrew word that means “benefit” in English. Some variables were created from a mixture of Hebrew and English words, e.g., deservedPinuks and pinuksUsed.

Analysing this data requires someone who is fluent in Hebrew and English. I am not a fluent, or even non-fluent, Hebrew speaker. My role in this debate is encouraging others, and at last I have some interesting data to show people.

The paper spends time showing how for personal preferences result in a wide selection of names being chosen by different people for the same quantity. I cannot think of any software engineering papers that have addressed this issue for variable names, but there is lots of evidence from other fields; also see figure 7.33.

Those interested in searching source code for the impact of native-language might like to look at the names of variables appearing as operands of the bitwise and logical operators. Some English words occur much more frequently in the names of these variable, compared to variables that are operands of arithmetic operators, e.g., flag, status, and signal. I predict that non-native English-speaking developers will make use of corresponding non-English words.

This is a short write-up of how a channel on Freenode was hijacked by staff, and how it was effectively deleted.

Background

The channel #photography was founded on Freenode in 2001, as a channel to discuss photography. Its single-hash status was challenged by Freenode staffer lilo, and it was changed to #photogeeks to comply with the requirement for single-hash channels to be associated with a project.

In 2006, the channel #photo was registered to discuss photography per se, and not gear. Quite soon the channel became social in nature, and it was hidden and a password set.

Over the years, about a dozen people used this channel as a social space to discuss everything under the sun.

Hijack

When the takeover of Freenode by Andrew Lee occurred, the channel was registered on Libera.chat, and on 2021-05-21, the topic was updated to say

09:07  -!- gustaf changed the topic of #photo to: this channel is now up and running on Libera.chat (same pwd), but this place is still the primary! | Discord bolthole - https://discord.gg/XXXX

On 2021-05-26, the channel was hijacked:

05:02  -!- freenodecom <~com@freenode/staff> has joined #photo
05:02  -!- mode/#photo (+o freenodecom) by freenodecom
05:02  -!- freenodecom changed the topic of #photo to: This channel has moved to ##photo. The topic is in violation of freenode policy: https://freenode.net/policies
05:02 <@freenodecom> This channel has been reopened with respect to the communities and new users. The topic is in violation of freenode policy: https://freenode.net/policies
05:02  -!- mode/#photo (+o freenodecom) by OperServ
05:02 <@freenodecom> The new channel is ##photo
05:02  -!- mode/#photo (-s+t) by ChanServ
05:02  -!- mode/#photo (+spimf ##photo) by freenodecom
05:02  -!- freenodecom <~com@freenode/staff> has left #photo ()
05:02  -!- mode/#photo (+f ##photo) by freenodecom
07:37 < gustaf> <abbr title="what the fsck">wtf</abbr>
07:37  -!- #photo Cannot send to nick/channel

(all times are in CEST).

A few hours later, Andrew Lee (rasengan) sent a network-wide wallop informing users that an attempt to enforce newly instituted rules against advertising other IRC networks had been overly broad and targeted more channels than intended.

It’s clear that our channel was included in this.

Users were encouraged to submit a request to Freenode staff to get their channels back.

Aftermath

The “regulars” of the channel were contacted via PM and informed that the Freenode channel was now closed. Most moved over to Libera. A few mentioned that they were permanently leaving Freenode.

As a good faith effort, prompted by Freenode promoters, I attempted to regain control of the channel at Freenode, but was informed that having a single-hash channel was not according to policy. The request was denied.

Discussion

I’ve written this post to present my side of the story. Over the last weeks, I’ve been told on Freenode that the widespread channel hijacking of 26 May 2021 (some reports say that 700+ channels were affected) was either not as widespread as reported, or “justified” to stem the flow of users to Libera.

I’m also a member of the channel #lobsters, which suffered the same fate. However, in that case, the project had officially moved, and the Freenode channel was locked. Based on Lee’s rationale, I actually find the hijack justified, as there were presumably people who would prefer to remain on Freenode and discuss the site there. However, note that there very little warning before this happened. There was no attempt to contact the project to present Freenode’s case as a better IRC host than Libera. Freenode instead unilaterally decided they knew better than the project’s themselves.

When Libera was announced, I did not feel that the urgency presented by the staff there was entirely justified. Never would I imagine that Andrew Lee would, within a week, exceed those warnings by a wide margin.

He and the current Freenode staff have proven that they cannot be trusted to be stewards of communities, by hijacking channels and disrupting them. They have proven to be incompetent, by affecting more channels than intended. They have proven to be discourteous, by requiring channel owners affected by their incompetence to apply, hat in hand and papers in order, for their channels to be reinstated. And finally, they’ve proven to be bad business people, by alienating their future customers and torching their future income stream.

Future

I’m nostalgic for my almost 17 year old Freenode account. But the more time passes, the more bitter I become. I’m going to hang around in some channels to see how things work out. I’m open to a more humble approach from Freenode staff and boosters. But if I feel I can’t be a part of a network that treats its users as peons to be exploited, I’m out.

We live on a street that's about 29 feet wide. There are parked cars on either side that take up about 7 feet each, which leaves 15 feet in the middle of the street for through traffic. There are hundreds or thousands of streets like this in San Francisco; here's a screenshot of one at random here.

Cars go really fast on our street - they'll gun it up to 35-40mph, but only when there's no traffic coming in the other direction. When traffic comes in the other direction they need to slow to steer between the oncoming car and the row of parked cars. Speeding cars are not safe for kids and they are leading sources of noise and air pollution.

If there are no cars coming, they can drive very quickly right down the middle of the street. So the obvious first step to slowing down traffic is don't let the cars drive in the middle of the street.

The SFMTA's program to address speeding involves adding a speed bump. A speed bump is very effective at slowing cars, but it's expensive and labor intensive. They only have the budget to address about 30-40 blocks a year.

What if we wanted to fix this for every residential block in the city? You'd want something that you could put in the very middle of the street, to prevent the behavior where cars drive very fast right down the middle. The ideal obstacle would be

1. cheap
2. would not cause any damage to the car
3. very visible
4. easy to move if necessary

I think a seven inch tall plastic soccer cone fits these criteria. You can buy them on Amazon for less than $1 each.

With a cone in the street, cars would either need to drive over it (undesirable) or around it, which would force them to slow down because there's a lot less space between the cones and the lane of parked cars.

My theory is that this would slow down cars a lot. It's cheap to test! Let me know if you are interested in trying it.

Measuring

To convince the MTA you'd want to do a test to see whether the cone is bringing down speeds. To do this, pick an hour during a weekday and count the car traffic — speeding vs. non-speeding, and then deploy the intervention the next day or next week during the same hour, and count the car traffic, speeding vs. non-speeding. You can then do a statistical test to determine if the results are significant.

You can buy a radar gun for about $100 to measure car speeds, which would be more accurate than eyeballing "speeding" vs. "not speeding" or have someone who doesn't know about the experiment (and can't see the cones) make the judgments.

This week I continued to progress in DuoLingo. My goal for this week was to learn definite conjugations of common verbs, but although I made some progress towards that end, I haven’t made as much progress as I had hoped. Instead I spent my past week drilling my understanding of the vocabulary I have learned so far, accusative and plural endings for nouns, and plural person indefinite verb conjugations (we see, you see, they see). I’ve gotten to a point where I think I need to practice what I know a little bit more before moving on to keep things from becoming too confusing.

I’m hoping that in the coming week I can get comfortable enough with my current knowledge to continue progressing. Outside of DuoLingo, I’ve started trying to translate some of the articles on the Hungarian National Geographic website using Glosbe as needed to look up the definition of words. It’s pretty difficult for me to read those still, but I’m hoping that the exposure to the language with pictures and context will be beneficial.

I’m still trying to find Hungarian children’s ebooks to read on my Kindle but I’m having trouble finding them. That might be because my Hungarian knowledge isn’t good enough yet to know how to search for them in Hungarian though. What I’d really like is to be able to find the Magic Treehouse series translated to Hungarian, or a similar book series. Something like the DK Eyewitness books would be good too.

Some Hungarian sentences

I figured as a little self exercise this week I’d post some pictures and try to write as much as I can about them with my current understanding of Hungarian. Native speakers who are reading this, please feel free to correct my sentences and translations below.

Ez egy piros autó. Azok ablakok az autó mögött vannak. Az autó Ford Mustang GT. Ez az autó egy Ford Mustang GT. Az autó kint áll. Az egy épület az autó mögött van. Az épület az autó mögött van. Az az autó szeretek Szeretem ezt az autót.

This is a red car. There are windows behind the car. The car is a Ford Mustang GT. The car stands (is parked) outside. There is a building behind the car. I like this car.

A férfi kint sétál. A férfi nem bent van. A fák között sétál. A férfi van egyedül. A férfi egyedül van. A férfi nem alacsony. Nem sétálok, hanem ülök. Te sétálsz vagy ülsz? Szeretsz sétálni?

The man walks outside. The man is not inside. He walks among/between the trees. The man is alone. The man is not short. I am not walking, but sitting. Do you walk or sit? Do you like to walk?

Resources

• Word order for adverbs of frequency - I was having trouble understanding where adverbs of frequency belong in a sentence and this blog post helped
• UPDATE: Just found this website, and I was able to read this article without much trouble! I’m excited that I’ve finally found something I can read that isn’t too difficult!

Corrections

Thanks to Teleonomix for the corrections to my Hungarian above.

I've been interested in a more self-reliant computer for some time. By that I mean, if you have a laptop from a typical company and the case or screen cracks, it's basically trash. The battery will eventually not hold a charge and usually the battery is specifically for that laptop so eventually they'll stop making them.

I want computer that if something goes, I could fix on my own. There's been some computers coming out lately that fit that.

Probably one of the most popular and closest to what I was interested in was the MNT Reform. This is a custom laptop that's completely open. Other's can use their design files so you could get new parts from third parties possibly. It uses 18650 batteries so replacing those is not an issue.

However, there wasn't much of an ecosystem around the MNT yet in terms of software or hardware. Also the price was too high, but it's understandable for a completely custom laptop.

There's also the idea of a cyberdeck, the computer used by hackers in cyberpunk stories. They come in all shapes, but the typical form is just a keyboard and a screen at the top. I'm not talking about a clam shell like a typical laptop, you just carry around the thing with a keyboard exposed.

That makes a great visual but I wanted a clam shell. There are also simple builds that are just a screen, bluetooth keyboard, and a Raspberry Pi. That's pretty DIY, but I wanted something less slapped together.

That's when I saw the Griz Sextant, and I knew that's what I wanted to build. It's repairable, modifiable, has a good form. And has a retro look which is even better.

The designer put a lot of work in to it, and did a great job. He released the files here.

This page will document my work on my own build. My hope is that I can contribute by recreating the build and see how it is for someone other than the designer to build it.

My first step was ordering everything from the BOM. That was straight forward, there were links for just what to buy with some alternatives. I noticed the switches for the keyboard were no longer available, but Amazon showed I could buy the same switches but in a different quantity so I went with that.

This will also be my first build of a custom keyboard, something else I've been interested in doing. That's why I'm going with the same switches, I'm not familiar enough yet with building keyboards to know that a different switch would work.

I also saw some of the HDMI cable parts weren't available at Adafruit. I'd prefer to support them over Amazon, but I want my parts soon. Also, it looked like I'd have to build the HDMI cable from seperate parts if I ordered from Adafruit which I don't see why I'd want to build my own cable.

This is also a good excuse to get my 3D printer operable again. I'm using a PowerSpec 3D printer from Micro Center. I found out you can get a really good deal on this if you buy refurbished. There would be nothing wrong with the printer itself, but it seems many people returned these so Micro Center would sell them refurbished at a big discount.

It turns out all that was wrong was the instructions missed a step to plug in a cable. That cable was labeled something like "D" and the only port it dangled right next to was also labeled "D". Some people missed that and returned it.

The biggest pain of this printer is calibrating the bed so it's the right length away from the print head around the entire bed. It's at least 8x8 which the Griz parts are for so it should work, but being that big it's even more important that it's calibrated correctly.

It will be annoying for all the parts to be delivered at different times, but I'll have printing the parts to keep me busy for the time being.

I did recieve my screws already which was impressively fast. But then I realized McMaster has a distribution center in Aurora, Ohio, about 30 minutes away from me.

I'll update this post with more as I go along. I'll post the updates to the RSS feed.

I'm not exactly sure how I first heard about the Vortex86 CPUs, I think it was either when seeing the demonstration video on KolibriOS project site showcasing the system running on a DMP EBOX machine, or when skimming NetBSD's identcpu.c code. Or did the discovery of the machine prompted me to check if the CPU would be correctly probed by the NetBSD's kernel?

For those interested, Wikipedia has an article retracing the history of the Vortex86 from its birth at Rise to our days.

Several DMP EBOX machines are available for sale at various specialized vendors, but new devices cost several hundreds of dollars which is prohibitive for such low spec systems. However, I was recently able to acquire a boxed older model on a local auction site for about $25: the EBOX 3300A-H, with a 1GHz CPU and 256MB of RAM, no less.

As I already mentioned, those machines are quite slow but they still do have a few things going for them:

• They are totally fanless, and the metal case finish is quite nice
• They are very low-power x86 embedded devices, and still being produced

I used a power meter to do measurements, and an idle system consumes 5.3W. Power consumption peaked at 6.4W when running the OpenSSL speed benchmark.

There is space for a 2.5" hard drive in the enclosure, but I don't have any IDE drives anymore so I opted to use old CompactFlash cards I had laying around. As a side note, it's actually exquisite to use those cards like glorified floppies :-)

For this post, I used a 1GB CompactFlash card and selected a minimal installation in sysinst.

The installed system takes 212M:

Filesystem         Size       Used      Avail %Cap Mounted on
/dev/wd0a          919M       212M       661M  24% /
kernfs             1.0K       1.0K         0B 100% /kern
ptyfs              1.0K       1.0K         0B 100% /dev/pts
procfs             4.0K       4.0K         0B 100% /proc
tmpfs               64M         0B        64M   0% /var/shm

On a freshly booted system, 15 processes are running and 26M of RAM are used:

load averages:  0.01,  0.00,  0.00;               up 0+00:48:26        14:48:28
16 processes: 15 sleeping, 1 on CPU
CPU states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Memory: 26M Act, 6460K Exec, 12M File, 195M Free
Swap: 

  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
    0 root      96    0     0K   26M usbevt     0:01  0.00%  0.00% [system]
  795 root      43    0  6160K 1628K CPU        0:00  0.00%  0.00% top
  555 root      85    0    12M 3472K wait       0:00  0.00%  0.00% login
  630 postfix   85    0    13M 3220K kqueue     0:00  0.00%  0.00% qmgr
  599 postfix   85    0    12M 3172K kqueue     0:00  0.00%  0.00% pickup
  575 root      85    0    13M 2304K kqueue     0:00  0.00%  0.00% master
  196 root      85    0  9780K 1960K kqueue     0:00  0.00%  0.00% syslogd
  583 root      85    0  6788K 1824K wait       0:00  0.00%  0.00% sh
  710 root      85    0  6276K 1448K nanoslp    0:00  0.00%  0.00% cron
  733 root      85    0  6108K 1396K ttyraw     0:00  0.00%  0.00% getty
  730 root      85    0  5720K 1392K ttyraw     0:00  0.00%  0.00% getty
  633 root      85    0  6104K 1388K ttyraw     0:00  0.00%  0.00% getty
  211 root      85    0  7316K 1360K kqueue     0:00  0.00%  0.00% dhcpcd
    1 root      85    0  6600K 1340K wait       0:00  0.00%  0.00% init
  689 root      85    0  5700K 1184K kqueue     0:00  0.00%  0.00% inetd
  402 root      84    0  5920K 1140K kqueue     0:00  0.00%  0.00% powerd

Here is the result of running cat /proc/cpuinfo on this device:

processor	: 0
vendor_id	: Vortex86 SoC
cpu family	: 5
model		: 2
model name	: Vortex86DX
stepping	: 2
cpu MHz		: 1000.05
apicid		: 0
initial apicid	: 0
fdiv_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu tsc cx8 
clflush size	: 0

For the record, OpenSSL speed benchmark results are available here.

System message buffer (dmesg output):

[     1.000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
[     1.000000]     2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
[     1.000000]     2018, 2019, 2020 The NetBSD Foundation, Inc.  All rights reserved.
[     1.000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
[     1.000000]     The Regents of the University of California.  All rights reserved.

[     1.000000] NetBSD 9.2 (GENERIC) #0: Wed May 12 13:15:55 UTC 2021
[     1.000000] 	mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/i386/compile/GENERIC
[     1.000000] total memory = 255 MB
[     1.000000] avail memory = 231 MB
[     1.000000] rnd: seeded with 66 bits
[     1.000000] timecounter: Timecounters tick every 10.000 msec
[     1.000000] Kernelized RAIDframe activated
[     1.000000] running cgd selftest aes-xts-256 aes-xts-512 done
[     1.000000] timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
[     1.000003] Generic PC
[     1.000003] mainbus0 (root)
[     1.000003] Firmware Error (ACPI): A valid RSDP was not found (20190405/tbxfroot-261)
[     1.000003] autoconfiguration error: acpi_probe: failed to initialize tables
[     1.000003] ACPI Error: Could not remove SCI handler (20190405/evmisc-312)
[     1.000003] cpu0 at mainbus0
[     1.000003] cpu0: Vortex86DX, id 0x522
[     1.000003] cpu0: package 0, core 0, smt 0
[     1.000003] pci0 at mainbus0 bus 0: configuration mode 1
[     1.000003] pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
[     1.000003] pchb0 at pci0 dev 0 function 0: vendor 17f3 product 6021 (rev. 0x02)
[     1.000003] vga0 at pci0 dev 3 function 0: vendor 18ca product 0020 (rev. 0x00)
[     1.000003] wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation)
[     1.000003] wsmux1: connecting to wsdisplay0
[     1.000003] drm at vga0 not configured
[     1.000003] rdcpcib0 at pci0 dev 7 function 0: vendor 17f3 product 6031 (rev. 0x02)
[     1.000003] rdcpcib0: watchdog timer configured.
[     1.000003] vte0 at pci0 dev 8 function 0: vendor 17f3 product 6040 (rev. 0x00)
[     1.000003] vte0: Ethernet address 00:1b:eb:22:16:5c
[     1.000003] vte0: interrupting at irq 10
[     1.000003] rdcphy0 at vte0 phy 1: R6040 10/100 media interface, rev. 1
[     1.000003] rdcphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
[     1.000003] ohci0 at pci0 dev 10 function 0: vendor 17f3 product 6060 (rev. 0x12)
[     1.000003] ohci0: interrupting at irq 11
[     1.000003] ohci0: OHCI version 1.0, legacy support
[     1.000003] usb0 at ohci0: USB revision 1.0
[     1.000003] ehci0 at pci0 dev 10 function 1: vendor 17f3 product 6061 (rev. 0x03)
[     1.000003] ehci0: interrupting at irq 11
[     1.000003] ehci0: BIOS has given up ownership
[     1.000003] ehci0: EHCI version 1.0
[     1.000003] ehci0: 1 companion controller, 2 ports: ohci0
[     1.000003] usb1 at ehci0: USB revision 2.0
[     1.000003] ohci1 at pci0 dev 11 function 0: vendor 17f3 product 6060 (rev. 0x12)
[     1.000003] ohci1: interrupting at irq 11
[     1.000003] ohci1: OHCI version 1.0, legacy support
[     1.000003] usb2 at ohci1: USB revision 1.0
[     1.000003] ehci1 at pci0 dev 11 function 1: vendor 17f3 product 6061 (rev. 0x03)
[     1.000003] ehci1: interrupting at irq 11
[     1.000003] ehci1: BIOS has given up ownership
[     1.000003] ehci1: EHCI version 1.0
[     1.000003] ehci1: 1 companion controller, 2 ports: ohci1
[     1.000003] usb3 at ehci1: USB revision 2.0
[     1.000003] rdcide0 at pci0 dev 12 function 0: RDC R1011 IDE controller (rev. 0x01)
[     1.000003] rdcide0: bus-master DMA support present
[     1.000003] rdcide0: primary channel configured to compatibility mode
[     1.000003] rdcide0: primary channel interrupting at irq 14
[     1.000003] atabus0 at rdcide0 channel 0
[     1.000003] rdcide0: secondary channel configured to compatibility mode
[     1.000003] rdcide0: secondary channel interrupting at irq 15
[     1.000003] atabus1 at rdcide0 channel 1
[     1.000003] isa0 at rdcpcib0
[     1.000003] pckbc0 at isa0 port 0x60-0x64
[     1.000003] attimer0 at isa0 port 0x40-0x43
[     1.000003] pcppi0 at isa0 port 0x61
[     1.000003] midi0 at pcppi0: PC speaker
[     1.000003] sysbeep0 at pcppi0
[     1.000003] isapnp0 at isa0 port 0x279
[     1.000003] attimer0: attached to pcppi0
[     1.000003] isapnp0: no ISA Plug 'n Play devices found
[     1.000003] timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
[     1.064509] uhub0 at usb1: NetBSD (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
[     1.064509] uhub0: 2 ports with 2 removable, self powered
[     1.064509] uhub1 at usb2: NetBSD (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
[     1.064509] uhub1: 2 ports with 2 removable, self powered
[     1.064509] uhub2 at usb3: NetBSD (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
[     1.064509] uhub2: 2 ports with 2 removable, self powered
[     1.064509] uhub3 at usb0: NetBSD (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
[     1.064509] uhub3: 2 ports with 2 removable, self powered
[     1.064509] IPsec: Initialized Security Association Processing.
[     3.914550] uaudio0 at uhub3 port 2 configuration 1 interface 0
[     3.914550] uaudio0: vendor 0d8c (0xd8c) C-Media USB Audio Device (0x08), rev 1.10/1.00, addr 2
[     3.934546] uaudio0: audio rev 1.00
[     3.934546] audio0 at uaudio0: playback, capture, full duplex, independent
[     3.934546] audio0: slinear_le:16 2ch 48000Hz, blk 11520 bytes (60ms) for playback
[     3.934546] audio0: slinear_le:16 1ch 48000Hz, blk 6000 bytes (62.5ms) for recording
[     3.934546] uhidev0 at uhub3 port 2 configuration 1 interface 3
[     3.934546] uhidev0: vendor 0d8c (0xd8c) C-Media USB Audio Device (0x08), rev 1.10/1.00, addr 2, iclass 3/0
[     3.944550] uhid0 at uhidev0: input=4, output=4, feature=0
[     4.054550] wd0 at atabus1 drive 0
[     4.054550] wd0: <Hitachi XX.V.3.5.0.0>
[     4.054550] wd0: drive supports 1-sector PIO transfers, LBA addressing
[     4.054550] wd0: 977 MB, 1987 cyl, 16 head, 63 sec, 512 bytes/sect x 2002896 sectors
[     4.064551] wd0: 32-bit data port
[     4.064551] wd0: drive supports PIO mode 4
[     4.064551] wd0(rdcide0:1:0): using PIO mode 4
[     4.084559] WARNING: 1 error while detecting hardware; check system log.
[     4.084559] boot device: wd0
[     4.084559] root on wd0a dumps on wd0b
[     4.094550] root file system type: ffs
[     4.094550] kern.module.path=/stand/i386/9.2/modules
[    20.764808] wsdisplay0: screen 1 added (80x25, vt100 emulation)
[    20.784809] wsdisplay0: screen 2 added (80x25, vt100 emulation)
[    20.794810] wsdisplay0: screen 3 added (80x25, vt100 emulation)
[    20.804812] wsdisplay0: screen 4 added (80x25, vt100 emulation)

A classic Australian movie: A nightmare chase through hell in a never-ending, unrequited daisy chain of desire… .

What is time? How do we measure a life? How many things did you get done today? What are you working on tomorrow!? Can you come hang out on Sunday!!?

Recently, and also always, I’ve been thinking about how to manage a list of todos. During an internal knowledge-sharing session at work I was prompted to reflect on all the different schemes I’ve tried:

• Random pieces of paper
• Notepad
• Windows sticky notes
• Tiddlywiki
• Mediawiki
• Jira
• phpBB and other misc BBS software
• Vim
• Vim quickfix window
• A terrible custom python script
• Vimwiki
• Text files
• Org-Mode (I gave up almost immediately)
• Emails to myself
• A private mailing list I made up
• Pomodoro
• Github issues
• Trello
• Notion
• Asana
• Readme files
• Taskwarrior
• Bullet journal (for <4 minutes)
• Physical notebooks
• Fancy notebooks
• Cheap notebooks
• Physical ‘kanban’ cards
• A whiteboard
• My head
• Nothing
• Random pieces of paper

Right now I’m back to a whiteboard (by luck I happen to have one behind me where I work), and ‘todo’ notes in code + github issues.

My friend Ruth told me something that perhaps is very obvious but was also surprisingly calming to me: You’ll die with items on your todo list. So don’t worry about it!

I think there’s a few ways to think about this:

1. The optimisic view: Amazing that I am lucky enough to have things I intend to do and look forward to!
2. The pessimistic view: Sad that I had goals that I was never able to achieve!
3. The Buddhist view: Your big mistake is having desires at all!

I want to live around option (3) and (1) at the moment, but of course option (2) is a very very strong driving force; and often necessary (for example, at work!); the only time I was brave enough to enact option (3) at a work setting was when I ran my own business!

Another interesting thing that came up as part of the work discussion was that in some ways what you want to do depends on your mood. No doubt! So … maybe mood can be a parameter?!

I think, secretly, for the rest of my life, I will believe that there is a True answer out there; an ultimate organisation scheme; the Four-Fold path to Life Efficiency: Right Editing, Right Organisation, Right Workflow and Right History.

But I also think that I should never think about this again. But on the other hand, I was inspired at the talk today; so I might give task-warrior another go.

But, you didn’t come here for that! Maybe you came here for advice? So, here it is (it’s not even my advice! Gala claims it’s her idea which is probably true):

If you can, don’t even write the thing down; just do it immediately, or, alternatively, convince yourself you don’t need to think about it now.

Maybe it’s part of my personality, but this works surprisingly well for me. Especially if it’s something that I’m working on with someone else; instead of delaying my part, I can just try and make as much progress as I can immediately, and send it back to the other person! Another way of saying this: I can offset storing my todo list to you!

I find this idea to be quite operaitonal; I often find myself using this rule to do something sooner rather than later.

Here’s one thing I know about myself: I can’t be trusted with lists. I get a bit obsessed with organising them. So I’m trying to avoid them these days.

@redvers has published an XML library for pony.

May 2020 | May 2019 | May 2018 | May 2017 | May 2016 | May 2015 | May 2014 | May 2013 | May 2012 | May 2011 | May 2010 | May 2009 | May 2008

Regular readers know that I am always on the lookout for software engineering data. One search technique is to feed a ‘magic’ phrase into a search engine, this can locate data hiding in plain sight. This week the magic phrase: “record of pomodoros” returned pages discussing two collections of daily Pomodoros worked, each over a year, plus several possible collections, i.e., not explicitly stated. My email requests for data have so far returned one of the collections, kindly made available by Alex Altair, and this post discusses Alex’s data (I have not discussed the data with Alex, who I’m hoping won’t laugh too loud at the conclusions I have reached).

Before analyzing data I always make predictions about what I expect to see. I know from the email containing the data that it consisted of two columns: date and Pomodoro’s worked, i.e., no record of task names. The first two predictions for this data were the two most common patterns seen in estimation data, i.e., use of round numbers, and a weekend-effect (most people don’t work during the weekend and the autocorrelation of the daily counts contain peaks at lags of 6 and 7). The third prediction was that over time the daily total Pomodoro counts would refine into counts for each of the daily tasks (I had looked at the first few lines of the data and seen totals for the daily Pomodoros worked.

The Renzo Pomodoro dataset is my only previous experience analysing Pomodoro data. Renzo created a list of tasks for the day, estimated the number of Pomodoros for each task would take, and recorded how many it actually took. For comparison, the SiP effort estimation dataset estimates software engineering tasks in hours.

Alex uses Pomodoros as a means of focusing his attention on the work to be done, and the recorded data is a measure of daily Pomodoro work done.

I quickly discovered that all my predictions were wrong, i.e., no obvious peaks showing use of round numbers, no weekend effect, and always daily totals. Ho-hum.

The round number effect is very prominent in estimates, but is not always visible in actuals; unless people are aiming to meet targets or following Parkinson’s law.

How many days had one Pomodoro worked, how many two Pomodoro, etc? The plot below shows the number of days for which a given number of Pomodoros were worked (the number of days with zero Pomodoros is not shown); note the axis are log scaled. The blue points are for all days in 2020, and the green points are all days in 2020+178 days of 2021. The red lines are two sets of two fitted power laws (code+data):

Why the sudden change of behavior after seven Pomodoro? Given a Pomodoro of 25-minutes (Alex says he often used this), seven of them is just under 3-hours, say half a day. Perhaps Alex works half a day, for every day of the week.

Why the change of behavior since the end of 2020 (i.e., exponent of left line changes from 0.3 to -0.1, exponent of right line is -3.0 in both cases)? Perhaps Alex is trying out another technique. The initial upward trend is consistent with the Renzo Pomodoro dataset.

The daily average Pomodoros worked is unchanged at around 5.6. The following plot shows daily Pomodoros worked over the 543 days, red line is a fitted loess model.

The weekend effect might not be present, but there is a strong correlation between adjacent days (code+data). The best fitting ARIMA model gives the equation: , where: is the Pomodoros worked on day , Pomodoros worked on the previous day, is white noise (e.g., a Normal distribution) with a zero mean and a standard deviation of 4 (in this case) on day , and the previous day’s noise (see section 11.10 of my book for technical time series details).

This model is saying that the number of Pomodoros worked today is strongly influenced by yesterday’s Pomodoro worked, modulated by a large random component that could be large enough to wipe out the previous days influence. Is this likely to be news to Alex, or to anybody looking at the plot of Pomodoros over time? Probably not.

For me, the purpose of data analysis is to find patterns of behavior that are of use to those involved in the processes that generated the data (for many academics, at least in software engineering, the purpose appears to be to find patterns that can be used to publish papers, and given enough searching, it is always possible to find patterns in data). What patterns of behavior might Alex be interested in?

Does more Pomodoro work get done at the start of the week, compared to the end of the week? The following heatmap is based on the number of week days on which a given number of Pomodoros were worked. The redder the region, the more likely that value is likely to occur (code+data):

There are certainly more days near the end of the week having little or no Pomodoro work, and the high Pomodoro work days appear to be nearer the start of the week. I need to find a statistical technique that quantifies these observations.

I think that the middle plot is the most generally useful, it illustrates how variable the work done during a day can be.

Is Alex’s Pomodoro work typical or unusual? We will have to wait for a lot more data before that question can be addressed.

If you are a Pomodoro user, and have ideas for possible patterns in the data, please let me know.

As always, pointers to more data, Pomodoro or otherwise, most welcome.

It has now been 2 weeks since I started learning Hungarian. In the past week I’ve used DuoLingo daily, attended a group Zoom lesson, and I attended a private Hungarian lesson last night. I’ve also done independent research into different aspects of the language that interest me, as they come up in my more structured learning. Some of the initial learning fever has worn off and I can feel myself starting to fall into a sustainable pace. I’m familiar with this feeling since it used to happen back when learning programming languages was new for me: I’d start off knowing nothing and feeling simultaneously overwhelmed and awestruck by the breadth and depth of information I was consuming. This caused (and still causes) my initial learning period to be very intense while I try to establish basic level understanding in a programming language’s syntax or a spoken language’s basic grammar and vocabulary.

My learning experience so far

For me, learning something brand new feels metaphorically like being hit by a giant wave of information and having no way to consume it. I then start to understand bits and pieces of it and eventually get to a point where I start to feel comfortable with some very small part of the language. For programming languages this was being able to write and understand my first chunk of meaningful code, usually a small program that responded to user input and maybe had a couple if statements for different control flow. For Hungarian that point was when I could start saying basic present tense sentences with a subject, verb, and object like János egy almát lát (János sees an apple). I see this small nucleus of understanding like a beachhead: it’s a starting point that I can use to explore in many different directions.

For instance, I can start exploring grammatical number using this basic framework. I know how to say “János sees an apple”, now I can also say János almákat lát (János sees apples). I can also try some different adjectives to describe the apple that János sees, for instance János egy nagy almát lát (János sees a large apple). I can play around with word order, saying János lát egy almát instead of János egy almát lát.

Let’s try out some new verbs in our kicsi (little) sentence. János egy almát vár (János waits for an apple). Kind of a nonsensical sentence, but it’s grammatically correct. We can also say János talál egy almát (János finds an apple).

These little forays into different aspects of the language while building on things I already know help me to steadily progress without getting overwhelmed. They also give me tangible goals that keep me from being discouraged by how much there is to learn and wanting to know all of it right now. If I focus on the vague concept of wanting to learn Hungarian, it immediately becomes overwhelming because mastering a language takes years. Instead I focus on figuring out how to say “János found an apple” or “János will find an apple”, and so on. These small goals make progression in the language seem relatively easy and rewarding.

A brief ramble about the difficulty of the Hungarian language

I’m writing this blog post as an exercise for myself, since teaching something is one of the best ways to learn, but also as a way to document my process of learning Hungarian in the hopes that it can help or encourage someone else. The Hungarian language specifically can be discouraging to start learning since Hungarian has a reputation for being impossible for English speakers to learn, or at least extremely difficult. And maybe it is difficult. Aside from taking a year of Latin in high school I have nothing to compare it to. Perhaps if I had learned an easier language first, Hungarian would seem more overwhelming, who knows. But what I do know is that this language is not impossible.

Something I find interesting is that language learning difficulty goes both ways. If Hungarian is difficult for English speakers, then English is also difficult for Hungarian speakers. My grandmother learned English in her teens, and spoke perfect English for the rest of her life with only the faintest of an accent. At this very moment there are thousands, maybe millions of Chinese speakers who are learning English and will succeed.

Why do English speakers get so overwhelmed by the difficulty of a language? Maybe it’s because we don’t have to learn another language. Maybe it has to do with the American notion of forcing our culture on others instead of learning and assimilating other cultures. I don’t really know. I do know I’m waxing political though, so I’ll end this ramble here.

Plans for my next week of learning

In the next week, I am hoping to be able to attend another group lesson, and of course will be continuing to use DuoLingo. My goal for the next week is to expand my Hungarian vocabulary and learn definite conjugations for verbs.

What is a definite conjugation? In Hungarian, different endings are used for verbs depending on whether the sentence is talking about something specific or something indefinite. So in the sentence a nő egy almát lát (the woman sees an apple), the object (egy almát or an apple) is indefinite. We aren’t talking about a specific apple, but an indefinite one.

On the other hand, the sentence a nő az almát látja (the woman sees the apple) has a definite object. We are talking about a specific apple this time. Because of this, the verb lát gets the ending -ja. You may say, “well Mark, it seems like you understand indefinite and definite verbs already”, and you’d be wrong, because it just so happens that I only understand the concept and I don’t know the endings for the different conjugations of verbs. I know the definite conjugation for “he/she/it sees” is látja, but I don’t know what the definite conjugation for “I see”, “we see”, etc is. And that will be one of my focuses for the coming week.

Resources

• Eszter Edl at FluentBox - Eszter is an excellent language teacher, and if you are trying to learn Hungarian I highly recommend that you attend one of her group lessons or set up private lessons with her.
• Hungarian: An Essential Grammar by Carol Rounds - Excellent Hungarian grammar book that I’ve started to read

The lessons from the history of the two companies are not at all obvious to me.

Read full article!

As mentioned in my previous post about playing with DJGPP and GCC 10 on DOS, I have been redoing my small character and color cycling effect in text mode. The original version in JavaScript can be seen here.

To understand why we can't access video memory directly and need to use the DPMI service to create a selector to access the required real-mode segment address, please refer to section 18.4 of the DJGPP FAQ.

The effect can be downloaded here. Sadly, the resulting binary is quite large (104K stripped!), and it requires CWSDPMI to run.

Here is the code:

#include <conio.h>
#include <dpmi.h>
#include <go32.h>
#include <pc.h>

void
wait_vbl()
{
	while(inp(0x3da) & 0x08);
	while(!(inp(0x3da) & 0x08));
}

int
main()
{
	short video = __dpmi_segment_to_descriptor(0xb800);
	unsigned char buffer[4000];

	int character = 0, color = 0;

	/* Define character and color arrays */
	char characters[] = { 0x5c, 0x7c, 0x2f, 0x2d };
	char colors[] = { 0xf, 0xb, 0x9, 0x1, 0x9, 0xb, 0xf };

	while (!kbhit()) {
		for (size_t offset = 0; offset < 4000; offset += 2) {
			/* Write character and color data */
			buffer[offset] = characters[character];
			buffer[offset + 1] = colors[color];

			/* Increment the color index */
			color = color + 1 < sizeof(colors) ? color + 1 : 0;
		}

		/* Increment the character index */
		character = character + 1 < sizeof(characters) ?
		    character + 1 : 0;

		/* Copy the buffer into text mode video memory */
		movedata(_my_ds(), (unsigned)buffer, video, 0, 4000);

		/* Wait for VBL */
		wait_vbl();
		wait_vbl();
		wait_vbl();
		wait_vbl();
		wait_vbl();
	}
}

Mantel concludes the trilogy about Thomas Cromwell as he reaches the pinnacle of power and then plummets precipitously.

As I’ve mentioned before about this trilogy, this is great historical fiction. Mantel deserves every ounce of praise for these.

Version 0.41.1 of ponylang/ponyc has been released!

Metastability and Distributed Systems

What if computer science had different parents?

There's no more time-honored way to get things working again, from toasters to global-scale distributed systems, than turning them off and on again. The reasons that works so well are varied, but one reason is especially important for the developers and operators of distributed systems: metastability.

I'll let the authors of Metastable Failures in Distributed Systems define what that means:

Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is removed.

What they're identifying here is a kind of stable down state, where the system is stable but not doing useful work, even though it's only being offered a load that it successfully handled sometime in the past.

One classic version of this problem involves queues. A system is ticking along nicely, and something happens. Could be a short failure, a spike of load, a deployment, or one of many other things. This causes queues to back up in the system, causing an increase in latency. That increased latency causes clients to time out before the system responds to them. Clients continue to send work, and the system continues to complete that work. Throughput is great. None of the work is useful, though, because clients aren't waiting for the results, so goodput is zero. The system is mostly stable in this state, and without an external kick, could continue going along that way indefinitely. Up, but down. Working, but broken.

In Metastable Failures in Distributed Systems, Bronson et al correctly observe that these types of failure modes are well-known¹ to the builders of large-scale systems:

By reviewing experiences from a decade of operating hyperscale distributed systems, we identify a class of failures that can disrupt them, even when there are no hardware failures, configuration errors, or software bugs. These metastable failures have caused widespread outages at large internet companies, lasting from minutes to hours. Paradoxically, the root cause of these failures is often features that improve the efficiency or reliability of the system.

The paper identifies a list of other triggers for these types of metastable failures, including retries, caching, slow error handling paths and emergent properties of load-balancing algorithms. That's a good list, but just scratching the surface of all of the possible causes of these 'down but up' states in distributed systems.

Is there a cure?

The disease is a serious one, but perhaps with the right techniques we can build systems that don't have these metastable states. Bronson et al propose approaching that in several ways:

We consider the root cause of a metastable failure to be the sustaining feedback loop, rather than the trigger. There are many triggers that can lead to the same failure state, so addressing the sustaining effect is much more likely to prevent future outages.

This isn't a controversial point, but is an important one: focusing on just fixing the triggering causes of issues causes us to fail to prevent similar issues with slightly different causes in future.

The rest of their proposed solutions are more debatable. Changing policy during overload introduces modal behavior that can be hard to reason about (and modes are bad). Prioritization and fairness are good if you can get them, but many systems can't, either because their workloads are complex interdependent graphs without clear priority order, or because the priority order is unpalatable to the business. Fast error paths and outlier hygiene are good, in an eat your broccoli kind of way.

The other two they cover that really resonate with me are organizational incentives and autoscaling. Autoscaling, again, is a good if you can get it kind of thing, but most applications can get it by building on top of modern cloud systems. Maybe even get it for free by building on serverless². On organizational incentives:

Optimizations that apply only to the common case exacerbate feedback loops because they lead to the system being operated at a larger multiple of the threshold between stable and vulnerable states.

Yes, precisely. This is a very important dynamic to understand, and design an organization around defeating⁴. One great example of this behavior is retries. If you're only looking at your day-to-day error rate metric, you can be lead to believe that adding more retries makes systems better because it makes the error rate go down. However, the same change can make systems more vulnerable, by converting small outages into sudden (and metastable) periods of internal retry storms. Your weekly loop where you look at your metrics and think about how to improve things may be making things worse.

Where do we go?

Knowing this problem exists, and having some tactics to fix certain versions of it, is useful. Even more useful would be to design systems that are fundamentally stable.

Can you predict the next one of these metastable failures, rather than explain the last one?

The paper lays out a couple of strategies here. The most useful one is a characteristic metric that gives insight into the state of the feedback loop that's holding the system down. This is the start of a line of thinking that treats large-scale distributed systems as control systems, and allows us to start applying the mathematical techniques of control theory and dynamical systems theory.

I believe that many of the difficulties we have in this area come from where computing grew up. Algorithms, data structures, discrete math, finite state machines, and the other core parts of the CS curriculum are only one possible intellectual and theoretical foundation for computing. It's interesting to think about what would be different in the way we teach CS, and the way we design and build systems, if we had instead chosen the mathematics of control systems and dynamical systems as the foundation. Some things would likely be harder. Others, like avoiding building metastable distributed systems, would likely be significantly easier.

In lieu of a time-travel-based rearrangement of the fundamentals of computing, I'm excited to see more attention being paid to this problem, and to possible solutions. We've made a lot of progress in this space over the last few decades, but there's a lot more research and work to be done.

Overall, Metastable Failures in Distributed Systems is an important part of a conversation that doesn't get nearly the attention it deserves in the academic or industrial literature. If I have any criticism, it's that the paper overstates its case for novelty. These kinds of issues are well known in the world of control systems, in health care, in operations research, and other fields. The organizational insights echo those of folks like Jens Rasmussen³. But it's a HotOS paper, and this sure is a hot topic, so I won't hold the lack of a rigorous investigation of the background against the authors.

If you build, operate, or research large-scale distributed systems, you should read this paper. There's also a good summary on Aleksey Charapko's blog.

Footnotes

1. For example, I wrote about part of this problem in Some risks of coordinating only sometimes, and talked about it at HPTS'19, although framed the issue as a bistability rather than metastability. Part of the thinking in that talk came from my own experience, and discussions of the topic in books like designing distributed control systems. It's a topic we've spent a lot of energy on at AWS over the last decade, although typically using different words.
2. Of course I'm heavily biased, but the big advantage of serverless is that most applications are small relative to the larger serverless systems they run on, and so have a lot of headroom to deal with sudden changes in efficiency. In practice, I think that building on higher-level abstractions is going to be the best way for most people to avoid problems like those described in the paper, most of the time.
3. Specifically the discussion of the "error margin" in Risk Management in a Dynamic Society, and how economic and labor forces push systems closer to the boundary of acceptable performance.
4. An organization and an economy. As we saw with supply-side shortages of things like masks early in the Covid pandemic, real-world systems are optimized for little excess capacity too, and optimized for the happy case.

Why don’t compiler projects for the R language make it into production use? The few that have been written have remained individual experimental products, e.g., RLLVMCompile.

Most popular languages attract many compiler implementations. I’m not saying that any of these implementations have more than a handful of users, that they implement the full language (a full implementation is not common), or that they fulfil any need other than their implementers desire to build something.

A commonly heard reason for the lack of production R compilers is that it is not worth the time and effort, because most of an R program’s time is spent in the library code which is written in a compiled language (e.g., C or Fortran). The fact that it is probably not worth the time and effort has not stopped people writing compilers for other languages, but then I think that the kind of people who use R tend not to be the kind of people who want to spend their time writing compilers. On the whole, they are the kind of people who are into statistics and data analysis.

Is it true that that most R programs spend most of their time executing library code? It’s certainly true for me. But I have noticed that a lot of the library functions executed by my code are written in R. Also, if somebody uses R for all their programming needs (it might be the only language they know), then their code might not be heavily library dependent.

I was surprised to read about Tierney’s byte code compiler, because his implementation is how I thought the R-core’s existing implementation worked (it does now). The internals of R is based on 1980s textbook functional techniques, and like many book implementations of the day, performance is dependent on the escape hatch of compiled code. R’s implementers wisely spent their time addressing user concerns, which revolved around statistics and visual presentation, i.e., not internal implementation technicalities.

Building an R compiler is easy, the much harder and time-consuming part is the runtime system.

Threaded code is a quick and simple approach to compiler implementation. R source gets mapped to a sequence of C function calls, with these functions proving a wrapper to library functions implementing the appropriate basic functionality, e.g., add two vectors. This approach has been the subject of at least one Master’s thesis. Thesis implementations rarely reach production use because those involved significantly underestimate the work that remains to be done, which is usually a lot more than the original implementation.

A simple threaded code approach provides a base for subsequent optimization, with the base having a similar performance to an interpreter. Optimizing requires figuring out details of the operations performed and replacing generic function calls with ones designed to be fast for specific cases, or even better replacing calls with inline code, e.g., adding short vectors of integers. There is a lot of existing work for scripting languages and a few PhD thesis researching R (e.g., Wang). The key technique is static analysis of R source.

Jan Vitek is running what appears to be the most active R compiler research group, at the moment e.g., the Ř project. Research can be good for uncovering language usage and trying out different techniques, but it is not intended to produce industry strength code. Lots of the fancy optimizations in early versions of the gcc C compiler started life as a PhD thesis, with the respective individual sometimes going on to spend a few years creating a production quality version for the released compiler.

The essential ingredient for building a production compiler is persistence. There are an awful lot of details that need to be sorted out (this is why research project code does not directly translate to production code, they ignore ‘minor’ details in order to concentrate on the ‘interesting’ research problem). Is there a small group of people currently beavering away on a production quality compiler for R? If there is, I can understand being discrete, on long-term projects it can be very annoying to have people regularly asking when the software is going to be released.

To have a life, once released, a production compiler needs to attract users, who are often loyal to their current compiler (because they know that their code works for this compiler); there needs to be a substantial benefit to entice people to switch. The benefit of compiling R to machine code, rather than interpreting, is performance. What performance improvement is needed to attract a viable community of users (there is always a tiny subset of users who will pay lots for even small performance improvements)?

My R code is rarely cpu bound, so I am not in the target audience, no matter what the speed-up. I don’t have any insight in the performance problems experienced by the R community, and have no idea whether a factor of two, five, ten or more would be enough.

I was recently amazed to discover that DJGPP was still being maintained, and had very recent versions of GCC and binutils available. I have not been doing any C programming on DOS in a very long time now, so I think the timing is right.

There is an installation program with an interface very similar to the good old Turbo Vision, which could be helpful in case one wants to install the full environment.

I'm only interested in using the C frontend for now, so I will do a manual installation.

We need the following components:

djdev205.zip    B  2,509,574  2015-10-18   DJGPP development kit 2.05
bnu2351b.zip    B  6,230,009  2021-01-16   GNU Binutils 2.35.1 binaries for DJGPP
gcc1030b.zip    B 42,027,946  2021-04-18   GNU GCC C compiler 10.3.0 for DJGPP V2
csdpmi7b.zip    B     71,339  2010-01-29   CS's DPMI Provider r7 Binaries

The development environment can be bootstrapped as follows:

mkdir -p ~/dos/djgpp
cd ~/dos/djgpp
wget https://www.delorie.com/pub/djgpp/current/v2/djdev205.zip
wget https://www.delorie.com/pub/djgpp/current/v2gnu/bnu2351b.zip
wget https://www.delorie.com/pub/djgpp/current/v2gnu/gcc1030b.zip
wget https://www.delorie.com/pub/djgpp/current/v2misc/csdpmi7b.zip
unzip djdev205.zip
unzip bnu2351b.zip
unzip gcc1030b.zip
unzip csdpmi7b.zip

When using FreeDOS, we need to add the following in AUTOEXEC.BAT:

set DJGPP=c:\djgpp\djgpp.env
set PATH=c:\djgpp\bin;%PATH%

Alternatively, when using DOSBox instead, we need the following in dosbox.conf:

[autoexec]
mount c ~/dos
path=c:\djgpp\bin
set DJGPP=c:\djgpp\djgpp.env
c:

Once we are done installing, this gives us GCC 10.3.0 and ld 2.35.1:

C:\>gcc --version
gcc.exe (GCC) 10.3.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

C:\>ld --version
GNU ld (GNU Binutils) 2.35.1
Copyright (C) 2020 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

To verify things are working properly, let's create a simple test program:

#include <stdio.h>

int
main()
{
	puts("Hello World!");

	return 0;
}

We then build and run it:

C:\>gcc hello.c -o hello

C:\>hello
Hello World!

Here is the output of running file on the executable:

HELLO.EXE: MS-DOS executable, COFF for MS-DOS, DJGPP go32 DOS extender

Let's build only an object file:

C:\>gcc -c hello.c

We can run nm on it to list symbols:

C:\>nm hello.o
00000000 b .bss
00000000 N .comment
00000000 d .data
00000000 t .text
0000000d T _main
         U _puts

And run objdump to display its content:

C:\>objdump -s hello.o

HELLO.o:     file format coff-go32

Contents of section .text:
 0000 48656c6c 6f20576f 726c6421 008d4c24  Hello World!..L$
 0010 0483e4f0 ff71fc55 89e55183 ec0483ec  .....q.U..Q.....
 0020 0c680000 0000e8d5 ffffff83 c410b800  .h..............
 0030 0000008b 4dfc89ec 5d8d61fc c3909090  ....M...].a.....
Contents of section .comment:
 0000 4743433a 2028474e 55292031 302e332e  GCC: (GNU) 10.3.
 0010 30000000                             0...            

We can also use objdump to disassemble the object file:

C:\>objdump -d hello.o

HELLO.o:     file format coff-go32

Disassembly of section .text:

00000000 <.text>:
   0:	48                      dec    %eax
   1:	65 6c                   gs insb (%dx),%es:(%edi)
   3:	6c                      insb   (%dx),%es:(%edi)
   4:	6f                      outsl  %ds:(%esi),(%dx)
   5:	20 57 6f                and    %dl,0x6f(%edi)
   8:	72 6c                   jb     76 <_main+0x69>
   a:	64 21 00                and    %eax,%fs:(%eax)

0000000d <_main>:
   d:	8d 4c 24 04             lea    0x4(%esp),%ecx
  11:	83 e4 f0                and    $0xfffffff0,%esp
  14:	ff 71 fc                pushl  -0x4(%ecx)
  17:	55                      push   %ebp
  18:	89 e5                   mov    %esp,%ebp
  1a:	51                      push   %ecx
  1b:	83 ec 04                sub    $0x4,%esp
  1e:	83 ec 0c                sub    $0xc,%esp
  21:	68 00 00 00 00          push   $0x0
  26:	e8 d5 ff ff ff          call   0 <.text>
  2b:	83 c4 10                add    $0x10,%esp
  2e:	b8 00 00 00 00          mov    $0x0,%eax
  33:	8b 4d fc                mov    -0x4(%ebp),%ecx
  36:	89 ec                   mov    %ebp,%esp
  38:	5d                      pop    %ebp
  39:	8d 61 fc                lea    -0x4(%ecx),%esp
  3c:	c3                      ret
  3d:	90                      nop
  3e:	90                      nop
  3f:	90                      nop

Besides the C frontend, the Ada, C++, Fortran, Objective C, and Pascal frontends are also available. While I used DOSBox to test the development environment and prepare this article, I've since moved to FreeDOS which I'm running natively on my ASUS Eee PC. I'm currently having a lot of fun redoing some small character and color cycling effects in text mode.

It’s been seven months since the last update article in October 2020. We’re still in a pandemic and living a more self-reliant, distanced, and introspective life. I’ve personally taken the stance of slowing down, to use this intermittent time to my advantage for personal growth. Slow living and the enjoyment of little things.

Let’s get to what I’ve been up to.

Psychology, Philosophy & Books

Language: culture
Explanation: The past few months I’ve been interesting in broadening my horizon, the so-called cognitive diversity. To do that I’ve tried to learn about other cultures, through translation tools, language, traveling by reading and viewing documentaries, or even watching “ancient” films.

One of the excellent series I’ve watched is called “Les Routes de L’Impossible”. It follows truck drivers from around the world as they move their load from one place to another. Transportation of goods, moving, is a great way to get acquainted with countries we didn’t know about.

Apart from the countless documentaries, I’ve also started a new hobby which consists of watching movies from the 1920-1930s era. They offer a crack through time, a telescope effect over new ways of thinking. It’s fascinating from an artistic point of view, as the medium was still relatively new, people were exploring the opportunities about what could be done with it, not bounded by templates, tropes, and budgets.
Arguably, I’ve found these movies more entertaining, perplexing, emotional, and enjoyable than the current ones, especially compared to Hollywood movies. They have more depth, an emphasis on the story-telling and not quick and dirty effects or cliché scenarios.

I’ve written some reviews of films I’ve watched on my gopher:

gopher://g.nixers.net/0/~vnm/classic_movies

In case you don’t have a gopher client, here’s a proxy that allows browsing gopher over the web. I am also considering adding a page on the blog with the same content.

When it comes to books, I haven’t read that many new books that were not technical.

• The Five Stages of Collapse: Survivors’ Toolkit by Dmitry Orlov
  This was more of a long rant on occidental cultures and economies than an actual book about collapse. It has many tidbits about indigenous groups and different ways of living, which was interesting.
• Das Kapital by Karl Marx
  A hard read, but worth reading. It splits everything that makes up what a “capital” really is, how it cannot disappear but inherent to transactions. Then defines “capitalist”, “capitalism”, and the concept of “surplus value”. The book then goes on a tirade about worker’s rights, especially in the textile industry with the 1847 Factories Act in the United Kingdom. It gives a novel perspective on things we take for granted.
• Waiting For Godot by Samuel Beckett
  An absurd metaphorical play that is instantly captivating. In its simplicity, you cannot but extract more meaning out of the actions of the characters.

Similarly, I’ve taken a liking to dadaist art, post-modernist, meta-modernist, surrealist, and shinkankakuha art.
I’ve watched multiple shorts, films, and visited a couple of online museums. This is worth the mind-opening experience, especially during these times, with friends through a video call.

Ascii Art & Art

Language: ASCII
Explanation: I’ve included, as part of the cultural journey, my ASCII artworks as an occasion to dive into history.

My latest set of pieces are about armours through the ages and geographical regions.
I’ve also updated my Redbubble account with them. Check it out!

With that, I’ve learned many of the historical biases, especial regarding the old “history is written by the victor”. It affects our perception of certain geographical areas on Earth and how we refer to them.

Learning

Language: brain
Explanation: Between October and December, the UNESCWA, in association with the Lebanese ministry of Labor, due to the dire situation, started a free access program to Coursera for Lebanese that registered on their platform. I did so, and went on a spree of courses. It was a hectic learning experience.
When I was done, I had completed over 20 certificates, both sub-courses, and specialization included.

Here’s some of them:

• University of Michigan — Model Thinking — certificate
• Stanford — Cryptography I — certificate
• University of Colorado Boulder — Effective Communication:Writing, Design, and Presentation — certificate
• BCG — The hidden value – Lean in manufacturing and services — certificate
• HEC Paris — Business Model Innovation — certificate
• HEC Paris — Inspirational Leadership:Leading with Sense — certificate
• University of Colorado — Secure Software Design specialization — certificate
• The University of North Carolina at Chapel Hill — Positive Psychology — certificate
• UCI — Fundamentals of Management — certificate
• UCI — The Art of Negotiation — certificate
• The University of Edinburgh — Philosophy, Science and Religion — certificate 1, certificate 2, certificate 3

As you can see, I went over multiple topics in a range of domains in a short span of time. The courses were all very interesting and added to my general knowledge. I might not use these in my day-to-day, but who would say no to more education and the chance to learn things we’ve always wanted to learn.

Articles & Writings

Language: pen
Explanation: I’ve written a book!
Ok, now that I’ve said it, I can explain.

I got interested in the subject of internet studies with an emphasis on narrative control. The content I found online was too descriptive of current events and didn’t take distance from them, a lot of them anchored in shock value without reflection.
So I took it upon myself to do my own mini-research, which turned into an article series, then a booklet called “Internet: Medium For Communication, Medium For Narrative Control” that I’ve released here.
This is probably the longest research I’ve done for anything, and I still felt as if it was barely scratching the surface of the topic.

I’ve later found that the booklet is also listed on Google Scholar, and so I’ve registered myself to be listed as the rightful author. See it live.

The cover, seen above, is an AI generated image that has been glitched using a single byte technique that I’ve mentioned in this article.

Apart from the book, I’ve also written 3 new articles since the last update:

• What Does It Take To Resolve A Hostname
  A transcript of a talk I’ve given for the nixers conf 2020.
• A Peek Into The Future Of Distros
  A reflection on the big projects that are moving the Linux ecosystem. Some people took it as a direct attack on their views, or as a representation of what I personally believe to be “a good development”. However, this was merely an observation of facts and not a personal opinion. I’m neither in favor, nor against the changes I’ve listed, I only see them happening.
• Making Sense of The Audio Stack On Unix
  An article that got some of attention about demystifying the audio stack on Unix. It covers a lot of ground in an approachable manner.

Unix, Nixers, Programming

Language: Unix
Explanation: Soon after the last update article, we organized a nixers conference, the very first one. It was a really fun experience that everyone enjoyed. It could have been a bit more of organized, but it was nice to try out.
We had four speakers and each presented some unique talk.

My own talk was about resolving hostnames, which was something I wanted to dive into for a long time, as it is convoluted. I hope I made it simpler.

On nixers, we’ve also started a book club. Phillbush is managing the thread. Here are the books we went over so far.

• Nixers Book Club - Book #1: The UNIX Programming Environment
• Nixers Book Club - Book #2: The UNIX-HATERS Handbook
• Nixers Book Club - Book #3: The Wayland Book
• Nixers Book Club - Book #4: The Art of UNIX Programming

I love the idea of reviewing the books, discussing, and reading them with others. It’s easier to grasp the ideas when you have others that are discovering them at the same time.

Our CTF team went to the Trend Micro finals in December, it was enjoyable but extremely hard. I haven’t looked much into CTFs since then, as it was only something I attempted on the side.

Additionally, I’ve been trying out the Rust language and the Nix package manager.

I’ve finished reading the Rust official book, and updated systemd-manager to support switching between session and service bus (more on dbus). This last one was something that motivated me to learn Rust, to test out my skills.

On the Nix pkg manager side, I’ve read the usual documentation, the nix-pills, and tried out things by myself.
I’ve found nix really nice and practical but I’ve ran into issues when downloading packages as it is extremely slow. The same goes for searching as it seems to be interpreting all expressions, so it’s easier to search online instead. These two are the same issues I’ve had when trying out port systems in the past, but they certainly didn’t have the advantages that nix has, such as rollbacks and reproducibility.

When it comes to audio specifically, after writing that audio stack article, I dig into multiple aspects that I’ve always wanted to.

• I made a small program called aconfdump after someone messaged me on IRC saying that the best explanation of ALSA configuration was in that post, but that they still found it difficult to understand how the configuration files got parsed. So I read a bit of the configurations and tools but couldn’t find how to dump the whole tree. Therefore, I wrote a mini tool to do it. However, the next day we saw that the upcoming ALSA version will have in the alsactl a dump-cfg option that does more or less the same thing.
• I found, when writing that article on audio, that pulseaudio restore database was something most had no idea about, was an unapproachable and uneditable binary blob. So I’ve written an editor and viewer for PulseAudio restoration DBs, pa-resto-edit.
• I started reading and trying out things with PipeWire. I’m planning to write an article specifically on this as I’ve found that the documentation is still extremely sparse. Even the documentation files are barely understood by anyone. I tried getting into the Rust binding to write a native client, however it isn’t ready yet, it’s still a work-in-progress, as Rust doesn’t create binding for inline functions, which constitute most of PipeWire code. I’m following on their IRC channel what is happening. I also had in mind to write a native client, but someone is already working on one called helvum, so I’m tagging along and trying to see how they go about it.
• To better understand PipeWire, I’ve taken a detour to learn Gstreamer. All the container format, codecs, and pipeline handling. I like how well documented and introspective the code and plugins are. Hopefully, PipeWire will soon follow the same route.

Mushrooms

Language: mycelium
Explanation: The online research I posted got referenced by some people discussing Lebanon biodiversity but later removed due to the work-in-progress.

I’ve also found a suillus mediterraneensis in my garden.

Other than this, I’ve implemented the Morel game on Tabletopia to be able to play even at distance. Give it a try here.

Life and other hobbies

Language: life
Explanation: My other hobbies are still there. I’ve tried a lot of cooking styles, different recipes, mastered the art of fried rice, had fun making bread, etc.. However, I haven’t updated my cooking section in a long time.

I’m still on a streak with the brain teaser app Elevate, however I’ve stopped my subscription since the banking situation in Lebanon and foreign currency restrictions have been in place.

My biggest hobby right now is my garden. I’ve been composting and planting different things. From tomatoes, cherry tomatoes, radish, carrots, zucchinis, rocket, persil, garden cress, cilantro, mint, different types of lettuces, cucumbers, hot and mild peppers, lemon, walnuts trees, loquat tree, common fig trees, lemon trees, berries, oranges, avocados, different types of beans, eggplants, a dozen olives trees, grape vines, and more…
It’s a hobby that is time consuming but rewarding. I love that when working from home I can just go in the garden directly after work and slowly, but certainly, take care of it.

Now

In retrospect that seems like plenty of stuff!
I got multiple plans ahead, countless things I want to explore. My most recent obsession has been with Gstreamer and PipeWire. I’m preparing an article about PipeWire, it’s still a work in progress though but if I ever need help the main devs are really friendly on IRC.

Aside from this, I want to keep that new hobby of watching old movies and enriching my perspectives by learning about other cultures and traditions.

Obviously, it’s gardening season so I won’t stop that until next October at least.

Next month, I’m planning to kick back the Nixers June events. We’ll have a workflow compilation and the week in the TTY.

I’m not so sure I’ll dive into nix or nixOs more, I’ll try from time to time.

There’s another topic, unrelated to tech I might explore and write about. Maybe it’ll give rise to another booklet. Some weird topic like Everyday philosophy. I’ve found I love writing in general.

On the tech side, I’m looking forward, after the audio topic, to go into more popular ones such as containers, eBPF, maybe continue with Wayland, flatpak, access control mechanisms, init systems or service managers, and randomness. These last two have been on my list for a while, especially randomness on Unix as I was planning to write about it before the Time on Unix article but left it aside when I did the Coursera courses.

As far as books are concerned, I haven’t read much because of the economic situation. Books are hard to come by and are now worth one fourth of the average salary. I’ve also refrained from ordering new ones as I used to ship them to my workplace and we’re all doing remote work now. Shipping to my house is too complicated with the lack of courier service and no addresses.

This is it!
As usual… If you want something done, no one’s gonna do it for you, use your own hands, even if it’s not much.
And let’s go for a beer together sometime, or just chill.

Attributions:

• Internet Archive Book Images, No restrictions, via Wikimedia Commons

This time I’m gonna address Django’s builtin authentication system, more specifically the ways we can build custom improvements over the already very solid foundations it provides.

The idea for this post came from reading an article summing up some considerations we should have when dealing with passwords. Most of those considerations are about what controls to implement (what “types” of passwords to accept) and how to securely store those passwords. By default Django does the following:

• Passwords are stored using PBKDF2. There are also other alternatives such as Argon2 and bcrypt, that can be defined in the setting PASSWORD_HASHERS.
• Every Django release the “strength”/cost of this algorithm is increased. For example, version 3.1 applied 216000 iterations and the last version (3.2 at the time of writing) applies 260000. The migration from one to another is done automatically once the user logs in.
• There are a set of validators that control the kinds of passwords allowed to be used in the system, such as enforcing a minimum length. These validators are defined on the setting AUTH_PASSWORD_VALIDATORS.

By default when we start a new project these are the included validators :

• UserAttributeSimilarityValidator
• MinimumLengthValidator
• CommonPasswordValidator
• NumericPasswordValidator

The names are very descriptive and I would say a good starting point. But as the article mentions the next step is to make sure users aren’t reusing previously breached passwords or using passwords that are known to be easily guessed (even when complying with the other rules). CommonPasswordValidator already does part of this job but with a very limited list (20000 entries).

Improving password validation

So for the rest of this post I will show you some ideas on how we can make this even better. More precisely, prevent users from using a known weak password.

1. Use your own list

The easiest approach, but also the more limited one, is providing your own list to `CommonPasswordValidator`, containing more entries than the ones provided by default. The list must be provided as a file with one entry in lower case per line. It can be set like this:

{
  "NAME": "django.contrib.auth.password_validation.CommonPasswordValidator",
  "OPTIONS": {"password_list_path": "<path_to_your_file>"}
}

2. Use zxcvbn-python

Another approach is to use an existing and well-known library that evaluates the password, compares it with a list of known passwords (30000) but also takes into account slight variations and common patterns.

To use zxcvbn-python we need to implement our own validator, something that isn’t hard and can be done this way:

# <your_app>/validators.py

from django.core.exceptions import ValidationError
from zxcvbn import zxcvbn


class ZxcvbnValidator:
    def __init__(self, min_score=3):
        self.min_score = min_score

    def validate(self, password, user=None):
        user_info = []
        if user:
            user_info = [
                user.email, 
                user.first_name, 
                user.last_name, 
                user.username
            ]
        result = zxcvbn(password, user_inputs=user_info)

        if result.get("score") < self.min_score:
            raise ValidationError(
                "This passoword is too weak",
                code="not_strong_enough",
                params={"min_score": self.min_score},
            )

    def get_help_text(self):
        return "The password must be long and not obvious"

Then we just need to add to the settings just like the other validators. It’s an improvement but we still can do better.

3. Use “have i been pwned?”

As suggested by the article, a good approach is to make use of the biggest source of leaked passwords we have available, haveibeenpwned.com.

The full list is available for download, but I find it hard to justify a 12GiB dependency on most projects. The alternative is to use their API (documentation available here), but again we must build our own validator.

# <your_app>/validators.py

from hashlib import sha1
from io import StringIO

from django.core.exceptions import ValidationError

import requests
from requests.exceptions import RequestException

class LeakedPasswordValidator:
    def validate(self, password, user=None):
        hasher = sha1(password.encode("utf-8"))
        hash = hasher.hexdigest().upper()
        url = "https://api.pwnedpasswords.com/range/"

        try:
            resp = requests.get(f"{url}{hash[:5]}")
            resp.raise_for_status()
        except RequestException:
            raise ValidationError(
                "Unable to evaluate password.",
                code="network_failure",
            )

        lines = StringIO(resp.text).readlines()
        for line in lines:
            suffix = line.split(":")[0]

            if hash == f"{hash[:5]}{suffix}":
                raise ValidationError(
                    "This password has been leaked before",
                    code="leaked_password",
                )

    def get_help_text(self):
        return "Use a different password"

Then add it to the settings.

Edit: As suggested by one reader, instead of this custom implementation we could use pwned-passwords-django (which does practically the same thing).

And for today this is it. If you have any suggestions for other improvements related to this matter, please share them in the comments, I would like to hear about them.

Today marks my first ever 7 day streak in DuoLingo. I have attempted to learn Hungarian before, but apparently not for 7 days in a row. During the past week I’ve learned some basic phrases and greetings, some of the basic grammar for the present tense singular, and I have a general sense of the word order rules. I can now confidently say én magyar diák vagyok (I am a hungarian student). Well, somewhat confidently… magyar is an adjective there and I’m not sure if it works the same as English where it can either mean “I am a student of Hungarian descent” or “I am a student of Hungarian”. Perhaps there is more I need to learn in order to say it correctly.

UPDATE: Magyarul tanulok is a better way to say this.

Things I learned this week about Hungarian

Here are some interesting things I learned this week about Hungarian.

Word order nuance

Hungarian word order is more flexible than English. However, it’s important to realize that this does not mean Hungarian word order doesn’t matter. Certain phrases cannot be jumbled. For instance, articles and adjectives seem to always come before the noun they modify, so te a fiatal fiú látsz (you see the young boy) cannot be rephrased as te fiú a látsz fiatal. However, the I believe the sentence can be rephrased as te látsz a fiatal fiú to place more emphasis on látz (see).

Which segues into what happens when you use different word ordering in Hungarian. The two places where the main emphasis is placed in a Hungarian sentence are the beginning of the sentence, and directly before the verb. In fact, the focus of words directly before the verb is so strong that it’s required for question words like mi (what) to be there. For instance, mi látsz? (what do you see?) can’t be rephrased as látsz mi?. So when you place a word in a different place in the sentence, you are usually not changing the literal meaning of the sentence, but you are changing the emphasis.

UPDATE: My understanding of this is incorrect Judit294350 (and jzsuzsi) kindly pointed out on the DuoLingo discussion boards:

“te a fiatal fiú látsz?” - Sorry several mistakes here.

1: You do not normally use the pronoun. If you do it is for emphasis and this sentence’s emphasis is “a fiatal fiú”

2: The direct object has t be put in the accusative “a fiatal fiút”

3: A definite diret object uses a different conjugation - the definite conjugation - which you would not have learnt yet - so látod not látsz.

4: I also have a sneaky suspicion it would be meglátod.

Optional pronouns (and sometimes verbs)

Hungarian doesn’t always require a pronoun when the verb already conveys that information. For instance, én látok egy almát and látok egy almát both mean “I see an apple”. The -ok suffix on lát tells us that the verb is in the first person, so adding én is optional.

In a very special case verbs can also be optional. This is the case for the verb van (he/she/it is) and only in the third person. So instead of saying piros autó van we can say piros autó, and both mean “the car is red”. We can’t do this for other conjugations of van, however, so jól vagyok (I am well) has to include vagyok. See update below.

UPDATE: jzuzsi pointed out that the second paragraph here is incorrect.

“In a very special case verbs can also be optional. This is the case for the verb van (he/she/it is) and only in the third person. So instead of saying piros autó van we can say piros autó, and both mean “the car is red”. We can’t do this for other conjugations of van, however, so jól vagyok (I am well) has to include vagyok.”

In this case it is not optional. You have to omit van. The car is red. Az autó piros.

You said “and both mean the car is red”…. actually none of them means that. “piros autó van” can mean “There is a red car” and “piros autó” = “red car”.

Genderless grammar

Hungarian does not have grammatical gender, so there aren’t different versions of verbs, nouns, pronouns, etc. for different genders. This also means that unlike many other languages (looking at you, Latin) random objects don’t have a gender. Hungarian specifies no gender by default, although you can specify that a noun is female by tacking on nő (woman) to the end. For instance, színész is actor, and színésznő is actress (literally “actor woman”).

Unexpected similarities to English

The Hungarian people (Magyars) descended from the Ugrians Ugrics (thank you csepcsenyi!), a people that lived near the Ural mountains. Some of these people moved west at some point. These people later split again, with one group, the Finns, going north, and the other group, the Magyars, settling in the Carpathian basin. As a result, the Hungarian language has very little in common with other European languages, it’s closest relatives being Khanty and Mansi which are spoken by remaining Uralic tribes to this day. Its closest relatives in Europe are Estonian and Finnish.

The reason I’m leading with that is to demonstrate why I’m surprised when Hungarian has similarities to English, since they are nowhere near each other in the family tree of languages.

One similarity I found interesting is that just as in English, székeket festek (I paint chairs) can either mean “I paint pictures of chairs” or “I paint actual chairs”.

Another interesting similarity is that just as in English we don’t put “a” before a word starting with a vowel, instead adding a consonant to make it “an”, in Hungarian the definite article a gets changed to az when it comes before a word starting with a vowel.

I’m not 100% sure on this, but from what I can tell “objects” predicates of van are not in the accusative case in Hungarian, just as “objects” predicates of “is” are not in the objective case in English. Whether Hungarian has predicate nominatives or if this is for some other grammatical reason is a point of future research for me.

UPDATE: jzuzsi confirmed that these are predicates and not objects in both English and Hungarian, and are in the nominative case.

There are also certain vocabulary similarities especially with modern words, like telefon meaning “telephone”, but these are less interesting to me since I’m pretty sure they’re loan words from the rest of Europe and really just show the influence English has on the world’s languages.

Pronunciation issues

I’m sure I’m mispronouncing many things and have a heavy American accent, but the most glaring things I notice about my pronunciation at the moment are the following:

• Te. This is pronounced “teh”, not “tay”. Years of pronouncing the Latin word “te” as “tay” are coming to haunt me.
• Rolled R’s. I can’t roll my r’s yet. I can do a voiced dental tap, but not a trill. I also struggle with r’s before some consonants, like vársz.
• Nyt. I’m not sure if I’m trying too hard or what, but I have a really hard time doing this sound. The Hungarian consonant ny is pronounced like the ny in Kenya without including the “a” (so not nyuh). I can pronounce it before vowels easily, and before some consonants, but for some reason when it comes before “t” I have a lot of trouble, so words like lányt become difficult for me.
• Az. This is pronounced similar to “ahz”, not like the English word “as”. For some reason I am occasionally mixing up these vowels, usually in az but sometimes in other words like tanár (teacher).

Studying techniques

There are a few studying techniques I am trying to employ to help with my understanding of Hungarian.

One technique is to try to visualize the concept of a word instead of tagging it with the English equivalent in my head. For instance, instead of thinking “tűz means fire” I say tűz while visualizing a campfire. I think to some extent, at least at first, some tagging does happen regardless, because I’m finding that with many words I still think the English word instead of simply understanding the Hungarian word. I hope that as I study, speak, and listen to Hungarian more that will gradually become less and less of an issue.

To help with memorizing words I’ve been doodling their meanings and then writing the word below. I keep a bullet journal, so in addition to these doodles I also keep notes of things I have trouble remembering.

I really want to start speaking Hungarian to a native speaker. There is a virtual group lesson series available that I plan to attend soon. I’d love to find a Discord server for Hungarian language learning, but I haven’t been able to find one. I may also try language exchange at some point, but my Hungarian skills aren’t good enough for that yet.

Things I wish I knew earlier

One thing I learned a couple days ago that I wish I knew way earlier is that the DuoLingo website has Hungarian tips which include grammar explanations, etc., but the DuoLingo mobile app only has tips for Spanish, French, and Chinese at the moment. So do yourself a favor and open the website and read the tips before you start a new bubble. It will save you a ton of trial and error.

Come on DuoLingo devs, you can do better than that.

Also, each DuoLingo exercise has a discussion thread which you can access after you complete the exercise. Be sure to check it out when you have questions about why an answer is right or wrong, or whether you can phrase a word a different way, or what a word actually means, or whatever. There are a few native speakers that seem to be pretty active, and I’ve usually gotten responses to my questions within hours. I’ve also found that many times my question has already been asked and answered very thoroughly.

Additional Language Learning Resources

I’ve been using the following resources in addition to DuoLingo:

• HungarianReference.com - though this website looks like it was made in 1999 and has an occasional English typo, it’s actually a very good resource, especially for grammar. Be aware that this website has Hungarian typos and inaccuracies as well and was not written by a native Hungarian speaker.
• Glosbe - a useful dual-language dictionary
• Wiktionary - good for looking up definitions for Hungarian words and also their conjugations and suffixes
• Drops - this seems to be a good vocabulary trainer. I like that it uses pictures to represent the Hungarian words instead of English words. I do about five minutes of these exercises a day.
• /r/hungarian - this is a great community of Hungarian learners and teachers/speakers. I’ve had a few positive experiences with various questions over the past week, as well as searching for answers to questions I had.

Credits for corrections

If I missed anyone else who critiqued this post, I apologize. Please see both the DuoLingo thread and the Reddit thread to see the kind help people put into correcting statements in this post. :)

Further reading

A non-exhaustive list of resources and articles that were referenced by people critiquing this post:

• On Emphasis and Word Order in Hungarian
• Structure and word order of a Hungarian sentence
• Once more on Hungarian Word Order
• Hungarian in focus

For most of the 2010s, the OpenBSD base system has been stuck with GCC 4.2.1. It was released in July 2007, imported into the OpenBSD source tree in October 2009, and became the default compiler on the amd64, i386, hppa, sparc64, socppc and macppc platforms in OpenBSD 4.8, released in November 2010. As specified in the commit message during import, this is the last version released under the GPLv2 license.

OpenBSD was not the only operating system sticking to GCC 4.2.1 for licensing reasons, FreeBSD did the same, and Mac OS X as well.

As a general rule, and this is not OpenBSD specific, being stuck with old compilers is problematic for several reasons:

• The main reason has to be newer C and C++ standards support. While the C standards committee is conservative and the language evolves slowly, the pace at which new C++ standards appear has been accelerating, with a new version emerging every 3 years.
• Another major reason is new architectures support. Not only new ISAs like ARMv8 and RISC-V, but also x86-64 microarchitecture updates.
• They are not getting bugfixes, nor new optimizations or advances in diagnostic (better warnings) and security features.

The latest point has been partially mitigated on OpenBSD, as several developers have worked on fixing OpenBSD related issues and backporting fixes, as detailed in Miod's excellent "Compilers in OpenBSD" post from 2013.

Regarding new architectures support, the more astute reader will know that all OpenBSD supported platforms are self-hosted and releases must be built using the base system compiler on real hardware. No cross-compilation, no emulators. The ARMv8 architecture was announced in 2011, a few years after GCC 4.2.1 was released. By the year 2016, 64-bit ARMv8 devices were getting widely available and more affordable. During the g2k16 hackathon, the Castle Inn pub had become a favorite meet-up point among OpenBSD developers, and the topic came up at one of the evening gatherings. I happened to be sitting nearby when patrick@ discussed with deraadt@ about the possibility of importing LLVM to make a future OpenBSD/arm64 porting effort possible, and Theo said that there was nothing blocking it. The next day, pascal@ mentioned he already had some Makefiles to replace the LLVM build system, and when someone then asked how long it would take to put them in shape for import, he said he didn't know, then smiled and said: "Let's find out! :-)". Before the end of the hackathon, he imported LLVM 3.8.1 along with Clang. Patrick's g2k16 hackathon report retraces the events and gives more technical details.

OpenBSD/arm64 became the first platform to use Clang as base system compiler and LLD as the default linker. Clang then became the default compiler on amd64 and i386 in July 2017, on armv7 in January 2018, on octeon in July 2019, on powerpc in April 2020, and finally on loongson in December 2020.

LLVM was updated regularly along the way up to the 8.0.1 version, which was the latest version released under the NCSA license. From then all later LLVM versions have been released under the Apache 2.0 license, which couldn't be included in OpenBSD. The project's copyright policy page details OpenBSD's stance on the license, and Mark Kettenis objection on the llvm-dev mailing list gives more background information.

While staying with LLVM 8.0.1 would not have been an immediate problem for the OpenBSD kernel and the base system userland which uses C99, the project also includes 3rd party C++ codebases for parts of the graphics stack and of course LLVM itself. Jonathan Gray hinted at the problem on the openbsd-misc mailing list, mentioning that not updating was becoming increasingly painful. The effect which can be observed in the 3rd party software ecosystem regarding newer C and C++ standards is that while C99 is still reigning supreme in C codebases, C++ codebases maintainers have been eager to adopt new C++ standards (and for good reasons). The recent RFC on cfe-dev about bumping toolchain requirements for LLVM to Clang 6.0 (released in March 2018) proves that LLVM is no exception. A compromise was thus inevitable, and LLVM 10.0.0 was imported into -current in August 2020.

At the time the OpenBSD 6.9 branch was created, the CVS tree contained LLVM 10.0.1, GCC 4.2.1, and GCC 3.3.6. However, it's important to understand that not all compilers are built on all platforms:

• Clang is the default compiler on amd64, arm64, armv7, i386, loongson, macppc, octeon, powerpc64, and riscv64.
• GCC 4.2.1 is still the default compiler on alpha, hppa, landisk, and sparc64.
• OpenBSD/luna88k is the only platform still using GCC 3.3.6, as m88k support was removed in GCC 3.4.

Following the OpenBSD 6.9 release, OpenBSD-current has been updated to LLVM 11.1.0 and GCC 4.2.1 is not built anymore on amd64. GCC 8.4.0 (released in March 2020) is available in the ports collection.

Among the remaining platforms still using GCC 4.2.1 as the default compiler, only sparc64 will be able to switch in the future. LLVM has a Sparc V9 backend and work has been done in OpenBSD to make the switch possible. For all the other remaining ones, there are no alpha, hppa, sh4, nor m88k backends in LLVM, and even if this changed in the future, the hardware is too slow to be able to self-host the compiler.

Regarding linkers, LLD is the default linker on amd64, arm64, armv7, i386, powerpc64, and riscv64. All other platforms still use GNU ld from binutils 2.17. Realistically, it should be possible to switch to LLD in the future on the following platforms: loongson, macppc, octeon, and sparc64.

At this point, all relevant architectures have modern and up-to-date toolchains, and we can look ahead in confidence on that front.

Due to JSON’s ubiquity, we end up reaching for JSON libraries regularly when we have a project that needs to exchange data with other systems. Whenever something becomes widespread and becomes an “infrastructure”, it turns into a black-box in people’s minds.

One reason JSON got so popular is the fact that it’s simple. It’s not the simplest solution to the problem, not by a long shot, but it’s flexible enough to solve a lot of problems without becoming too large. In this post we’ll be making a JSON serializer in Python that can serialize arbitrary nested data structures in a few lines of code. And more importantly, every part should be understandable and self-contained.

Below is the default implementation of the encoder, which fails with an “Unknown type” error. We will implement encoding for the types shared by Python and JSON, and have this error as a fallback in case people try to encode something weird.

from functools import singledispatch

@singledispatch
def encode(value):
    raise Exception(f"Unknown type: {type(value)}")

Serializing None

This is probably the easiest type to encode, as it involves no logic whatsoever. JSON has a null type that represents a value that is intentionally missing. The None keyword in Python is the equivalent. When we see a value of NoneType, we should encode it as the 4-bytes null.

@encode.register(type(None))
def encode_none(_):
    return "null"

print(encode(None))

null

Serializing booleans

The boolean values True and False in Python can be encoded as true and false respectively. This is also pretty straightforward.

@encode.register(bool)
def encode_bool(value):
    if value:
        return "true"
    return "false"

print(encode(True))
print(encode(False))

true
false

Serializing numbers

The way Python formats numbers by default is actually suitable for JSON. So encoding int’s and float’s is pretty easy, just call str(x) on the value.

@encode.register(int)
@encode.register(float)
def encode_number(val):
    return str(val)

print(encode(42))
print(encode(123.45))

42
123.45

Serializing strings

Strings are a list of characters enclosed in double quotes ("). Aside from some special characters, everything else can be encoded as-is. The characters we’ll escape are double-quotes and newlines.

@encode.register(str)
def encode_str(val):
    result = '"'
    for c in val:
        if c == '"':
            result += r'\"'
        elif c == '\n':
            result += r'\n'
        else:
            result += c
    result += '"'
    return result

print(encode("Hello, world!"))
print(encode('Hello "World"'))
print(encode("""Hello
world!"""))

"Hello, world!"
"Hello \"World\""
"Hello\nworld!"

Serializing lists

Lists in JSON are comma-separated values enclosed in [ and ]. An empty list becomes [], a list with one element is encoded as [42], and a list with two elements can be written as [42, 43]. Of course, you are not limited to only integers. Any JSON type, including other lists and dictionaries, can be a list member.

@encode.register(list)
def encode_list(l):
    vals = ','.join(map(encode, l))
    return f"[{vals}]"

print(encode([]))
print(encode([1]))
print(encode([1, 2, 3]))
print(encode([1, 2.0, "3"]))
print(encode([1, 2, [3, 4], 5, 6, False, True]))

[]
[1]
[1,2,3]
[1,2.0,"3"]
[1,2,[3,4],5,6,false,true]

Serializing dictionaries

Dictionaries are similar to lists, but instead of a list of values, we have a list of key-value pairs. Before we get to the dictionary part, let’s start by encoding key-value pairs in the suitable format.

In JSON, key value pairs are formatted as key:value.

def encode_key_value(t):
    k, v = t
    return f"{encode(k)}:{encode(v)}"

print(encode_key_value((1, 2)))
print(encode_key_value(("key", 1234)))
print(encode_key_value(("this is a list", [1, 2, 3])))

1:2
"key":1234
"this is a list":[1,2,3]

Looks good so far. To turn key-value pairs into a dictionary, we encode them the same way as the list, but instead of wrapping them in [ and ], we wrap them in { and }.

@encode.register(dict)
def encode_dict(d):
    vals = ','.join(map(encode_key_value, d.items()))
    return '{' + vals + '}'

print(encode({"Name": "Leo", "BirthYear": 1998, "Website": "www.gkbrk.com"}))

{"Name":"Leo","BirthYear":1998,"Website":"www.gkbrk.com"}

Jó napot!

I have decided to finally make a serious attempt at learning Hungarian. I will be trying to blog in Hungarian occasionally, and then posting the English translation in the second half of the post.

Also, in case anyone is curious, I am using the Hungarian 101-key keyboard layout. To set this up in Windows 10, go to Start > Language Settings, click Add preferred language, and select Hungarian. Once the language pack finishes downloading, click the language, then Options, and remove the default QWERTZ keyboard. Click Add a keyboard and select Hungarian 101-key, QWERTY.

You should now be able to switch between English and Hungarian keyboards by using the keyboard switcher in the taskbar.

Viszlát!

A software estimate is a prediction about the future. Software developers were not the first people to formalize processes for making predictions about the future. Starting in the last 1940s, the RAND Corporation’s Delphi project created what became known as the Delphi method, e.g., An Experiment in Estimation, and Construction of Group Preference Relations by Iteration.

In its original form experts were anonymous; there was a “… deliberate attempt to avoid the disadvantages associated with more conventional uses of experts, such as round-table discussions or other milder forms of confrontation with opposing views.”, and no rules were given for the number of iterations. The questions involved issues whose answers involved long term planning, e.g., how many nuclear weapons did the Soviet Union possess (this study asked five questions, which required five estimates). Experts could provide multiple answers, and had to give a probability for each being true.

One of those involved in the Delphi project (Helmer-Hirschberg) co-founded the Institute for the Future, which published reports about the future based on answers obtained using the Delphi method, e.g., a 1970 prediction of the state-of-the-art of computer development by the year 2000 (Dalkey, a productive member of the project, stayed at RAND).

The first application of Delphi to software estimation was by Farquhar in 1970 (no pdf available), and Boehm is said to have modified the Delphi process to have the ‘experts’ meet together, rather than be anonymous, (I don’t have a copy of Farquhar, and my copy of Boehm’s book is in a box I cannot easily get to); this meeting together form of Delphi is known as Wideband Delphi.

Planning poker is a variant of Wideband Delphi.

An assessment of Delphi by Sackman (of Grant-Sackman fame) found that: “Much of the popularity and acceptance of Delphi rests on the claim of the superiority of group over individual opinions, and the preferability of private opinion over face-to-face confrontation.” The Oracle at Delphi was one person, have we learned something new since that time?

Group dynamics is covered in section 3.4 of my Evidence-based software engineering book; resource estimation is covered in section 5.3.

The likelihood that a group will outperform an individual has been found to depend on the kind of problem. Is software estimation the kind of problem where a group is likely to outperform an individual? Obviously it will depend on the expertise of those in the group, relative to what is being estimated.

What does the evidence have to say about the accuracy of the Delphi method and its spinoffs?

When asked to come up with a list of issues associated with solving a problem, groups generate longer lists of issues than individuals. The average number of issues per person is smaller, but efficient use of people is not the topic here. Having a more complete list of issues ought to be good for accurate estimating (the validity of the issues is dependent on the expertise of those involved).

There are patterns of consistent variability in the estimates made by individuals; some people tend to consistently over-estimate, while others consistently under-estimate. A group will probably contain a mixture of people who tend to over/under estimate, and an iterative estimation process that leads to convergence is likely to produce a middling result.

By how much do some people under/over estimate?

The multiplicative factor values (y-axis) appearing in the plot below are from a regression model fitted to estimate/actual implementation time for a project involving 13,669 tasks and 47 developers (data from a study Nichols, McHale, Sweeney, Snavely and Volkmann). Each vertical line, or single red plus, is one person (at least four estimates needed to be made for a red plus to occur); the red pluses are the regression model’s multiplicative factor for that person’s estimates of a particular kind of creation task, e.g., design, coding, or testing. Points below the grey line are overestimation, and above the grey line the underestimation (code+data):

What is the probability of a Delphi estimate being more accurate than an individual’s estimate?

If we assume that a middling answer is more likely to be correct, then we need to calculate the probability that the mix of people in a Delphi group produces a middling estimate while the individual produces a more extreme estimate.

I don’t have any Wideband Delphi estimation data (or rather, I only have tiny amounts); pointers to such data are most welcome.

Audio from the May 11, 2021 Pony development sync is available.

I was received into the Catholic Church on the Easter Vigil of 2019.

This post is the AArch64 counterpart of my "Speedbuilding LLVM/Clang in 5 minutes" article.

After publishing and sharing the previous post URL with some friends on IRC, I was asked if I wanted to try doing the same on a 160 cores ARM machine. Finding out what my answer was is left as an exercise to the reader :-)

The system I'm using for this experiment is a BM.Standard.A1.160 bare-metal machine from Oracle Cloud, which has a dual-socket motherboard with two 80 cores Ampere Altra CPUs, for a total 160 cores, and 1024 GB of RAM. This is to the best of my knowledge the fastest AArch64 server machine available at this time.

The system is running Oracle Linux Server 8.3 with up-to-date packages and kernel.

The full result of cat /proc/cpuinfo is available here.

uname -a
Linux benchmarks 5.4.17-2102.201.3.el8uek.aarch64 #2 SMP Fri Apr 23 09:42:46 PDT 2021 aarch64 aarch64 aarch64 GNU/Linux

Let's start by installing required packages:

dnf in clang git lld

Unfortunately the CMake version available in the packages repository (3.11.4) is too old to build the main branch of the LLVM Git repository, and Ninja is not available either.

Let's bootstrap Pkgsrc to build and install them:

git clone https://github.com/NetBSD/pkgsrc.git
cd pkgsrc/bootstrap
./bootstrap --make-jobs=160 --unprivileged

===> bootstrap started: Wed May 12 12:23:34 GMT 2021
===> bootstrap ended:   Wed May 12 12:26:08 GMT 2021

We then need to add ~pkg/bin and ~pkg/sbin to the path:

export PATH=$PATH:$HOME/pkg/bin:$HOME/pkg/sbin

For faster Pkgsrc builds, we can edit ~/pkg/etc/mk.conf and add:

MAKE_JOBS=              160

Let's build and install CMake and Ninja:

cd ~/pkgsrc/devel/cmake
bmake install package clean clean-depends

cd ~/pkgsrc/devel/ninja-build
bmake install package clean clean-depends

The compiler used for the builds is Clang 10.0.1:

clang --version
clang version 10.0.1 (Red Hat 10.0.1-1.0.1.module+el8.3.0+7827+89335dbf)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /bin

Regarding linkers, we are using GNU ld and GNU Gold from binutils 2.30, and LLD 10.0.1.

GNU ld version 2.30-79.0.1.el8
GNU gold (version 2.30-79.0.1.el8) 1.15
LLD 10.0.1 (compatible with GNU linkers)

For all the following runs, I'm building from the Git repository main branch commit cf4610d27bbb5c3a744374440e2fdf77caa12040. The build directory is of course fully erased between each run.

commit cf4610d27bbb5c3a744374440e2fdf77caa12040
Author: Victor Huang <wei.huang@ibm.com>
Date:   Wed May 12 10:56:54 2021 -0500

I'm not sure what the underlying storage is, but with 1 TB of RAM there is no reason not to use a ramdisk.

mkdir /mnt/ramdisk
mount -t tmpfs -o size=32g tmpfs /mnt/ramdisk
cd /mnt/ramdisk

To get a baseline, let's do a full release build on this machine:

cd llvm-project
mkdir build
cd build

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        ../llvm

time make -j160
real    7m3.226s
user    403m28.362s
sys     6m41.331s

By default, CMake generates Makefiles. As documented in the "Getting Started with the LLVM System" tutorial, most LLVM developers use Ninja.

Let's switch to generating Ninja build files, and using ninja to build:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -GNinja ../llvm

time ninja
[4182/4182] Linking CXX executable bin/c-index-test

real    4m20.403s
user    427m27.118s
sys     7m2.320s

By default, GNU ld is used for linking. Let's switch to using gold:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -DLLVM_USE_LINKER=gold \
        -GNinja ../llvm

time ninja
[4182/4182] Linking CXX executable bin/c-index-test

real    4m1.062s
user    427m1.648s
sys     6m58.282s

LLD has been a viable option for some years now. Let's use it:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -DLLVM_USE_LINKER=lld \
        -GNinja ../llvm

time ninja
[4182/4182] Linking CXX executable bin/clang-scan-deps

real    3m58.476s
user    428m3.807s
sys     7m14.418s

Using GNU gold instead of GNU ld results in noticeably faster builds, and switching to LLD shaves a few mores seconds from the build.

If we want to build faster, we can make some compromises and start stripping the build by removing some components.

Let's start by disabling additional architecture support:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -DLLVM_USE_LINKER=lld \
        -DLLVM_TARGETS_TO_BUILD="AArch64" \
        -GNinja ../llvm

time ninja
[3195/3195] Linking CXX executable bin/c-index-test

real    3m10.312s
user    326m54.898s
sys     5m24.770s

We can verify the resulting Clang binary only supports AArch64 targets:

bin/clang --print-targets
  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_32 - AArch64 (little endian ILP32)
    aarch64_be - AArch64 (big endian)
    arm64      - ARM64 (little endian)
    arm64_32   - ARM64 (little endian ILP32)

Let's go further and disable the static analyzer and the ARC Migration Tool:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -DLLVM_USE_LINKER=lld \
        -DLLVM_TARGETS_TO_BUILD="AArch64" \
        -DCLANG_ENABLE_STATIC_ANALYZER=OFF \
        -DCLANG_ENABLE_ARCMT=OFF \
        -GNinja ../llvm

time ninja
[3146/3146] Creating library symlink lib/libclang-cpp.so

real    3m6.474s
user    319m25.914s
sys     5m20.924s

Let's disable building some LLVM tools and utils:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -DLLVM_USE_LINKER=lld \
        -DLLVM_TARGETS_TO_BUILD="AArch64" \
        -DCLANG_ENABLE_STATIC_ANALYZER=OFF \
        -DCLANG_ENABLE_ARCMT=OFF \
        -DLLVM_BUILD_TOOLS=OFF \
        -DLLVM_BUILD_UTILS=OFF \
        -GNinja ../llvm

time ninja
[2879/2879] Creating library symlink lib/libclang-cpp.so

real    2m59.659s
user    298m47.482s
sys     4m57.430s

Compared to the previous build, the following binaries were not built: FileCheck, count, lli-child-target, llvm-jitlink-executor, llvm-PerfectShuffle, not, obj2yaml, yaml2obj, and yaml-bench.

We are reaching the end of our journey here. At this point, we are done stripping out things.

Let's disable optimizations and do a last run:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -DLLVM_USE_LINKER=lld \
        -DLLVM_TARGETS_TO_BUILD="AArch64" \
        -DCLANG_ENABLE_STATIC_ANALYZER=OFF \
        -DCLANG_ENABLE_ARCMT=OFF \
        -DLLVM_BUILD_TOOLS=OFF \
        -DLLVM_BUILD_UTILS=OFF \
        -DCMAKE_CXX_FLAGS_RELEASE="-O0" \
        -GNinja ../llvm

time ninja
[2879/2879] Linking CXX executable bin/c-index-test

real    2m37.003s
user    231m53.133s
sys     4m56.675s

So this is it, this machine can build a full LLVM/Clang release build in a bit less than four minutes, and a stripped down build with optimizations disabled in two minutes. Two minutes. This is absolutely mind-blowing… The future is now!

This post is a spiritual successor to my "Building LLVM on OpenBSD/loongson" article, in which I retraced my attempts to build LLVM 3.7.1 on MIPS64 in a RAM constrained environment.

After reading the excellent "Make LLVM fast again", I wanted to revisit the topic, and see how fast I could build a recent version of LLVM and Clang on modern x86 server hardware.

The system I'm using for this experiment is a CCX62 instance from Hetzner, which has 48 dedicated vCPUs and 192 GB of RAM. This is the fastest machine available in their cloud offering at the moment.

The system is running Fedora 34 with up-to-date packages and kernel.

The full result of cat /proc/cpuinfo is available here.

uname -a
Linux benchmarks 5.11.18-300.fc34.x86_64 #1 SMP Mon May 3 15:10:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Let's start by installing required packages:

dnf in clang cmake git lld ninja-build

The compiler used for the builds is Clang 12.0.0:

clang --version
clang version 12.0.0 (Fedora 12.0.0-0.3.rc1.fc34)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

Regarding linkers, we are using GNU ld and GNU Gold from binutils 2.35.1, and LLD 12.0.0.

GNU ld version 2.35.1-41.fc34
GNU gold (version 2.35.1-41.fc34) 1.16
LLD 12.0.0 (compatible with GNU linkers)

For all the following runs, I'm building from the Git repository main branch commit 831cf15ca6892e2044447f8dc516d76b8a827f1e. The build directory is of course fully erased between each run.

commit 831cf15ca6892e2044447f8dc516d76b8a827f1e
Author: David Spickett <david.spickett@linaro.org>
Date:   Wed May 5 11:49:35 2021 +0100

To get a baseline, let's do a full release build on this machine:

cd llvm-project
mkdir build
cd build

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        ../llvm

time make -j48
real    11m19.852s
user    436m30.619s
sys     12m5.724s

By default, CMake generates Makefiles. As documented in the "Getting Started with the LLVM System" tutorial, most LLVM developers use Ninja.

Let's switch to generating Ninja build files, and using ninja to build:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -GNinja ../llvm

time ninja
[4182/4182] Generating ../../bin/llvm-readelf

real    10m13.755s
user    452m16.034s
sys     12m7.584s

By default, GNU ld is used for linking. Let's switch to using gold:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -DLLVM_USE_LINKER=gold \
        -GNinja ../llvm

time ninja
[4182/4182] Generating ../../bin/llvm-readelf

real    10m13.405s
user    451m35.029s
sys     11m57.649s

LLD has been a viable option for some years now. Let's use it:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -DLLVM_USE_LINKER=lld \
        -GNinja ../llvm

time ninja
[4182/4182] Generating ../../bin/llvm-readelf

real    10m12.710s
user    451m12.444s
sys     12m12.634s

During tests on smaller build machines, I had observed that using GNU gold or LLD instead of GNU ld resulted in noticeably faster builds. This doesn't seem to be the case on this machine. We end up with a slightly faster build by using LLD, but not by a large margin at all.

If we want to build faster, we can make some compromises and start stripping the build by removing some components.

Let's start by disabling additional architecture support:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -DLLVM_USE_LINKER=lld \
        -DLLVM_TARGETS_TO_BUILD="X86" \
        -GNinja ../llvm 

time ninja
[3196/3196] Generating ../../bin/llvm-readelf

real    7m55.531s
user    344m56.462s
sys     8m53.970s

We can verify the resulting Clang binary only supports x86 targets:

bin/clang --print-targets
  Registered Targets:
    x86    - 32-bit X86: Pentium-Pro and above
    x86-64 - 64-bit X86: EM64T and AMD64

Let's go further and disable the static analyzer and the ARC Migration Tool:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -DLLVM_USE_LINKER=lld \
        -DLLVM_TARGETS_TO_BUILD="X86" \
        -DCLANG_ENABLE_STATIC_ANALYZER=OFF \
        -DCLANG_ENABLE_ARCMT=OFF \
        -GNinja ../llvm 

time ninja
[3147/3147] Generating ../../bin/llvm-readelf

real    7m42.299s
user    334m47.916s
sys     8m44.704s

Let's disable building some LLVM tools and utils:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -DLLVM_USE_LINKER=lld \
        -DLLVM_TARGETS_TO_BUILD="X86" \
        -DCLANG_ENABLE_STATIC_ANALYZER=OFF \
        -DCLANG_ENABLE_ARCMT=OFF \
        -DLLVM_BUILD_TOOLS=OFF \
        -DLLVM_BUILD_UTILS=OFF \
        -GNinja ../llvm

time ninja
[2880/2880] Generating ../../bin/llvm-readelf

real    7m21.016s
user    315m42.127s
sys     8m9.377s

Compared to the previous build, the following binaries were not built: FileCheck, count, lli-child-target, llvm-jitlink-executor, llvm-PerfectShuffle, not, obj2yaml, yaml2obj, and yaml-bench.

We are reaching the end of our journey here. At this point, we are done stripping out things.

Let's disable optimizations and do a last run:

cmake   -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_PROJECTS=clang \
        -DLLVM_USE_LINKER=lld \
        -DLLVM_TARGETS_TO_BUILD="X86" \
        -DCLANG_ENABLE_STATIC_ANALYZER=OFF \
        -DCLANG_ENABLE_ARCMT=OFF \
        -DLLVM_BUILD_TOOLS=OFF \
        -DLLVM_BUILD_UTILS=OFF \
        -DCMAKE_CXX_FLAGS_RELEASE="-O0" \
        -GNinja ../llvm 

time ninja
[2880/2880] Linking CXX executable bin/c-index-test

real    5m37.225s
user    253m18.515s
sys     9m2.413s

That's it. Five minutes. Don't try this at home :-)

New releases are available, including ponylang/ponyc 0.41.0 and seantallen-org/lori 0.1.0. Interested in helping maintain ponyup?

If 100 people estimate the time needed to implement a feature, in software, what is the expected variability in the estimates?

Studies of multiple implementations of the same specification suggest that standard deviation of the mean number of lines across implementations is 25% of the mean (based on data from 10 sets of multiple implementations, of various sizes).

The plot below shows lines of code against the number of programs (implementing the 3n+1 problem) containing that many lines (red line is a Normal distribution fitted by eye, code and data):

Might any variability in the estimates for task implementation be the result of individuals estimating their own performance (which is variable)?

To the extent that an estimate is based on a person’s implementation experience, a developer’s past performance will have some impact on their estimate. However, studies have found a great deal of variability between individual estimates and their corresponding performance.

One study asked 14 companies to bid on implementing a system (four were eventually chosen to implement it; see figure 5.2 in my book). The estimated elapsed time varied by a factor of ten. Until the last week this was the only study of this question for which the data was available (and may have been the only such study).

A study by Alhamed and Storer investigated crowd-sourcing of effort estimates, structured by use of planning poker. The crowd were workers on Amazon’s Mechanical Turk, and the tasks estimated came from the issue trackers of JBoss, Apache and Spring Integration (using issues that had been annotated with an estimate and actual time, along with what was considered sufficient detail to make an estimate). An initial set of 419 issues were whittled down to 30, which were made available, one at a time, as a Mechanical Turk task (i.e., only one issue was available to be estimated at any time).

Worker estimates were given using a time-based category (i.e., the values 1, 4, 8, 20, 40, 80), with each value representing a unit of actual time (i.e., one hour, half-day, day, half-week, week and two weeks, respectively).

Analysis of the results from a pilot study were used to build a model that detected estimates considered to be low quality, e.g., providing a poor justification for the estimate. These were excluded from any subsequent iterations.

Of the 506 estimates made, 321 passed the quality check.

Planning poker is an iterative process, with those making estimates in later rounds seeing estimates made in earlier rounds. So estimates made in later rounds are expected to have some correlation with earlier estimates.

Of the 321 quality check passing estimates, 153 were made in the first-round. Most of the 30 issues have 5 first-round estimates each, one has 4 and two have 6.

Workers have to pick one of five possible value as their estimate, with these values being roughly linear on a logarithmic scale, i.e., it is not possible to select an estimate from many possible large values, small values, or intermediate values. Unless most workers pick the same value, the standard deviation is likely to be large. Taking the logarithm of the estimate maps it to a linear scale, and the plot below shows the mean and standard deviation of the log of the estimates for each issue made during the first-round (code+data):

The wide spread in the standard deviations across a spread of mean values may be due to small sample size, or it may be real. The only way to find out is to rerun with larger sample sizes per issue.

Now it has been done once, this study needs to be run lots of times to measure the factors involved in the variability of developer estimates. What would be the impact of asking workers to make hourly estimates (they would not be anchored by experimenter specified values), or shifting the numeric values used for the categories (which probably have an anchoring effect)? Asking for an estimate to fix an issue in a large software system introduces the unknown of all kinds of dependencies, would estimates provided by workers who are already familiar with a project be consistently shifted up/down (compared to estimates made by those not familiar with the project)? The problem of unknown dependencies could be reduced by giving workers self-contained problems to estimate, e.g., the 3n+1 problem.

The crowdsourcing idea is interesting, but I don’t think it will scale, and I don’t see many companies making task specifications publicly available.

To mimic actual usage, research on planning poker (which appears to have non-trivial usage) needs to ensure that the people making the estimates are involved during all iterations. What is needed is a dataset of lots of planning poker estimates. Please let me know if you know of one.

Sooner or later everybody that works with computers will have to deal with software licenses. Newcomers usually assume that software is either open-source (aka free stuff) or proprietary, but this is a very simplistic view of the world and wrong most of the time.

This topic can quickly become complex and small details really matter. You might find yourself using a piece of software in a way that the license does not allow.

There are many types of open-source licenses with different sets of conditions, while you can use some for basically whatever you want, others might impose some limits and/or duties. If you aren’t familiar with the most common options take look at choosealicense.com.

This is also a topic that was recently the source of a certain level of drama in the industry, when companies that usually released their software and source code with a very permissive license opted to change it, in order to protect their work from certain behaviors they viewed as abusive.

In this post I share my current approach regarding the licenses of the computer programs I end up releasing as FOSS (Free and Open Source Software).

Let’s start with libraries, that is, packages of code containing instructions to solve specific problems, aimed to be used by other software developers in their own apps and programs. On this case, my choice is MIT, a very permissive license which allows it to be used for any purpose without creating any other implications for the end result (app/product/service). In my view this is exactly the aim an open source library should have.

The next category is “apps and tools”, these are regular computer programs aimed to be installed by the end user in his computer. For this scenario, my choice is GPLv3. So I’m providing a tool with the source code for free, that the user can use and modify as he sees fit. The only thing I ask for is: if you modify it in any way, to make it better or address a different scenario, please share your changes using the same license.

Finally, the last category is “network applications”, which are computer programs that can be used through the network without having to install them on the local machine. Here I think AGPLv3 is a good compromise, it basically says if the end user modifies the software and let his users access it over the network (so he doesn’t distribute copies of it), he is free to do so, as long as he shares is changes using the same license.

And this is it. I think this is a good enough approach for now (even though I’m certain it isn’t a perfect fit for every scenario). What do you think?

Just-in-time programming is a workflow for creating a program top-down, while a program is running. This is typical in Smalltalk environments like Squeak.

In this post, I want to describe an instruction level variant of this.

I think the best way to describe instruction level just-in-time programming (IL-JIT programming) is by showing how it works. I'll start with the classic Fibonacci example (Rosetta code has many implementations). We'll implement Fibonacci while evaluating fib(3).

Even in a fully distributed organisation, teams are often clustered in timezones to facilitate collaboration. Another model is the deliberate spreading of teams to enable Follow the Sun workflows, which can also improve pager rotations.

In a distributed team, onboarding new team members can be especially difficult. New onboardees lack the context and institutional knowledge required to work effectively in an independent manner. Here are some strategies which I have found ease this process, specifically in the field of software engineering.

Onboarding Buddies

It is standard practice to designate an onboarding buddy for a new onboardee, a person in the same team who can take care of them during the first few weeks, answer questions, and pair a lot. It is not always possible to have this onboarding buddy in the same timezone, and in these cases it is a good idea to have an existing employee in the same timezone as a designated point of contact, regardless of team affiliations.

Making the Most of the Overlap

When onboarding someone in different timezones, everything has to be planned around the overlap in office hours. Start by clearly identifying this overlap. It is most important for the onboardee to be able to onboard and work effectively outside of this overlap.

To optimise the value gotten out of the overlap, it is best spent with synchronous conversations. Other tasks such as code reviews or planning work should be moved outside the overlap unless there is value in doing it synchronously, for example by sharing additional context.

If the overlap is very small, short screencasts can be medium-cost, high-bandwidth method to transfer context asynchronously. These can be recorded and watched outside the overlap, but convey much of the same context as pairing could, albeit without the ability to actually interject questions.

Enabling Self-Directed Learning

To enable the onboardee to learn on the job, there needs to be a pool of work items for them to pick up tasks from. These items need be narrowly scoped and include much more detail and context than they ordinarily would, to reduce the risk of the onboardee getting stuck.

Tasks should be sized “just right,” that is not larger than two or three days, but also not shorter than a single day. Very small tasks encourage the onboardee to start working on several tasks in parallel while waiting for code reviews, increasing cognitive load.

A library of self-directed training material is a must-have for every organisation, distributed or not, but can also serve as a fallback if the onboardee gets stuck on their current task and has to wait for the next overlap.

Just Because You Can Does Not Mean You Should

While asking colleagues for ideas & feedback, a sentiment I have heard more than once was “if you can, avoid scattering your team.” Just because you can spread a team literally around the globe does not mean you should do this. At my current day job we have employees around the globe, but generally try to keep teams within two timezone regions, for example within the North American east coast and Europe, or the west coast and APAC, so that we have an overlap of at least two to three hours.

There was once a young lady named Bella
Who fancied herself at Coachella
She wanted a blog post
Instead got a blog roast
I don’t know how to end this poem wella.

Note: A version of this blog post is available as an interactive Jupyter Notebook hosted on Google Colab.

The Wikidata relation doctoral advisor (P184) links researchers to the advisor or advisors who supervised their Ph.D. Note that coverage is currently a bit spotty. But, it's at least a superset of the data in the Mathematics Genealogy Project (because MGP data was imported into Wikidata), plus some from various other sources, such as parsing Wikipedia infoboxes, and manual additions (I've added quite a bit myself).

A nice thing about Wikidata is that it's queryable with SPARQL. Here is a query that finds all my academic "ancestors":

SELECT ?ancestorLabel WHERE {
  wd:Q65921654 wdt:P184+ ?ancestor.
  ?ancestor wdt:P31 wd:Q5.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Brief explanation:

• Q65921654 is the Wikidata ID for me (Mark J. Nelson).
• P184+ means one or more applications of the doctoral advisor relation.
• ?ancestor is a variable
• We only consider possible ancestors who are (P31) humans (Q5). This shouldn't really be necessary, but I sometimes get spurious results if it's omitted.
• The wikibase:label service adds some magic that grabs the labels for variables: for any variable ?foo, ?fooLabel will be Wikidata's label for that item (in this case in English). That's so we get people's names back instead of Qxxxx codes.

You can edit and run queries like this interactively over at Wikidata, or programmatically from any language that can make HTTP requests.

Academic ancestor tree

That's a flat list of ancestors. But how do they relate? It might be nice to draw out the tree. To do that, we need to save a little more data: who's connected to who. (The query below uses some techniques adapted from this Stackoverflow answer by Joshua Taylor.) Specifically, we want every link in the ancestry tree. In the query below, ?ancestor2 is a direct advisor of ?ancestor1, and ?ancestor1 is either myself, or one of my academic ancestors:

SELECT ?ancestor1Label ?ancestor2Label WHERE {
  wd:Q65921654 wdt:P184* ?ancestor1.
  ?ancestor1 wdt:P184 ?ancestor2.
  ?ancestor1 wdt:P31 wd:Q5.
  ?ancestor2 wdt:P31 wd:Q5.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Either way, we now we have enough information to draw an ancestor tree using graphviz!

Mine ends up being pretty giant, so I won't show it here. One of my advisors, Charles Isbell, has an absolutely huge ancestor tree that trails off into long chains of medieval mathematicians that the Mathematics Genealogy Project has meticulously chronicled. But here's the other half of my academic ancestor tree, the one starting at my advisor Michael Mateas:

Common academic ancestor tree

So far we've been querying ancestors of a specific person. There are of course a lot more ways to slice and dice this big doctoral-advisor graph in Wikidata. Another one I find interesting: do two people have a common ancestor?

Here's one way to pull that out of SPARQL, again borrowing an idea from something Joshua Taylor posted on Stackoverflow. It's a bit hairier than the previous queries.

SELECT ?ancestor1aLabel ?ancestor2aLabel ?ancestor1bLabel ?ancestor2bLabel WHERE {
  # ancestors of the first person leading to a common ancestor (or ancestors)
  wd:Q65921654 wdt:P184* ?ancestor1a.
  ?ancestor1a wdt:P184 ?ancestor2a.
  ?ancestor2a wdt:P184* ?common_ancestor.
  # ancestors of the second person leading to a common ancestor (or ancestors)
  wd:Q105669257 wdt:P184* ?ancestor1b.
  ?ancestor1b wdt:P184 ?ancestor2b.
  ?ancestor2b wdt:P184* ?common_ancestor.
  # stop at the common ancestor(s) rather than retrieving their own ancestors
  FILTER NOT EXISTS {
    wd:Q65921654 wdt:P184* ?intermediate_ancestor.
    wd:Q105669257 wdt:P184* ?intermediate_ancestor.
    ?intermediate_ancestor wdt:P184 ?common_ancestor.
  }
  ?ancestor1a wdt:P31 wd:Q5.
  ?ancestor2a wdt:P31 wd:Q5.
  ?ancestor1b wdt:P31 wd:Q5.
  ?ancestor2b wdt:P31 wd:Q5.
  ?common_ancestor wdt:P31 wd:Q5.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

The result:

We weren't sure we had one, but it turns out that we do have a common academic ancestor, Roger Schank.

Naturally, all these queries depend on the data being in Wikidata. Anyone can add data there, so if you or your advisor are missing (or are in Wikidata but the advisor link isn't there), it's possible to go add it.

* * *

I've written some command-line Python scripts implementing the above queries:

• https://github.com/anadrome/academic-ancestors

For Mosaic, my open source schematic entry and simulation tool, I’m reevaluating the choice of GUI toolkit, and along with that the implementation language. The original plan was to use Rust and GTK, but I’m slowly coming around to the idea that if its feasible performance wise to make a web app, that there is great value in that.

On the one hand IC designers are a conservative bunch and it’s great to have a good desktop app, but on the other hand EDA tools can be notoriously hard to set up and manage to the point where most commercial IC design happens via remote desktop. Having a the option to a web app allows you to have a snappy interface without all the hassle, and as eFabless demonstrated allows people to do IC design without signing an NDA.

I kind of want the option to run as a desktop app or as a web app. So what are my options? I think the most common one is to use Electron, which everyone likes to hate on, but in fact I’m typing up this blogpost in an Electron app and I can’t complain. Another option would be Flutter, which has web and desktop targets. And finally, some desktop GUI toolkits can be ran in a browser, which I’ll almost dismiss right away because they don’t behave like a proper web app.

This last point is worth reiterating on, because people keep suggesting canvas based systems such as Dear ImGui and Makepad. These systems break basic expectations like being able to select text, tab through form fields, search, use onscreen keyboards, screen readers… the list is endless. I will not write software that breaks basic expectations and is inaccessible to some people.

My plan is to make some very simple proof of concepts in various GUI toolkits to see how they stack up. I plan to look at

• Flutter (Not Electron!)
• TypeScript + React
• Elm
• Rust + Yew
• ClojureScript + Reagent
• Kotlin + React

I will consider the following criteria

• Ease of use
• Performance
• Not breaking basic shit
• Churn

By implementing the following tasks that I deem critical for my use as a schematic entry and simulation tool

• Schematic: Draw a bunch of SVGs and lines that can be panned and zoomed.
• Scripting: Interface with an IPython kernel for a minial notebook style interface.
• Plotting: Plot a simulation result with a lot of traces and time points that can be panned and zoomed.

Flutter

On the surface it seems like the dream combo. Instead of a bloated Electron app you get a snappy desktop app that can also render to the web. It’s not all roses though, it’s originally a mobile toolkit, and it kind of shows. The desktop target it still in beta, missing advanced features such as “file picker” and “menu bars”, who needs ‘em right?

Flutter has a nice gallery where you can browse around some example apps. I opened the email app and you can’t select the text in an email. Talk about “breaking basic shit”. To continue on that point, I tried to access the app with a screen reader and on Android I just could not. On Linux I managed to push the “enable accessibility” button, but all the actions in the email app are just “push button clickable” without further description. They do bring up the on-screen keyboard in the compose window and increasing the magnification in Firefox did work, so there are some things they managed not to break.

On my desktop performance seemed good, but on my phone it is super janky. (jank seems to be the favorite word in the Flutter community) Curious to see how performance will hold up with my benchmarks.

Schematic

The code for drawing a hundred mosfets was quite okay. I added the flutter_svg library, added an a mosfet to the assets, and generated a hundred SVG widgets inside a Positioned inside a Stack inside a InteractiveViewer for the pan/zoom action. I don’t particularly care about all the details for the moment, I just want to see how easy it is and how well it performs.

Pretty easy, but honestly it doesn’t perform that well. In release mode, with a 100 mosfets, it feels like it’s dragging along with like 15fps. In profiling mode it instantly crashes. It actually seems faster in debugging mode though?! According to the profiler it’s running at 30fps when I’m panning around.

Plotting

Of course step one is to locate a library. I checked out several. The first was very fancy but doesn’t seem to offer panning and zooming. The next gave a compile error because it’s not null safe and I’m not a savage. The next is a commercial package which is a instant no-go for an open source project. Then I posted on Stack Overflow and moved on. I also had a look at the Discord server but it seems like the #help channel is just noobs like me asking questions with no one giving answers.

Notebook

There are two ways to go about embedding a IPython notebook. One would be to use a webview, which is ruled out because it only supports iOS and Android. The other is to implement it from scratch with Websockets, which is what I will do.

I figured I could use the code_editor package, but was once again foiled by null safety. My understanding is that Flutter 2 does not allow nullable types by default, so packages that have not yet been updated can’t be used with it. I’m sure there are ways to deal with this but for a quick test I just counted my losses and used a plain text field.

The async http and websocket code to connect to the IPython kernel was pleasant, and after a bit of fiddling, I managed to write this simple demo app that once you enter a token and connect, you can execute Python code.

So while I got a good impression of async programming, I wasn’t able to get syntax highlighting to work, and am worried about how I will handle anything other than plaintext output. One of the main powers of notebooks is being able to embed figures and even interactive widgets. Without a Webview to reuse parts of the Jupyter frontend this becomes a difficult proposition.

Summary

Ease of use is quite okay. Installation was smooth, VS Code integration is amazing, hot code reload is sweet, programming in Dart is more or less fine, and working with async websockets was also fine.

Performance was quite disappointing, with the app unable to keep up the frame rate when panning around with 100 SVGs on the screen. I was unable to test the performance of the plotting libraries because I was unable to find any that support interactive panning and zooming.

Breaking basic shit: yes. The fact that you can’t select text in an email app says it all. Accessibility is also not great on the web, and in this beta completely absent when building for Linux.

In terms of churn, the ecosystem seems to be in the middle of a big migration to Flutter 2. Flutter for desktop is also still in beta, so will likely see some breaking changes when it comes to using plugins for file pickers and menu bars. On the other hand, with a significant community and the weight of Google behind it, Flutter is not likely to disappear anytime soon, and is on the whole a lot less fragmented than the JS ecosystem.

React + TypeScript

Basically I blinked and the entire ecosystem changed. Are Grunt and Gulp not a thing anymore? Is Yarn the new npm? I would very much like to not rewrite my whole app every year. But hey, React and TypeScript seem to be still around and going strong so that’s something.

I tried to figure out how to set this up and kinda just got lost in boilerplate. I was about to just go back to Makefiles and script tags when someone pointed out Next.js which allows you to create an app with a simple yarn create next-app. I’m sure it has a bunch of things I don’t need but at least it gets me a good setup with little hassle.

I had a look at nextron for making an Electron app with Next.js but it crashed while downloading the internet. Aint nobody got time for that. I’ll start with a plain web app.

Next.js worked as advertised and had me editing a React app in TypeScript with hot reload in no time. TypeScript is basically just JavaScript with type annotations, so it’s fine but not amazing.

Schematic

The start was pretty great. I made a for loop that inserts a bunch of <image> tags into an <svg> tag, and boom a hundred mosfets on the screen. Then I tried to add scrolling but I forgot how CSS works. I wrapped the SVG in a <div> with overflow: scroll but if I set the size to 100% it becomes as big as the <svg> and if I make it a fixed width, well, it’s a fixed width. I should probably learn flexbox properly (I’m from the float days) but for now I just went with a fixed size.

Holy shit it’s fast. I cranked it up to 5000 mosfets and it’s still smooth as butter. Browsers are amazing.

Now for the zooming part, I figured I could listen for wheel events and if control is pressed, prevent the default behaviour. But no such luck. After a bit of searching I gave up and just used a range input. Hooking up the range input I ran into some JS this weirdness already, oh well… The range input just sets transform: scale(value); which works well enough, but does not maintain scroll position. Proper implementation left as an exercise for my future self.

Even zooming is quite amazingly fast. Only when zoomed all the way out does it struggle a tiny bit with redrawing 5000 mosfets, but it’s super smooth when zoomed in.

Plotting

It seems Plotly is a widely used plotting library for the JS world, so I went with that. It even comes with a nice React wrapper, and can also be used in IPython notebooks. For using with TypeScript it’s helpful to install the @types/react-plotly.js dev dependency to make the type checker happy.

But then I got bitten HARD by my decision to use Next.js for getting up and running quickly. You see, Next.js tries to be all fancy with sever side rendering, and Plotly does NOT like this. I’m not particularly interested in any of the features Next.js offers, I just don’t want to deal with all the boilerplate of setting it up manually. In the end it turns out you have to use some dynamic import loophoole to avoid SSR and the resulting “document not found” errors from Plotly trying to access the DOM.

From there it’s smooth sailing to generate a bunch of data points and plot them. Subplots were a small puzzle, but actually fine. The interactive interface is very good, easy to pan and zoom around.

In terms of performance, I first made 10 traces of 100k points, which works fine. But increasing it to 10 traces of 1M points sends performance completely off a cliff. The interactive cursor updates with a few frames per second, panning and zooming hangs for a second, but once you start dragging it’s actually smooth until you release.

I did a bit of profiling, and it’s looping over the entire 10M points where it could probably be O(log(n)) by using some spatial index. I reported an issue, where the maintainers suggested using scattergl instead, which improved panning and zooming. Then I added hovermode: false to work around the last issue, and that gives very performant results all around.

In short, plotting works very well, but Plotly could use some improvements for very large datasets.

Notebook

I went ahead and asked on the Jupyter mailing list what would be the best way to integrate it. I was pointed to an example that reuses Jupyter components to build a custom notebook. It was also suggested I could embed my entire app as an extension into Jupyter Lab, which is not as crazy as it sounds as Jupyter Lab has grown into much more than just Python notebooks.

For now I’ll just try to follow the example though. I tried with Yarn and that gave errors but then I tried with npm install; npm run build and it worked. Don’t ask me… Npm told me there are 28 vulnerabilities (1 moderate, 27 high), so packages must be upgraded and the churn must continue.

It’s a bit of a different setup though, where the JS is compiled and then served with a Python script that is a subclass of Jupyter server. That takes care of the kernels of course, but doesn’t give you the fancy hot reload. I wonder if could somehow rig hot reloading into the bundled JS served by Python, but that’s for later. For now I just want to see if I can interact with the notebook from the outside.

I found it quite hard to find good documentation on how to interface with the notebook, so I settled for just poking around a bit. The commands.ts file provides some guidance for what sort of actions are available. Call me barbaric, but the easiest way for me to play with the code was to assign the NotebookPanel and NotebookAction to the window object so I can access them from the browser console.

> nb.content.activeCellIndex = 0
0
> nb.content.activeCellIndex = 3
3
> nba.run(nb.content, nb.context.sessionContext)
Promise {<pending>}
> nb.content.activeCell.model.outputs.get(0).data["application/vnd.jupyter.stdout"]
"test\n"

I think that’s sufficient for now to know that it is possible to use Jupyter Lab as plain TypeScript modules and interact with the notebook cells in interesting ways. The actual integration is going to be a bit of a puzzle though.

Summary

This whole setup honestly gave me quite a bit of puzzles, surprises, and headaches. Next.js gave me an easy start with React, TypeScript, and hot module reload, but server side rendering bit me when I tried to use Plotly. I will probably want to move to a pure React setup, but it won’t be enjoyable to set up. HTML makes it very easy to draw arbitrary boxes and media, but on the flip side, making conventional widgets and interactions requires a lot more manual effort. Like how Flutter just has a pan-zoom container, while I had a big struggle to make my SVGs zoom in and out.

Performance was honestly super impressive compared to Flutter. Where Flutter compiled as a native Linux app struggled to pan and zoom 100 mosfets, Firefox just kept going with 5k of them. I wasn’t able to compare plotting performance because Flutter just doesn’t seem to have a suitable plotting library, but the results with Plotly were quite impressive. Granted, I had to switch to a canvas backend and disable hover behavior, but 10M points is no joke.

HTML allows you to select text! Amazing, I know, right? React wins the doesn’t-break-basic-shit badge. Of course, unless you as the author break basic shit. Don’t do that.

Honestly the “churn” item is there because of the JS ecosystem. Somehow they managed to build an ecosystem on top of a language that’s backwards compatible decades into the past, where as soon as you look away for a second your code bitrots away. I hope that by sticking to the tried and true, I can avoid some of that. The big libraries seem to be here to stay.

Elm

For the remaining browser based solutions I will take a more lightweight approach. Obviously all of them inherit the same performance characteristics when it comes to rendering HTML. The plotting and notebook benchmarks are mainly about integrating libraries. So for the browser based solutions I’ll be mainly looking at the strength of the package ecosystem or interop with the JS ecosystem.

And frankly Elm just doesn’t cut it. They take a pretty hardball stance on interop, requiring the package ecosystem to be pure Elm, and only allowing top-level “ports” to talk to JS code. I had a look at plotting/charting libraries in Elm and none of them have been updated in the last 3 years.

Rust + Yew

The main concerns for me here are compile times and once again interop. I will try to plot a thing and see how it goes. In Rust the dominant plotting library seems to be Plotters, but it’s fairly low-level and does not have built in pan/zoom. Instead I’ll try using Plotly again.

Interop in Yew is similar to calling a C function. There are example templates for webpack and parcel, so you can just npm install things. I went with the parcel one, which told me once more there are a billion outdated dependencies and security vulnerabilities. If you want to use the JS ecosystem you have to deal with the JS ecosystem. Obvious but true.

Running the example template is easy enough. It even reloads and recompiles automatically, though it’s not hot module reload, just a page refresh. On this small example, compile times are short, not sure it’ll stay that way. A tendency of JSX-like Rust macros is that they require an ever-increasing recursion limit and result in slow compile times.

After several hours of trying to integrate Plotly by fiddling with annotations, I can confidently say: screw that.

ClojureScript

I used to do a lot of Clojure some years ago but never really got into ClojureScript. Clojure has always had good interop, and a functional programming approach without static typing. Not having static types reduces friction, but makes debugging harder.

In particular, Clojure has mostly useful default behaviour for handing nil rather than forcing you to handle Maybe results, until you get a null pointer exception and you’re in for a mystery hunt. The tooling seems not as mature as some other languages. There seem to be two major Clojure/ClojureScript VS Code extensions, neither of which supports debugging ClojureScript. The on I tried complained about me using VIM mode…

I’m once more just going to try to integrate Plotly and see how it goes.

The official website lists several abandoned projects which is not a good look. I asked about it on Slack and and they maintain that once software is done it requires no updates ever. It’s perfect, don’t touch. Which is honestly kinda refreshing compared to JS land where you have to burn everything to the ground perpetually. But you also have to be realistic about it, if a project has a bunch of open issues and PRs it’s maybe not as perfect as you say.

At first I went with Leiningen and Figwheel based on my experience from a few years back, but along the way learned from friendly community members that these days tools-deps is the standard way to bundle apps and shadow-cljs is the new hotness when it comes to ClojureScript tooling. Both Figwheel and shadow-cljs offer a nicely integrated development environment with state preserving hot code reload and a REPL into the running app but shadow-cljs has better integration with npm.

I created a new project with npx create-cljs-project cljsplotly and from there added Reagent (a react wrapper) and copied some example code. Compiling and running the project is as simple as npx shadow-cljs watch frontend, and its not even like there is a ton of boilerplate, just the simple config files for npm and shadow-cljs. Then I added react-plotly.js as a regular npm dependency. After that I was able to import the library with ["react-plotly.js$default" :as PlotComponent], and define it as a reagent component with (def plot (r/adapt-react-class PlotComponent)), which can then simply be used like so [plot {:data (clj->js [{:x [1 2 3 4], :y [1 2 3 4], :type "scatter"}])}].

Kotlin

I wasn’t even planning on covering this option, but it popped up along the way and I figured I might as well check it out. I mainly know Kotlin as a very pragmatic “better java”, and have used it once in this capacity. Apparently it now also supports native and JS targets, with what looks to be favorable options.

Right away the first struggle is the tooling. The setup page only refers to IntelliJ with some mentions of the actual Gradle stuff. I’m sure you can use plain Gradle to do it, but for getting started, not ideal. It seems that the VS Code plugins are comparatively basic. So I guess I’m installing IntelliJ and figure out another workflow later if needed.

Setting up a project was easy. From there things got a bit confusing. There is a debugger, but it debugs the JVM server process. You can use the browser debugger, but sourcemaps aren’t working for me. (this turns out to be because you have to us the LEGACY compiler for that) It has hot code reload, but not state preserving, it just reloads the page.

Installing an npm dependency is pretty easy, but then you run into the exact same problems as with Rust where you have to add some externs with magic annotations to declare the JS interface. The tutorial for adding a React component has the following helpful information on the matter

Because JavaScript imports/exports isn’t the simplest topic, it can sometimes be tricky to find the correct combination between annotations to get the Kotlin compiler on the same page as us.

Again, I tried for a good while and asked on Slack. A day later a suggestion came in and after a bith more back and forth I managed to get it to work. I still do not understand why or how, and how I would go about future JS interactions. I also had a bunch of trouble just making a JS object to pass to the component. In short the interop story isn’t very obvious.

Conclusion

I had high hopes for Flutter, but it was pretty disappointing across the board. Poor performance, poor accessibility, poor libraries.

Browser performance when drawing 5k SVGs was amazing, and so was Plotly performance when drawing 10M points, once I used the right settings. So the web is definitely the way forward.

React is good, TypeScript is a minor improvement over JavaScript, but honestly the worst part is the tooling and quirky browser APIs. If I have to, I’m sure I can make it work. Not sure it’ll be the most enjoyable thing in the world though.

I dismissed Elm for lack of libraries. Rust and Kotlin seemed interesting, but their hot code reload does not maintain state, there is no REPL, and I had a lot of trouble making JS interop work. I’m sure they work well for some people, but I just could not make them work well for me.

ClojureScript was a very pleasant homecoming. I had kind of stopped using Clojure because a lot of what I do these days is either scientific Python/Julia scripts or embedded C/C++/Rust, but it’s a nice language. The community was very helpful and welcoming, with some old friends still around. It has by far the nicest toolchain and workflow of all the compile-to-js systems I tried, and the only one that can come close to TypeScript in JS interop.

So in the end I think the competition is between ClojureScript and TypeScript. I had a lot more fun with ClojureScript, but of course it’s a less widely used language. TypeScript, for better or worse, is more mainstream and closer to JavaScript.

Version 0.40.0 of ponylang/ponyc has been released! Also, the ‘Add String.add iso proposal’ RFC is now in final comment period.

The European Commission is updating the EU Machinery Directive, which covers the sale of machinery products within the EU. The updates include wording to deal with intelligent robots, and what the commission calls AI software (contained in machinery products).

The purpose of the initiative is to: “… (i) ensuring a high level of safety and protection for users of machinery and other people exposed to it; and (ii) establishing a high level of trust in digital innovative technologies for consumers and users, …”

What is AI software, and how is it different from non-AI software?

Answering these questions requires knowing what is, and is not, AI. The EU defines Artificial Intelligence as:

• ‘AI system’ means a system that is either software-based or embedded in hardware devices, and that displays behaviour simulating intelligence by, inter alia, collecting and processing data, analysing and interpreting its environment, and by taking action, with some degree of autonomy, to achieve specific goals;
• ‘autonomous’ means an AI system that operates by interpreting certain input, and by using a set of predetermined instructions, without being limited to such instructions, despite the system’s behaviour being constrained by and targeted at fulfilling the goal it was given and other relevant design choices made by its developer;

‘Simulating intelligence’ sounds reasonable, but actually just moves the problem on, to defining what is, or is not, intelligence. If intelligence is judged on an activity by activity bases, will self-driving cars be required to have the avoidance skills of a fly, while other activities might have to be on par with those of birds? There is a commission working document that defines: “Autonomous AI, or artificial super intelligence (ASI), is where AI surpasses human intelligence across all fields.”

The ‘autonomous’ component of the definition is so broad that it covers a wide range of programs that are not currently considered to be AI based.

The impact of the proposed update is that machinery products containing AI software are going to incur expensive conformance costs, which products containing non-AI software won’t have to pay.

Today it does not cost companies to claim that their systems are AI based. This will obviously change when a significant cost is involved. There is a parallel here with companies that used to claim that their beauty products provided medical benefits; the Federal Food and Drug Administration started requiring companies making such claims to submit their products to the new drug approval process (which is hideously expensive), companies switched to claiming their products provided “… the appearance of …”.

How are vendors likely to respond to the much higher costs involved in selling products that are considered to contain ‘AI software’?

Those involved in the development of products labelled as ‘safety critical’ try to prevent costs escalating by minimizing the amount of software treated as ‘safety critical’. Some of the arguments made for why some software is/is not considered safety critical can appear contrived (at least to me). It will be entertaining watching vendors, who once shouted “our products are AI based”, switching to arguing that only a tiny proportion of the code is actually AI based.

A mega-corp interested in having their ‘AI software’ adopted as an industry standard could fund the work necessary for the library/tool to be compliant with the EU directives. The cost of initial compliance might be within reach of smaller companies, but the cost of maintaining compliance as the product evolves is something that only a large company is likely to be able to afford.

The EU’s updating of its machinery directive is the first step towards formalising a legal definition of intelligence. Many years from now there will be a legal case that creates what later generation will consider to be the first legally accepted definition.

Apr 2020 | Apr 2019 | Apr 2018 | Apr 2017 | Apr 2016 | Apr 2015 | Apr 2014 | Apr 2013 | Apr 2012 | Apr 2011 | Apr 2010 | Apr 2009

Remember 2020? Yeah, I know. Well here is the list you’ve been waiting for.

Previous lists are here: 2019, 2018 part 1, 2018 part 2, 2017, 2016 and 2015

As usual: a you get couple of words and a YouTube video. All killer, no filler. I promise.

Note: I only noticed when I compiled the list that a lot of songs are about death…

Let’s kick it off with Molchat Doma. What’s that? Glad you asked! Molchat Doma is russian doomer music and the perfect soundtrack for 2020. Their album Etazhi is probably my most played album of 2020. It’s a rare album in its complete tonal consistency and overall freshness. This next track Volny is one of the highlights. The video alone should receive all sorts of prizes. Enjoy, and don’t worry there is more Molchat Doma on the list.

Small song, country feel. 272 views. Steph Copeland ft. Chris Wong. When I lay my head to rest, there will grow a garden from my chest.

This next song is quite the trip! If the first few notes don’t grab you by the throat, maybe Ben’s performance will. This is actually three songs into one. Take your time, this is a very intense performance. And do stick around for what happens after four minutes.

I like me some Bruce and you know it. Bruce is like a machine and delivers, always. And the title track for the docu/album is as Springsteen as they come. Banger!

While we’re on the subject of old rockers with guitars, let me just share this rendition of an old folk song. I can picture this song to be the last song in every Irish pub before closing time (you know, pre-covid etc.). You cannot not love this.

Ok last old guy with a guitar. I can’t quite put my finger on it but the remaster of Pocahontas is much better than the original?? It’s the tempo, maybe. I don’t know. Anyway here is Neil Young singing he wished he had a thousand pelts so he could sleep with Pocahontas (?!), read the wiki if you want to know more. It’s quite the trip.

Lets pick up the tempo. I am a sucker for urgent sounding songs like this. The pacing here is terrific. And only 278 views. How?! This song is STRAIGHT FIRE. Time! Let the room untouched!

Talking about urgency, Fontaines D.C. know everything about that. The band with multiple entries on last year’s list just keeps producing one great track after. This band is everything a band should be. And if this next track doesn’t convince you, I am sorry for you.

One of the few women on the list. Katie Pruitt. Poppy but she has skills. I like it! The Fleetwood Mac is strong in this one.

Back to the UK. If you are still wondering where Arctic Monkeys got their inspiration, look no further. I was thoroughly surprised when I found out this song actually predates the Arctic Monkeys because this song is MOST definitely the inspiration for Teddy Picker. Not the other way around. Alex Turner you cheeky bastard!

Let’s stay in the UK shall we. Bands like the Enemy, with clear tracebacks to the Jam can only spring from the UK. A quintessentially British song.

Here is a very special song by Empress Camelian. This song has everything you would want from a laptop artist. Or any artist for that matter. And this is exactly the kind of song I hope to find when I scour the internet. Intimate and melodic. You’ll be surprised.

Ok I lied, here is another old guy with a guitar. Bob Dylan. Of course. Dylan released his 39th studio album in 2020 Rough and Rowdy Ways. And a few alternate takes and reissues surfaced. Good stuff for fans of the man that’ll turn 80 (!) soon. But the one song that had everybody talking, because of its length and content, was Murder Most Foul. There is a lot to unpack here, because not only is it a Bob Dylan song, it’s also almost 17 minutes. Clear your schedule please.

Ok so this is a Dutch track, but I can’t not include this. These boys hit all the right notes. The guitar is definitely a callback to best Dutch popbands of the 80s. But don’t be fooled, there is more here than what it looks like, and it already looks just terrific.

Paul Simon is and will always be one of my favorite songwriters. This song was written just after Nixon got elected and is probably one of his most political songs. And I only recently learned what makes the melody for American Tune so great. Can you hear it?

I am sorry what’s that? You want sci-fi m83-ish sounds? Say no more.

I promised a little bit more of Molchat Doma. Well here it is, for the acquired taste palette. This track is just completely bonkers, but in a good way, image what that sounds like live.

Oh, Bright Eyes how I love thee. This guy can hardly do wrong and just keeps churning out instant classics. And this is a fantastic live rendition of a fantastic song. Handsdown the best Covid performance I saw this year. For the daily double bonus: can you tell who the drummer is?

This track was removed from Youtube. So I’ll share another rather similar one.

You know what, that is a GREAT question Jason Isbell?! What have *I* done.

Declan McKenna was on my list in 2015, when he was just a 16 year old kid, because the talent was so obviously apparent. This next track — 5 years on — proves that his talent has not been wasted. He already has quite the following and is on his way to becoming a bona fide popstar. This is his 2020 hit. Poppy, sure. Delicious, yes. You can’t hide talent like this.

Save the best for last. This is hands down the best song of 2020 for me (Molchat Doma takes home the album award). Wolf Parade. The video, well it’s… something and the artist went ALL out. The song itself is a grower. And I admit It didn’t click on my first listen. But hidden in this song is something quite remarkable. There is no greater build up to an outro break in all of 2020 — or maybe even the decade — than the 3:28 mark in this video. Just when you start wondering where this song is going, the essence of this track lays it soul bare. It is astounding and it’s the absolute pinnacle of an already absolutely incredible song. One for the ages. Turn it up to 11. Please do also check out the some of the live versions. The live performances really punctuate what this song is about. Drums, keys, guitar and a singer who doesn’t hold back. A delight to watch. This is what it is about. To say that he loves you, would be unfair towards you.

The post 2020 Music Discoveries appeared first on Jan van den Berg.

• Internet: Medium For Communication, Medium For Narrative Control
• Conclusion & Bibliography

• Introduction
• Part 1: The Artifacts And Spaces
  In this part we'll describe the important artifacts and places. Going over these essential, but basic, pieces is mandatory to understand how they come into play as tools.
  • Defining and Understanding Propaganda, Influence, And Persuasion
  • Social media — The Democratization of Speech
  • Data & Metadata: Personal Data & Models
  • Memes & Cults
• Part 2: The Actors and Incentives
  In this part we'll go over how the previous elements are put into work by the different actors, who these actors are, what are their incentives, and the new dynamics.
  • New Economies: Information Economy, Attention Economy, Data Economy, Surveillance Economy
  • Internet, In Between The Corporate, Private, And Public Spheres
  • State Actors: PSYOP, Narrative Warfare, And Weaponized Tech
• Part 3: Biases & Self
  In this part we'll try to understand why we are prone to manipulation, why they work so effectively or not on us, and who is subject to them.
  • Cognitive Biases
  • Cultural Ambiguities & Insecurities
• Part 4: The Big Picture
  In this part we'll put forward the reasons why we should care about what is happening in the online sphere. Why it's important to pay attention to it and the effects it could have at the scale of societies, and individuals. This part will attempt to give the bigger picture of the situation.
  • The Formatted Individual
  • Truth & Trust Crises
  • Mass Hypnosis or Mass Psychosis
• Part 5: Adapting
  In this concluding part we'll go over the multiple solutions that have been proposed or tried to counter the negative aspects of the internet.
  • Free Market, Let It Solve Itself
  • Legality, Transparency, Accountability, The Nations Intervene
  • Technical Solutions, Wars and Patches
  • Education, Literacy, and Reality
• Conclusion & Bibliography

Our mini internet study has come to an end.
In this series we’ve seen the new artifacts and spaces introduced by the internet, the actors using them, from the new economies, to netizens in between, to state actors. We also reflected and tried to understand why we can be susceptible to biases and why we have so much difficulties with online interactions. Next we’ve looked at the big picture by diving into subjects such as paralysis, neoliberalism, the truth and trust crises, and a future glance at mass hypnosis and psychosis. Finally, in the last part of the series, we’ve seen four type of solutions: market and economy or laissez-faire, legal path with governments being involved for transparency and accountability, technical software solutions, and web literacy as education and maturity to learn to live in the information society — a post-modern or meta-modern society.

I hope you’ve learned as much as I did during this series. We’ve broadly covered quite a lot of topics, from sociology, psychology, computer science, art, memetics, history, warfare studies, politics, and more.

Let me know if it has been as helpful as it has been for me.

This series can also be found in PDF booklet format.

Bibliography

• 1% rule
• 1957 High Schoolers Debate- Nigeria’s Boniface Offokaja, Ethiopia, Ghana and South Africa.
• 1958 High school exchange students - Ghana, India, France, Israel. “American High Schools” pt 2.
• 4 myths about open source in government
• 533 million Facebook users’ phone numbers and personal data have been leaked online
• 6 ways technology has changed the mindset of modern employees forever!
• Active participants and lurkers in online discussion groups: an exploratory analysis of focus group interviews and observation
• A Day in the Life of Your Data
• Adoption of free and open-source software by public institutions
• Advertising Models vs. Subscription Models
• A huge database of Facebook users’ phone numbers found online
• Algorithms, bots, and political communication in the US 2016 election: The challenge of automated political communication for election law and administration
• All Hail the Meme, The New King of Political Communication
• Analysis Paralysis (Wikipedia)
• Apple, Its Control Over the iPhone, The Internet, And The Metaverse
• Apple says App Tracking Transparency feature will launch in ‘early spring’ with iOS 14 update
• Apple’s cooperation with authoritarian governments
• A Practical Guide to Data Privacy Laws by Country
• Are We Automating Racism?
• A study on Internet addiction and its relation to psychopathology and self-esteem among college students. Industrial Psychiatry Journal. 2018;27(1):61-66. doi:10.4103/ipj.ipj_61_17
• Attention economy (Wikipedia)
• Attention management (Wikipedia)
• Australia Passes Law Making Facebook, Google Pay for News
• Banner Ads Considered Harmful
• BBC Online Briefing Spring 2012: The Participation Choice
• Big Data useful for mass surveillance (Wikipedia)
• BIG DATA WHAT IS IT, HOW IS IT COLLECTED AND HOW MIGHT LIFE INSURERS USE IT? - The Actuary Magazine
• Black Propaganda (Wikipedia)
• Booking.com fined €475,000 for reporting data breach too late
• Brave New World
• Bringing it all together
• Cambridge Analytica’s parent pleads guilty to breaking UK data law
• Cancel culture
• Center For Media Literacy
• Charles Taylor’s The Ethics of Authenticity (Chapters 1-5)
• Charles Taylor’s the malaise of modernity
• Charting a course towards a more privacy-first web
• Citizen Journalism (Wikipedia)
• Communication studies — Wikipedia
• Countering Disinformation: Russia’s Infowar in Ukraine
• Counterpropaganda (Wikipedia)
• Cross Culture Frustration
• Cross Culture Frustration.
• Culture war
• Cyberpolitics
• Dark Pattern
• Dark Patterns after the GDPR: Scraping Consent Pop-ups and Demonstrating their Influence
• Dark Pattern (Wikipedia)
• Data Protection Act 2018
• Data Protection Commissioner (DPC)
• DEFENCE STRATEGIC COMMUNICATIONS Winter 2015 - The official journal of the NATO Strategic Communications Centre of Excellence - IT’S TIME TO EMBRACE MEMETIC WARFARE - Jeff Giesea
• Democracy of the Gullible 2020 DW Documentary
• Democracy of the Gullible 2020 DW Documentary (YT)
• Despotism 1946
• Different People Use the Internet Differently
• DigiD
• Digital Cosmopolitans — Ethan Zuckerman — Talks at Google
• Digital Identity
• Digital Identity in China: Today & Tomorrow
• Digital Identity: Towards Shared Principles for Public and Private Sector Cooperation
• Digital literacy in the classroom. How important is it?
• Digital news consumers unlikely to pay for content and increasingly block ads
• Disinformation and ‘fake news’: Interim Report
• DoD Open Source Software (OSS) FAQ
• EFF - Electronic Frontier Foundation
• EFF (Wikipedia)
• Electronic identification
• Electronics right to repair
• Estonian Electronic Identity Card and Its Security Challenges
• Ethical anti-design, or designing products that people can’t get addicted to.
• Ethics of the Attention Economy: The Problem of Social Media Addiction
• Exclusive: PayPal launches crypto checkout service
• Exclusive: Visa moves to allow payment settlements using cryptocurrency
• Exploring the Utility of Memes for U.S. Government Influence Campaigns
• Facebook hopes tiny labels on posts will stop users confusing satire with reality
• Facebook’s data-sharing deals exposed
• Facebook’s GDPR consent bypass via a “contractual advertisement duty” before Austrian Supreme Court, with high potential for a reference to the CJEU
• Facebook test adds contextual ‘labels’ to popular pages
• Facebook v Apple: the looming showdown over data tracking and privacy
• factcheck.org
• Factfulness
• Find Out Where Apple, Facebook, Google, Twitter and Other Tech Giants Are Sending Your Data
• Flarum Forums made simple
• Forget the new iPhones: Apple’s best product is now privacy
• Free Speech Is Not the Same As Free Reach
• FSF - The Free Software Foundation (FSF)
• Gatekeeping in Online Communities
• GDPR
• GDPR (Wikipedia)
• George Orwell, Aldous Huxley : “1984” ou “Le meilleur des mondes” ?
• GlobalPlatform Root of Trust
• Glue Comic
• GNU
• Google Is Testing Its Controversial New Ad Targeting Tech in Millions of Browsers. Here’s What We Know
• Google Phases Out Use of Apple IDFA as New iOS 14 Ad Tracking Notifications Are Set for Early Spring
• Google misled consumers about the collection and use of location data
• Google says it’s confident about alternatives to browser cookies that track you for ads
• Google’s top security teams unilaterally shut down a counterterrorism operation
• Google to Stop Selling Ads Based on Your Specific Web Browsing
• GSMA Identity programme
• GSMA IoT SAFE
• Hardware Security Module
• Hashtag activism
• HBR, 2010, The Transparent Supply Chain
• Hidden Hope For Control
• Hitting the Books: How biased AI can hurt users or boost a business’s bottom line
• homophily
• Honest Ads Act — USA
• How Estonia is Pioneering the Digital Identity Space
• How Facebook Incubated the Insurrection
• How memes are becoming the new frontier of information warfare
• How memes got weaponized: A short history
• How much of the internet is fake
• How Much of the Internet Is Fake? Turns Out, a Lot of It, Actually.
• How Neoliberal Policy Shaped the Internet—and What to Do About It Now
• How PR Sold the War in the Persian Gulf
• How The Biggest Tech Companies Spent Half A Billion Dollars Lobbying Congress
• How The Subscription Economy Is Disrupting The Traditional Business Model
• How to control people’s thoughts with word
• How to have better arguments online
• How to overcome Phone Addiction
• How To Spot Fake News
• How to Teach Digital Literacy Skills
• How U.S. media lost the trust of the public
• HOW WILL WECHAT ELECTRONIC ID SYSTEM WORK?
• Identity in a Multi-polar World and China’s 2035 Standards
• Identity management
• I Have Come to Bury Ayn Rand
• I’m Offended: How The Social Justice Warriors Made Thinking Dangerous
• Inceptionism: Going Deeper into Neural Networks
• In Defense Of Social Justice
• India’s billion digital identities
• India’s new intermediary liability and digital media regulations will harm the open internet
• India Targets Climate Activists With the Help of Big Tech
• Information ethics in the context of information Literacy
• Information overload - Wikipedia
• Information pollution (Wikipedia)
• Information Revolution (Wikipedia)
• Information Society (Wikipedia)
• In Over our heads (The mental demands of modern society)
• Inside Aadhaar, India’s massive digital identity program
• Inside a viral website
• Intellectual dark web
• Internet culture
• Internet meme and Political Discourse: A study on the impact of internet meme as a tool in communicating political satire
• Introduction to Secure Elements
• Is a Mass Psychosis the Greatest Threat to Humanity?
• Is China The Roadmap — Or Red Flag — For Digital ID Verification?
• Is Hypnosis Real? Here’s What Science Says
• Is India’s digital ID system, Aadhaar, a tech solution for a socio-economic problem?
• Is Total Mind Control Possible?
• Is your government requesting user data from tech giants?
• It’s Not Cancel Culture — It’s A Platform Failure.
• It’s time to embrace memetic warfare
• Just Don’t Call It Privacy
• Keeping Track of Your Things
• KEY DRIVERS OF SUBSCRIPTION-BASED ECONOMY
• Lack of competition in ad tech affecting publishers, advertisers and consumers
• La fabrique de l’ignorance — ARTE
• La paresse: avantage évolutif ou péché capital ? - Dans la tête d’un paresseux
• L’homme formaté, manipulations commerciales, médiatiques et professionnelles - Hacking social
• L’homme formaté, manipulations commerciales, médiatiques et professionnelles
• Liberté de réception
• List of cognitive biases (Wikipedia)
• Living Like It’s 99: No Social Media, No Smartphone
• Loot boxes linked to problem gambling in new research
• Lurker
• Maslow’s hierarchy of needs (Wikipedia)
• Mass psychogenic illness
• Mass Surveillance in China
• Mass surveillance in the United Kingdom
• Mass Surveillance in the United Kingdom
• Meaningness
• Measures Governments Can Use to Promote Free Software
• Memes in Digital Culture
• Memes: Persuasive Political Warfare
• Memetic Engineering (Wikipedia)
• MEMETICS: A DANGEROUS IDEA
• MEMETICS — A GROWTH INDUSTRY IN US MILITARY OPERATIONS
• Memetics (Wikipedia)
• Memetic warfare (Wikipedia)
• Micropayments can unlock “an entire economy” between ads and subscriptions
• Microtransaction
• Mind Reading and Mind Control Technologies Are Coming
• Minister hints at UK departure from GDPR
• MIPB April-June 2010 issue - Memetic Warfare: The Future of War by First Lieutenant Brian J. Hancock
• Mitigating Absolutist Thinking
• Modernizing Canada’s Privacy Act
• Most of What You Read on the Internet is Written by Insane People
• Muslims reel over a prayer app that sold user data: ‘A betrayal from within our own community’
• My Phone Was Spying on Me, so I Tracked Down the Surveillants
• Neoliberalism (Wikipedia)
• Neuroscience—and the new weapons of the mind
• New China Data Privacy Standard Looks More Far-Reaching than GDPR
• New human rights to protect against ‘mind hacking’ and brain data theft proposed
• New media
• Nineteen Eighty-Four: A Novel
• No Gates, No Keepers — The technology industry is hurt at every level by toxic gatekeeping
• NRCFOSS
• Nuances Of Aadhaar: India’s Digital Identity, Identification System And ID
• NZGOAL
• Online Privacy Law: France
• On Mob Justice in the Tech Industry
• ON THE NATURE OF THE INTERNET — Leslie Daigle
• On the Origins of Memes by Means of Fringe Web Communities
• Open access (Wikipedia)
• Open access in New Zealand
• Open Source for Government
• Organized Persuasive Communication — A new conceptual framework for research on public relations, propaganda and promotional culture
• Organized Persuasive Communication: A new conceptual framework for research on public relations, propaganda and promotional culture
• Others in Mind: Social Origins of Self-Consciousness
• Passion Economy — What is the passion economy?
• PAYING FOR PRIVACY AND THE PERSONAL DATA ECONOMY
• Percolation Theory
• Personal Data Protection Bill, 2019
• Persuasion Basics - From Psychology Today
• Persuasive Technology (Wikipedia)
• Platforms Must Pay for Their Role in the Insurrection
• POLITICAL MEDIA MEMES’ PERSUASION AND FUNCTIONING IN SOCIAL MEDIA
• Privacy Act (Canada)
• Privacy activist Max Schrems claims Google Advertising ID on Android is unlawful
• Privacy as a luxury commodity
• Privacy as a product: The argument for trading your personal data to get a discount on a car
• Privacy as a right or as a commodity in the online world: the limits of regulatory reform and self-regulation
• Product Placement (Wikipedia)
• Promotional Culture – Critical Outlook
• Propaganda and Persuasion (Fourth Edition) — What Is Propaganda, and How Does It Differ From Persuasion
• Propaganda, Disinformation, & Other Influence Efforts: The Modern Information Environment
• Propaganda (Wikipedia)
• Psychic Life and the Power of Words
• Psychological Operations: The Need to Understand the Psychological Plane of Warfare
• Psychological operations (United States) (PSYOP)
• Psychological Warfare
• Racial Discrimination in Face Recognition Technology
• Reading in the Age of Constant Distraction
• Re-directing socialist persuasion through affective reiteration: a discourse analysis of ‘Socialist Memes’ on the Chinese internet
• Requests to delist content under European privacy law
• Right to Internet access
• Right to Repair Europe
• Rise of Subscriptions and the Fall of Advertising
• Schools Teach ‘Cyber Hygiene’ to Combat Phishing, Identity Theft
• Screw it, I’ll host it myself
• Self-inflicted injuries surge among tween and early teen girls
• Self-sovereign identity
• Sex, Drugs, and Cults. An evolutionary psychology perspective on why and how cult memes get a drug-like hold on people, and what might be done to mitigate the effects
• Share of U.S. adults using social media, including Facebook, is mostly unchanged since 2018
• Sheryl Sandberg and Top Facebook Execs Silenced an Enemy of Turkey to Prevent a Hit to the Company’s Business
• Silicon Valley’s POWER GRAB - Can Big Tech Be Broken Up?
• SIM, eSIM and Secure Elements
• Singer’s Six Conditions of Mind Control
• Slacktivism
• Smart doorbell
• Snopes
• Snowflake (slang)
• Social justice warrior
• Social Justice Warriors and Narcissism—What Is the Connection?
• Social Media and Social Problems: A Complex Link
• Social media as a public utility
• Social media as a public utility (Wikipedia)
• Social media as a utility
• Social Media Marketing (Wikipedia)
• Social Media, News Consumption, and Polarization: Evidence from a Field Experiment
• Spin (propaganda) (Wikipedia)
• STATE-SPONSORED TROLLING How Governments Are Deploying Disinformation as Part of Broader Digital Harassment Campaigns
• Strategic Influence — Public Diplomacy, Counterpropaganda, And Political Warfare
• SUBSCRIPTION ECONOMY: BUSINESS PERSPECTIVE
• Subscription economy to get a real boost from advertising fiasco
• Sunstein on the Internet and Political Polarization
• Surrealism Museums
• Surveillance Capitalism (Wikipedia)
• Tech Companies Are Profiling Us From Before Birth
• Tech companies use “persuasive design” to get us hooked. Psychologists say it’s unethical.
• Ted Chiang: Fears of Technology Are Fears of Capitalism
• THE AFFECT AND EFFECT OF INTERNET MEMES:ASSESSING PERCEPTIONS AND INFLUENCE OF ONLINE USER-GENERATED POLITICAL DISCOURSE AS MEDIA
• THE BIG POTENTIAL OF BIG DATA - Forbes Insights
• THE BIOLOGY OF DISINFORMATION — memes, media viruses, and cultural inoculation - IFTF
• The Digital Citizenship Curriculum: Digital Literacy, Cyber Hygiene and More
• The digital language divide — How does the language you speak shape your experience of the internet?
• The digital language divide
• The Dream of Neural Networks
• The Ethics of Authenticity
• The Future of Group Messaging
• The future of your privacy: universal right, luxury, commodity or utility?
• The Hitchhiker’s Guide to Online Anonymity
• The Impact of the Internet on Society: A Global Perspective
• The Internet and Diversity
• The internet didn’t kill counterculture—you just won’t find it on Instagram
• The Internet Way of Networking: Defining the critical properties of the Internet
• The internet will continue to make life better
• The Little-Known Data Broker Industry Is Spending Big Bucks Lobbying Congress
• The Mass Psychosis and the Demons of Dostoevsky
• The Meme Machine
• The Micropayment Economy: What It Is, and Why It Matters
• The Nature and Content of a New-Generation War
• The network of fake foreign media
• The ongoing rise of big tech
• The “online brain”: how the Internet may be changing our cognition
• The People Behind Online Trolling, Shaming & Cancel Culture
• The Psychology Behind Social Media Lurkers
• The Righteous Mind — Jonathan Haidt
• The Self, Metaperceptions, and Self-Transformation
• The Social Media Mindset
• The Strange World of Social Justice Warriors
• The Subscription Economy: A Win-Win for Brands and Affiliates
• The subscription economy: how to focus your business model on subscribers, not just products
• The Tech Industry’s War on Kids
• The Threat Of Mind Control, Psychotronic Weapons
• The Trivium and “The Semiotic Iceberg”: A Paradigm for Reading and Responding to Texts
• The “we live in a society” meme, explained
• The Whole Web Pays For Google And Facebook To Be Free
• The world’s most valuable resource is no longer oil, but data
• This is what happens when ICE asks Google for your user information
• Time to regulate AI that interprets human emotions
• Total war (Wikipedia)
• Towards a more sustainable single market for business and consumers (2020/2021(INI))
• TUTORIAL: MILITARY MEMETICS — Dr. Robert Finkelstein
• TUTORIAL: MILITARY MEMETICS — Dr. Robert Finkelstein (pdf)
• UICC
• UK Government Service Design Manual
• Understanding Max Weber’s ‘Iron Cage’
• Using Memes To Improve Climate Change Communication
• U.S. Supreme Court rebuffs Facebook appeal in user tracking lawsuit
• US to require would-be immigrants to turn over social media handles
• Virtue signalling
• Viruses of the Mind
• ‘Weaponized Ad Technology’: Facebook’s Moneymaker Gets a Critical Eye
• Web Literacy
• We can have democracy or we can have Facebook, but we can’t have both
• Welcome To The Passion Economy
• What Does Privacy Really Mean Under Surveillance Capitalism?
• What Estonia’s digital ID scheme can teach Europe
• What is China’s Social Credit System and What Does it Mean for Online Identity?
• What is hypnosis and how might it work?
• What Is Propaganda, and How Does It Differ From Persuasion? Propaganda and Persuasion (Fourth Edition)
• What is Social Cooling?
• Whatsapp and the domestication of users
• What’s Data Privacy Law In Your Country?
• What’s Going on With France’s Online Hate Speech Law?
• Who Controls The Media? You’d Be Surprised
• Who runs the world… Big Tech?
• Why Is Facebook Launching An All-Out War On Apple’s Upcoming iPhone Update?
• Why popular YouTubers are building their own sites
• Why You Should Use It to Protect Your Privacy, Forbes
• New IP
• Woke
• The Narrative and Its Discontents
• Above the Narrative
• Confirmation Bias in Action
• World War Meme
• You Can’t Censor Away Extremism (or Any Other Problem)
• Your Addiction to Outrage is Ruining Your Life
• Zero Data App
• “Discussion Paper: An Analysis of the ‘New IP’ Proposal to the ITU-T.” Internet Society, 3 June 2020, https://www.internetsociety.org/resources/doc/2020/discussion-paper-an-analysis-of-the-new-ip-proposal-to-the-itu-t/.
• Factfulness by Rossling
• Floridi, Luciano. The onlife manifesto: Being human in a hyperconnected era. Springer Nature, 2015.
• Brunton, Finn. “Constitutive interference: Spam and online communities.” Representations 117.1 (2012): 30-58.
• Immunity to change: How to overcome it and unlock the potential in yourself and your organization by Robert Kegan & Lisa Laskow Lahey
• In Over our heads (The mental demands of modern society) by Robert Kegan
• Others in Mind: Social Origins of Self-Consciousness by Philippe Rochat
• Serrano CC, Muñoz SJF, Brusca I. A Multivariate Study of Internet Use and the Digital Divide*. Social Science Quarterly (Wiley-Blackwell). 2018;99(4):1409-1425. doi:10.1111/ssqu.12504
• Stavropoulos V, Griffiths MD, Burleigh TL, Kuss DJ, Doh YY, Gomez R. Flow on the Internet: a longitudinal study of Internet addiction symptoms during adolescence. Behaviour & Information Technology. 2018;37(2):159-172. doi:10.1080/0144929X.2018.1424937
• Study of internet addiction in children with attention-deficit hyperactivity disorder and normal control. Industrial Psychiatry Journal. 2018;27(1):110-114. doi:10.4103/ipj.ipj_47_17
• The Righteous Mind by Jonathan Haidt
• Thinking, Fast and slow by Kahneman

• Introduction
• Part 1: The Artifacts And Spaces
  In this part we'll describe the important artifacts and places. Going over these essential, but basic, pieces is mandatory to understand how they come into play as tools.
  • Defining and Understanding Propaganda, Influence, And Persuasion
  • Social media — The Democratization of Speech
  • Data & Metadata: Personal Data & Models
  • Memes & Cults
• Part 2: The Actors and Incentives
  In this part we'll go over how the previous elements are put into work by the different actors, who these actors are, what are their incentives, and the new dynamics.
  • New Economies: Information Economy, Attention Economy, Data Economy, Surveillance Economy
  • Internet, In Between The Corporate, Private, And Public Spheres
  • State Actors: PSYOP, Narrative Warfare, And Weaponized Tech
• Part 3: Biases & Self
  In this part we'll try to understand why we are prone to manipulation, why they work so effectively or not on us, and who is subject to them.
  • Cognitive Biases
  • Cultural Ambiguities & Insecurities
• Part 4: The Big Picture
  In this part we'll put forward the reasons why we should care about what is happening in the online sphere. Why it's important to pay attention to it and the effects it could have at the scale of societies, and individuals. This part will attempt to give the bigger picture of the situation.
  • The Formatted Individual
  • Truth & Trust Crises
  • Mass Hypnosis or Mass Psychosis
• Part 5: Adapting
  In this concluding part we'll go over the multiple solutions that have been proposed or tried to counter the negative aspects of the internet.
  • Free Market, Let It Solve Itself
  • Legality, Transparency, Accountability, The Nations Intervene
  • Technical Solutions, Wars and Patches
  • Education, Literacy, and Reality
• Conclusion & Bibliography

Attributions: Philotheus, Symbola Christiana, Frankfurt, 1677

• Internet: Medium For Communication, Medium For Narrative Control
• Part 5 — Adapting
• Section 4 — Education, Literacy, and Reality

• Introduction
• Part 1: The Artifacts And Spaces
  In this part we'll describe the important artifacts and places. Going over these essential, but basic, pieces is mandatory to understand how they come into play as tools.
  • Defining and Understanding Propaganda, Influence, And Persuasion
  • Social media — The Democratization of Speech
  • Data & Metadata: Personal Data & Models
  • Memes & Cults
• Part 2: The Actors and Incentives
  In this part we'll go over how the previous elements are put into work by the different actors, who these actors are, what are their incentives, and the new dynamics.
  • New Economies: Information Economy, Attention Economy, Data Economy, Surveillance Economy
  • Internet, In Between The Corporate, Private, And Public Spheres
  • State Actors: PSYOP, Narrative Warfare, And Weaponized Tech
• Part 3: Biases & Self
  In this part we'll try to understand why we are prone to manipulation, why they work so effectively or not on us, and who is subject to them.
  • Cognitive Biases
  • Cultural Ambiguities & Insecurities
• Part 4: The Big Picture
  In this part we'll put forward the reasons why we should care about what is happening in the online sphere. Why it's important to pay attention to it and the effects it could have at the scale of societies, and individuals. This part will attempt to give the bigger picture of the situation.
  • The Formatted Individual
  • Truth & Trust Crises
  • Mass Hypnosis or Mass Psychosis
• Part 5: Adapting
  In this concluding part we'll go over the multiple solutions that have been proposed or tried to counter the negative aspects of the internet.
  • Free Market, Let It Solve Itself
  • Legality, Transparency, Accountability, The Nations Intervene
  • Technical Solutions, Wars and Patches
  • Education, Literacy, and Reality
• Conclusion & Bibliography

• Web and internet literacy
• Biases and Human Nature
• Our Online Social Lives
• Disconnecting to Better Reconnect
• Learning to Live in The Information Era

We can attempt to patch things, use tools and software as countermeasures, to add laws and regulations, or to let the market decide, but in the end we are at the center of everything. If we are facing difficulties with the medium it’s because we haven’t matured enough to handle it properly. Right now we’re still in the process of trying to grasp how it works, in an apprenticeship stage.

Knowing our tools, how to use the internet medium properly, is a foundational skill, as important as any other in our information society. This goes under the name of digital, internet, or web literacy: the ability to read, write, and participate properly on the medium, in its full extent. It’s an application of the broader information literacy.

Literacy is important because it shapes societies. The degree of literacy of people affects directly what they’re able to imagine doing and actually do on a medium. By providing these skills to more people, teaching them how to use the internet, it would make the access to participation more equitable. It would ensure that there would be more representation of diverse ideas, geographies, languages, and cultures. Consequently, it would reduce the digital divide and increase the cognitive diversity of the netizens.

At the moment, as we’ve seen before, some topics are reserved to the elites. For example, we’ve dived into privacy as a luxury in one of the previous sections. Bringing internet-literacy to everyone, as a human right, would bridge the class gaps.
In 2016 the UN started considering the right to connect as a human right and some countries have adopted it in their laws. However, education on how to use the medium is as important but often dismissed.

Different organizations have thought this through and came up with pedagogical curriculums for the digital literacy core skills needed.
These are skills that are part of information literacy in general but applied to the internet medium. They fall in these categories:

• Seeking information
• Verifying information
• Using information
• Collaborating and participating
• Ethics
• Security and protection

The Mozilla foundation has created a web literacy curriculum that sets the base for the creation of courses that educators can give. They have three broad sections: write, read, and participate. These are subdivided into particular digital skills such as: designing, coding, composing, revising, remixing, connecting, protecting, open practices, contributing, sharing, evaluating, synthesizing, navigating, and searching.

Instead of jumping without knowing — letting users discover everything by themselves — this curriculum could be taught in schools, or as part of trainings easily available to everyone willing to learn.
Such digital citizenship curriculum would help everyone navigate the internet effectively, communicate on it using a variety of methods and tools to a range of different audiences, and have the adequate critical thinking to be able to evaluate information and arguments, identify patterns and connections, and construct meaningful knowledge that can be applied online and in the real world.

Additionally, this curriculum could have an emphasis on understanding the dynamics of the environment, namely: the actors, economies, and algorithms we’ve seen through this series.
Awareness is key to empower us against the influential algorithms, the curation engines of the internet, so that we can use them to our advantage.

Inherently part of this dynamic are the last point above of security and protection, which rhyme with anonymity, lack of trust, and the truth crisis we’ve seen before.

Security also goes along with privacy, implying tackling social cooling problems. Teaching people about cyberbullying, digital footprints, e-safety, and cyber hygiene as soon as possible is a must to protect ourselves online. As we said, people can, after getting a basic overview of the topics, follow recommendations and news from the Electronic Frontier Foundation (EFF) and other non-profit organizations.

One of the most important aspect of information literacy on the internet is making sure of the veracity of what we are consuming. Some organizations and institutions related to libraries, scholars, and media literacy as a whole have put forward good practices that can be taught to everyone to critically assess and navigate the internet space.
The IFLA (International Federation of Library Associations and Institutions) has redacted a document on how to spot fake news and published it initially on https://factcheck.org, to later translate it into 45 languages for accessibility.
Similarly, the Center For Media Literacy has also put forward key questions to ask every time we encounter dubious messages.

A critical part of fact checking is to know our own biases, to understand our own vulnerabilities.
Being aware of the existence and inner workings of the techniques that can be used to abuse our biases is the best defence against them. It provides the basis to quickly notice when we’re being tricked by messages and take some distance. Be it propaganda or marketing schemes, an intimate knowledge of our automatic impulses, training to be primed to know the coercive nature of these messages, make us proactive instead of reactive.

Outrage and shock are two emotional reactions that are employed to get us caught in the tornado of rampage. Noticing how anger doesn’t make our opinion valid, nor more righteous, is essential. Our emotions and opinions shouldn’t always mix.

Many of these biases are temporal and only work for a short period of time, this is why they often seek immediacy, scarcity, and speed. Slowing down media sharing and consumption practices is a good method to avoid falling for them.
The slow thinking way of processing information, compared with fast thinking, would also counter the speed property of the internet medium.

The author Hans Rossling has written about the declinism mindset, how the media change our perceptions, and how to fight these biases in his book Factfulness. The book takes three sides: realization that we do not see the world as is, awareness and identification of our instincts, and how to fight and control them by changing the way we see things.
Documents and teaching materials for educators can be found on the gapminder’s website.

Biases and instincts are short-term, on the long-term we should consider habits related to our surroundings, how we participate online and create communities. Just like it’s essential to learn the dynamics of the environment, it’s also essential to understand and learn how social interactions work on the internet.

This includes grasping what social biases we are prone to — the influence of our virtual surroundings — a good calibration of what this surrounding consists of, and how they are linked.

To calibrate, we can deliberately determine and examine the gatekeeping level of the community we are part of, or the ones we intend on joining or creating. This can be used to bring back a certain quality and trust.

The calibration can come in the form of segregation or separation of our online interactions and profiles. Instead of partaking on platforms that force us to integrate all of our lives, we have to select the ones that help us distinguish between the multiple facets and activities we do.
In the real world we go to different places with different people to do different things, we can do the same virtually. This would help mitigate the phenomena of private life becoming public life, social cooling, and overall paralysis.

The influence of our digital surrounding comes from various sides, all of them wanting to put us in a box, as a well-defined templatized individual. This could be because of marketing, the recommendation and curation engines, or simply the influencers and people we keep up-to-date with.

As with most of what we’ve mentioned so far, knowledge is the best way to avoid being manipulated to fit a narrative. We could train ourselves in metamemetics thinking to be able to recognize the content of the memeplexes, take some distance to criticise them. This would include stopping seeing memes as infotainment and to see them in their real light.
Above all, we need to learn to separate our definition of self from what we consume and share.

We could take this further and distance ourselves completely, to disconnect as a way of reconnecting.

Many are seeking interactions on the internet as an escapism from the difficulties of their lives. To feel safe between the walls of filter bubbles.
Some of these malaises emerge from the cultural insecurities and lacuna we’ve dived into in other sections.

Yet, some studies have found that these insecurities are reduced, along with fewer biases and reaction to outrage, and better communication, when we pull the plug for a while to do offline activities.

In particular, the most effective studies have looked at performing offline pro-social activities such as helping NGOs. Keep in mind that this should be sustained long enough so that it isn’t linked to any online social justice, mob-justice, internet karma points, or the virtue signalling we discussed earlier.

On one side, it works because we get far away from outrage and the negativity that is prominent online. It brings a more positive balance to what we feel instead of being consumed by our screens and their constantly urgent and pressing issues that we sense we are forced to react to.

On another side, it works because it takes us back to raw reality, with its sense of community away from neoliberalism and individualism. Maybe it shows how real human connection cannot be replicated online, but only the chemical reactions.
Nearly everything distinctive about human nature can be seen as a form of cooperation, and taking part in offline activities brings back our humanity.

Ironically, this is the same mechanism that is used to deprogram people from cults or addictive habits. To create a support group around them, with real connection, bonding, support, and orientation.

It could also be because taking time offline and doing non-profit work changes our neoliberal views, if our societies are inundated with them. We can think again of society as an organism which we are part of, with its flaws, and accepting each member with their minute differences.

This last point is probably what triggers our issues on the internet, the malaise and cultural gaps we talked about but that are not addressed.
Learning to live in the information era and the societies shaped around it needs the acceptance of multi-culturality and ambiguity. It requires a post-modern mindset, or even a meta-modern mindset, while most are equipped with a modern one.

For that, we have to make peace with post-modernism and what it implies, without reverting, because of our insecurities, to a cozy pre-modern mindset of tribalism. Some call this transition meta-modernism, or neo-modernism and post-postmodernity.
This means mitigating absolutist thinking, mitigating our inner urge to grasp for reassurances that would reinforce our beliefs when we find ourselves in the absence of certainty. The post-modern world doesn’t owe us certainty, on the contrary, it disintegrates absolutisms, annihilate dichotomy thinking (black-and-white mindset), removes categorical imperatives, and all the expectations on oneself and others. For post-modernity and meta-modernity, these are obstacle to our growth, they are our immunity to change and it makes us uncomfortable when we have to let go.
Absolutist thinking can significantly contribute to disorders of mood and affect, which can, in turn, negatively impact our quality of life because the world itself isn’t absolute and doesn’t bend to our preconceived notions. They are considered cognitive distortions in modern psychology since most of reality and everyday life takes place in the gray area, in between extremes.

This also applies to the trend of over-rationality and over-objectivism, where everything has metrics and is calculated. This over-emphasis on clear definitions and delineations, with no in-betweens, is another kind of absolutist thinking.

Navigating risk and uncertainty is a never ending endeavor. No amount of reassurance will ever quell all of the anxieties we have in our lifetimes.

This transition from modernity isn’t new, it’s been happening for almost a hundred years now and is only accelerated by the new attributes the internet brings.
This is clearly seen through some Western and Eastern art movements.

Modernism was about individualism, the conquest of the single person against the world, independence, personal ownership, but also a general malaise about our place in the world, nihilism, existentialism, fordism, seeing humans as tools, consumption, the psychoanalysts such as Freud, and the marketers such as Bernays.
In this transition, initially out of the absurdity of consumption, sprung art movements such as Dada. It had objects at its center and rejected all logic and reasons, the non-sense of all the machineries.

This extended into movements such as surrealism, shinkankakuha, and abstract expressionism that now had in the center the internal human experience and its reflection on the world. It portrayed a space between dreams and reality, using psychic automatism as a tool to extract thoughts, an inner discovery of our own senses and connection with our environment. The Surrealists and the shinkankakuha movement were fascinated by dreams, desire, magic, sexuality, and the revolutionary power of artworks to transform how we understand the world.

All of these movements were prequels to today’s post-modernism and meta-modernism. Our malaise was put unto art that expressed our infinite smallness, squeezing out our unconscious thought for us to admire and discover new visions, always reinterpreting, always breaking assumptions.
They express the tension we have between structure and non-structure, in any domain, even vis-à-vis language with topics such semiotics.

Presently, we have neural networks dreaming, algorithms that go into inceptions called deep dreams. We are the spectators of this space, the guides which gets back the reflection of their own dreams through a machine.
Yet, dreams rely on associations, even if unthinkable. They learn from these templatized, memeplexes, absolutist thinking.
Again, we should learn to distance our definition of self from what we consume, or offered to consume.

To tackle this we have to fix our cultural gaps, to get our act together. Do everything we can to increase our cognitive diversity, to have more global voices, to look for bridge figures, to search for in-betweens be it geographical, ideological, cultural, or others.

We can build internet realities that take into consideration our complexity, our diversity, our various and myriads of ways of expression who and what we are, and how we want to live. To accept the multiple facets of human ingenuity and creativity.
We can deliberately dabble with this way of thinking, this “meaningness”. The acceptance of nebulosity and fluidity, to learn to find meaning in uncertainty, to learn inter-sectional, cross-cultural connectivity. We can find our gaps and fix our weaknesses within this space, in others, creating a collage of humanness.

However, this poses multiples forms of challenges to move out of our comfort zones and expand our world views. Societies move the burden on the individuals and ask them to provide the structure and system for themselves instead of delegating it to some absolute authority.

Yet as the meme trickled into other online spaces, the line “we live in a society” – originally intended to be an enlightened statement which denounced the many flaws and contradictions of society – instead turned into a piece of satire.

The online world and discussions that bridge cultures make issues more apparent. These controversial topics need to be addressed openly, not through mobs, but maturely. Each on their own.

Some authors and psychologists have written endlessly on the individual’s effort to make sense of experiences, the meaning-making process, metaperception, neo-perception, and self-transformation.
From the “Mental demands of modern society” from Kegan, to Piaget subject-object relations, to Erik Erikson “ego identities”, to Abraham Maslow “self-actualization”.

Learning to live in the informational society is being deliberately open and motivated to possibilities from the intersectionality of all the experiences and ways of seeing the world.
We’re learning to communicate on a global scale while coexisting with algorithms.

This concludes our review of how we can mature to better adapt to the internet medium. We’ve started by exploring educational curriculum such as web and internet literacy, so that we can be properly prepared. We’ve said this includes reading, writing, and participating. Next, we’ve said this curriculum should also take into consideration awareness to biases, knowledge being the best way to tackle them. Later, we’ve mentioned long-term adaptation, social adaptation to the internet, which include learning to manage our connections online by segregating them properly and not being put in templatized boxes. After that, we’ve examined completely disconnecting as a way to better reconnect. We’ve seen some studies about helping NGOs and how it reduced biases and our reaction to online outrage. Finally, we’ve dived into what it means to be a netizen of the information era and societies, to live in post-modernity reaching meta-modernity. We’ve discussed some of the artistic transitions and how they reflect our malaise and continuous improvement at including ourselves in this intersectional and cross-cultural world that has never been as connected as today.

• Introduction
• Part 1: The Artifacts And Spaces
  In this part we'll describe the important artifacts and places. Going over these essential, but basic, pieces is mandatory to understand how they come into play as tools.
  • Defining and Understanding Propaganda, Influence, And Persuasion
  • Social media — The Democratization of Speech
  • Data & Metadata: Personal Data & Models
  • Memes & Cults
• Part 2: The Actors and Incentives
  In this part we'll go over how the previous elements are put into work by the different actors, who these actors are, what are their incentives, and the new dynamics.
  • New Economies: Information Economy, Attention Economy, Data Economy, Surveillance Economy
  • Internet, In Between The Corporate, Private, And Public Spheres
  • State Actors: PSYOP, Narrative Warfare, And Weaponized Tech
• Part 3: Biases & Self
  In this part we'll try to understand why we are prone to manipulation, why they work so effectively or not on us, and who is subject to them.
  • Cognitive Biases
  • Cultural Ambiguities & Insecurities
• Part 4: The Big Picture
  In this part we'll put forward the reasons why we should care about what is happening in the online sphere. Why it's important to pay attention to it and the effects it could have at the scale of societies, and individuals. This part will attempt to give the bigger picture of the situation.
  • The Formatted Individual
  • Truth & Trust Crises
  • Mass Hypnosis or Mass Psychosis
• Part 5: Adapting
  In this concluding part we'll go over the multiple solutions that have been proposed or tried to counter the negative aspects of the internet.
  • Free Market, Let It Solve Itself
  • Legality, Transparency, Accountability, The Nations Intervene
  • Technical Solutions, Wars and Patches
  • Education, Literacy, and Reality
• Conclusion & Bibliography

References

• Propaganda (Wikipedia)
• THE AFFECT AND EFFECT OF INTERNET MEMES:ASSESSING PERCEPTIONS AND INFLUENCE OF ONLINE USER-GENERATED POLITICAL DISCOURSE AS MEDIA
• Using Memes To Improve Climate Change Communication
• THE BIOLOGY OF DISINFORMATION memes, media viruses, and cultural inoculation - IFTF
• Democracy of the Gullible 2020 DW Documentary
• Free Speech Is Not the Same As Free Reach
• Sex, Drugs, and Cults. An evolutionary psychology perspective on why and how cult memes get a drug-like hold on people, and what might be done to mitigate the effects
• MEMETICS: A DANGEROUS IDEA
• How to have better arguments online
• Factfulness
• Your Addiction to Outrage is Ruining Your Life
• No Gates, No Keepers — The technology industry is hurt at every level by toxic gatekeeping
• Information ethics in the context of information Literacy
• Digital literacy in the classroom. How important is it?
• Schools Teach ‘Cyber Hygiene’ to Combat Phishing, Identity Theft
• You Can’t Censor Away Extremism (or Any Other Problem)
• Center For Media Literacy
• How to Teach Digital Literacy Skills
• Mitigating Absolutist Thinking
• The Digital Citizenship Curriculum: Digital Literacy, Cyber Hygiene and More
• I Have Come to Bury Ayn Rand
• The “we live in a society” meme, explained
• The internet didn’t kill counterculture—you just won’t find it on Instagram
• How To Spot Fake News
• factcheck.org
• Cross Culture Frustration.
• EFF
• Who Controls The Media? You’d Be Surprised
• Digital Cosmopolitans, Ethan Zuckerman, Talks at Google
• Snopes
• Shinkankakuha
• Web Literacy
• Surrealism Museums
• Metamodernism
• Right to Internet access
• The Dream of Neural Networks
• Inceptionism: Going Deeper into Neural Networks
• The Self, Metaperceptions, and Self-Transformation
• Meaningness
• A gigantic chart that explains absolutely everything
• Fluidity: a preview
• Thinking, Fast and slow by Kahneman
• Others in Mind: Social Origins of Self-Consciousness by Philippe Rochat
• In Over our heads (The mental demands of modern society) by Robert Kegan
• Immunity to change: How to overcome it and unlock the potential in yourself and your organization by Robert Kegan & Lisa Laskow Lahey
• Factfulness by Rossling
• The Righteous Mind by Jonathan Haidt

Attributions: Origin of the ‘Primum Mobile’, from: Robert Fludd, Philosophia Sacra, Frankfurt, 1626

This is a bit late since some of the deadlines have already passed, but nonetheless, here are the deadlines I know of for 2021 editions of the conferences I track for my lists of institutions and researchers active in technical games research. Plus two journal special issues.

Deadlines still in the future (as of April 26):

Deadlines that have passed:

I might make this a regularly maintained feature in the future, but this is a one-off list for now. Partly so I can stop having to repeatedly search for the deadlines.

There are some other deadlines omitted from the tables above, like workshops, tutorial proposals, doctoral consortia, etc. Click through for the complete CfPs and deadlines.

Audio from the April 20, 2021 Pony development sync call is available.

A new paper investigates “… the feasibility of stealthily introducing vulnerabilities in OSS via hypocrite commits (i.e., seemingly beneficial commits that in fact introduce other critical issues).” Their chosen Open source project was the Linux kernel, and they submitted three patches to the kernel review process.

This interesting idea blew up in their faces, when the kernel developers deduced that they were being experimented on (they obviously don’t have a friend on the inside). The authors have come out dodging and weaving.

What can be learned by reading the paper?

Firstly, three ‘hypocrite commits’ is not enough submissions to do any meaningful statistical analysis. I suspect it’s a convenience sample, a common occurrence in software engineering research. The authors sell three as a proof-of-concept.

How many of the submitted patches passed the kernel review process?

The paper does not say. The first eight pages provide an introduction to the Open source development model, the threat model for introducing vulnerabilities, and the characteristics of vulnerabilities that have been introduced (presumably by accident). This is followed by 2.5 pages of background and setup of the experiment (labelled as a proof-of-concept).

The paper then switches (section VII) to discussing a different, but related, topic: the lifetime of (unintended) vulnerabilities in patches that had been accepted (which I think should have been the topic of the paper. This interesting discussion is 1.5 pages; also see The life and death of statically detected vulnerabilities: An empirical study, covered in figure 6.9 in my book.

The last two pages discuss mitigation, related work, and conclusion (“…a proof-of-concept to safely demonstrate the practicality of hypocrite commits, and measured and quantified the risks.”; three submissions is not hard to measure and quantify, but the results are not to be found in the paper).

Having the paper provide the results (i.e., all three commits spotted, and a very negative response by those being experimented on) would have increased the chances of negative reviewer comments.

Over the past few years I have started noticing this kind of structure in software engineering papers, i.e., extended discussion of an interesting idea, setup of experiment, and cursory or no discussion of results. Many researchers are willing to spend lots of time discussing their ideas, but are unwilling to invest much time in the practicalities of testing them. Some reviewers (who decide whether a paper is accepted to publication) don’t see anything wrong with this approach, e.g., they accept these kinds of papers.

Software engineering research remains a culture of interesting ideas, with evidence being an optional add-on.

• Internet: Medium For Communication, Medium For Narrative Control
• Part 5 — Adapting
• Section 3 — Technical Solutions, Wars and Patches

• Introduction
• Part 1: The Artifacts And Spaces
  In this part we'll describe the important artifacts and places. Going over these essential, but basic, pieces is mandatory to understand how they come into play as tools.
  • Defining and Understanding Propaganda, Influence, And Persuasion
  • Social media — The Democratization of Speech
  • Data & Metadata: Personal Data & Models
  • Memes & Cults
• Part 2: The Actors and Incentives
  In this part we'll go over how the previous elements are put into work by the different actors, who these actors are, what are their incentives, and the new dynamics.
  • New Economies: Information Economy, Attention Economy, Data Economy, Surveillance Economy
  • Internet, In Between The Corporate, Private, And Public Spheres
  • State Actors: PSYOP, Narrative Warfare, And Weaponized Tech
• Part 3: Biases & Self
  In this part we'll try to understand why we are prone to manipulation, why they work so effectively or not on us, and who is subject to them.
  • Cognitive Biases
  • Cultural Ambiguities & Insecurities
• Part 4: The Big Picture
  In this part we'll put forward the reasons why we should care about what is happening in the online sphere. Why it's important to pay attention to it and the effects it could have at the scale of societies, and individuals. This part will attempt to give the bigger picture of the situation.
  • The Formatted Individual
  • Truth & Trust Crises
  • Mass Hypnosis or Mass Psychosis
• Part 5: Adapting
  In this concluding part we'll go over the multiple solutions that have been proposed or tried to counter the negative aspects of the internet.
  • Free Market, Let It Solve Itself
  • Legality, Transparency, Accountability, The Nations Intervene
  • Technical Solutions, Wars and Patches
  • Education, Literacy, and Reality
• Conclusion & Bibliography