Planet Crustaceans

This is a Planet instance for lobste.rs community feeds. To add/update an entry or otherwise improve things, fork this repo.

January 22, 2021

Gustaf Erikson (gerikson)

11,000 dead in Sweden January 22, 2021 02:15 PM

Jan van den Berg (j11g)

Merge two images in Windows from right-click context menu January 22, 2021 11:16 AM

  1. Download and install ImageMagick.
  2. Go to Windows Explorer and type sendto in the address bar. This will open the following path:

    C:\Users\<username>\AppData\Roaming\Microsoft\Windows\SendTo

    The files here will be available as actions from the Windows “Send to” right-click context menu.
  3. Create a new (text) file in this directory and add the following line:

    magick.exe %1 %2 -resize "x%%[fx:max(u.h,v.h)]" +append -set filename: "COMBINED-%%k" "%%[filename:].jpg"

    This is essentially what will be executed when you right click two images anywhere and select the action.
    %1 and %2 are the two files
    – the resize parameters makes sure the two images line up correctly, from here
    +append means the images will be merged side by side — horizontally — as opposed to vertically
    filename uses %k to pass some random value to the generated new filename. Otherwise it would overwrite already existing files with the same name. By generating something unique this doesn’t happen. The word COMBINED is chosen by me, you can change this to whatever you like.

    This line has extra % this is necessary when you run this script as a batch script. If you want to run it from the commandline by hand, you need to remove the double %% and replace them for single %.
  4. Name this file Merge two images side by side.bat or any name you like as long it ends with .bat, so Windows knows to execute it.
  5. Done!

And the result looks like this. Click image or here.

The post Merge two images in Windows from right-click context menu appeared first on Jan van den Berg.

Marc Brooker (mjb)

The Fundamental Mechanism of Scaling January 22, 2021 12:00 AM

The Fundamental Mechanism of Scaling

It's not Paxos, unfortunately.

A common misconception among people picking up distributed systems is that replication and consensus protocols—Paxos, Raft, and friends—are the tools used to build the largest and most scalable systems. It's obviously true that these protocols are important building blocks. They're used to build systems that offer more availability, better durability, and stronger integrity than a single machine. At the most basic level, though, they don't make systems scale.

Instead, the fundamental approach used to scale distributed systems is avoiding co-ordination. Finding ways to make progress on work that doesn't require messages to pass between machines, between clusters of machines, between datacenters and so on. The fundamental tool of cloud scaling is coordination avoidance.

A Spectrum of Systems

With this in mind, we can build a kind of spectrum of the amount of coordination required in different system designs:

Coordinated These are the kind that use paxos, raft, chain replication or some other protocol to make a group of nodes work closely together. The amount of work done by the system generally scales with the offered work (W) and the number of nodes (N), something like O(N * W) (or, potentially, worse under some kinds of failures).

Data-dependent Coordination These systems break their workload up into uncoordinated pieces (like shards), but offer ways to coordinate across shards where needed. Probably the most common type of system in this category is sharded databases, which break data up into independent pieces, but then use some kind of coordination protocol (such as two-phase commit) to offer cross-shard transactions or queries. Work done can vary between O(W) and O(N * W) depending on access patterns, customer behavior and so on.

Leveraged Coordination These systems take a coordinated system and build a layer on top of it that can do many requests per unit of coordination. Generally, coordination is only needed to handle failures, scale up, redistribute data, or perform other similar management tasks. In the happy case, work done in these kinds of systems is O(W). In the bad case, where something about the work or environment forces coordination, they can change to O(N * W) (see Some risks of coordinating only sometimes for more). Despite this risk, this is a rightfully popular pattern for building scalable systems.

Uncoordinated These are the kinds of systems where work items can be handled independently, without any need for coordination. You might think of them as embarrassingly parallel, sharded, partitioned, geo-partitioned, or one of many other ways of breaking up work. Uncoordinated systems scale the best. Work is always O(W).

This is only one cut through a complex space, and some systems don't quite fit1. I think it's still useful, though, because by building a hierarchy of coordination we can think clearly about the places in our systems that scale the best and worst. The closer a system is to the uncoordinated end the better it will scale, in general.

Other useful tools

There are many other ways to approach this question of when coordination is necessary, and how that influences scale.

The CAP theorem2, along with a rich tradition of other impossibility results3, places limits on the kinds of things systems can do (and, most importantly, the kinds of things they can offer to their clients) without needing coordination. If you want to get into the details there, the breakdown in Figure 2 of Highly Available Transactions: Virtues and Limitations is pretty clear. I like it because it shows us both what is possible, and what isn't.

The CALM theorem4 is very useful, because it provides a clear logical framework for whether particular programs can be run without coordination, and something of a path for constructing programs that are coordination free. If you're going to read just one distributed systems paper this year, you could do a lot worse than Keeping CALM.

Harvest and Yield is another way to approach the problem, by thinking about when systems can return partial results4. This is obviously a subtle topic, because the real question is when your clients and customers can accept partial results, and how confused they will be when they get them. At the extreme end, you start expecting clients to write code that can handle any subset of the full result set. Sometimes that's OK, sometimes it sends them down the same rabbit hole that CALM takes you down. Probably the hardest part for me is that partial-result systems are hard to test and operate, because there's a kind of mode switch between partial and complete results and modes make life difficult. There's also the minor issue that there are 2N subsets of results, and testing them all is often infeasible. In other words, this is a useful too, but it's probably best not to expose your clients to the full madness it leads to.

Finally, we can think about the work that each node needs to do. In a coordinated system, there is generally one or more nodes that do O(W) work. In an uncoordinated system, the ideal node does O(W/N) work, which turns into O(1) work because N is proportional to W.

Footnotes

  1. Like systems that coordinate heavily on writes by mostly avoid coordination on reads. CRAQ is one such system, and a paper that helped me fall in love with distributed systems. So clever, and so simple once you understand it.
  2. Best described by Brewer and Lynch.
  3. See, for example, Nancy Lynch's 1989 paper A Hundred Impossibility Proofs for Distributed Computing. If there were a hundred of these in 1989, you can imagine how many there are now, 32 years later. Wow, 1989 was 32 years ago. Huh.
  4. I wrote a post about it back in 2014.

January 21, 2021

Sevan Janiyan (sevan)

Forcing an Xcode command line tools reinstall in order to update January 21, 2021 04:38 PM

Lost some time debugging a build issue which I was unable to reproduce. Turns out I was on an older version of clang despite both of us running the same version of macOS Catalina. Though you install the command line tools using xcode-select --install, there’s no way to force a reinstall with the tool as …

January 20, 2021

Gustaf Erikson (gerikson)

Our long international nightmare is over (for now) January 20, 2021 07:34 PM

For four years, I have managed to avoid to hear all but a few sentences spoken by the late unlamented 45th president of the United States. I don’t know if I’d managed four more years.

Trump is of course just the symptom, not the disease, and he’s just the tip of an iceberg of stewing resentment and irrationality. I’m not convinced today’s and tomorrow’s leaders have what it takes to melt it. Perhaps it will all be boiled away like life itself when the planet heats up.

January 19, 2021

Kevin Burke (kb)

Decline the 15 Minute Post-Vaccination Waiting Period January 19, 2021 05:36 PM

In very rare cases, the Pfizer and Moderna vaccines will cause the person being vaccinated to have an allergic reaction. When I say very rare, I mean it; the chances are a few in a million, or about the same of picking a specific resident of Newark, New Jersey at random out of the phone book.

Because of this chance, the pharmacy/hospital/vaccination site will ask you to wait around for 15 minutes after getting the shot so they can check whether you have an allergic reaction. Most places (scroll through the list on Vaccinate CA) administering the shot are doing so indoors in windowless rooms. People - right now, seniors and others with high exposure to COVID - are being asked to wait for 15 minutes in crowded waiting rooms.

However - waiting in a cramped room indoors is exactly how COVID spreads! Sure, most people are probably wearing masks. But the new B.117 variant is more dangerous and, anecdotally, can get past a cloth mask much more easily. Right after getting the vaccine, but before it has kicked in, we are asking people to huddle in basically the ideal location for transmitting COVID, all to avoid a miniscule risk of an allergic reaction. Not only is this extremely dangerous but it's a huge waste of vaccine - if we know you are going to get COVID today, we shouldn't give you protection against COVID starting a week from now.

The risk of not spotting someone who has an allergic reaction must be weighed against the risk of transmitting COVID. Right now about 3% of the US population is infected with COVID. So about 1 in every 30 people in a vaccination waiting room (likely higher due to selection effects) will be infected with COVID and transmit it to others. About 1-2% of people who get COVID will die from it and more will be hospitalized. Contrast this with about a 1 in ~180,000 risk of an allergic reaction. It's just not comparable.

If you are offered the vaccine, and the waiting area is indoors, I would urge you to decline to wait. Explain that waiting indoors with other unvaccinated1 people is not safe, and then wait outside. You can call someone on the phone for 15 minutes who can monitor you for side effects. Or, walk back in after 15 minutes, tell the pharmacist you are OK, and then leave.

You are not breaking the law by doing this and you are aware of the risks. The more people that choose to do this, the more pressure we can put on vaccinators and public health agencies to end this dangerous practice, and offer waiting areas that are well spaced outdoors.

So many people are dying every day and a vaccine is so close now that small changes will have a huge impact on the total number of people hospitalized and dead. Please share this post with someone you know who is getting vaccinated soon.

Thanks to Michael Story for bringing this issue up.

1. The vaccine takes a week to kick in so for all intents and purposes you are still unvaccinated minutes after you have received the shot.

Bogdan Popa (bogdan)

Running Racket CS on iOS January 19, 2021 08:25 AM

A couple of weeks ago, I started working on getting Racket CS to compile and run on iOS and, with a lot of guidance from Matthew Flatt, I managed to get it working (with some caveats). Those changes have now been merged, so I figured I’d write another one of these guides while the information is still fresh in my head.

January 18, 2021

Robin Schroer (sulami)

Traps to Avoid When Reviewing Code Changes January 18, 2021 12:00 AM

Reviewing code changes is an underappreciated art. It is part of most software engineers’ daily routine, but as an industry we do little towards developing it as a skill, even though it contributes directly to the quality of the software we produce.

The LGTM Trap

Characterised by the eponymous review comment, this trap can have different root causes, all of them resulting in the rubber-stamping of a bad pull request.

First of all, conducting proper code reviews is difficult and mentally exhausting. It is important to take breaks when conducting long or consecutive reviews. Resist the temptation to just “get something in” because it has been open for a while, or because someone else is blocked. Avoid including anything that you already know will need fixing up later, this road leads to broken windows. This also means your comment-change-test cycle should be as fast as possible to encourage fixing even the smallest issues before merging.

If a critical issue makes it past your review, you should investigate how it got missed, and what you can do to catch similar issues in the future. This way you can build your own checklist to use during reviews. You can also ask someone you trust to conduct a supplementary review, or even try pairing with them on some reviews.

The Human Linter Trap

Engineering time is expensive, and the focus required for good reviews is hard to maintain, so minimising the time required for a review is key. This is why we should avoid automatable tasks in code reviews. Prime examples include linting and enforcing a style guide. A pre-commit hook or a CI job can do either of these much more efficiently than a human reviewer ever could.

Beyond the efficiency gains, this also avoids the clutter resulting from many small comments pointing out typos, bad indentation, or non-idiomatic code, and lets both the reviewer and the author focus on more important issues.

The Implementation Trap

Tests can not only be useful to ensure correctness of our code, they can also help us during review. Tests exercise the interface of our code, ideally without giving us too much of an idea of the implementation. As a general rule, you want to be reviewing changes from the outside in.

This forces you to actually understand the test code, assert that the checks performed actually match the contract you expect the code to abide by, and catch potential holes in test coverage. It also allows you to judge the interface provided, as awkward tests often hint at sub-optimally factored code.

The Iceberg Trap

78 ths of an iceberg are famously below the water line and functionally invisible. Similarly some of the most important parts to pay attention to during reviews are not visible in the diff. This can range from introducing some avoidable duplication because the author was not aware of existing code with the same functionality, all the way to production outages because a remote piece of code made an assumption that does not hold anymore.

It can be helpful to checkout the change locally and look at it in the context of the entire code base instead of in isolation. Asking others familiar with the code base or related ones to have a cursory look can also uncover a wide range of problems quickly.

The Rube Goldberg Trap

Just because you can, it does not mean you should. And sometimes there is a better solution than the one proposed.

To review a change, it is important to agree on the problem to solve. Ask the author to supply a problem statement if it is not presented in the change. Only once you understand the problem statement you can evaluate the quality of a solution. The solution could be over- or under-engineered, implemented in the wrong place, or you could even disagree with the problem statement altogether.

January 17, 2021

Derek Jones (derek-jones)

Software effort estimation is mostly fake research January 17, 2021 08:53 PM

Effort estimation is an important component of any project, software or otherwise. While effort estimation is something that everybody in industry is involved with on a regular basis, it is a niche topic in software engineering research. The problem is researcher attitude (e.g., they are unwilling to venture into the wilds of industry), which has stopped them acquiring the estimation data needed to build realistic models. A few intrepid people have risked an assault on their ego and talked to people in industry, the outcome has been, until very recently, a small collection of tiny estimation datasets.

In a research context the term effort estimation is actually a hang over from the 1970s; effort correction more accurately describes the behavior of most models since the 1990s. In the 1970s models took various quantities (e.g., estimated lines of code) and calculated an effort estimate. Later models have included an estimate as input to the model, producing a corrected estimate as output. For the sake of appearances I will use existing terminology.

Which effort estimation datasets do researchers tend to use?

A 2012 review of datasets used for effort estimation using machine learning between 1991-2010, found that the top three were: Desharnias with 24 papers (29%), COCOMO with 19 papers (23%) and ISBSG with 17. A 2019 review of datasets used for effort estimation using machine learning between 1991 and 2017, found the top three to be NASA with 17 papers (23%), the COCOMO data and ISBSG were joint second with 16 papers (21%), and Desharnais was third with 14. The 2012 review included more sources in its search than the 2019 review, and subjectively your author has noticed a greater use of the NASA dataset over the last five years or so.

How large are these datasets that have attracted so many research papers?

The NASA dataset contains 93 rows (that is not a typo, there is no power-of-ten missing), COCOMO 63 rows, Desharnais 81 rows, and ISBSG is licensed by the International Software Benchmarking Standards Group (academics can apply for a limited time use for research purposes, i.e., not pay the $3,000 annual subscription). The China dataset contains 499 rows, and is sometimes used (there is no mention of a supercomputer being required for this amount of data ;-).

Why are researchers involved in software effort estimation feeding tiny datasets from the 1980s-1990s into machine learning algorithms?

Grant money. Research projects are more likely to be funded if they use a trendy technique, and for the last decade machine learning has been the trendiest technique in software engineering research. What data is available to learn from? Those estimation datasets that were flogged to death in the 1990s using non-machine learning techniques, e.g., regression.

Use of machine learning also has the advantage of not needing to know anything about the details of estimating software effort. Everything can be reduced to a discussion of the machine learning algorithms, with performance judged by a chosen error metric. Nobody actually looks at the predicted estimates to discover that the models are essentially producing the same answer, e.g., one learner predicts 43 months, 2 weeks, 4 days, 6 hours, 47 minutes and 11 seconds, while a ‘better’ fitting one predicts 43 months, 2 weeks, 2 days, 6 hours, 27 minutes and 51 seconds.

How many ways are there to do machine learning on datasets containing less than 100 rows?

A paper from 2012 evaluated the possibilities using 9-learners times 10 data-prerocessing options (e.g., log transform or discretization) times 7-error estimation metrics giving 630 possible final models; they picked the top 10 performers.

This 2012 study has not stopped researchers continuing to twiddle away on the option’s nobs available to them; anything to keep the paper mills running.

To quote the authors of one review paper: “Unfortunately, we found that very few papers (including most of our own) paid any attention at all to properties of the data set.”

Agile techniques are widely used these days, and datasets from the 1990s are not applicable. What datasets do researchers use to build Agile effort estimation models?

A 2020 review of Agile development effort estimation found 73 papers. The most popular data set, containing 21 rows, was used by nine papers. Three papers used simulated data! At least some authors were going out and finding data, even if it contains fewer rows than the NASA dataset.

As researchers in business schools have shown, large datasets can be obtained from industry; ISBSG actively solicits data from industry and now has data on 9,500+ projects (as far as I can tell a small amount for each project, but that is still a lot of projects).

Are there any estimates on Github? Some Open source projects use JIRA, which includes support for making estimates. Some story point estimates can be found on Github, but the actuals are missing.

A handful of researchers have obtained and released estimation datasets containing thousands of rows, e.g., the SiP dataset contains 10,100 rows and the CESAW dataset contains over 40,000 rows. These datasets are generally ignored, perhaps because when presented with lots of real data researchers have no idea what to do with it.

Caius Durling (caius)

Cleaning BODUM Bistro Coffee Grinder January 17, 2021 11:41 AM

I’ve owned a BODUM Bistro Coffee Grinder for a number of years, and aside from occasionally running rice through to clean the grinding surfaces haven’t had any issues with it. Recently bought some new beans which are much oilier than ones I usually get, and after running most of them through ended up with the grinder failing to work.

The failure was the mechanism sounding like it was okay for roughly a second, then the motor straining under load before what sounded like plastic gears jumping teeth. At this point I turned it off. Running it with an empty hopper worked fine, adding anything (either beans or ground coffee) to the hopper caused the load issue. On the third attempt it also had stopped self-feeding from the hopper, and trying to gently push beans/grounds through caused the stoppage above.

David Hagman has previously torn down his grinder and posted a video on YouTube showing the internals. I couldn’t see any obvious part of the internals that would be related to the failure I was experiencing, so I decided to start with cleaning it out and then continue with a strip down if that didn’t reveal anything.

To start with the hopper came off, then the top half of the burr grinder lifts out vertically leaving a worm screw standing proud. I first started gently tapping the grinder unit upside down to free any stuck coffee, then escalated to a small bottle brush. There was still enough grounds stuck around the base of the screw mechanism I couldn’t reach with a narrow brush, so I switched to a bamboo barbequeue stick to loosen grounds and then tip them out.

After clearing out most of the base of the worm gear, whilst I had the unit upside down tapping out the loosened grounds I looked up the chute the grounds fall down into the jar normally to find it was blocked solid with ground coffee. Some gentle rodding with the skewer to break it up and eventually I could see from the grinder mechanism through to the end of the chute.

Once that was clear and everything removable had been thoroughly cleaned and dried, I reassembled and ran rice through it a few times starting with a really coarse grind and fed the result back through the hopper, getting finer each grind. Grinder now works flawlessly, and I guess lesson learned about checking the chute to make sure it’s clear more frequently.

January 15, 2021

Gonçalo Valério (dethos)

Django Friday Tips: Permissions in the Admin January 15, 2021 05:55 PM

In this year’s first issue of my irregular Django quick tips series, lets look at the builtin tools available for managing access control.

The framework offers a comprehensive authentication and authorization system that is able to handle the common requirements of most websites without even needing any external library.

Most of the time, simple websites only make use of the “authentication” features, such as registration, login and logout. On more complex systems only authenticating the users is not enough, since different users or even groups of users will have access to distinct sets of features and data records.

This is when the “authorization” / access control features become handy. As you will see they are very simple to use as soon as you understand the implementation and concepts behind them. Today I’m gonna focus on how to use these permissions on the Admin, perhaps in a future post I can address the usage of permissions on other situations. In any case Django has excellent documentation, so a quick visit to this page will tell you what you need to know.

Under the hood

Simplified Entity-Relationship diagram of Django's authentication and authorization features.ER diagram of Django’s “auth” package

The above picture is a quick illustration of how this feature is laid out in the database. So a User can belong to multiple groups and have multiple permissions, each Group can also have multiple permissions. So a user has a given permission if it is directly associated with him or or if it is associated with a group the user belongs to.

When a new model is added 4 permissions are created for that particular model, later if we need more we can manually add them. Those permissions are <app>.add_<model>, <app>.view_<model>, <app>.update_<model> and <app>.delete_<model>.

For demonstration purposes I will start with these to show how the admin behaves and then show how to implement an action that’s only executed if the user has the right permission.

The scenario

Lets image we have a “store” with some items being sold and it also has a blog to announce new products and promotions. Here’s what the admin looks like for the “superuser”:

Admin view, with all models being displayed.The admin showing all the available models

We have several models for the described functionality and on the right you can see that I added a test user. At the start, this test user is just marked as regular “staff” (is_staff=True), without any permissions. For him the admin looks like this:

A view of Django admin without any model listed.No permissions

After logging in, he can’t do anything. The store manager needs the test user to be able to view and edit articles on their blog. Since we expect in the future that multiple users will be able to do this, instead of assigning these permissions directly, lets create a group called “editors” and assign those permissions to that group.

Only two permissions for this group of users

Afterwards we also add the test user to that group (in the user details page). Then when he checks the admin he can see and edit the articles as desired, but not add or delete them.

Screenshot of the Django admin, from the perspective of a user with only No “Add” button there

The actions

Down the line, the test user starts doing other kinds of tasks, one of them being “reviewing the orders and then, if everything is correct, mark them as ready for shipment”. In this case, we don’t want him to be able to edit the order details or change anything else, so the existing “update” permissions cannot be used.

What we need now is to create a custom admin action and a new permission that would let specific users (or groups) execute that action. Lets start with the later:

class Order(models.Model):
    ...
    class Meta:
        ...
        permissions = [("set_order_ready", "Can mark the order as ready for shipment")]

What we are doing above, is telling Django there is one more permission that should be created for this model, a permission that we will use ourselves.

Once this is done (you need to run manage.py migrate), we can now create the action and ensure we check that the user executing it has the newly created permission:

class OrderAdmin(admin.ModelAdmin):
    ...
    actions = ["mark_as_ready"]

    def mark_as_ready(self, request, queryset):
        if request.user.has_perm("shop.set_order_ready"):
            queryset.update(ready=True)
            self.message_user(
                request, "Selected orders marked as ready", messages.SUCCESS
            )
        else:
            self.message_user(
                request, "You are not allowed to execute this action", messages.ERROR
            )

    mark_as_ready.short_description = "Mark selected orders as ready"

As you can see, we first check the user as the right permission, using has_perm and the newly defined permission name before proceeding with the changes.

And boom .. now we have this new feature that only lets certain users mark the orders as ready for shipment. If we try to execute this action with the test user (that does not have yet the required permission):

No permission assigned, no action for you sir

Finally we just add the permission to the user and it’s done. For today this is it, I hope you find it useful.

eta (eta)

Setting the tone in a group is very important January 15, 2021 12:00 AM

Various people have various different opinions on how they should present themselves. Personally, I’d consider myself quite a sensitive person; I’d like to think I try quite hard to take the feelings and circumstances of other people into account when talking to them, although doing so is by no means easy or automatic (and I often fail at this, sometimes quite badly). Partially, this is because I often have a lot of feelings and circumstances going in my life myself which I’d like other people to attempt to take into account – in fact, especially nowadays, I’d argue that the overwhelming majority of people have some topics or areas that make them uncomfortable, or that it’d be possible to upset them with a correctly targeted remark.

It’s very hard to judge what might upset or offend someone, simply because there can be a lot of stuff going on behind the scenes that people just don’t tell others about (it’s almost as if having an entire country where not talking about stuff is the norm could lead to significant problems!). That said, you can at least attempt to be reactive – it’s sometimes possible to detect that something you’ve said or done wasn’t received well, and try to do less of that thing in future.


There are, of course, cultures and groups where this sort of thing is very much not the norm. Some people – and groups of people – attempt to act in a sort of “macho” / “tough guy” sort of way, where one is supposed to pretend that one doesn’t really have feelings, or that one is immune to whatever life might throw in one’s way. This, of course, is obviously false – everyone has feelings! – but it suits them to conduct themselves in this manner, because, at the end of the day, talking about one’s feelings can be very hard.

Maybe you weren’t brought up in an environment where people do that; maybe you never saw your parents, or your school friends, cry, or be angry, or show feelings toward things, because they thought they had to be ‘strong’). Maybe you had people express too much of their feelings in your past and really got put off by it (as in, maybe you were distressed by other people having issues and now have decided that ‘burdening’ others with your feelings isn’t a good idea). Maybe you don’t have any friends you really trust enough to be able to confide in, perhaps because they’re all tough guys, or perhaps because you have issues trusting people for some other reason – especially in our modern society, that reason can often be loneliness, an epidemic that people don’t actually realise the severity of.

But anyway, you don’t want to talk about your feelings. I get that, because I don’t really want to either1. Being in that position, while it’s not really a good thing for you long-term, isn’t really wrong; you aren’t hurting anyone except yourself there (although if you read the previous paragraph and found it hit a bit too close to home, you should probably watch this video).


It’s not uncommon for large-ish (i.e. more than 3 or 4) groups of people to have leaders, whether explicitly or implicitly allocated. Usually there are one or two people who do a lot of the talking – who appear to set the tone and the rules of engagement for the rest of the group. This doesn’t always have to happen, of course, but what I’m saying is it probably happens more than you realise. (It can also become painfully obvious when the leaders step out to go and do something else for a bit and you end up with a bunch of people who don’t really know what they should be doing with themselves, which is always a fun scenario!)

As you probably gathered from the title, what I really want to emphasise is the whole “setting the tone” aspect of being a leader. This turns out to be important in a bunch of ways that aren’t immediately obvious. I’d hypothesize that a large part of the issues people who aren’t cishet white males face in STEM fields, and especially programming / IT, are down to this factor; the groups people tend to hang out in are implicitly using a bunch of norms that probably aren’t very inclusive (think people making slightly inappropriate cheeky comments about women they fancy, but also more subtle mannerisms and ways of communicating that tend to only be shared by people from a certain background that make it harder for people not from that background to communicate). A lot of ink has been spilled about this2 in terms of how companies should try to avoid it when creating teams at work, because it’s a real problem.

The macho people from earlier can face real issues with this sort of thing. Often, their tough-guy behaviour implicitly sets the tone for the groups of people they find themselves in (which usually end up being filled with people who don’t really care, or are also trying to be macho). To avoid opening themselves up to the potential of insecurity, they often tend to do this more forcefully – terms like “pussy”, “wimp”, et al. are often employed for this purpose, wherein such people attempt to claim that people who do have feelings, are afraid of things, etc. are somehow ‘weaker’ than them.


The astute reader will think that I’m writing this because someone did that to me and I’m angry about it, and this blog post is my way of rationalizing their behaviour and asserting that I’m actually a better person than them. And they’d be partially right!3

It’s more nuanced than that, however. The thing that’s actually really sad about these sorts of situations is that the people responsible for creating the harmful no-feelings-allowed environment are often the people most in need of a way to express their feelings (as implied earlier). And what they’ve managed to do by creating such an environment is ensure they most likely won’t be able to do that thing with those people – if they try, it could get awkward (since the others aren’t really happy having a more ‘deep’ conversation, and that’s why they’re in the group), or they might find themselves met with a surprising lack of sympathy (because others actually did have problems and got humiliated for them).

I don’t even think you can blame these people, either. They’ve just found themselves in a situation that most likely isn’t even their fault, and they don’t really know what they should do to cope with it. If anything, it’s probably society that teaches them to behave in this way – and that’s just a sad, sad situation that’s not exactly easy to fix4.


  1. No, it suits me to write pseudo-intellectual blog posts that nobody reads that vaguely hint at a whole bunch of screwed up stuff going on. 

  2. Like a lot of the claims on this blog, this one is unsubstantiated. I think it’s true though… 

  3. I mean I really don’t post often, so someone has to have annoyed me for things to get this bad. 

  4. That said, I’ve seen advertising campaigns that try! I think some biscuit company teamed up with some mental health charity to promote the idea of having a cuppa and a chat about your problems with your friends, which is absolutely a good thing (even if Big Biscuit ends up profiting from it) 

January 14, 2021

Gustaf Erikson (gerikson)

719: Number splitting January 14, 2021 06:13 PM

I’m a bit fired up after Advent of Code, so thought I’d start by picking off some “easy” recent PE problems.

My first brute-force attempt found a solution after 4.5 hours, but access to the forum showed much smarter solutions so that’s what I’ve added to the repo.

10,000 dead in Sweden January 14, 2021 02:48 PM

Maxwell Bernstein (tekknolagi)

Inline caching January 14, 2021 12:00 AM

Inline caching is a popular technique for runtime optimization. It was first introduced in 1984 in Deutsch & Schiffman’s paper Efficient implementation of the smalltalk-80 system [PDF] but has had a long-lasting legacy in today’s dynamic language implementations. Runtimes like the Hotspot JVM, V8, and SpiderMonkey use it to improve the performance of code written for those virtual machines.

In this blog post, I will attempt to distill the essence of inline caching using a small and relatively useless bytecode interpreter built solely for this blog post. The caching strategy in this demo is a technique similar to the ideas from Inline Caching meets Quickening [PDF] in that it caches function pointers instead of making use of a JIT compiler.

Background

In many compiled programming languages like C and C++, types and attribute locations are known at compile time. This makes code like the following fast:

#include "foo.h"

Foo do_add(Foo left, Foo right) {
  return left.add(right);
}

The compiler knows precisely what type left and right are (it’s Foo) and also where the method add is in the executable. If the implementation is in the header file, it may even be inlined and do_add may be optimized to a single instruction. Check out the assembly from objdump:

0000000000401160 <_Z6do_add3FooS_>:
  401160:	48 83 ec 18          	sub    $0x18,%rsp
  401164:	89 7c 24 0c          	mov    %edi,0xc(%rsp)
  401168:	48 8d 7c 24 0c       	lea    0xc(%rsp),%rdi
  40116d:	e8 0e 00 00 00       	callq  401180 <_ZN3Foo3addES_>
  401172:	48 83 c4 18          	add    $0x18,%rsp
  401176:	c3                   	retq

All it does is save the parameters to the stack, call Foo::add, and then restore the stack.

In more dynamic programming languages, it is often impossible to determine at runtime startup what type any given variable binding has. We’ll use Python as an example to illustrate how dynamism makes this tricky, but this constraint is broadly applicable to Ruby, JavaScript, etc.

Consider the following Python snippet:

def do_add(left, right):
    return left.add(right)

Due to Python’s various dynamic features, the compiler cannot in general know what type left is and therefore what code to run when reading left.add. This program will be compiled down to a couple Python bytecode instructions that do a very generic LOAD_METHOD/CALL_METHOD operation:

>>> import dis
>>> dis.dis("""
... def do_add(left, right):
...     return left.add(right)
... """)
[snip]
Disassembly of <code object do_add at 0x7f0b40cf49d0, file "<dis>", line 2>:
  3           0 LOAD_FAST                0 (left)
              2 LOAD_METHOD              0 (add)
              4 LOAD_FAST                1 (right)
              6 CALL_METHOD              1
              8 RETURN_VALUE

>>> 

This LOAD_METHOD Python bytecode instruction is unlike the x86 mov instruction in that LOAD_METHOD is not given an offset into left, but instead is given the name "add". It has to go and figure out how to read add from left’s type — which could change from call to call.

In fact, even if the parameters were typed (which is a new feature in Python 3), the same code would be generated. Writing left: Foo means that left is a Foo or a subclass.

This is not a simple process like “fetch the attribute at the given offset specified by the type”. The runtime has to find out what kind of object add is. Maybe it’s just a function, or maybe it’s a property, or maybe it’s some custom descriptor protocol thing. There’s no way to just turn this into a mov!

… or is there?

Runtime type information

Though dynamic runtimes do not know ahead of time what types variables have at any given opcode, they do eventually find out when the code is run. The first time someone calls do_add, LOAD_METHOD will go and look up the type of left. It will use it to look up the attribute add and then throw the type information away. But the second time someone calls do_add, the same thing will happen. Why don’t runtimes store this information about the type and the method and save the lookup work?

The thinking is “well, left could be any type of object — best not make any assumptions about it.” While this is technically true, Deutsch & Schiffman find that “at a given point in code, the receiver is often the same class as the receiver at the same point when the code was last executed”.

Note: By receiver, they mean the thing from which the attribute is being loaded. This is some Object-Oriented Programming terminology.

This is huge. This means that, even in this sea of dynamic behavior, humans actually are not all that creative and tend to write functions that see only a handful of types at a given location.

The Smalltalk-80 paper describes a runtime that takes advantage of this by adding “inline caches” to functions. These inline caches keep track of variable types seen at each point in the code, so that the runtime can make optimization decisions with that information.

Let’s take a look at how this could work in practice.

A small example

I put together a small stack machine with only a few operations. There are very minimal features to avoid distracting from the main focus: inline caching. Extending this example would be an excellent exercise.

Objects and types

The design of this runtime involves two types of objects (ints and strs). Objects are implemented as a tagged union, but for the purposes of this blog post the representation does not matter very much.

typedef enum {
  kInt,
  kStr,
} ObjectType;

typedef struct {
  ObjectType type;
  union {
    const char *str_value;
    int int_value;
  };
} Object;

These types have methods on them, such as add and print. Method names are represented with an enum (Symbol) though strings would work just as well.

typedef enum {
  kAdd,
  kPrint,

  kUnknownSymbol,
} Symbol;

The representation of type information isn’t super important. Just know that there is a function called lookup_method and that it is very slow. Eventually we’ll want to cache its result.

Method lookup_method(ObjectType type, Symbol name);

Let’s see how we use lookup_method in the interpreter.

Interpreter

There’s no way to call these methods directly. For the purposes of this demo, the only way to call these methods is through purpose-built opcodes. For example, the opcode ADD takes two arguments. It looks up kAdd on the left hand side and calls it. PRINT is similar.

There are only two other opcodes, ARG and HALT.

typedef enum {
  // Load a value from the arguments array at index `arg'.
  ARG,
  // Add stack[-2] + stack[-1].
  ADD,
  // Pop the top of the stack and print it.
  PRINT,
  // Halt the machine.
  HALT,
} Opcode;

Bytecode is represented by a series of opcode/argument pairs, each taking up one byte. Only ARG needs an argument; the other instructions ignore theirs.

Let’s look at a sample program.

byte bytecode[] = {/*0:*/ ARG,   0,
                   /*2:*/ ARG,   1,
                   /*4:*/ ADD,   0,
                   /*6:*/ PRINT, 0,
                   /*8:*/ HALT,  0};

This program takes its two arguments, adds them together, prints the result, and then halts the interpreter.

You may wonder, “how is it that there is an instruction for loading arguments but no call instruction?” Well, the interpreter does not support calls. There is only a top-level function, eval_code. It takes an object, evaluates its bytecode with the given arguments, and returns. Extending the interpreter to support function calls would be another good exercise.

The interpreter implementation is a fairly straightforward switch statement. Notice that it takes a representation of a function-like thing (Code) and an array of arguments. nargs is only used for bounds checking.

typedef unsigned char byte;

typedef struct {
  ObjectType key;
  Method value;
} CachedValue;

typedef struct {
  // Array of `num_opcodes' (op, arg) pairs (total size `num_opcodes' * 2).
  byte *bytecode;
  int num_opcodes;
  // Array of `num_opcodes' elements.
  CachedValue *caches;
} Code;

static unsigned kBytecodeSize = 2;

void eval_code_uncached(Code *code, Object *args, int nargs) {
  int pc = 0;
#define STACK_SIZE 100
  Object stack_array[STACK_SIZE];
  Object *stack = stack_array;
#define PUSH(x) *stack++ = (x)
#define POP() *--stack
  while (true) {
    Opcode op = code->bytecode[pc];
    byte arg = code->bytecode[pc + 1];
    switch (op) {
      case ARG:
        CHECK(arg < nargs && "out of bounds arg");
        PUSH(args[arg]);
        break;
      case ADD: {
        Object right = POP();
        Object left = POP();
        Method method = lookup_method(left.type, kAdd);
        Object result = (*method)(left, right);
        PUSH(result);
        break;
      }
      case PRINT: {
        Object obj = POP();
        Method method = lookup_method(obj.type, kPrint);
        (*method)(obj);
        break;
      }
      case HALT:
        return;
      default:
        fprintf(stderr, "unknown opcode %d\n", op);
        abort();
    }
    pc += kBytecodeSize;
  }
}

Both ADD and PRINT make use of lookup_method to find out what function pointer corresponds to the given (type, symbol) pair. Both opcodes throw away the result. How sad. Let’s figure out how to save some of that data. Maybe we can use the caches slot in Code.

Inline caching strategy

Since the Smalltalk-80 paper tells us that the receiver type is unlikely to change from call to call at a given point in the bytecode, let’s cache one method address per opcode. As with any cache, we’ll have to store both a key (the object type) and a value (the method address).

There are several states that the cache could be in when entering an opcode:

  1. If it is empty, look up the method and store it in the cache using the current type as a cache key. Use the cached value.
  2. If it has an entry and the entry is for the current type, use the cached value.
  3. Last, if it has an entry and the entry is for a different type, invalidate the cache. Repeat the same steps as in the empty case.

This is a simple monomorphic (one element) implementation that should give us most of the performance. A good exercise would be to extend this cache system to be polymorphic (multiple elements) if the interpreter sees many types. For that you will want to check out Optimizing Dynamically-Typed Object-Oriented Languages With Polymorphic Inline Caches by Hölzle, Chambers, and Ungar.

For the purposes of this inline caching demo, we will focus on caching lookups in ADD. This is a fairly arbitrary choice in our simple runtime, since the caching implementation will not differ between opcodes.

Inline caching implementation

Let’s think back to this CachedValue *caches array.

typedef struct {
  ObjectType key;
  Method value;
} CachedValue;

This looks like it’ll suit us just fine. Each element has a key and a value. Each Code object has an array of these, one per opcode.

Let’s see what changed in ADD.

void eval_code_cached(Code *code, Object *args, int nargs) {
  // ...
#define CACHE_AT(pc) code->caches[(pc) / kBytecodeSize]
  while (true) {
    // ...
    switch (op) {
      // ...
      case ADD: {
        Object right = POP();
        Object left = POP();
        CachedValue cached = CACHE_AT(pc);
        Method method = cached.value;
        if (method == NULL || cached.key != left.type) {
          // Case 1 and 3
          method = lookup_method(left.type, kAdd);
          CACHE_AT(pc) = (CachedValue){.key = left.type, .value = method};
        }
        Object result = (*method)(left, right);
        PUSH(result);
        break;
      }
      // ...
    }
    pc += kBytecodeSize;
  }
}

Now instead of always calling lookup_method, we do two quick checks first. If we have a cached value and it matches, we use that instead. So not much, really, except for the reading and writing to code->caches.

Run a test program and see results

Let’s put it all together for some satisfying results. We can use the sample program from earlier that adds its two arguments.

We’ll call it four times. The first time we will call it with integer arguments, and it will cache the integer method. The second time, it will use the cached integer method. The third time, we will call it with string arguments and it will cache the string method. The fourth time, it will use the cached string method.

int main() {
  byte bytecode[] = {/*0:*/ ARG,   0,
                     /*2:*/ ARG,   1,
                     /*4:*/ ADD,   0,
                     /*6:*/ PRINT, 0,
                     /*8:*/ HALT,  0};
  Object int_args[] = {
      new_int(5),
      new_int(10),
  };
  Object str_args[] = {
      new_str("hello "),
      new_str("world"),
  };
  Code code = new_code(bytecode, sizeof bytecode / kBytecodeSize);
  void (*eval)() = eval_code_cached;
  eval(&code, int_args, ARRAYSIZE(int_args));
  eval(&code, int_args, ARRAYSIZE(int_args));
  eval(&code, str_args, ARRAYSIZE(str_args));
  eval(&code, str_args, ARRAYSIZE(str_args));
}

And if we run that, we see:

laurel% ./a.out
int: 15
int: 15
str: "hello world"
str: "hello world"

Which superficially seems like it’s working, at least. 5 + 10 == 15 and "hello " + "world" == "hello world" after all.

To get an insight into the behavior of the caching system, I added some logging statements. This will help convince us that the cache code does the right thing.

laurel% ./a.out
updating cache at 4
int: 15
using cached value at 4
int: 15
updating cache at 4
str: "hello world"
using cached value at 4
str: "hello world"

Hey-ho, looks like it worked.

Conclusion

Inline caches can be a good way to speed up your bytecode interpreter. I hope this post helped you understand inline caches. Please write me if it did not.

Exploring further

There are a number of improvements that could be made to this very simple demo. I will list some of them below:

  • Rewrite generic opcodes like ADD to type-specialized opcodes like ADD_INT. These opcodes will still have to check the types passed in, but can make use of a direct call instruction or even inline the specialized implementation. This technique is mentioned in Efficient Interpretation using Quickening [PDF] and is used in by the JVM.
  • Not all opcodes require caches, yet we allocate for them anyway. How can we eliminate these wasted cache slots?
  • Actually store caches inline in the bytecode, instead of off in a side table.
  • Instead of storing cached function pointers, build up a sort of “linked list” of assembly stubs that handle different type cases. This is also mentioned in the Deutsch & Schiffman paper. See for example this excellent post by Matthew Gaudet that explains it with pseudocode.
  • Figure out what happens if the runtime allows for mutable types. How will you invalidate all the relevant caches?

Maybe I will even write about them in the future.

January 11, 2021

Jan van den Berg (j11g)

Freedom of speech is not freedom of reach January 11, 2021 05:37 PM

The man with access to the nuclear launch codes has been deemed unfit for Twitter. And the country that doesn’t believe universal healthcare is a human right, all of a sudden believes access to Twitter should be an inalienable right. Interesting times!

This week more Americans died from Covid than on 9/11, the Iraq war and the Afghanistan war combined, and that fact isn’t even in the top 10 news stories right now.

The news is dominated by the insurrection and the presidents’ direct incitement of it. And the subsequent (social) media bans that followed. And most notably his account suspension on Twitter.

Most news seems to be focused on the Twitter ban — his preferred outlet — and this has made a lot of people angry, specifically the ones being so called silenced. Which is strange: because I don’t know why GOP politicians are upset about the president losing his Twitter account. They’ve never seen any of his tweets anyway–at least, that’s what they told reporters every time they were asked, right before they ran away.

But it’s not just Twitter. Also Facebook, Google, Apple and Amazon are banning the president, and other apps are pulled and online stores are closed etc. This has people arguing that their first amendment right is violated. In doing so they fail to understand that first amendement is not just to protect the president from his people, but to protect the people from the president.

The relation the USA has to freedom of speech is uniquely American and the complicated nuances and general differences between tolerance and freedom of speech, are paradoxical. But nonetheless: freedom of speech does not entail freedom of reach.

Just like the second amendment was written with slow loading muskets in mind — one round every few minutes — it is now abused to argue the right to automatic weapons that fire hundreds of rounds per minute. And the same is true for the first amendment which was developed to — maybe — reach 40-50 people by standing on a scaffold in a park, is now being abused to argue the right to broadcast opinions to 80 million people. These are clearly different things.

Conflicted

Over the last few days I have heard several arguments against or for banning. Let’s look at some of them.

  • They waited until the democrats had the majority, weak!

Well, it seems the platforms waited until the election was officially called and the electoral college had spoken. Imagine what would have happened if they blocked the president before it? That would have been a much more impactful decision. And then you could have really argued they influenced elections. Now the platforms at least think this argument can never be thrown at them. I take it a lot of lawyers have looked at the timing of this decision.

There have been plenty of reasons already to block the president, citing Terms of Service violations. But if you don’t do it out of the gate (i.e. four years ago) it becomes increasingly more difficult to pick a good time. So we now had to watch and escalate this whole thing steadily for four years.

  • Twitter silenced the president!

Well, it is the president himself who chose to make Twitter his default media outlet. The person with access to every news channel and newspaper in the world chose Twitter, the outrage amplifier, as his biggest news outlet and contact with the people.

Sure Twitter has silenced him, but he still has plenty of other ways to reach people. This proves however that Twitter is not a right, it’s a privilege and it has rules.

This being said, other well known dictators still do have a Twitter account. The difference might be direct incitation?

Still, you can say plenty about the power Twitter yields and the inherit risks involved. Same goes for Facebook et al. of course. They do have great power (too much), and therefore great responsibility. And I do believe regulation should be in place, but that’s another topic.

  • Twitter is a private company, they can do whatever they want!

Well, this is true (the section 230 discussion aside). And this is also how free and open markets should work. He is still entitled to his opinion and spreading this wherever he wants (see above). So we’re not watching “censorship” we’re watching an open source, free market approach to safety measures on the web.

I’ll say this though, Twitter is the de facto pulse of society, whereas Facebook is the personal newspaper and I am willing to state when something is de facto that it has inherent responsibilities following from that. But clearly there are lines and they have chosen to draw the line. As is their right.

That doesn’t mean you can’t feel conflicted about the whole situation. Which I do.

Who dis?

This all being said there is just an incredible amount of complaining about cancel culture, from people that actually tried to cancel the election and the democracy.

The good news is that a test of a secure democracy isn’t whether mobs storm the seat of government. The test of a secure democracy is whether democratic processes survive and continue *in spite of* mobs storming the seat of government. And democracy is proving itself secure in the USA.

In the end what happened was no surprise, at least if you had eyes and ears. And this is not about who the president is, we know who he is, this about who America is. If you want to know who he is, there is an hour long tape of someone who is out of options and plain and simple wants to cheat.

And we can all see and hear with our own eyes what happened. And no, it is not a media narrative.

Now what?


Part of the damage is done. These companies missed the chance to change course years ago. There is no separating Facebook, Twitter, and YouTube from what happened on Wednesday.

The insurrection took longer than necessary, and it sure took time before law enforcement showed up. But the president waited so long, because he was waiting for it to work. And he deliberately took his time. This was the Trump coup. And he got exposed.

The insurrection was a blatant grab to seize power, but it was also to bully and frighten people and to literally terrorize people.

Accountability here is important. Because every time the president hasn’t been held accountable, he’s gotten worse. Every time.

Some people ask: Why would you impeach and convict a president who has only a few days left in office? The answer: Precedent. It must be made clear that no president, now or in the future, can lead an insurrection against the U.S. government.

And there are other reasons of course, the president would loose a lot of benefits.

But even with the president out of the equation, 147 Republicans voted to overturn election results. The USA is in deep trouble. And most Capitol stormers themselves seem deeply troubled. And white. It’s frightening.

One of the indigestible facts of the USA is that most of its terrorism and nearly all its mass shootings are committed by mostly conservative-leaning white men…earnestly committed to their white supremacist-misogynist identity politics.

So there is a lot of work to be done, before we can discuss healing.

To end on a positive note, fortunately stories like these also have heros.

(This post is constructed by assembling tweets from my timeline into a more or less coherent story. I’ve hardly typed any words. For readability all blue links are referenced tweets. Twitter is great.)

The post Freedom of speech is not freedom of reach appeared first on Jan van den Berg.

Nikola Plejić (nikola)

2020: A Year In Review January 11, 2021 03:30 PM

I liked going back to my previous "year in review" post, so let's keep it up. I particularly liked how funny it looks in retrospect. Having good sense of humor is important.

So, 2020... Whew. Oh boy.

The Ugly

COVID-19 happened very early on. Luckily, it did not seriously affect anyone close to me, but it did change everyone's approach to living.

In March, Zagreb got hit by a fairly strong M=5.3 earthquake. The city and its surroundings are still recovering: the renovation project is painfully slow, and the leadership was dreadful. Me and my partner were dislocated for a few months, but have since returned to our rental.

In December, Petrinja got hit by a devastating M=6.4 earthquake which basically destroyed several smaller cities, a lot of villages, and caused widespread damage in central Croatia. The aftershocks are still strong, and if Zagreb is any indicator, the recovery will be ongoing for years to come.

It's been a traumatic year all-around, and I'm sure we'll all be dealing with it for quite a while.

"Academics"

In 2018, I have started pursuing a BSc in Physics at The Open University. This year, I finally graduated — with first class honors. It was probably the pinnacle of the year, and one of my prouder achievements.

I was very happy with the distance learning approach. While I don't see face-to-face teaching disappearing from the face of the Earth any time soon, I believe there's much to explore and experiment within that approach to teaching.

I will be applying to Georgia Tech's Online MS in Computer Science in the Fall 2021 semester. I seem to be on a roll, and it'd be a shame to stop now.

Travel

COVID-19 meant I was mostly roaming around Croatia: Vis, Prvić, Fužine, Ozalj, Plitvice. I very much did not mind that.

Books

This year, I've read a disproportional amount of fiction written in my mother tongue. I very much did not mind that, either. A couple, regardless of genre, that I found particularly impactful:

  • The Judgment of Richard Richter and W by Igor Štiks
  • Selected Poems by Constantinos P. Cavafy
  • Huzur by Martina Mlinarević Sopta
  • Meho by Amin Kaplan
  • Permanent Record by Edward Snowden
  • Patterns of Software by Richard P. Gabriel
  • Homo Faber by Max Frisch
  • Blue Horses by Mary Oliver

Music

I was slightly less mindful of music this year than usual. However, here's a few albums I particularly enjoyed, in alphabetical order with Bandcamp or other indie-ish purchase links where available:

Other

Photography

I bought a new camera (a Canon EOS M6 Mark II) and took a fair amount of photos. I don't do much conventionally creative work so I'm enjoying playing around with the camera and post-processing — with various degrees of success. It's also a great motivator to go outside and move.

I really liked this one:

Downhill

For the first time ever, I made a photobook of photos taken during a trip I went to with my partner in 2018! It turned out really well, and I think I'll be doing more of these in the future.

A picture of a photobook.

Soldering

Last year, one of my plans was to learn some electronics and embedded programming, and I've managed to at least get that going by buying a Lily58L keyboard kit and assembling the thing! I love it, and it's the keyboard I now use daily. Here's me trying really hard not to burn myself:

A picture of me soldering.

Thanks to Lovro for both teaching me how to solder & find my way around the project, as well as for taking this picture! It was an incredibly fun feat.

Politics

In the recent election, Možemo!, the green/left political party featuring yours truly as a member, ended up with four seats in the Croatian Parliament. As limited as it is, this was a remarkable result, and a refreshing change to the parliamentary structure.

Plans for 2021

This time I'll try to keep it simple. Throughout 2021, I'd like to:

  • get vaccinated against COVID-19;
  • get into Georgia Tech's MS in CS program;
  • read, listen to music;
  • take some pictures, preferably while travelling;
  • survive, preferably without any serious long-term consequences.

January 10, 2021

Derek Jones (derek-jones)

My new kitchen clock January 10, 2021 08:22 PM

After several decades of keeping up with the time, since November my kitchen clock has only been showing the correct time every 12-hours. Before I got to buy a new one, I was asked what I wanted to Christmas, and there was money to spend :-)

Guess what Santa left for me:

Hermle Ravensburg clock.

The Hermle Ravensburg is a mechanical clock, driven by the pull of gravity on a cylindrical 1kg of Iron (I assume).

Setup requires installing the energy source (i.e., hang the cylinder on one end of a chain), attach clock to a wall where there is enough distance for the cylinder to slowly ‘fall’, set the time, add energy (i.e., pull the chain so the cylinder is at maximum height), and set the pendulum swinging.

The chain is long enough for eight days of running. However, for the clock to be visible from outside my kitchen I had to place it over a shelf, and running time is limited to 2.5 days before energy has to be added.

The swinging pendulum provides the reference beat for the running of the clock. The cycle time of a pendulum swing is proportional to the square root of the distance of the center of mass from the pivot point. There is an adjustment ring for fine-tuning the swing time (just visible below the circular gold disc of the pendulum).

I used my knowledge of physics to wind the center of mass closer to the pivot to reduce the swing time slightly, overlooking the fact that the thread on the adjustment ring moved a smaller bar running through its center (which moved in the opposite direction when I screwed the ring towards the pivot). Physics+mechanical knowledge got it right on the next iteration.

I have had the clock running 1-second per hour too slow, and 1-second per hour too fast. Current thinking is that the pendulum is being slowed slightly when the cylinder passes on its slow fall (by increased air resistance). Yes dear reader, I have not been resetting the initial conditions before making a calibration run 😐

What else remains to learn, before summer heat has to be adjusted for?

While the clock face and hands may be great for attracting buyers, it has serious usability issues when it comes to telling the time. It is difficult to tell the time without paying more attention than normal; without being a few feet of the clock it is not possible to tell the time by just glancing at it. The see though nature of the face, the black-on-black of the end of the hour/minute hands, and the extension of the minute hand in the opposite direction all combine to really confuse the viewer.

A wire cutter solved the minute hand extension issue, and yellow fluorescent paint solved the black-on-black issue. Ravensburg clock with improved user interface, framed by faded paint of its predecessor below:

Ravensburg clock with improved user interface.

There is a discrete ting at the end of every hour. This could be slightly louder, and I plan to add some weight to the bell hammer. Had the bell been attached slightly off center, fine volume adjustment would have been possible.

January 07, 2021

Gustaf Erikson (gerikson)

8,000 dead in Sweden January 07, 2021 08:50 PM

9,000 dead in Sweden January 07, 2021 08:49 PM

Also a coup or something in the US…

Derek Jones (derek-jones)

Likelihood of a fault experience when using the Horizon IT system January 07, 2021 03:34 PM

It looks like the UK Post Office’s Horizon IT system is going to have a significant impact on the prosecution of cases that revolve around the reliability of software systems, at least in the UK. I have discussed the evidence illustrating the fallacy of the belief that “most computer error is either immediately detectable or results from error in the data entered into the machine.” This post discusses what can be learned about the reliability of a program after a fault experience has occurred, or alleged to have occurred in the Horizon legal proceedings.

Sub-postmasters used the Horizon IT system to handle their accounts with the Post Office. In some cases money that sub-postmasters claimed to have transferred did not appear in the Post Office account. The sub-postmasters claimed this was caused by incorrect behavior of the Horizon system, the Post Office claimed it was due to false accounting and prosecuted or fired people and sometimes sued for the ‘missing’ money (which could be in the tens of thousands of pounds); some sub-postmasters received jail time. In 2019 a class action brought by 550 sub-postmasters was settled by the Post Office, and the presiding judge has passed a file to the Director of Public Prosecutions; the Post Office may be charged with instituting and pursuing malicious prosecutions. The courts are working their way through reviewing the cases of the sub-postmasters charged.

How did the Post Office lawyers calculate the likelihood that the missing money was the result of a ‘software bug’?

Horizon trial transcript, day 1, Mr De Garr Robinson acting for the Post Office: “Over the period 2000 to 2018 the Post Office has had on average 13,650 branches. That means that over that period it has had more than 3 million sets of monthly branch accounts. It is nearly 3.1 million but let’s call it 3 million and let’s ignore the fact for the first few years branch accounts were weekly. That doesn’t matter for the purposes of this analysis. Against that background let’s take a substantial bug like the Suspense Account bug which affected 16 branches and had a mean financial impact per branch of £1,000. The chances of that bug affecting any branch is tiny. It is 16 in 3 million, or 1 in 190,000-odd.”

That 3.1 million comes from the calculation: 19-year period times 12 months per year times 13,650 branches.

If we are told that 16 events occurred, and that there are 13,650 branches and 3.1 million transactions, then the likelihood of a particular transaction being involved in one of these events is 1 in 194,512.5. If all branches have the same number of transactions, the likelihood of a particular branch being involved in one of these 16 events is 1 in 853 (13650/16 -> 853); the branch likelihood will be proportional to the number of transactions it performs (ignoring correlation between transactions).

This analysis does not tell us anything about the likelihood that 16 events will occur, and it does not tell us anything about whether these events are the result of a coding mistake or fraud.

We don’t know how many of the known 16 events are due to mistakes in the code and how many are due to fraud. Let’s ask the question: What is the likelihood of one fault experience occurring in a software system that processes a total of 3.1 million transactions (the number of branches is not really relevant)?

The reply to this question is that it is not possible to calculate an answer, because all the required information is not specified.

A software system is likely to contain some number of coding mistakes, and given the appropriate input any of these mistakes may produce a fault experience. The information needed to calculate the likelihood of one fault experience occurring is:

  • the number of coding mistakes present in the software system,
  • for each coding mistake, the probability that an input drawn from the distribution of input values produced by users of the software will produce a fault experience.

Outside of research projects, I don’t know of any anyone who has obtained the information needed to perform this calculation.

The Technical Appendix to Judgment (No.6) “Horizon Issues” states that there were 112 potential occurrences of the Dalmellington issue (paragraph 169), but does not list the number of transactions processed between these ‘issues’ (which would enable a likelihood to be estimated for that one coding mistake).

The analysis of the Post Office expert, Dr Worden, is incorrect in a complicated way (paragraphs 631 through 635). To ‘prove’ that the missing money was very unlikely to be the result of a ‘software bug’, Dr Worden makes a calculation that he claims is the likelihood of a particular branch experiencing a ‘bug’ (he makes the mistake of using the number of known events, not the number of unknown possible events). He overlooks the fact that while the likelihood of a particular branch experiencing an event may be small, the likelihood of any one of the branches experiencing an event is 13,630 times higher. Dr Worden’s creates complication by calculating the number of ‘bugs’ that would have to exist for there to be a 1 in 10 chance of a particular branch experiencing an event (his answer is 50,000), and then points out that 50,000 is such a large number it could not be true.

As an analogy, let’s consider the UK National Lottery, where the chance of winning the Thunderball jackpot is roughly 1 in 8-million per ticket purchased. Let’s say that I bought a ticket and won this week’s jackpot. Using Dr Worden’s argument, the lottery could claim that my chance of winning was so low (1 in 8-million) that I must have created a counterfeit ticket; they could even say that because I did not buy 0.8 million tickets, I did not have a reasonable chance of winning, i.e., a 1 in 10 chance. My chance of winning from one ticket is the same as everybody else who buys one ticket, i.e., 1 in 8-million. If millions of tickets are bought, it is very likely that one of them will win each week. If only, say, 13,650 tickets are bought each week, the likelihood of anybody winning in any week is very low, but eventually somebody will win (perhaps after many years).

The difference between the likelihood of winning the Thunderball jackpot and the likelihood of a Horizon fault experience is that we have enough information to calculate one, but not the other.

The analysis by the defence team produced different numbers, i.e., did not conclude that there was not enough information to perform the calculation.

Is there any way that the information needed to calculate the likelihood of a fault experience occurring?

In theory fuzz testing could be used. In practice this is probably completely impractical. Horizon is a data driven system, and so a copy of the database would need to be used, along with a copy of all the Horizon software. Where is the computer needed to run this software+database? Yes, use of the Post Office computer system would be needed, along with all the necessary passwords.

Perhaps if we wait long enough, a judge will require that one party make all the software+database+computer+passwords available to the other party.

José Padilla (jpadilla)

Figuring out the home network January 07, 2021 05:26 AM

After moving in to our new home 🏠, one of the first things I did was sign up for gigabit internet from @GoNetspeed.

The ONT is in a corner down in our basement and my office is in the second floor at the opposite side.

Follow me as I figure shit out…

Obviously I’m not going to work right next to the ONT all the time, so I at least need a wireless router…

@gcollazo sold me on @Ubiquiti UniFi _things_. We all need these sorta-over-the-top systems to manage our home network after all.

After watching a few videos and reading a few blogs, I bought a Dream Machine.

Basically a cute all-in-one access point, 4-port switch, and Wi-Fi router, with a shit ton of features and settings I know little about with a nice UI/UX.

Surprisingly, first floor is kinda set. Signal on the second floor sucked. I bought a AP BeaconHD W-Fi MeshPoint.

Basically a cute night light with a seamless integration with the Dream Machine that extends Wi-Fi coverage.

Speed tests in my office gave me a nice 300Mbps. Not enough! Where’s the other 700Mbps!?

I knew I needed to get some wiring done.

This house is already wired for phone ☎ and cable 📺 in every room. All of that ends up back on the basement, right next to the ONT.

I tried following a few of those cables, yanked on some, and noticed they were stapled in a couple of places.

I called it quits at that point and thought of hiring an expert to do what I wanted.

Fast forward a month and decided to revisit this during the holiday break 🎄.

I had gone up to the attic once or twice before. Its nice and “finished” only on top of my office. I thought about venturing to the other 3/4 of the space, but was kinda scared I’d just fall through the ceiling and make a horrible mistake 😂

I read up and watched some videos that gave me an idea of how to actually walk up there, so I did! I walked the whole thing a couple of times and got comfortable up there, you know, not falling through and all that

I started tracing some of the cables across and took a look at the corner where I imagined some of the cables from the basement came up. Jackpot!

Found some conduits that I hoped went into the basement(they did). One had two coaxial cables and one phone cable.

I found some roll of cable up there that was disconnected, so I tied a metal door stopper I had lying around, put a crap ton of tape and threw it down the empty conduit. Down all the stairs for the nth time.

There it was! It reached the basement!

This gave me the extra confidence that I could maybe do this with the minimum number of tools, holes, and mistakes. Given that I don’t even own a drill or anything like that. I have my trusty Ikea toolset, one other screwdriver, and a utility knife.

Went to a home improvement store, got 1000ft of CAT6 cable, because I have no real idea how much I’d need.

Also got a few other things I had seen I would need to actually terminate and test the cables.

I thought I’d give a try creating a 3ft cable first. After like an hour of trying to align cables and getting them on the connector, I noticed I had pass-thru connectors but the wrong crimping tool. Apparently you can still just use a utility knife(I have that!) to cut em.

First-timer luck, tested the cable and it passed!

Ok so managed to get one pull line through from the attic to the basement, why stop at one I thought. Lets do more, future me will thank me…

Mistake número 1: this just ended up being tangled a pain in the ass when pulling the actual CAT6 cable up.

Mistake número 2: got multiple pull lines, but no way to identify them. How the hell am I supposed to know which I need to pull up!?

Had to get @anapadilla_ to help me with this mess.

After quite a struggle a pulled the damn thing up to the attic! 🎉

Got it across the attic and up to the conduit thingy that went down into my office.

Mistake número 3: attached the pull string to the cable with some crappy knot and a bunch of tape. Together with the all the strings, and too much pulling, they came off.

Watched some more videos about useful knots for pulling cable, reattached to another pull string and bam💥

After another hour I got the cable terminated into a keystone jack and replaced the wall plate.

Another hour to terminate the other end in the basement. Moment of truth, got the tester hooked up….

Passed! ✅

I bought two UniFi 6 Lite APs, a UniFi Switch PoE 16, a panel, and rack.

Which I’ll install, eventually, right after I buy a drill I guess.

https://store.ui.com/products/unifi-switch-16-150w

https://store.ui.com/products/unifi-ap-6-lite

https://www.computercablestore.com/6u-wall-mount-cabinet-101-series-18-inches-deep

https://www.monoprice.com/product?p_id=39751

At some point, I’ll also try running an ethernet cable down to a few other key places where there’s a coaxial cable already.

Oh and maybe get into the UniFi Protect stuff for some cameras and doorbell.

Originally tweeted by José Padilla (@jpadilla_) on January 6, 2021.

Andreas Zwinkau (qznc)

Influence (review) January 07, 2021 12:00 AM

Humans rely on mental shortcuts and that is exploited.

Read full article!

January 06, 2021

Marc Brooker (mjb)

Quorum Availability January 06, 2021 12:00 AM

Quorum Availability

It's counterintuitive, but is it right?

In our paper Millions of Tiny Databases, we say this about the availability of quorum systems of various sizes:

As illustrated in Figure 4, smaller cells offer lower availability in the face of small numbers of uncorrelated node failures, but better availability when the proportion of node failure exceeds 50%. While such high failure rates are rare, they do happen in practice, and a key design concern for Physalia.

And this is what Figure 4 looks like:

The context here is that a cell is a Paxos cluster, and the system needs a majority quorum for the cluster to be able to process requests1. A cluster of one box needs one box available, five boxes need three available and so on. The surprising thing here is the claim that having smaller clusters is actually better if the probability of any given machine failing is very high. The paper doesn't explain it well, and I've gotten a few questions about it. This post attempts to do better.

Let's start by thinking about what happens for a cluster of one machine (n=1), in a datacenter of N machines (for very large N). We then fail each machine independently with probability p. What is the probability that our one machine failed? That's trivial: it's p. Now, let's take all N machines and put them into a cluster of n=N. What's the probability that a majority of the cluster is available? For large N, it's 1 for p < 0.5, and 0 for p > 0.5. If less than half the machines fail, less than half have failed. If more than half the machines fail, more than half have failed. Ok?

Notice how a cluster size of 1 is worse than N up until p = 0.5 then better after. Peleg and Wool say:

... for 0 < p < ½ the most available NDC2 is shown to be the "democracy" (namely, the minimal majority system), while the "monarchy" (singleton system) is least available. Due to symmetry, the picture reverses for ½ < p < 1.

Here, the minimal majority system is the one I'd call a majority quorum, and is used by Physalia (and, indeed, most Paxos implementations). The monarchy is where you have one leader node.

What about real practical cluster sizes like n=3, 5, and 7? There are three ways we can do this math. In The Availability of Quorum Systems, Peleg and Wool derive closed-form solutions to this problem3. Our second approach is to observe that the failures of the nodes are Bernoulli trials with probability p, and therefore we can read the answer to "what is the probability that 0 or 1 of 3 fail for probability p" from the distribution function of the binomial distribution. Finally, we can be lazy and do it with Monte Carlo. That's normally my favorite method, because it's easier to include correlation and various "what if?" questions as we go.

Whichever way you calculate it, what do you expect it to look like? For small n you may expect it to be closer in shape to n=1, and for large n you may expect it to approach the shape of n=N. If that's what you expect, you'd be right.

I'll admit that I find this result deeply deeply counter-intuitive. I think it's right, because I've approached it multiple ways, but it still kind of bends my mind a little. That may just be me. I've discussed it with friends and colleagues over the years, and they seem to think it matches their intuition. It's counter-intuitive to me because it suggests that smaller n (smaller clusters, or smaller cells in Physalia's parlance) is better for high p! If you think a lot of your boxes are going to fail, you may get better availability (not durability, though) from smaller clusters.

Weird.

Correlation to the rescue!

It's not often that my statistical intuition is saved by introducing correlation, but in this case it helps. I'd argue that, in practice, that you only lose machines in an uncorrelated Bernoulli trial way for small p. Above a certain p, it's likely that the failures have some shared cause (power, network, clumsy people, etc) and so the failures are likely to be correlated in some way. In which case, we're back into the game we're playing with Physalia of avoiding those correlated failures by optimizing placement.

In many other kinds of systems, like ones you deploy across multiple datacenters (we'd call that regional in AWS, deployed across multiple availability zones), you end up treating the datacenters as units of failure. In that case, for 3 datacenters you'd pick something like n=9 because you can keep quorum after the failure of an entire datacenter (3 machines) and any one other machine. As soon as there's correlation, the math above is mostly useless and the correlation's cause is all that really matters.

Availability also isn't the only thing to think about with cluster size for quorum systems. Durability, latency, cost, operations, and contention on leader election also come into play. Those are topics for another post (or section 2.2 of Millions of Tiny Databases).

Updates

JP Longmore sent me this intuitive explanation, which makes a lot of sense:

Probability of achieving a quorum will increase when removing 2 nodes from a cluster, each with failure rate p>.5, since on average you're removing 2 bad nodes instead of 2 good nodes. Other cases with 1 good node & 1 bad node don't change the outcome (quorum/not). Repeat reasoning till N=1 or all remaining nodes have p<=.5 (if failure rate isn’t uniform).

Footnotes

  1. Physalia uses a very naive Paxos implementation, intentionally optimized for testability and simplicity. The quorum intersection requirements of Paxos (or Paxos-like protocols) are more subtle than this, and work like Heidi Howard et al's Flexible Paxos has been pushing the envelope here recently. Flexible Paxos: Quorum intersection revisited is a good place to start.
  2. Here, an NDC is a non-dominated coterie, and a coterie is a set of groups of nodes (like {{a, b}, {b, c}, {a, c}}). See Definition 2.2 in How to Assign Votes in a Distributed System for the technical definition of domination. What's important, though, is that for each dominated coterie there's a non-dominated coterie that provides the same mutual exclusion properties, but superior availability under partitions. The details are not particularly important here, but are very interesting if you want to do tricky things with quorum intersection.
  3. Along with a whole lot of other interesting facts about quorums, majority quorums and other things. It's a very interesting paper. Another good read in this space is Garcia-Molina and Barbara's How to Assign Votes in a Distributed System, which both does a better job than Peleg and Wool of defining the terms it uses, but also explores the general idea of assigning votes to machines, rather than simply forming quorums of machines. As you read it, it's worth remembering that it predates Paxos, and many of the terms might not mean what you expect.

Andreas Zwinkau (qznc)

Made to Stick (review) January 06, 2021 12:00 AM

Sticky ideas are simple, unexpected, concrete, credentialed, emotional, and a story.

Read full article!

January 05, 2021

Andreas Zwinkau (qznc)

The Three Owners of an Interface January 05, 2021 12:00 AM

An interface is owned above, below, or at the same layer.

Read full article!

January 04, 2021

Gustaf Erikson (gerikson)

Advent of Code 2020 wrapup January 04, 2021 03:05 PM

This was a good year, in my opinion. It was a return to the 2015-2017 era, before the unfortunate experiment in collective puzzle-making in 2018, and the “armature” of the intcode puzzles in 2019. These were fun, and impressive, but if you missed out on doing them there was ~45% of the puzzles gone.

I managed to solve all puzzles before the end of the year, with a personal score of 48 out of 50, which is the best I’ve done since 2016.

AoC has grown hugely, even compared to last year. Here’s how many solved both parts of day 1 and day 25 (essentially solving every puzzle):

Year 2 stars day 1 2 stars day 25 Personal best ranking
for 2 stars
2015 50,358 3,521 422 (day 16)
2019 101,125 3,137 2,111 (day 23)
2020 148,468 11,285 5,040 (day 18)

Favorite puzzle was probably day 20, where we had to do some image recognition. I didn’t finish this on the day it was released but had some fun polishing it later.

Andreas Zwinkau (qznc)

The Culture Code (review) January 04, 2021 12:00 AM

Grow an effective team through building safety, sharing vulnerability, and establishing purpose.

Read full article!

January 03, 2021

Derek Jones (derek-jones)

What impact might my evidence-based book have in 2021? January 03, 2021 10:36 PM

What impact might the release of my evidence-based software engineering book have on software engineering in 2021?

Lots of people have seen the book. The release triggered a quarter of a million downloads, or rather it getting linked to on Twitter and Hacker News resulted in this quantity of downloads. Looking at the some of the comments on Hacker News, I suspect that many ‘readers’ did not progress much further than looking at the cover. Some have scanned through it expecting to find answers to a question that interests them, but all they found was disconnected results from a scattering of studies, i.e., the current state of the field.

The evidence that source code has a short and lonely existence is a gift to those seeking to save time/money by employing a quick and dirty approach to software development. Yes, there are some applications where a quick and dirty iterative approach is not a good idea (iterative as in, if we make enough money there will be a version 2), the software controlling aircraft landing wheels being an obvious example (if the wheels don’t deploy, telling the pilot to fly to another airport to see if they work there is not really an option).

There will be a few researchers who pick up an idea from something in the book, and run with it; I have had a couple of emails along this line, mostly from just starting out PhD students. It would be naive to think that lots of researchers will make any significant changes to their existing views on software engineering. Planck was correct to say that science advances one funeral at a time.

I’m hoping that the book will produce a significant improvement in the primitive statistical techniques currently used by many software researchers. At the moment some form of Wilcoxon test, invented in 1945, is the level of statistical sophistication wielded in most software engineering papers (that do any data analysis).

Software engineering research has the feeling of being a disjoint collection of results, and I’m hoping that a few people will be interested in starting to join the dots, i.e., making connections between findings from different studies. There are likely to be a limited number of major dot joinings, and so only a few dedicated people are needed to make it happen. Why hasn’t this happened yet? I think that many academics in computing departments are lifestyle researchers, moving from one project to the next, enjoying the lifestyle, with little interest in any research results once the grant money runs out (apart from trying to get others to cite it). Why do I think this? I have emailed many researchers information about the patterns I have found in the data they sent me, and a common response is almost completely disinterest (some were interested) in any connections to other work.

What impact do you think ‘all’ the evidence presented will have?

Andreas Zwinkau (qznc)

The Heart of Change (review) January 03, 2021 12:00 AM

Change initiatives succeed in stages building on each other.

Read full article!

January 02, 2021

Bogdan Popa (bogdan)

neko.app January 02, 2021 12:00 AM

I was watching Systems with JT the other day and he demoed a hobby operating system called skiftOS. During the demo he ran one of the built-in apps called “neko” which looks like a clone of an old Windows “pet” program I remember from my childhood, also called “neko” (or “neko32”). It’s a really simple program: when you start it up, a cute little kitten shows up on your screen and starts running around, trying to catch your mouse cursor.

Andreas Zwinkau (qznc)

Turn the Ship Around! (review) January 02, 2021 12:00 AM

The best book about empowerment. Turns a submarine from worst to best.

Read full article!

December 31, 2020

Gustaf Erikson (gerikson)

Advent of Code 2020 December 31, 2020 12:56 PM

Completed all puzzles on 2020-12-30

Project website: Advent of Code 2020.

Previous years: 2015, 2016, 2017, 2018. 2019.

I use Perl for all the solutions.

Most assume the input data is in a file called input.txt in the same directory as the file.

Ratings (new for 2020)

I’m giving each puzzle a subjective rating between 1 and 5. This is based on difficulty, “fiddliness” and how happy I am with my own solution.

A note on scoring

I score my problems to mark where I’ve finished a solution myself or given up and looked for hints. A score of 2 means I solved both the daily problems myself, a score of 1 means I looked up a hint for one of the problems, and a zero score means I didn’t solve any of the problems myself.

My goals for this year (in descending order of priority):

  • get 40 stars or more (75%)
  • solve all problems up until day 15 without any external input
  • solve all problems within 24 hours of release

Final score: 48/50

Link to Github repo.

Day 1 - Day 2 - Day 3 - Day 4 - Day 5 - Day 6 - Day 7 - Day 8 - Day 9 - Day 10 - Day 11 - Day 12 - Day 13 - Day 14 - Day 15 - Day 16 - Day 17 - Day 18 - Day 19 - Day 20 - Day 21 - Day 22 - Day 23 - Day 24 - Day 25

Day 1 - Report Repair

Day 1 - complete solution

Not much to say about this. I used a hash to keep track of the “rest” of the values when comparing.

Apparently this (or at least part 2) is the 3SUM problem which is considered “hard”. I accidentally implemented the standard solution in part 1 so props for that I guess.

I still believe firing up the Perl interpreter and loading the actual file takes longer than just solving part 2 with two nested loops.

Beginning programmer example: loops and hashes/dicts/maps.

Puzzle rating: 3/5

Score: 2

Day 2 - Password Philosophy

Day 2 - complete solution

Despite actually being awake and caffienated when the puzzle dropped, I still managed to mess this up. Extra annoying when it’s basically tailor-made for Perl.

Here’s a partial list

  • messed up the initial regex
  • assigned $min and $max to the same value
  • messed up the comparison range in part 1
  • off-by-one error in the indexing in part 2
  • in part 2, tried to use the sum() function from List::Utils but forgot to use parentheses around the array

Beginning programmer example: parsing input, exclusive OR.

Puzzle rating: 3/5

Score: 2

Day 3 - Toboggan Trajectory

Day 3 - complete solution Day 3 - alternative solution

Veterans of previous AoC’s will get pathfinding flashbacks from this problem’s description, but it turns out it’s a bit simpler - as can be expected for day 3.

I decided before coding to not store the map explicitely as individual coordinates, instead just storing the rows as a text string and unpacking via split when needed.

Another decision was to work with the test input first to confirm my algorithm. That way it would be easier to, for example, print out the rows in case I needed to visually debug.

Beginning programmer example: dealing with infinite repeats using mod.

Puzzle rating: 4/5

Score: 2

Day 4 - Passport Processing

Day 4 - complete solution

This is a “M4P” problem - Made for Perl.

Nothing really to say about this. Set the $/ variable to an empty string to import the records as paragraphs.

I used a dispatch table to avoid a giant if/then/else statement.

Beginning programmer example: regexps! (and handling mult-line input).

Puzzle rating: 3/5

Score: 2

Day 5 - Binary Boarding

Day 5 - complete solution

I have a massive binary blind spot, so I just knew there was going to be a CS-appropriate simple solution to this. But I just followed the instructions and got the right answer in the end anyway.

Beginning programmer example: binary.

Puzzle rating: 3/5

Score: 2

Day 6 - Custom Customs

Day 6 - complete solution

Another easy problem, and during the weekend too! <anxiety intensifies>

My first stab at part 1 contained one specific data structure, that I had to tear out for part 2. After submitting the solution I realized the first solution could work for both.

Puzzle rating: 4/5

Score: 2

Day 7 - Handy Haversacks

Day 7 - complete solution

As was expected, difficulty has ramped up a bit.

Can’t really explain what I did here … the main idea was using BFS to scan the “table”, but part 2 was basically me fiddling around with terms until the test cases passed.

Puzzle rating: 4/5

Score: 2

Day 8 - Handheld Halting

Day 8 - complete solution

This year’s register rodeo, but with a fun twist.

Part 2 was solved by brute forcing every solution, it still took only ~0.3s to find the answer.

Puzzle rating: 4/5

Score: 2

Day 9 - Encoding Error

Day 9 - complete solution

An ok puzzle. It pays to read what’s sought carefully…

Puzzle rating: 4/5

Score: 2

Day 10 - Adapter Array

Day 10 - complete solution

This was a tough one! I was too impatient to really start to optimize after making a solution that solved the two example files, so I “cheated” and looked for inspiration. Full credit in source.

As a bonus I learned about postfix dereferencing in Perl.

Puzzle rating: 4/5

Score: 1.

Day 11 - Seating System

Day 11 - complete solution Day 11 - part 1 Day 11 - part 2

Ugh, got burned by Perl’s negative indices on arrays which messed up part 2. I rewrote it using hashrefs instead.

Puzzle rating: 2/5, mostly because we’ve seen this sort of puzzle before and I don’t enjoy them that much.

Score: 2

Day 12 - Rain Risk

Day 12 - complete solution Day 12 - part 1 Day 12 - part 2

A not too taxing problem.

I don’t like having to re-write the guts of part 1 to solve part 2, however.

Update: after some perusal of the subreddit I realized it was easy enough to run both solutions in one pass, so I rewrote part 2 to handle that.

Puzzle rating: 3/5.

Score: 2

Day 13 - Shuttle Search

Day 13 - complete solution

A tough Sunday problem.

Everyone on the internet figured out that this was an implementation of the Chinese Remainder Theorem, but I followed the logic of a commenter on Reddit (credit in source) and I’m quite proud of the solution.

Puzzle rating: 4/5

Score: 2

Day 14 - Docking Data

Day 14 - complete solution Day 14 - part 1 Day 14 - part 2

A bit of a fiddly problem, which I solved by only dealing with strings and arrays. Bit manipulation is for CS weenies.

I got good help from today’s solutions megathread in the AoC subreddit.

Puzzle rating: 3/5

Score: 2

Day 15 - Rambunctious Recitation

Day 15 - part 1 Day 15 - part 2

This looked complicated at first glance but wasn’t hard to implement.

My part 1 code solves part 2, given a powerful enough computer (I had to use my Raspberry Pi 4). However it takes very long on my standard VPS, so I re-implemented a solution from /u/musifter on Reddit. Credit in source.

Puzzle rating: 3/5

Score: 2

Day 16 - Ticket Translation

Day 16 - complete solution

I realized this was basically the same as 2018D16, but I had a hard time wrapping my head around how to lay out the gathering of required info. A bit of a slog.

Puzzle rating: 3/5

Score: 2

Day 17 - Conway Cubes

Day 17 - part 1 Day 17 - part 2

What if Conway’s game of life - but in more dimensions?!

Not too hard, but not too entertaining either.

Puzzle rating: 3/5

Score: 2

Day 18 - Operator Order

Day 18 - complete solution

A CS staple. So I didn’t feel bad for googling “shunting-yard algorithm” and cribbing a solution from Rosettacode.org. Same for the RPN evaluation algorithm, but I found a much more straightforward implementation on Perlmonks. Credits in source.

I wonder how many CS grads nowadays have even seen a shunting-yard. The nerds in the MIT model railroad society had, of course, and Djikstra too.

Puzzle rating: 3/5

Score: 2

Day 19 - Monster Messages

Day 19 - part 1

For part 1, I tried using Parse::RecDescent and managed to get a kludgy solution, but without really knowing what I was doing.

Skipped part 2 for this one.

Puzzle rating: 3/5

Score: 1

Day 20 - Jurassic Jigsaw

Day 20 - complete solution

This year’s most involved problem. In the end it’s not that difficult, but there are a lot of moving parts.

I’m happy that the code I wrote first (to manipulate grids for part 1) was useful for part 2 too.

Puzzle rating: 4/5

Score: 2

Day 21 - Allergen Assessment

Day 21 - complete solution

Not my favorite puzzle this year. I had the right idea on how to go about it but had to look around for practical solutions.

Puzzle rating: 2/5, mostly because I’m cranky today.

Score: 2

Day 22 - Crab Combat

Day 22 - complete solution

I took a “free card” for part 2 today. Credit in source.

Puzzle rating: 3/5

Score: 1

Day 23 - Crab Cups

Day 23 - part 1 Day 23 - part 2

Part one saw me getting very fiddly with splice, an effort that was not appropriate for part 2…

Score: 2

Day 24 - Lobby Layout

Day 24 - complete solution

A fun little problem. I reused my hex grid logic from 2017D11 and the technique from this year’s day 17 to find the solution.

Puzzle rating: 4/5

Score: 2

Day 25 - Combo Breaker

Day 25 - complete solution

Nice and simple Christmas day puzzle.

I love the denoument of our hero’s journey. Luckily for him, I don’t have all the stars, so I guess I’ll have to stay in the resort until I can finish all the puzzles!

Puzzle rating: 3/5

Score: 2

December 30, 2020

Jan van den Berg (j11g)

Podcast: Donald Knuth Lectures on Things a Computer Scientist Rarely Talks About December 30, 2020 09:45 AM

I recently read ‘Things a Computer Scientist Rarely Talks About’ by Donald Knuth from 2001. Recommended reading if you like reading about how a world-renowned computer scientist wrote a book about how he wrote a book that deals with another book! Sounds recursive 😏

That last book is of course the bible and the book Knuth wrote about it is ‘3:16 Bible Texts Illuminated ‘ — published in 1991. And in 1999 Knuth gave a series of lectures at MIT ‘on the general subject of relations between faith and science’. In these lectures he explains how he went about writing this book and the thought process involved. So the lectures make for an enjoyable deep dive on creating such a book and how Knuth’s brain works, paired with discussions and insights on religion and things like eternity and finiteness.

And it is this series of lectures that are bundled together in ‘Things a Computer Scientist Rarely Talks About’ — almost verbatim. But, the lectures have also always been available as audio files (sadly no visuals) on Knuth’s homepage. And I listened to those a few years back, and as I read this book I was reminded that I had created a RSS feed for these files, effectively creating a Knuth podcast!

This is a later picture of Donald Knuth and not from the 1999 lectures. I added the text, of course in the only possible font.
(I have no copyright on this picture and couldn’t find out who did actually. Feel free to drop me a line if I can accredit you, or if you want it changed.)

I mostly created the file for myself to have the convenience of listening to the lectures in a podcast player. But I have also dropped the link to the XML file here and there over the years, and I noticed 607 unique IP addresses hit this link this month alone! There are only six lectures and one panel discussion and never any new content, so I am not sure what these numbers mean, if they mean anything at all.

But I also remembered I had never blogged about this, until now. So without further ado here is the link:

https://j11g.com/knuth.xml

You can add this to your favorite podcast player. I have added the feed to Overcast myself so it looks like this which is nice.

Having the audiofiles available in a podcast player enables you to track progress, speed up/down parts and have an enhanced audio experience.

I do remember writing an email (no hyphen) to Knuth’s office and I received a nice reply that they thought it was ‘a great idea’, and they were actually also thinking of starting their own podcast ‘based on these materials’. However I haven’t found any link to this yet, so for now it is just this.

If you are more into video, here is a great conversation Donald Knuth had with Lex Fridman last year. Published exactly a year ago to this day. The video is not embeddable but you can click the image to go there. Recommended.

0.jpg (480×360)

The post Podcast: Donald Knuth Lectures on Things a Computer Scientist Rarely Talks About appeared first on Jan van den Berg.

December 29, 2020

Dan Luu (dl)

Against essential and accidental complexity December 29, 2020 12:00 AM

In the classic 1986 essay, No Silver Bullet, Fred Brooks argued that there is, in some sense, not that much that can be done to improve programmer productivity. His line of reasoning is that programming tasks contain a core of essential/conceptual1 complexity that's fundamentally not amenable to attack by any potential advances in technology (such as languages or tooling). He then uses an Ahmdahl's law argument, saying that because 1/X of complexity is essential, it's impossible to ever get more than a factor of X improvement via technological improvements.

Towards the end of the essay, Brooks claims that at least 1/2 (most) of complexity in programming is essential, bounding the potential improvement remaining for all technological programming innovations combined to, at most, a factor of 22:

All of the technological attacks on the accidents of the software process are fundamentally limited by the productivity equation:

Time of task = Sum over i { Frequency_i Time_i }

If, as I believe, the conceptual components of the task are now taking most of the time, then no amount of activity on the task components that are merely the expression of the concepts can give large productivity gains.

Let's see how this essential complexity claim holds for a couple of things I did recently at work:

  • scp from a bunch of hosts to read and download logs, and then parse the logs to understand the scope of a problem
  • Query two years of metrics data from every instance of every piece of software my employer has, for some classes of software and then generate a variety of plots that let me understand some questions I have about what our software is doing and how it's using computer resources

Logs

If we break this task down, we have

  • scp logs from a few hundred thousand machines to a local box
    • used a Python script for this to get parallelism with more robust error handling than you'd get out of pssh/parallel-scp
    • ~1 minute to write the script
  • do other work while logs download
  • parse downloaded logs (a few TB)
    • used a Rust script for this, a few minutes to write (used Rust instead of Python for performance reasons here — just opening the logs and scanning each line with idiomatic Python was already slower than I'd want if I didn't want to farm the task out to multiple machines)

In 1986, perhaps I would have used telnet or ftp instead of scp. Modern scripting languages didn't exist yet (perl was created in 1987 and perl5, the first version that some argue is modern, was released in 1994), so writing code that would do this with parallelism and "good enough" error handling would have taken more than an order of magnitude more time than it takes today. In fact, I think just getting semi-decent error handling while managing a connection pool could have easily taken an order of magnitude longer than this entire task took me (not including time spent downloading logs in the background).

Next up would be parsing the logs. It's not fair to compare an absolute number like "1 TB", so let's just call this "enough that we care about performance" (we'll talk about scale in more detail in the metrics example). Today, we have our choice of high-performance languages where it's easy to write, fast, safe code and harness the power of libraries (e.g., a regexp library3) that make it easy to write a quick and dirty script to parse and classify logs, farming out the work to all of the cores on my computer (I think Zig would've also made this easy, but I used Rust because my team has a critical mass of Rust programmers).

In 1986, there would have been no comparable language, but more importantly, I wouldn't have been able to trivially find, download, and compile the appropriate libraries and would've had to write all of the parsing code by hand, turning a task that took a few minutes into a task that I'd be lucky to get done in an hour. Also, if I didn't know how to use the library or that I could use a library, I could easily find out how I should solve the problem on StackOverflow, which would massively reduce accidental complexity. Needless to say, there was no real equivalent to Googling for StackOverflow solutions in 1986.

Moreover, even today, this task, a pretty standard programmer devops/SRE task, after at least an order of magnitude speedup over the analogous task in 1986, is still nearly entirely accidental complexity.

If the data were exported into our metrics stack or if our centralized logging worked a bit differently, the entire task would be trivial. And if neither of those were true, but the log format were more uniform, I wouldn't have had to write any code after getting the logs; rg or ag would have been sufficient. If I look for how much time I spent on the essential conceptual core of the task, it's so small that it's hard to estimate.

Query metrics

We really only need one counter-example, but I think it's illustrative to look at a more complex task to see how Brooks' argument scales for a more involved task. If you'd like to skip this lengthy example, click here to skip to the next section.

We can view my metrics querying task as being made up of the following sub-tasks:

  • Write a set of Presto SQL queries that effectively scan on the order of 100 TB of data each, from a data set that would be on the order of 100 PB of data if I didn't maintain tables that only contain a subset of data that's relevant
    • Maybe 30 seconds to write the first query and a few minutes for queries to finish, using on the order of 1 CPU-year of CPU time
  • Write some ggplot code to plot the various properties that I'm curious about
    • Not sure how long this took; less time than the queries took to complete, so this didn't add to the total time of this task

The first of these tasks is so many orders of magnitude quicker to accomplish today that I'm not even able to hazard a guess to as to how much quicker it is today within one or two orders of magnitude, but let's break down the first task into component parts to get some idea about the ways in which the task has gotten easier.

It's not fair to port absolute numbers like 100 PB into 1986, but just the idea of having a pipeline that collects and persists comprehensive data analogous to the data I was looking at for a consumer software company (various data on the resource usage and efficiency of our software) would have been considered absurd in 1986. Here we see one fatal flaw in the concept of accidental essential complexity providing an upper bound on productivity improvements: tasks with too much accidental complexity wouldn't have even been considered possible. The limit on how much accidental complexity Brooks sees is really a limit of his imagination, not something fundamental.

Brooks explicitly dismisses increased computational power as something that will not improve productivity ("Well, how many MIPS can one use fruitfully?", more on this later), but both storage and CPU power (not to mention network speed and RAM) were sources of accidental complexity so large that they bounded the space of problems Brooks was able to conceive of.

In this example, let's say that we somehow had enough storage to keep the data we want to query in 1986. The next part would be to marshall on the order of 1 CPU-year worth of resources and have the query complete in minutes. As with the storage problem, this would have also been absurd in 19864, so we've run into a second piece of non-essential complexity so large that it would stop a person from 1986 from thinking of this problem at all.

Next up would be writing the query. If I were writing for the Cray-2 and wanted to be productive, I probably would have written the queries in Cray's dialect of Fortran 77. Could I do that in less than 300 seconds per query? Not a chance; I couldn't even come close with Scala/Scalding and I think it would be a near thing even with Python/PySpark. This is the aspect where I think we see the smallest gain and we're still well above one order of magnitude here.

After we have the data processed, we have to generate the plots. Even with today's technology, I think not using ggplot would cost me at least 2x in terms of productivity. I've tried every major plotting library that's supposedly equivalent (in any language) and every library I've tried either has multiple show-stopping bugs rendering plots that I consider to be basic in ggplot or is so low-level that I lose more than 2x productivity by being forced to do stuff manually that would be trivial in ggplot. In 2020, the existence of a single library already saves me 2x on this one step. If we go back to 1986, before the concept of the grammar of graphics and any reasonable implementation, there's no way that I wouldn't lose at least two orders of magnitude of time on plotting even assuming some magical workstation hardware that was capable of doing the plotting operations I do in a reasonable amount of time (my machine is painfully slow at rendering the plots; a Cray-2 would not be able to do the rendering in anything resembling a reasonable timeframe).

The number of orders of magnitude of accidental complexity reduction for this problem from 1986 to today is so large I can't even estimate it and yet this problem still contains such a large fraction of accidental complexity that it's once again difficult to even guess at what fraction of complexity is essential. To write it all down all of the accidental complexity I can think of would require at least 20k words, but just to provide a bit of the flavor of the complexity, let me write down a few things.

  • SQL; this is one of those things that's superficially simple but actually extremely complex
    • Also, Presto SQL
  • Arbitrary Presto limits, some of which are from Presto and some of which are from the specific ways we operate Presto and the version we're using
    • There's an internal Presto data structure assert fail that gets triggered when I use both numeric_histogram and cross join unnest in a particular way. Because it's a waste of time to write the bug-exposing query, wait for it to fail, and then re-write it, I have a mental heuristic I use to guess, for any query that uses both constructs, whether or not I'll hit the bug and I apply it to avoid having to write two queries. If the heuristic applies, I'll instead write a more verbose query that's slower to execute instead of the more straightforward query
    • We partition data by date, but Presto throws this away when I join tables, resulting in very large and therefore expensive joins when I join data across a long period of time even though, in principle, this could be a series of cheap joins; if the join is large enough to cause my query to blow up, I'll write what's essentially a little query compiler to execute day-by-day queries and then post-process the data as necessary instead of writing the naive query
      • There are a bunch of cases where some kind of optimization in the query will make the query feasible without having to break the query across days (e.g., if I want to join host-level metrics data with the table that contains what cluster a host is in, that's a very slow join across years of data, but I also know what kinds of hosts are in which clusters, which, in some cases, lets me filter hosts out of the host-level metrics data that's in there, like core count and total memory, which can make the larger input to this join small enough that the query can succeed without manually partitioning the query)
    • We have a Presto cluster that's "fast" but has "low" memory limits a cluster that's "slow" but has "high" memory limits, so I mentally estimate how much per-node memory a query will need so that I can schedule it to the right cluster
    • etc.
  • When, for performance reasons, I should compute the CDF or histogram in Presto vs. leaving it to the end for ggplot to compute
  • How much I need to downsample the data, if at all, for ggplot to be able to handle it, and how that may impact analyses
  • Arbitrary ggplot stuff
    • roughly how many points I need to put in a scatterplot before I should stop using size = [number] and should switch to single-pixel plotting because plotting points as circles is too slow
    • what the minimum allowable opacity for points is
    • If I exceed the maximum density where you can see a gradient in a scatterplot due to this limit, how large I need to make the image to reduce the density appropriately (when I would do this instead of using a heatmap deserves its own post)
    • etc.
  • All of the above is about tools that I use to write and examine queries, but there's also the mental model of all of the data issues that must be taken into account when writing the query in order to generate a valid result, which includes things like clock skew, Linux accounting bugs, issues with our metrics pipeline, issues with data due to problems in the underlying data sources, etc.
  • etc.

For each of Presto and ggplot I implicitly hold over a hundred things in my head to be able to get my queries and plots to work and I choose to use these because these are the lowest overhead tools that I know of that are available to me. If someone asked me to name the percentage of complexity I had to deal with that was essential, I'd say that it was so low that there's no way to even estimate it. For some queries, it's arguably zero — my work was necessary only because of some arbitrary quirk and there would be no work to do without the quirk. But even in cases where some kind of query seems necessary, I think it's unbelievable that essential complexity could have been more than 1% of the complexity I had to deal with.

Revisiting Brooks on computer performance, even though I deal with complexity due to the limitations of hardware performance in 2020 and would love to have faster computers today, Brooks wrote off faster hardware as pretty much not improving developer productivity in 1986:

What gains are to be expected for the software art from the certain and rapid increase in the power and memory capacity of the individual workstation? Well, how many MIPS can one use fruitfully? The composition and editing of programs and documents is fully supported by today’s speeds. Compiling could stand a boost, but a factor of 10 in machine speed would surely . . .

But this is wrong on at least two levels. First, if I had access to faster computers, a huge amount of my accidental complexity would go away (if computers were powerful enough, I wouldn't need complex tools like Presto; I could just run a query on my local computer). We have much faster computers now, but it's still true that having faster computers would make many involved engineering tasks trivial. As James Hague notes, in the mid-80s, writing a spellchecker was a serious engineering problem due to performance constraints.

Second, (just for example) ggplot only exists because computers are so fast. A common complaint from people who work on performance is that tool X has somewhere between two and ten orders of magnitude of inefficiency when you look at the fundamental operations it does vs. the speed of hardware today5. But what fraction of programmers can realize even one half of the potential performance of a modern multi-socket machine? I would guess fewer than one in a thousand and I would say certainly fewer than one in a hundred. And performance knowledge isn't independent of other knowledge — controlling for age and experience, it's negatively correlated with knowledge of non-"systems" domains since time spent learning about the esoteric accidental complexity necessary to realize half of the potential of a computer is time spent not learning about "directly" applicable domain knowledge. When we look software that requires a significant amount of domain knowledge (e.g., ggplot) or that'slarge enough that it requires a large team to implement (e.g., IntelliJ6), the vast majority of it wouldn't exist if machines were orders of magnitude slower and writing usable software required wringing most of the performance out of the machine. Luckily for us, hardware has gotten much faster, allowing the vast majority of developers to ignore performance-related accidental complexity and instead focus on all of the other accidental complexity necessary to be productive today.

Faster computers both reduce the amount of accidental complexity tool users run into as well as the amount of accidental complexity that tool creators need to deal with, allowing more productive tools to come into existence.

Summary

To summarize, Brooks states a bound on how much programmer productivity can improve. But, in practice, to state this bound correctly, one would have to be able to conceive of problems that no one would reasonably attempt to solve due to the amount of friction involved in solving the problem with current technologies.

Without being able to predict the future, this is impossible to estimate. If we knew the future, it might turn out that there's some practical limit on how much computational power or storage programmers can productively use, bounding the resources available to a programmer, but getting a bound on the amount of accidental complexity would still require one to correctly reason about how programmers are going to be able to use zillions times more resources than are available today, which is so difficult we might as well call it impossible.

Moreover, for each class of tool that could exist, one would have to effectively anticipate all possible innovations. Brooks' strategy for this was to look at existing categories of tools and state, for each, that they would be ineffective or that they were effective but played out. This was wrong not only because it underestimated gains from classes of tools that didn't exist yet, weren't yet effective, or he wasn't familiar with (e.g., he writes off formal methods, but it doesn't even occur to him to mention fuzzers, static analysis tools that don't fully formally verify code, tools like valgrind, etc.) but also because Brooks thought that every class of tool where there was major improvement was played out and it turns out that none of them were (e.g., programming languages, which Brooks wrote just before the rise of "scripting languages" as well as just before GC langauges took over the vast majority of programming).

In some sense, this isn't too different from when we looked at Unix and found the Unix mavens saying that we should write software like they did in the 70s and that the languages they invented are as safe as any language can be. Long before computers were invented, elders have been telling the next generation that they've done everything that there is to be done and that the next generation won't be able to achieve more. Even without knowing any specifics about programming, we can look at how well these kinds of arguments have held up historically and have decent confidence that the elders are not, in fact, correct this time.

Looking at the specifics with the benefit of hindsight, we can see that Brooks' 1986 claim that we've basically captured all the productivity gains high-level languages can provide isn't too different from an assembly language programmer saying the same thing in 1955, thinking that assembly is as good as any language can be7 and that his claims about other categories are similar. The main thing these claims demonstrate are a lack of imagination. When Brooks referred to conceptual complexity, he was referring to complexity of using the conceptual building blocks that Brooks was familiar with in 1986 (on problems that Brooks would've thought of as programming problems). There's no reason anyone should think that Brooks' 1986 conception of programming is fundamental any more than they should think that how an assembly programmer from 1955 thought was fundamental. People often make fun of the apocryphal "640k should be enough for anybody" quote, but Brooks saying that, across all categories of potential productivity improvement, we've done most of what's possible to do, is analogous and not apocryphal!

We've seen that, if we look at the future, the fraction of complexity that might be accidental is effectively unbounded. One might argue that, if we look at the present, these terms wouldn't be meaningless. But, while this will vary by domain, I've personally never worked on a non-trivial problem that isn't completely dominated by accidental complexity, making the concept of essential complexity meaningless on any problem I've worked on that's worth discussing.

Thanks to Peter Bhat Harkins, Ben Kuhn, Yuri Vishnevsky, Chris Granger, Wesley Aptekar-Cassels, Lifan Zeng, Scott Wolchok, Martin Horenovsky, @realcmb, Kevin Burke, Aaron Brown, and Saul Pwanson for comments/corrections/discussion.


  1. The accidents I discuss in the next section. First let us consider the essence

    The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract, in that the conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed.

    I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared to the conceptual errors in most systems.

    [return]
  2. Curiously, he also claims, in the same essay, that no individual improvement can yield a 10x improvement within one decade. While this technically doesn't contradict his Ahmdal's law argument plus the claim that "most" (i.e., at least half) of complexity is essential/conceptual, it's unclear why he would include this claim as well.

    When Brooks revisited his essay in 1995 in No Silver Bullet Refired, he claimed that he was correct by using the weakest form of the three claims he made in 1986, that within one decade, no single improvement would result in an order of magnitude improvement. However, he did then re-state the strongest form of the claim he made in 1986 and made it again in 1995, saying that this time, no set of technological improvements could improve productivity more than 2x, for real:

    It is my opinion, and that is all, that the accidental or representational part of the work is now down to about half or less of the total. Since this fraction is a question of fact, its value could in principle be settled by measurement. Failing that, my estimate of it can be corrected by better informed and more current estimates. Significantly, no one who has written publicly or privately has asserted that the accidental part is as large as 9/10.

    By the way, I find it interesting that he says that no one disputed this 9/10ths figure. Per the body of this post, I would put it at far above 9/10th for my day-to-day work and, if I were to try to solve the same problems in 1986, the fraction would have been so high that people wouldn't have even conceived of the problem. As a side effect of having worked in hardware for a decade, I've also done work that's not too different from what some people faced in 1986 (microcode, assembly & C written for DOS) and I would put that work as easily above 9/10th as well.

    Another part of his follow-up that I find interesting is that he quotes Harel's "Biting the Silver Bullet" from 1992, which, among other things, argues that that decade deadline for an order of magnitude improvement is arbitrary. Brooks' response to this is

    There are other reasons for the decade limit: the claims made for candidate bullets all have had a certain immediacy about them . . . We will surely make substantial progress over the next 40 years; an order of magnitude over 40 years is hardly magical.

    But by Brooks' own words when he revisits the argument in 1995, if 9/10th of complexity is essential, it would be impossible to get more than an order of magnitude improvement from reducing it, with no caveat on the timespan:

    "NSB" argues, indisputably, that if the accidental part of the work is less than 9/10 of the total, shrinking it to zero (which would take magic) will not give an order of magnitude productivity improvement.

    Both his original essay and the 1995 follow-up are charismatically written and contain a sort of local logic, where each piece of the essay sounds somewhat reasonable if you don't think about it too hard and you forget everything else the essay says. As with the original, a pedant could argue that this is technically not incoherent — after all, Brooks could be saying:

    • at most 9/10th of complexity is accidental (if we ignore the later 1/2 claim, which is the kind of suspension of memory/disbelief one must do to read the essay)
    • it would not be surprising for us to eliminate 100% of accidental complexity after 40 years

    While this is technically consistent (again, if we ignore the part that's inconsistent) and is a set of claims one could make, this would imply that 40 years from 1986, i.e., in 2026, it wouldn't be implausible for there to be literally zero room for any sort of productivity improvement from tooling, languages, or any other potential source of improvement. But this is absurd. If we look at other sections of Brooks' essay and combine their reasoning, we see other inconsistencies and absurdities.

    [return]
  3. Another issue that we see here is Brooks' insistence on bright-line distinctions between categories. Essential vs. accidental complexity. "Types" of solutions, such as languages vs. "build vs. buy", etc.

    Brooks admits that "build vs. buy" is one avenue of attack on essential complexity. Perhaps he would agree that buying a regexp package would reduce the essential complexity since that would allow me to avoid keeping all of the concepts associated with writing a parser in my head for simple tasks. But what if, instead of buying regexes, I used a language where they're bundled into the standard library or is otherwise distributed with the language? Or what if, instead of having to write my own concurrency primitives, those are bundled into the language? Or for that matter, what about an entire HTTP server? There is no bright-line distinction between what's in a library one can "buy" (for free in many cases nowadays) and one that's bundled into the language, so there cannot be a bright-line distinction between what gains a language provides and what gains can be "bought". But if there's no bright-line distinction here, then it's not possible to say that one of these can reduce essential complexity and the other can't and maintain a bright-line distinction between essential and accidental complexity (in a response to Brooks, Harel argued against there being a clear distinction in a response, and Brooks' response was to say that there there is, in fact, a bright-line distinction, although he provided no new argument).

    Brooks' repeated insistence on these false distinctions means that the reasoning in the essay isn't composable. As we've already seen in another footnote, if you take reasoning from one part of the essay and apply it alongside reasoning from another part of the essay, it's easy to create absurd outcomes and sometimes outright contradictions.

    I suspect this is one reason discussions about essential vs. accidental complexity are so muddled. It's not just that Brooks is being vague and handwave-y, he's actually not self-consistent, so there isn't and cannot be a coherent takeaway. Michael Feathers has noted that people are generally not able to correct identify essential complexity; as he says, One person’s essential complexity is another person’s accidental complexity.. This is exactly what we should expect from the essay, since people who have different parts of it in mind will end up with incompatible views.

    This is also a problem when critisizing Brooks. Inevitably, someone will say that what Brooks really meant was something completely different. And that will be true. But Brooks will have meant something completely different while also having meant the things he said that I mention. In defense of the view I'm presenting in the body of the text here, it's a coherent view that one could have had in 1986. Many of Brooks' statements don't make sense even when considered as standalone statements, let alone when cross-referenced with the rest of his essay. For example, the statement that no single development will result in an order of magnitude improvement in the next decade. This statement is meaningless as Brooks does not define and no one can definitively say what a "single improvement" is. And, as mentioned above, Brooks' essay reads quite oddly and basically does not make sense if that's what he's trying to claim. Another issue with most other readings of Brooks is that those are positions that are also meaningless even if Brooks had done the work to make them well defined. Why does it matter if one single improvement or two result in an order of magnitude improvement. If it's two improvements, we'll use them both.

    [return]
  4. Let's arbitrarily use a Motorola 68k processor with an FP co-processor that could do 200 kFLOPS as a reference for how much power we might have in a consumer CPU (FLOPS is a bad metric for multiple reasons, but this is just to get an idea of what it would take to get 1 CPU-year of computational resources, and Brooks himself uses MIPS as a term as if it's meaningful). By comparison, the Cray-2 could achieve 1.9 GFLOPS, or roughly 10000x the performance (I think actually less if we were to do a comparable comparison instead of using non-comparable GFLOPS numbers, but let's be generous here). There are 525600 / 5 = 105120 five minute periods in a year, so to get 1 CPU year's worth of computation in five minutes we'd need 105120 / 10000 = 10 Cray-2s per query, not including the overhead of aggregating results across Cray-2s.

    It's unreasonable to think that a consumer software company in 1986 would have enough Cray-2s lying around to allow for any random programmer to quickly run CPU years worth of queries whenever they wanted to do some data analysis. One sources claims that 27 Cray-2s were ever made over the production lifetime of the machine (1985 to 1990). Even if my employer owned all of them and they were all created by 1986, that still wouldn't be sufficient to allow the kind of ad hoc querying capacity that I have access to in 2020.

    Today, someone at a startup can even make an analogous argument when comparing to a decade ago. You used to have to operate a cluster that would be prohibitively annoying for a startup to operate unless the startup is very specialized, but you can now just use Snowflake and basically get Presto but only pay for the computational power you use (plus a healthy markup) instead of paying to own a cluster and for all of the employees necessary to make sure the cluster is operable.

    [return]
  5. I actually run into one of these every time I publish a new post. I write my posts in Google docs and then copy them into emacs running inside tmux running inside Alacritty. My posts are small enough to fit inside L2 cache, so I could have 64B/3.5 cycle write bandwidth. And yet, the copy+paste operation can take ~1 minute and is so slow I can watch the text get pasted in. Since my chip is working super hard to make sure the copy+paste happens, it's running at its full non-turbo frequency of 4.2Ghz, giving it 76.8GB/s of write bandwidth. For a 40kB post, 1 minute = 666B/s. 76.8G/666 =~ 8 orders of magnitude left on the table. [return]
  6. In this specific case, I'm sure somebody will argue that Visual Studio was quite nice in 2000 and ran on much slower computers (and the debugger was arguably better than it is in the current version). But there was no comparable tool on Linux, nor was there anything comparable to today's options in the VSCode-like space of easy-to-learn programming editor that provides programming-specific facilities (as opposed to being a souped up version of notepad) without being a full-fledged IDE. [return]
  7. And by the way, this didn't only happen in 1955. I've worked with people who, this century, told me that assembly is basically as productive as any high level language. This probably sounds ridiculous to almost every reader of this blog, but if you talk to people who spend all day writing microcode or assembly, you'll occasionally meet somebody who believes this.

    Thinking that the tools you personally use are as good as it gets is an easy trap to fall into.

    [return]

December 27, 2020

Derek Jones (derek-jones)

Source code discovery, skipping over the legal complications December 27, 2020 10:27 PM

The 2020 US elections introduced the issue of source code discovery, in legal cases, to a wider audience. People wanted to (and still do) check that the software used to register and count votes works as intended, but the companies who wrote the software wouldn’t make it available and the courts did not compel them to do so.

I was surprised to see that there is even a section on “Transfer of or access to source code” in the EU-UK trade and cooperation agreement, agreed on Christmas Eve.

I have many years of experience in discovering problems in the source code of programs I did not write. This experience derives from my time as a compiler implementer (e.g., a big customer is being held up by a serious issue in their application, and the compiler is being blamed), and as a static analysis tool vendor (e.g., managers want to know about what serious mistakes may exist in the code of their products). In all cases those involved wanted me there, I could talk to some of those involved in developing the code, and there were known problems with the code. In court cases, the defence does not want the prosecution looking at the code, and I assume that all conversations with the people who wrote the code goes via the lawyers. I have intentionally stayed away from this kind of work, so my practical experience of working on legal discovery is zero.

The most common reason companies give for not wanting to make their source code available is that it contains trade-secrets (they can hardly say that it’s because they don’t want any mistakes in the code to be discovered).

What kind of trade-secrets might source code contain? Most code is very dull, and for some programs the only trade-secret is that if you put in the implementation effort, the obvious way of doing things works, i.e., the secret sauce promoted by the marketing department is all smoke and mirrors (I have had senior management, who have probably never seen the code, tell me about the wondrous properties of their code, which I had seen and knew that nothing special was present).

Comments may detail embarrassing facts, aka trade-secrets. Sometimes the code interfaces to a proprietary interface format that the company wants to keep secret, or uses some formula that required a lot of R&D (management gets very upset when told that ‘secret’ formula can be reverse engineered from the executable code).

Why does a legal team want access to source code?

If the purpose is to check specific functionality, then reading the source code is probably the fastest technique. For instance, checking whether a particular set of input values can cause a specific behavior to occur, or tracing through the logic to understand the circumstances under which a particular behavior occurs, or in software patent litigation checking what algorithms or formula are being used (this is where trade-secret claims appear to be valid).

If the purpose is a fishing expedition looking for possible incorrect behaviors, having the source code is probably not that useful. The quantity of source contained in modern applications can be huge, e.g., tens to hundreds of thousands of lines.

In ancient times (i.e., the 1970s and 1980s) programs were short (because most computers had tiny amounts of memory, compared to post-2000), and it was practical to read the source to understand a program. Customer demand for more features, and the fact that greater storage capacity removed the need to spend time reducing code size, means that source code ballooned. The following plot shows the lines of code contained in the collected algorithms of the Transactions on Mathematical Software, the red line is a fitted regression model of the form: LOC approx e^{0.0003Day}(code+data):

Lines of code contained in the collected algorithms of the Transactions on Mathematical Software, over time.

How, by reading the source code, does anybody find mistakes in a 10+ thousand line program? If the program only occasionally misbehaves, finding a coding mistake by reading the source is likely to be very very time-consuming, i.e, months. Work it out yourself: 10K lines of code is around 200 pages. How long would it take you to remember all the details and their interdependencies of a detailed 200-page technical discussion well enough to spot an inconsistency likely to cause a fault experience? And, yes, the source may very well be provided as a printout, or as a pdf on a protected memory stick.

From my limited reading of accounts of software discovery, the time available to study the code may be just days or maybe a week or two.

Reading large quantities of code, to discover possible coding mistakes, are an inefficient use of human time resources. Some form of analysis tool might help. Static analysis tools are one option; these cost money and might not be available for the language or dialect in which the source is written (there are some good tools for C because it has been around so long and is widely used).

Character assassination, or guilt by innuendo is another approach; the code just cannot be trusted to behave in a reasonable manner (this approach is regularly used in the software business). Software metrics are deployed to give the impression that it is likely that mistakes exist, without specifying specific mistakes in the code, e.g., this metric is much higher than is considered reasonable. Where did these reasonable values come from? Someone, somewhere said something, the Moon aligned with Mars and these values became accepted ‘wisdom’ (no, reality is not allowed to intrude; the case is made by arguing from authority). McCabe’s complexity metric is a favorite, and I have written how use of this metric is essentially accounting fraud (I have had emails from several people who are very unhappy about me saying this). Halstead’s metrics are another favorite, and at least Halstead and others at the time did some empirical analysis (the results showed how ineffective the metrics were; the metrics don’t calculate the quantities claimed).

The software development process used to create software is another popular means of character assassination. People seem to take comfort in the idea that software was created using a defined process, and use of ad-hoc methods provides an easy target for ridicule. Some processes work because they include lots of testing, and doing lots of testing will of course improve reliability. I have seen development groups use a process and fail to produce reliable software, and I have seen ad-hoc methods produce reliable software.

From what I can tell, some expert witnesses are chosen for their ability to project an air of authority and having impressive sounding credentials, not for their hands-on ability to dissect code. In other words, just the kind of person needed for a legal strategy based on character assassination, or guilt by innuendo.

What is the most cost-effective way of finding reliability problems in software built from 10k+ lines of code? My money is on fuzz testing, a term that should send shivers down the spine of a defense team. Source code is not required, and the output is a list of real fault experiences. There are a few catches: 1) the software probably to be run in the cloud (perhaps the only cost/time effective way of running the many thousands of tests), and the defense is going to object over licensing issues (they don’t want the code fuzzed), 2) having lots of test harnesses interacting with a central database is likely to be problematic, 3) support for emulating embedded cpus, even commonly used ones like the Z80, is currently poor (this is a rapidly evolving area, so check current status).

Fuzzing can also be used to estimate the numbers of so-far undetected coding mistakes.

Ponylang (SeanTAllen)

Last Week in Pony - December 27, 2020 December 27, 2020 09:54 PM

Pony 0.38.2 has been released! Audio from the December 15 sync call is available.

Jeff Carpenter (jeffcarp)

2020 Year in Review December 27, 2020 12:00 AM

2020. What can I say? It was a year I don’t think any of us need to live again. Here’s a brief overview of my year. Life In February on Valentine’s day, Elva and I adopted Noona(!), an 8 year old Siberian Husky. She’s the chillest, sweetest dog. Due to lockdown starting 2 weeks later, I’m sure she thinks that spending all day at the house giving her attention is normal.

December 26, 2020

Pete Corey (petecorey)

MongoDB Lookup Aggregation December 26, 2020 12:00 AM

Recently, I received an email from a reader asking for tips on writing a MongoDB aggregation that aggregated the layers of a tree, stored in separate collections, into a single document:

Hi Pete,

I had a question related to your article on MongoDB object array lookup aggregations.

I’m working on something similar, but with a small difference. Imagine I have three collections that represent the different layers of a tree. A is the root. B are the children of A, and C are the children of B. Each child holds the ID of its parent in a parentId field.

The end goal is to write an aggregation that fleshes out every layer of the tree:

{
  _id
  B: [
    {
      _id
      parentId
      C: [
        {
          _id, 
          parentId
        }
      ]
    }
  ]
}

How should I approach this? Thanks.

Hello friend,

I feel your pain. Writing MongoDB aggregation feels like an under-documented dark art. In newer versions of Mongo you can write sub-pipelines under lookups. I think this will get you where you want to go:

db.getCollection('a').aggregate([
  {
    $lookup: {
      from: 'b',
      let: { "id": '$_id' },
      as: 'b',
      pipeline: [
        { $match: { $expr: { $eq: ['$$id', '$parentId'] } } },
        {
          $lookup: {
            from: 'c',
            let: { "id": '$_id' },
            as: 'c',
            pipeline: [
              { $match: { $expr: { $eq: ['$$id', '$parentId'] } } },
            ]
          }
        }
      ]
    }
  }
]);

You can keep adding sub-piplines until you get as deep as you need.

I hope that helps.

Pete

December 23, 2020

Oliver Charles (ocharles)

Monad Transformers and Effects with Backpack December 23, 2020 12:00 AM

A good few years ago Edward Yang gifted us an implementation of Backpack - a way for us to essentially abstract modules over other modules, allowing us to write code independently of implementation. A big benefit of doing this is that it opens up new avenues for program optimization. When we provide concrete instantiations of signatures, GHC compiles it as if that were the original code we wrote, and we can benefit from a lot of specialization. So aside from organizational concerns, Backpack gives us the ability to write some really fast code. This benefit isn’t just theoretical - Edward Kmett gave us unpacked-containers, removing a level of indirection from all keys, and Oleg Grenrus showed as how we can use Backpack to “unroll” fixed sized vectors. In this post, I want to show how we can use Backpack to give us the performance benefits of explicit transformers, but without having library code commit to any specific stack. In short, we get the ability to have multiple interpretations of our program, but without paying the performance cost of abstraction.

The Problem

Before we start looking at any code, let’s look at some requirements, and understand the problems that come with some potential solutions. The main requirement is that we are able to write code that requires some effects (in essence, writing our code to an effect interface), and then run this code with different interpretations. For example, in production I might want to run as fast as possible, in local development I might want further diagnostics, and in testing I might want a pure or in memory solution. This change in representation shouldn’t require me to change the underlying library code.

Seasoned Haskellers might be familiar with the use of effect systems to solve these kinds of problems. Perhaps the most familiar is the mtl approach - perhaps unfortunately named as the technique itself doesn’t have much to do with the library. In the mtl approach, we write our interfaces as type classes abstracting over some Monad m, and then provide instances of these type classes - either by stacking transformers (“plucking constraints”, in the words of Matt Parson), or by a “mega monad” that implements many of these instances at once (e.g., like Tweag’s capability) approach.

Despite a few annoyances (e.g., the “n+k” problem, the lack of implementations being first-class, and a few other things), this approach can work well. It also has the potential to generate a great code, but in practice it’s rarely possible to achieve maximal performance. In her excellent talk “Effects for Less”, Alexis King hits the nail on the head - despite being able to provide good code for the implementations of particular parts of an effect, the majority of effectful code is really just threading around inside the Monad constraint. When we’re being polymorphic over any Monad m, GHC is at a loss to do any further optimization - and how could it? We know nothing more than “there will be some >>= function when you get here, promise!” Let’s look at this in a bit more detail.

Say we have the following:

foo :: Monad m => m Int
foo = go 0 1_000_000_000
  where
    go acc 0 = return acc
    go acc i = return acc >> go (acc + 1) (i - 1)

This is obviously “I needed an example for my blog” levels of contrived, but at least small. How does it execute? What are the runtime consequences of this code? To answer, we’ll go all the way down to the STG level with -ddump-stg:

$wfoo =
    \r [ww_s2FA ww1_s2FB]
        let {
          Rec {
          $sgo_s2FC =
              \r [sc_s2FD sc1_s2FE]
                  case eqInteger# sc_s2FD lvl1_r2Fp of {
                    __DEFAULT ->
                        let {
                          sat_s2FK =
                              \u []
                                  case +# [sc1_s2FE 1#] of sat_s2FJ {
                                    __DEFAULT ->
                                        case minusInteger sc_s2FD lvl_r2Fo of sat_s2FI {
                                          __DEFAULT -> $sgo_s2FC sat_s2FI sat_s2FJ;
                                        };
                                  }; } in
                        let {
                          sat_s2FH =
                              \u []
                                  let { sat_s2FG = CCCS I#! [sc1_s2FE]; } in  ww1_s2FB sat_s2FG;
                        } in  ww_s2FA sat_s2FH sat_s2FK;
                    1# ->
                        let { sat_s2FL = CCCS I#! [sc1_s2FE]; } in  ww1_s2FB sat_s2FL;
                  };
          end Rec }
        } in  $sgo_s2FC lvl2_r2Fq 0#;

foo =
    \r [w_s2FM]
        case w_s2FM of {
          C:Monad _ _ ww3_s2FQ ww4_s2FR -> $wfoo ww3_s2FQ ww4_s2FR;
        };

In STG, whenever we have a let we have to do a heap allocation - and this code has quite a few! Of particular interest is the what’s going on inside the actual loop $sgo_s2FC. This loop first compares i to see if it’s 0. In the case that’s it’s not, we allocate two objects and call ww_s2Fa. If you squint, you’ll notice that ww_s2FA is the first argument to $wfoo, and it ultimately comes from unpacking a C:Monad dictionary. I’ll save you the labor of working out what this is - ww_s2Fa is the >>. We can see that every iteration of our loop incurs two allocations for each argument to >>. A heap allocation doesn’t come for free - not only do we have to do the allocation, the entry into the heap incurs a pointer indirection (as heap objects have an info table that points to their entry), and also by merely being on the heap we increase our GC time as we have a bigger heap to traverse. While my STG knowledge isn’t great, my understanding of this code is that every time we want to call >>, we need to supply it with its arguments. This means we have to allocate two closures for this function call - which is basically whenever we pressed “return” on our keyboard when we wrote the code. This seems crazy - can you imagine if you were told in C that merely using ; would cost time and memory?

If we compile this code in a separate module, mark it as {-# NOINLINE #-}, and then call it from main - how’s the performance? Let’s check!

module Main (main) where

import Foo

main :: IO ()
main = print =<< foo
$ ./Main +RTS -s
1000000000
 176,000,051,368 bytes allocated in the heap
       8,159,080 bytes copied during GC
          44,408 bytes maximum residency (1 sample(s))
          33,416 bytes maximum slop
               0 MB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0     169836 colls,     0 par    0.358s   0.338s     0.0000s    0.0001s
  Gen  1         1 colls,     0 par    0.000s   0.000s     0.0001s    0.0001s

  INIT    time    0.000s  (  0.000s elapsed)
  MUT     time   54.589s  ( 54.627s elapsed)
  GC      time    0.358s  (  0.338s elapsed)
  EXIT    time    0.000s  (  0.000s elapsed)
  Total   time   54.947s  ( 54.965s elapsed)

  %GC     time       0.0%  (0.0% elapsed)

  Alloc rate    3,224,078,302 bytes per MUT second

  Productivity  99.3% of total user, 99.4% of total elapsed

OUCH. My i7 laptop took almost a minute to iterate a loop 1 billion times.

A little disclaimer: I’m intentionally painting a severe picture here - in practice this cost is irrelevant to all but the most performance sensitive programs. Also, notice where the let bindings are in the STG above - they are nested within the loop. This means that we’re essentially allocating “as we go” - these allocations are incredibly cheap, and the growth to GC is equal trivial, resulting in more like constant GC pressure, rather than impending doom. For code that is likely to do any IO, this cost is likely negligible compared to the rest of the work. Nonetheless, it is there, and when it’s there, it’s nice to know if there are alternatives.

So, is the TL;DR that Haskell is completely incapable of writing effectful code? No, of course not. There is another way to compile this program, but we need a bit more information. If we happen to know what m is and we have access to the Monad dictionary for m, then we might be able to inline >>=. When we do this, GHC can be a lot smarter. The end result is code that now doesn’t allocate for every single >>=, and instead just gets on with doing work. One trivial way to witness this is to define everything in a single module (Alexis rightly points out this is a trap for benchmarking that many fall into, but for our uses it’s the behavior we actually want).

This time, let’s write everything in one module:

module Main ( main ) where

And the STG:

lvl_r4AM = CCS_DONT_CARE S#! [0#];

lvl1_r4AN = CCS_DONT_CARE S#! [1#];

Rec {
main_$sgo =
    \r [void_0E sc1_s4AY sc2_s4AZ]
        case eqInteger# sc1_s4AY lvl_r4AM of {
          __DEFAULT ->
              case +# [sc2_s4AZ 1#] of sat_s4B2 {
                __DEFAULT ->
                    case minusInteger sc1_s4AY lvl1_r4AN of sat_s4B1 {
                      __DEFAULT -> main_$sgo void# sat_s4B1 sat_s4B2;
                    };
              };
          1# -> let { sat_s4B3 = CCCS I#! [sc2_s4AZ]; } in  Unit# [sat_s4B3];
        };
end Rec }

main2 = CCS_DONT_CARE S#! [1000000000#];

main1 =
    \r [void_0E]
        case main_$sgo void# main2 0# of {
          Unit# ipv1_s4B7 ->
              let { sat_s4B8 = \s [] $fShowInt_$cshow ipv1_s4B7;
              } in  hPutStr' stdout sat_s4B8 True void#;
        };

main = \r [void_0E] main1 void#;

main3 = \r [void_0E] runMainIO1 main1 void#;

main = \r [void_0E] main3 void#;

The same program compiled down to much tighter loop that is almost entirely free of allocations. In fact, the only allocation that happens is when the loop terminates, and it’s just boxing the unboxed integer that’s been accumulating in the loop.

As we might hope, the performance of this is much better:

$ ./Main +RTS -s
1000000000
  16,000,051,312 bytes allocated in the heap
         128,976 bytes copied during GC
          44,408 bytes maximum residency (1 sample(s))
          33,416 bytes maximum slop
               0 MB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0     15258 colls,     0 par    0.031s   0.029s     0.0000s    0.0000s
  Gen  1         1 colls,     0 par    0.000s   0.000s     0.0001s    0.0001s

  INIT    time    0.000s  (  0.000s elapsed)
  MUT     time    9.402s  (  9.405s elapsed)
  GC      time    0.031s  (  0.029s elapsed)
  EXIT    time    0.000s  (  0.000s elapsed)
  Total   time    9.434s  (  9.434s elapsed)

  %GC     time       0.0%  (0.0% elapsed)

  Alloc rate    1,701,712,595 bytes per MUT second

  Productivity  99.7% of total user, 99.7% of total elapsed

Our time in the garbage collector dropped by a factor of 10, from 0.3s to 0.03. Our total allocation dropped from 176GB (yes, you read that right) to 16GB (I’m still not entirely sure what this means, maybe someone can enlighten me). Most importantly our total runtime dropped from 54s to just under 10s. All this from just knowing what m is at compile time.

So GHC is capable of producing excellent code for monads - what are the circumstances under which this happens? We need, at least:

  1. The source code of the thing we’re compiling must be available. This means it’s either defined in the same module, or is available with an INLINABLE pragma (or GHC has chosen to add this itself).

  2. The definitions of >>= and friends must also be available in the same way.

These constraints start to feel a lot like needing whole program compilation, and in practice are unreasonable constraints to reach. To understand why, consider that most real world programs have a small Main module that opens some connections or opens some file handles, and then calls some library code defined in another module. If this code in the other module was already compiled, it will (probably) have been compiled as a function that takes a Monad dictionary, and just calls the >>= function repeatedly in the same manner as our original STG code. To get the allocation-free version, this library code needs to be available to the Main module itself - as that’s the module that choosing what type to instantiate ‘m’ with - which means the library code has to have marked that code as being inlinable. While we could add INLINE everywhere, this leads to an explosion in the amount of code produced, and can sky rocket compilation times.

Alexis’ eff library works around this by not being polymorphic in m. Instead, it chooses a concrete monad with all sorts of fancy continuation features. Likewise, if we commit to a particular monad (a transformer stack, or maybe using RIO), we again avoid this cost. Essentially, if the monad is known a priori at time of module compilation, GHC can go to town. However, the latter also commits to semantics - by choosing a transformer stack, we’re choosing a semantics for our monadic effects.

With the scene set, I now want to present you with another approach to solving this problem using Backpack.

A Backpack Primer

Vanilla GHC has a very simple module system - modules are essentially a method for name-spacing and separate compilation, they don’t do much more. The Backpack project extends this module system with a new concept - signatures. A signature is like the “type” of a module - a signature might mention the presence of some types, functions and type class instances, but it says nothing about what the definitions of these entities are. We’re going to (ab)use this system to build up transformer stacks at configuration time, and allow our library to be abstracted over different monads. By instantiating our library code with different monads, we get different interpretations of the same program.

I won’t sugar coat - what follows is going to pretty miserable. Extremely fun, but miserable to write in practice. I’ll let you decide if you want to inflict this misery on your coworkers in practice - I’m just here to show you it can be done!

A Signature for Monads

The first thing we’ll need is a signature for data types that are monads. This is essentially the “hole” we’ll rely on with our library code - it will give us the ability to say “there exists a monad”, without committing to any particular choice.

In our Cabal file, we have:

library monad-sig
  hs-source-dirs:   src-monad-sig
  signatures:       Control.Monad.Signature
  default-language: Haskell2010
  build-depends:    base

The important line here is signatures: Control.Monad.Signature which shows that this library is incomplete and exports a signature. The definition of Control/Monad/Signature.hsig is:

signature Control.Monad.Signature where

data M a
instance Functor M
instance Applicative M
instance Monad M

This simply states that any module with this signature has some type M with instances of Functor, Applicative and Monad.

Next, we’ll put that signature to use in our library code.

Libary Code

For our library code, we’ll start with a new library in our Cabal file:

library business-logic
  hs-source-dirs:   lib
  signatures:       BusinessLogic.Monad
  exposed-modules:  BusinessLogic
  build-depends:
    , base
    , fused-effects
    , monad-sig

  default-language: Haskell2010
  mixins:
    monad-sig requires (Control.Monad.Signature as BusinessLogic.Monad)

Our business-logic library itself exports a signature, which is really just a re-export of the Control.Monad.Signature, but we rename it something more meaningful. It’s this module that will provide the monad that has all of the effects we need. Along with this signature, we also export the BusinessLogic module:

{-# language FlexibleContexts #-}
module BusinessLogic where

import BusinessLogic.Monad ( M )
import Control.Algebra ( Has )
import Control.Effect.Empty ( Empty, guard )

businessCode :: Has Empty sig M => Bool -> M Int
businessCode b = do
  guard b
  return 42

In this module I’m using fused-effects as a framework to say which effects my monad should have (though this is not particularly important, I just like it!). Usually Has would be applied to a type variable m, but here we’re applying it to the type M. This type comes from BusinessLogic.Monad, which is a signature (you can confirm this by checking against the Cabal file). Other than that, this is all pretty standard!

Backpack-ing Monad Transformers

Now we get into the really fun stuff - providing implementations of effects. I mentioned earlier that one possible way to do this is with a stack of monad transformers. Generally speaking, one would write a single newtype T m a for each effect type class, and have that transformer dispatch any effects in that class, and to lift any effects from other classes - deferring their implementation to m.

We’re going to take the same approach here, but we’ll absorb the idea of a transformer directly into the module itself. Let’s look at an implementation of the Empty effect. The Empty effect gives us a special empty :: m a function, which serves the purpose of stopping execution immediately. As a monad transformer, one implementation is MaybeT:

newtype MaybeT m a = MaybeT { runMaybeT :: m (Maybe a) }

But we can also write this using Backpack. First, our Cabal library:

library fused-effects-empty-maybe
  hs-source-dirs:   src-fused-effects-backpack
  default-language: Haskell2010
  build-depends:
    , base
    , fused-effects
    , monad-sig

  exposed-modules: Control.Carrier.Backpack.Empty.Maybe
  mixins:
    monad-sig requires (Control.Monad.Signature as Control.Carrier.Backpack.Empty.Maybe.Base)

Our library exports the module Control.Carrier.Backpack.Empty.Maybe, but also has a hole - the type of base monad this transformer stacks on top of. As a monad transformer, this would be the m parameter, but when we use Backpack, we move that out into a separate module.

The implementation of Control.Carrier.Backpack.Empty.Maybe is short, and almost identical to the body of Control.Monad.Trans.Maybe - we just change any occurrences of m to instead refer to M from our .Base module:

{-# language BlockArguments, FlexibleContexts, FlexibleInstances, LambdaCase,
      MultiParamTypeClasses, TypeOperators, UndecidableInstances #-}

module Control.Carrier.Backpack.Empty.Maybe where

import Control.Algebra
import Control.Effect.Empty
import qualified Control.Carrier.Backpack.Empty.Maybe.Base as Base

type M = EmptyT

-- We could also write: newtype EmptyT a = EmptyT { runEmpty :: MaybeT Base.M a }
newtype EmptyT a = EmptyT { runEmpty :: Base.M (Maybe a) }

instance Functor EmptyT where
  fmap f (EmptyT m) = EmptyT $ fmap (fmap f) m

instance Applicative EmptyT where
  pure = EmptyT . pure . Just
  EmptyT f <*> EmptyT x = EmptyT do
    f >>= \case
      Nothing -> return Nothing
      Just f' -> x >>= \case
        Nothing -> return Nothing
        Just x' -> return (Just (f' x'))

instance Monad EmptyT where
  return = pure
  EmptyT x >>= f = EmptyT do
    x >>= \case
      Just x' -> runEmpty (f x')
      Nothing -> return Nothing

Finally, we make sure that Empty can handle the Empty effect:

instance Algebra sig Base.M => Algebra (Empty :+: sig) EmptyT where
  alg handle sig context = case sig of
    L Empty -> EmptyT $ return Nothing
    R other -> EmptyT $ thread (maybe (pure Nothing) runEmpty ~<~ handle) other (Just context)

Base Monads

Now that we have a way to run the Empty effect, we need a base case to our transformer stack. As our transformer is now built out of modules that conform to the Control.Monad.Signature signature, we need some modules for each monad that we could use as a base. For this POC, I’ve just added the IO monad:

library fused-effects-lift-io
  hs-source-dirs:   src-fused-effects-backpack
  default-language: Haskell2010
  build-depends:    base
  exposed-modules:  Control.Carrier.Backpack.Lift.IO
module Control.Carrier.Backpack.Lift.IO where
type M = IO

That’s it!

Putting It All Together

Finally we can put all of this together into an actual executable. We’ll take our library code, instantiate the monad to be a combination of EmptyT and IO, and write a little main function that unwraps this all into an IO type. First, here’s the Main module:

module Main where

import BusinessLogic
import qualified BusinessLogic.Monad

main :: IO ()
main = print =<< BusinessLogic.Monad.runEmptyT (businessCode True)

The BusinessLogic module we’ve seen before, but previously BusinessLogic.Monad was a signature (remember, we renamed Control.Monad.Signature to BusinessLogic.Monad). In executables, you can’t have signatures - executables can’t be depended on, so it doesn’t make sense for them to have holes, they must be complete. The magic happens in our Cabal file:

executable test
  main-is:          Main.hs
  hs-source-dirs:   exe
  build-depends:
    , base
    , business-logic
    , fused-effects-empty-maybe
    , fused-effects-lift-io
    , transformers

  default-language: Haskell2010
  mixins:
    fused-effects-empty-maybe (Control.Carrier.Backpack.Empty.Maybe as BusinessLogic.Monad) requires (Control.Carrier.Backpack.Empty.Maybe.Base as BusinessLogic.Monad.Base),
    fused-effects-lift-io (Control.Carrier.Backpack.Lift.IO as BusinessLogic.Monad.Base)

Wow, that’s a mouthful! The work is really happening in mixins. Let’s take this step by step:

  1. First, we can see that we need to mixin the fused-effects-empty-maybe library. The first (X as Y) section specifies a list of modules from fused-effects-empty-maybe and renames them for the test executable that’s currently being compiled. Here, we’re renaming Control.Carrier.Backpack.Empty.Maybe as BusinessLogic.Monad. By doing this, we satisfy the hole in the business-logic library, which was otherwise incomplete.

  2. But fused-effects-empty-maybe itself has a hole - the base monad for the transformer. The requires part lets us rename this hole, but we’ll still need to plug it. For now, we rename Control.Carrier.Backpack.Empty.Maybe.Base).

  3. Next, we mixin the fused-effects-lift-io library, and rename Control.Carrier.Backpack.Lift.IO to be BusinessLogic.Monad.Base. We’ve now satisfied the hole for fused-effects-empty-maybe, and our executable has no more holes and can be compiled.

We’re Done!

That’s “all” there is to it. We can finally run our program:

$ cabal run
Just 42

If you compare against businessCode you’ll see that we got passed the guard and returned 42. Because we instantiated BusinessLogic.Monad with a MaybeT-like transformer, this 42 got wrapped up in Just.

Is This Fast?

The best check here is to just look at the underlying code itself. If we add

{-# options -ddump-simpl -ddump-stg -dsuppress-all #-}

to BusinessLogic and recompile, we’ll see the final code output to STDERR. The core is:

businessCode1
  = \ @ sig_a2cM _ b_a13P eta_B1 ->
      case b_a13P of {
        False -> (# eta_B1, Nothing #);
        True -> (# eta_B1, lvl1_r2NP #)
      }

and the STG:

businessCode1 =
    \r [$d(%,%)_s2PE b_s2PF eta_s2PG]
        case b_s2PF of {
          False -> (#,#) [eta_s2PG Nothing];
          True -> (#,#) [eta_s2PG lvl1_r2NP];
        };

Voila!

Conclusion

In this post, I’ve hopefully shown how we can use Backpack to write effectful code without paying the cost of abstraction. What I didn’t answer is the question of whether or you not you should. There’s a lot more to effectful code than I’ve presented, and it’s unclear to me whether this approach can scale to the needs. For example, if we needed something like mmorph’s MFunctor, what do we do? Are we stuck? I don’t know! Beyond these technical challenges, it’s clear that Backpack here is also not remotely ergonomic, as is. We’ve had to write five components just to get this done, and I pray for any one who comes to read this code and has to orientate themselves.

Nonetheless, I think this an interesting point of the effect design space that hasn’t been explored, and maybe I’ve motivated some people to do some further exploration.

The code for this blog post can be found at https://github.com/ocharles/fused-effects-backpack.

Happy holidays, all!

December 21, 2020

Sevan Janiyan (sevan)

LFS, round #4 December 21, 2020 09:49 PM

Haven’t made any progress for a couple of weeks but things came together and instrumenting libc works as expected. One example demonstrated in section 12.2.2, chapter 12 of the BPF Performance book is attempting to instrument bash compiled without frame pointers where you only see a call to the read function. Compiling with -fno-omit-frame-pointer produces …

December 20, 2020

Derek Jones (derek-jones)

Many coding mistakes are not immediately detectable December 20, 2020 10:04 PM

Earlier this week I was reading a paper discussing one aspect of the legal fallout around the UK Post-Office’s Horizon IT system, and was surprised to read the view that the Law Commission’s Evidence in Criminal Proceedings Hearsay and Related Topics were citing on the subject of computer evidence (page 204): “most computer error is either immediately detectable or results from error in the data entered into the machine”.

What? Do I need to waste any time explaining why this is nonsense? It’s obvious nonsense to anybody working in software development, but this view is being expressed in law related documents; what do lawyers know about software development.

Sometimes fallacies become accepted as fact, and a lot of effort is required to expunge them from cultural folklore. Regular readers of this blog will have seen some of my posts on long-standing fallacies in software engineering. It’s worth collecting together some primary evidence that most software mistakes are not immediately detectable.

A paper by Professor Tapper of Oxford University is cited as the source (yes, Oxford, home of mathematical orgasms in software engineering). Tapper’s job title is Reader in Law, and on page 248 he does say: “This seems quite extraordinarily lax, given that most computer error is either immediately detectable or results from error in the data entered into the machine.” So this is not a case of his words being misinterpreted or taken out of context.

Detecting many computer errors is resource intensive, both in elapsed time, manpower and compute time. The following general summary provides some of the evidence for this assertion.

Two events need to occur for a fault experience to occur when running software:

  • a mistake has been made when writing the source code. Mistakes include: a misunderstanding of what the behavior should be, using an algorithm that does not have the desired behavior, or a typo,
  • the program processes input values that interact with a coding mistake in a way that produces a fault experience.

That people can make different mistakes is general knowledge. It is my experience that people underestimate the variability of the range of values that are presented as inputs to a program.

A study by Nagel and Skrivan shows how variability of input values results in fault being experienced at different time, and that different people make different coding mistakes. The study had three experienced developers independently implement the same specification. Each of these three implementations was then tested, multiple times. The iteration sequence was: 1) run program until fault experienced, 2) fix fault, 3) if less than five faults experienced, goto step (1). This process was repeated 50 times, always starting with the original (uncorrected) implementation; the replications varied this, along with the number of inputs used.

How many input values needed to be processed, on average, before a particular fault is experienced? The plot below (code+data) shows the numbers of inputs processed, by one of the implementations, before individual faults were experienced, over 50 runs (sorted by number of inputs needed before the fault was experienced):

Number of inputs processed before particular fault experienced

The plot illustrates that some coding mistakes are more likely to produce a fault experience than others (because they are more likely to interact with input values in a way that generates a fault experience), and it also shows how the number of inputs values processed before a particular fault is experienced varies between coding mistakes.

Real-world evidence of the impact of user input on reported faults is provided by the Ultimate Debian Database, which provides information on the number of reported faults and the number of installs for 14,565 packages. The plot below shows how the number of reported faults increases with the number of times a package has been installed; one interpretation is that with more installs there is a wider variety of input values (increasing the likelihood of a fault experience), another is that with more installs there is a larger pool of people available to report a fault experience. Green line is a fitted power law, faultsReported=1.3*installs^{0.3}, blue line is a fitted loess model.

Number of inputs processed before particular fault experienced

The source containing a mistake may be executed without a fault being experienced; reasons for this include:

  • the input values don’t result in the incorrect code behaving differently from the correct code. For instance, given the made-up incorrect code if (x 8) (i.e., 8 was typed rather than 7), the comparison only produces behavior that differs from the correct code when x has the value 7,
  • the input values result in the incorrect code behaving differently than the correct code, but the subsequent path through the code produces the intended external behavior.

Some of the studies that have investigated the program behavior after a mistake has deliberately been introduced include:

  • checking the later behavior of a program after modifying the value of a variable in various parts of the source; the results found that some parts of a program were more susceptible to behavioral modification (i.e., runtime behavior changed) than others (i.e., runtime behavior not change),
  • checking whether a program compiles and if its runtime behavior is unchanged after random changes to its source code (in this study, short programs written in 10 different languages were used),
  • 80% of radiation induced bit-flips have been found to have no externally detectable effect on program behavior.

What are the economic costs and benefits of finding and fixing coding mistakes before shipping vs. waiting to fix just those faults reported by customers?

Checking that a software system exhibits the intended behavior takes time and money, and the organization involved may not be receiving any benefit from its investment until the system starts being used.

In some applications the cost of a fault experience is very high (e.g., lowering the landing gear on a commercial aircraft), and it is cost-effective to make a large investment in gaining a high degree of confidence that the software behaves as expected.

In a changing commercial world software systems can become out of date, or superseded by new products. Given the lifetime of a typical system, it is often cost-effective to ship a system expected to contain many coding mistakes, provided the mistakes are unlikely to be executed by typical customer input in a way that produces a fault experience.

Beta testing provides selected customers with an early version of a new release. The benefit to the software vendor is targeted information about remaining coding mistakes that need to be fixed to reduce customer fault experiences, and the benefit to the customer is checking compatibility of their existing work practices with the new release (also, some people enjoy being able to brag about being a beta tester).

  • One study found that source containing a coding mistake was less likely to be changed due to fixing the mistake than changed for other reasons (that had the effect of causing the mistake to disappear),
  • Software systems don't live forever; systems are replaced or cease being used. The plot below shows the lifetime of 202 Google applications (half-life 2.9 years) and 95 Japanese mainframe applications from the 1990s (half-life 5 years; code+data).

    Number of software systems having a given lifetime, in days

Not only are most coding mistakes not immediately detectable, there may be sound economic reasons for not investing in detecting many of them.

December 19, 2020

Jeff Carpenter (jeffcarp)

How to clone a Google Source Repository in Working Copy iOS December 19, 2020 09:00 AM

I recently went through this process and couldn’t find a guide (though I swear one existed at some point in the past). Here’s how to clone a git repository from Google Source Repositories in the Working Copy iOS or iPadOS app: Navigate to https://source.cloud.google.com/ Pick out the repo you want to clone and open the clone dialog (it looks like a “+” icon) Go to the “Manually generated credentials” tab Click on “Generate and store your Git credentials” Go through the authentication flow You should be prompted to copy a shell command onto your clipboard Go to the Working Copy iOS app Go to Settings (the gear icon) > Authentication Cookies Tap the “+” icon and import from clipboard You should now be able to clone the repository using the https://source.

December 16, 2020

Pierre Chapuis (catwell)

Personal news, 2020 edition December 16, 2020 01:50 PM

I haven't posted anything here for 6 months so I thought it would be a good idea to post a personal news update before the end of 2020.

Shutting down Chilli

I haven't posted about it here yet, but my co-founder Julien and me decided to shut down Chilli at the end of last year, in agreement with eFounders. Basically, some of our initial hypotheses about the state of the market we were in (french SMBs) were wrong and the business couldn't be profitable enough.

In retrospect, it was an interesting experience. I really appreciated being part of eFounders, I do recommend it for people who want to start a B2B SaaS company.

However, I wanted to get out of my comfort bubble too much with this one, by tackling a problem in a market I didn't know. Because of that, I had to rely on others regarding core business choices, and while that was fine with me initially it ended up being frustrating in the end when things didn't work well and I couldn't help much. So if I start another company someday, it will be one where I know the problem domain better.

Joining Inch

After we shut down Chilli, I decided to join a small startup editing a SaaS CRM and ticketing solution for property managers called Inch, only available in the French market for now.

I didn't pick Inch randomly, I have known one of the founders for years since he was a Moodstocks user! Fun fact: at the time, I had met him and he was looking for a technical co-founder for his project. I told him he should learn to code instead and gave him Ruby sample code... and this is why I am back to Rails today. Karma? :)

Anyway, I had been following Inch with interest since its creation, because it is the company I like: solving a real need in a market where tools were either terrible or non-existent. Now they have a unique place in the market and some interesting technical challenges to solve, so I decided to join and help.

Starting a family

And here comes the last piece of news: despite all the madness that happened this year, the biggest change for me is that I am now a father! My son was born last week, and I took a holiday until the end of the year to spend time with him and his mother. I don't intend to post too much personal news here, but this one deserved it. =)

Awn Umar (awn)

rosen: censorship-resistant proxy tunnel December 16, 2020 12:00 AM

GitHub: https://github.com/awnumar/rosen

Many governments and other well-resourced actors around the world implement some form of censorship to assert control over the flow of information on the Internet, either because it is deemed “sensitive” or because it is inconvenient for those with power.

Suppose there is some adversary, Eve, that wants to prevent users from accessing some content. There are many ways of implementing such censorship but they broadly fall into one of two categories: endpoint-based or flow-fingerprinting attacks.

A user attempting to access censored material through an adversarial Internet service provider.

A user attempting to access censored material through an adversarial Internet service provider.

Eve could maintain a list of banned services and refuse to serve any network request she receives if it is on this list. Here Eve is deciding based on the destination of the request: this is endpoint-based censorship. In response a user, Alice, could use a VPN or TOR to disguise the destination so that from Eve’s perspective, the destination will appear to be the VPN server or the TOR entry node instead of the censored service.

This is a working solution in many places, but Eve is not beaten. In response she can add the IP addresses of known VPN providers as well as public TOR nodes to her blocklist, reasoning that only a user who wants to bypass her blocking efforts would use these services. Alice could then setup her own VPN server or access the TOR network through a non-public TOR bridge that is not blocked.

Eve could actively probe the servers that Alice connects to in order to find out if they are TOR entry nodes, for example, but apart from this she has stretched endpoint-based censorship to its limits. An alternative is to censor a connection based on characteristics of the network flow instead of its destination: this is flow-fingerprinting. This is usually accomplished using some kind of deep packet inspection engine that can detect and group traffic into protocols and applications. With this capability Eve can block any traffic that is detected as a proxy regardless of whether any particular server is known.

An adversary using a deep packet inspection engine to decide whether to censor traffic.

An adversary using a deep packet inspection engine to decide whether to censor traffic.

To bypass this technique, Alice must disguise the fingerprint of her traffic so that a DPI engine does not classify it as a blocked protocol or application. There are a few approaches to this:

  1. Randomisation. The goal here is to make the traffic look indistinguishable from randomness, or put another way, to make it look like “nothing”. This would successfully hide which category traffic belongs to, but a lack of a fingerprint is a fingerprint itself and that’s a vulnerability.

    Examples of randomising obfuscators include Obfsproxy and ScrambleSuit.

  2. Mimicry. Instead of making traffic look like random noise, mimicry-based obfuscation makes packets look like they belong to a specific protocol or application that is assumed to be unblocked. For example, StegoTorus and SkypeMorph produce traffic that looks like HTTP and Skype, respectively, but they are prohibitively slow.

    Another option is LibFTE which is roughly a cryptographic cipher that produces ciphertext conforming to a given regular expression. DPI engines also commonly use regular expressions so with LibFTE it is possible to precisely force misclassification of a protocol.

    Mimicry only tries to make packet payloads look like some cover protocol and so the syntax and semantics of the overall network flow can deviate substantially from the protocol specification or any known implementation. This makes mimicry-based obfuscators easily detectable and results in the approach being fundamentally flawed.

  3. Tunnelling. A tunnelling obfuscator encapsulates traffic within some cover protocol using an actual implementation of the cover protocol instead of simply trying to mimic the way its packets look. An example is meek which uses HTTPS to communicate between the client and the server and domain fronting to hide the true destination of the traffic, but since domain fronting relied on an undocumented feature of major CDNs, it no longer works.

    Tunnelling obfuscators have to be careful to look like a specific and commonly used implementation of a cover protocol since a custom implementation may be distinguishable. China and Iran managed to distinguish TOR TLS and non-TOR TLS first-hop connections even though TOR used a real implementation of TLS to tunnel traffic.

An important metric to consider is the false positive detection rate associated with each method. This is the proportion of traffic that a DPI engine falsely detects as coming from an obfuscation tool. A high false-positive rate results in lots of innocent traffic being blocked which will cause frustration for ordinary users. Therefore the goal of an obfuscator should be to look as much like innocent traffic as possible to maximise the “collateral damage” of any attempted censorship. Overall, it seems like tunnelling is the best approach.

This brings us to Rosen, a modular, tunnelling proxy that I have developed as part of my ongoing masters thesis. It currently only implements HTTPS as a cover protocol, but this has been tested against nDPI and a commercial DPI engine developed by Palo Alto Networks, both of which detected TOR traffic encapsulated by Rosen as ordinary HTTPS. The goals of Rosen are:

  1. Unobservability. It should be difficult to distinguish obfuscated traffic from innocent background traffic using the same protocol.
  2. Endpoint-fingerprinting resistance. It should be difficult to use active probing to ascertain that a given server is actually a proxy server. This is accomplished by responding as a proxy if and only if a valid key is provided and falling back to some default behaviour otherwise. For example, the HTTPS implementation serves some static content in this case.
  3. Modularity. It should be relatively easy to add support for another cover protocol or configure the behaviour of an existing protocol to adapt to changing adversarial conditions. This is facilitated by a modular architecture.
  4. Compatibility. It should be possible to route most application traffic through the proxy. This is why a SOCKS interface was chosen, but TUN support is also a goal.
  5. Usability. It should be easy to use.
High-level overview of Rosen's architecture.

High-level overview of Rosen's architecture.

HTTPS was chosen as the first cover protocol to be implemented as it provides confidentiality, authenticity, and integrity; and it is ubiquitous on the Internet making it infeasible for an adversary to block. The implementation is provided by the Go standard library and most configuration options are set to their defaults so that it blends in with other applications. There is a option to disable TLS 1.3 as it could be blocked by some nation-state firewalls I was informed that censors are blocking ESNI specifically. The server will automatically provision a TLS certificate from LetsEncrypt and the client pins LetsEncrypt’s root by default.

It’s difficult to know how effective this truly is without further battle-testing by security researchers and users, but we can theorise to some extent.

  1. Endpoint-based censorship. Users are able to setup Rosen on their own servers behind their own domains so there is no generic firewall rule that can block all of them. An adversary could instead try to actively probe a Rosen server in order to detect it.

    One option is to provide a key and detect a timing difference as the server checks it. The delta I measured between providing a 32 byte key and not providing a key is 29ns (on an AMD Ryzen 3700X). Since network requests have a latency in the milliseconds, I assume this attack is practically infeasible.

    A simpler attack is to look at the static files that the HTTPS server responds with. If the user does not replace the default files with their own, an easy distinguishing attack is possible. This could be easier to avoid with a different protocol. For example, if an incorrect SSH password is provided to an SSH server, it simply refuses the connection and there are no other obvious side-effects for an adversary to analyse.

  2. Flow-fingerprinting. The cover protocol uses the standard library implementation of HTTPS which should be widely used by many different applications in various contexts. Default cipher suites are chosen and other aspects of the implementation are deliberately very typical.

    However, this does not cover the behaviour of Rosen clients. For example, HTTP requests to an ordinary website are usually a lot smaller than responses. Also, an adversary could compare the traffic between Alice and a Rosen HTTPS server with the static content available on that server to ascertain if something else is going on.

    To handle these attacks, the protocol could use some kind of random padding, limit the size and frequency of round trips, or replace the static decoy handler with a custom one that has different traffic characteristics.

    Timing patterns are particularly of importance. Currently the client waits a random interval between 0 and 100ms before polling the server for data. This choice was made to minimise latency but it is not typical of an ordinary website. Analysing timing patterns is what allowed researchers to detect meek, for example. There’s no evidence that this attack is employed by real-world censors, but a configuration flag that implements a tradeoff between performance and behaving “more typically” will be added in the future.

If you have the capability to test out Rosen, especially if you are behind a firewall that implements censorship, I would greatly appreciate you telling me about about your experiences at my Email address (available on GitHub and on this website’s homepage). If you want to contribute, you can open an issue or pull request on the project’s GitHub page.

December 14, 2020

Caius Durling (caius)

Disable Google Autoupdater on macOS December 14, 2020 04:14 PM

From reading Chrome is Bad, it seems in some situations the updater (also known as keystone) can chew up CPU cycles. Whilst I’m not 100% convinced keystone continuously chews CPU, its launchctl configuration suggests it runs at least once an hour. Given I don’t use Chrome as my main browser, this is undesirable behaviour for me.

With that in mind, I’ve decided to disable the background services rather than delete Chrome entirely. (I need it occasionally.) Stopping/unloading the services and fettling the config files to do nothing achieves this aim (and stops Chrome re-enabling them next launch), whilst leaving Chrome fully functional when needed.

  1. Unload the currently loaded services

    launchctl unload -w ~/Library/LaunchAgents/com.google.keystone.xpcservice.plist
    launchctl unload -w ~/Library/LaunchAgents/com.google.keystone.agent.plist
    
  2. Empty the config files, so if launchd ever tries to launch them they’ll just error out

    echo > ~/Library/LaunchAgents/com.google.keystone.xpcservice.plist
    echo > ~/Library/LaunchAgents/com.google.keystone.agent.plist
    
  3. Change ownership and permissions of these files so only root can write to the files

    chmod 644 ~/Library/LaunchAgents/com.google.keystone.xpcservice.plist
    chmod 644 ~/Library/LaunchAgents/com.google.keystone.agent.plist
    sudo chown root ~/Library/LaunchAgents/com.google.keystone.xpcservice.plist
    sudo chown root ~/Library/LaunchAgents/com.google.keystone.agent.plist
    

Now when I want to update Chrome once in a blue moon when I need it, I can navigate to chrome://settings/help to update (or from the UI, Chrome -> About Chrome.)

December 13, 2020

Derek Jones (derek-jones)

Survival rate of WG21 meeting attendance December 13, 2020 10:55 PM

WG21, the C++ Standards committee, has a very active membership, with lots of people attending the regular meetings; there are three or four meetings a year, with an average meeting attendance of 67 (between 2004 and 2016).

The minutes of WG21 meetings list those who attend, and a while ago I downloaded these for meetings between 2004 and 2016. Last night I scraped the data and cleaned it up (or at least the attendee names).

WG21 had its first meeting in 1992, and continues to have meetings (eleven physical meetings at the time or writing). This means the data is both left and right censored; known as interval censored. Some people will have attended many meetings before the scraped data starts, and some people listed in the data may not have attended another meeting since.

What can we say about the survival rate of a person being a WG21 attendee in the future, e.g., what is the probability they will attend another meeting?

Most regular attendees are likely to miss a meeting every now and again (six people attended all 30 meetings in the dataset, with 22 attending more than 25), and I assumed that anybody who attended a meeting after 1 January 2015 was still attending. Various techniques are available to estimate the likelihood that known attendees were attending meetings prior to those in the dataset (I’m going with what ever R’s survival package does). The default behavior of R’s Surv function is to handle right censoring, the common case. Extra arguments are needed to handle interval censored data, and I think I got these right (I had to cast a logical argument to numeric for some reason; see code+data).

The survival curves in days since 1 Jan 2004, and meetings based on the first meeting in 2004, with 95% confidence bounds, look like this:


Meeting survival curve of WG21 attendees.

I was expecting a sharper initial reduction, and perhaps wider confidence bounds. Of the 374 people listed as attending a meeting, 177 (47%) only appear once and 36 (10%) appear twice; there is a long tail, with 1.6% appearing at every meeting. But what do I know, my experience of interval censored data is rather limited.

The half-life of attendance is 9 to 10 years, suspiciously close to the interval of the data. Perhaps a reader will scrape the minutes from earlier meetings :-)

Within the time interval of the data, new revisions of the C++ standard occurred in 20072011 and 2014; there had also been a new release in 2003, and one was being worked on for 2017. I know some people stop attending meetings after a major milestone, such as a new standard being published. A fancier analysis would investigate the impact of standards being published on meeting attendance.

People also change jobs. Do WG21 attendees change jobs to ones that also require/allow them to attend WG21 meetings? The attendee’s company is often listed in the minutes (and is in the data). Something for intrepid readers to investigate.

Ponylang (SeanTAllen)

Last Week in Pony - December 13, 2020 December 13, 2020 09:07 PM

Version 0.2.2 of ponylang/http_server has been released.

Gonçalo Valério (dethos)

Mirroring GitHub Repositories December 13, 2020 03:58 PM

Git by itself is a distributed version control system (a very popular one), but over the years organizations started to rely on some internet services to manage their repositories and those services eventually become the central/single source of truth for their code.

The most well known service out there is GitHub (now owned by Microsoft), which nowadays is synonymous of git for a huge amount of people. Many other services exist, such as Gitlab and BitBucked, but GitHub gained a notoriety above all others, specially for hosting small (and some large) open source projects.

These centralized services provide many more features that help managing, testing and deploying software. Functionality not directly related to the main purpose of git.

Relying on these central services is very useful but as everything in life, it is a trade-off. Many large open source organizations don’t rely on these companies (such as KDE, Gnome, Debian, etc), because the risks involved are not worth the convenience of letting these platforms host their code and other data.

Over time we have been witnessing some of these risks, such as your project (and all the related data) being taken down without you having any chance to defend yourself (Example 1 and Example 2). Very similar to what some content creators have been experiencing with Youtube (I really like this one).

When this happens, your or your organizations don’t lose the code itself since you almost certainly have copies on your own devices (thanks to git), but you lose everything else, issues, projects, automated actions, documentation and essentially the known used by URL of your project.

Since Github is just too convenient to collaborate with other people, we can’t just leave. In this post I explain an easy alternative to minimize the risks described above, that I implemented myself after reading many guides and tools made by others that also tried to address this problem before.

The main idea is to automatically mirror everything in a machine that I own and make it publicly available side by side with the GitHub URLs, the work will still be done in Github but can be easily switched over if something happens.

The software

To achieve the desired outcome I’ve researched a few tools and the one that seemed to fit all my requirements (work with git and be lightweight) was “Gitea“. Next I will describe the steps I took.

The Setup

This part was very simple, I just followed the instructions present on the documentation for a docker based install. Something like this:

version: "3"

networks:
  gitea:
    external: false

services:
  server:
    image: gitea/gitea:latest
    container_name: gitea
    environment:
      - USER_UID=1000
      - USER_GID=1000
    restart: always
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"

If you are doing the same, don’t copy the snippet above. Take look here for updated instructions.

Since my website is not supposed to have much concurrent activity, using an SQLite database is more than enough. So after launching the container, I chose this database type and made sure I disabled the all the functionality I won’t need.

Part of Gitea's configuration page. Server and Third-Party Service Settings.Part of the Gitea’s configuration page

After this step, you should be logged in as an admin. The next step is to create a new migration on the top right menu. We just need to choose the “Github” option and continue. You should see the below screen:

Screenshot of the page that lets the users create a new migration/mirror in Gitea.Creating a new Github migration/mirror in Gitea.

If you choose This repository will be a mirror option, Gitea will keep your repository and wiki in sync with the original, but unfortunately it will not do the same for issues, labels, milestones and releases. So if you need that information, the best approach is to uncheck this field and do a normal migration. To keep that information updated you will have to repeat this process periodically.

Once migrated, do the same for your other repositories.

Conclusion

Having an alternative with a backup of the general Github data ended up being quite easy to set up. However the mirror feature would be much more valuable if it included the other items available on the standard migration.

During my research for solutions, I found Fossil, which looks very interesting and something that I would like to explore in the future, but at the moment all repositories are based on Git and for practical reasons that won’t change for the time being.

With this change, my public repositories can be found in:

December 07, 2020

Ponylang (SeanTAllen)

Last Week in Pony - December 6, 2020 December 07, 2020 03:57 AM

The audio recording of the December 1, 2020 Pony development sync call is available.

December 06, 2020

Derek Jones (derek-jones)

Christmas books for 2020 December 06, 2020 10:54 PM

A very late post on the interesting books I read this year (only one of which was actually published in 2020). As always the list is short because I did not read many books and/or there is lots of nonsense out there, but this year I have the new excuses of not being able to spend much time on trains and having my own book to finally complete.

I have already reviewed The Weirdest People in the World: How the West Became Psychologically Peculiar and Particularly Prosperous, and it is the must-read of 2020 (after my book, of course :-).

The True Believer by Eric Hoffer. This small, short book provides lots of interesting insights into the motivational factors involved in joining/following/leaving mass movements. Possible connections to software engineering might appear somewhat tenuous, but bits and pieces keep bouncing around my head. There are clearer connections to movements going mainstream this year.

The following two books came from asking what-if questions about the future of software engineering. The books I read suggesting utopian futures did not ring true.

“Money and Motivation: Analysis of Incentives in Industry” by William Whyte provides lots of first-hand experience of worker motivation on the shop floor, along with worker response to management incentives (from the pre-automation 1940s and 1950s). Developer productivity is a common theme in discussions I have around evidence-based software engineering, and this book illustrates the tangled mess that occurs when management and worker aims are not aligned. It is easy to imagine the factory-floor events described playing out in web design companies, with some web-page metric used by management as a proxy for developer productivity.

Labor and Monopoly Capital: The Degradation of Work in the Twentieth Century by Harry Braverman, to quote from Wikipedia, is an “… examination the nature of ‘skill’ and the finding that there was a decline in the use of skilled labor as a result of managerial strategies of workplace control.” It may also have discussed management assault of blue-collar labor under capitalism, but I skipped the obviously political stuff. Management do want to deskill software development, if only because it makes it easier to find staff, with the added benefit that the larger pool of less skilled staff increases management control, e.g., low skilled developers knowing they can be easily replaced.

Gokberk Yaltirakli (gkbrk)

Twitch GraphQL API December 06, 2020 09:00 PM

As I mentioned in the last Status Update, I was working on a small project that used Twitch data. The data I needed was not fancy or complex. I only needed the Live/Online status of the channel, the stream title, and the stream start time. As I am only looking for things that appear on the channel page, I thought I could parse the page HTML. To my disappointment, Twitch had turned their website into a slow, single-page application.

This threw a wrench in the plans, but it wasn’t too bad. I looked at the network tab, hoping to find some endpoints that provided the data I wanted. Instead of something REST-like, I saw that almost all the requests were going to a single endpoint. The endpoint was https://gql.twitch.tv/gql, which hinted at this being a GraphQL API.

I was not familiar with GraphQL, as I had never used it before. From what I understood, it was like SQL and it lets you query and fetch the data that you need in a structured format. I looked at some online examples and came up with a query to test. When I attempted to run the query; I, unfortunately, encountered an error. It turns out, their GraphQL endpoint needed an API token called Client-ID. If you send a request without this header, you will get the error below.

{
 'error': 'Bad Request',
 'status': 400,
 'message': 'The "Client-ID" header is missing from the request.'
}

A website or app that uses an API needs to either hard-code the tokens or provide another endpoint to fetch them. Twitch embeds this in a JavaScript snippet on the pages. You can grab this with a small regular expression. This token has stayed unchanged for a long time. But it is still a good idea to fetch it every time, in case they decide to change it at some point. Below is a code snippet that can grab this token. While the code snippet is in Python, you can port it to other languages without difficulty. In fact, the original version I wrote was a shell script.

import requests
import re

homepage = requests.get("https://www.twitch.tv").text
client_id = re.search('"Client-ID" ?: ?"(.*?)"', homepage).group(1)

Here is an example script that outputs the number of followers for a user. Please note that you can send many queries with a single POST to a GraphQL endpoint. This will result in better performance.

#!/usr/bin/env python3
import requests
import re
import json

# Fetch client-id

homepage = requests.get("https://www.twitch.tv").text
client_id = re.search('"Client-ID" ?: ?"(.*?)"', homepage).group(1)


def get_followers(username):
    query = 'query {user(login: "%s") { followers { totalCount } } }' % username

    resp = requests.post(
        "https://gql.twitch.tv/gql",
        data=json.dumps([{"query": query}]),
        headers={"AClient-ID": client_id},
    )
    return resp.json()[0]["data"]["user"]["followers"]["totalCount"]


print(get_followers("tsoding"))

December 03, 2020

Nikola Plejić (nikola)

org the ultimate API client December 03, 2020 05:20 PM

There's a whole genre of software I like to call "GUIs for curl": Postman, PatchGirl, and others. They aim at making web developers' lives easier by providing nice graphical interfaces for executing HTTP requests. I never got around to using any of them since they either had clunky UIs or were not free software. Thus, I defaulted to using curl in a shell.

Grantedly, with time, I came up with a wishlist of things I wish I could do while playing around with other people's HTTP APIs and testing my own. This boils down to the abilities to:

  • actually execute HTTP requests
  • parse, analyze and extract data from API responses
  • "compose" API requests by using parts of the response of one API request in the body of another
    • this is most notable during authentication: somewhere early on in your workflow you'll usually get an access token that you need to pass in each subsequent request
  • "bookmark" (persist) well-formulated API calls
  • annotate queries
  • tag the queries
  • share your queries with others

I tend to spend my working days in Emacs. Unlike the usual Emacs person, I don't really live in Emacs so I don't mind an occasional context shift, but a decent integrated solution is alwasy appreciated. As it turns out, org-mode checks all the boxes (as it usually does).

Executing HTTP requests

You could most certainly use request.el to send HTTP requests out into the world. However, I just use an org-babel "shell" block and curl like a barbarian. I believe curl is the de-facto standard HTTP client, and a lot of people are familiar with it. This approach has the extra benefit of transferring nicely into the command-line should it be necessary.

Here's a sample HTTP POST request:

* Example API Request: Authentication

  #+NAME: auth
  #+begin_src shell :cache no :results verbatim
    curl -X POST \
      --data "grant_type=password&username=nikola&password=hunter22&scope=all" \
      http://localhost:8080/v1/auth
  #+end_src

  #+RESULTS[...]: auth
  : {"access_token": "f744cb5c55d2946c4jzb3d7d61fb4850967.dae42e4e4d40", "expires": "..."}
  #+end_src

CTRL-c CTRL-c (or leader leader for evil folks) executes the block and outputs the result which can then be used further. Which leads us to...

Parsing, analyzing and extracting data in API responses

Most APIs these days dump JSON or XML on you, thus having results in the form of plain strings isn't very useful.

Let's say we're only interested in the access_token property of the response to the request executed above. Again, you could very legitimately manipulate that data in elisp, but since we're in the shell, we can pipe the data to another process:

  #+NAME: auth
  #+begin_src shell :cache no :results verbatim
    curl -X POST \
      --data "grant_type=password&username=nikola&password=hunter22&scope=all" \
      http://localhost:8080/v1/auth | jq .access_token
  #+end_src

  #+RESULTS[...]: auth
  : f744cb5c55d2946c4jzb3d7d61fb4850967.dae42e4e4d40
  #+end_src

This uses jq, a nice little tool for manipulating JSON documents on the command line. There are similar tools for XML/XPath as well.

Using API results as API parameters

Using some wonderful features of org-babel, you can trivially use results of previous commands in subsequent code blocks:

  #+begin_src shell :var t=auth :results verbatim :cache no
    curl \
        -H "Authorization: Bearer ${t}" \
        http://localhost:8080/v1/endpoint | jq .
  #+end_src

This will capture the result of our auth code block into a variable called t, which will then be available within the code block as a regular variable in your language of choice. org-babel will keep track of dependencies and offer to execute any code blocks that are necessary to figure out all of the variables used by the current one.

This has other handy use-cases as well. Need to generate some JSON for your API requests? Just do it in your favorite language:

  #+NAME: json_data
  #+begin_src python :results verbatim :python python3
    import json
    import random

    return json.dumps({"param1": random.randint(0, 100)})
  #+end_src

  #+RESULTS: json_data
  : {"param1": 49}

  #+begin_src shell :var data=json_data :var t=token :results verbatim :cache no
    curl \
        -X POST \
        -H "Authorization: Bearer ${t}" \
        -H "Content-Type: application/json" \
        --data "${data}" \
        https://httpbin.org/post | jq .
  #+end_src

Bookmarking, tagging and annotating API calls

org-mode files are text files, so each entry is automatically "persisted" when you save the file. Everything outside of code blocks is regular org, and org works well for taking notes! You can embed images, add tables, annotate, whatever. For tagging, I just use org's built in tagging facilities at the heading level.

Sharing

org-mode files are plain text files, so anyone can open them in their prefered text editor and have a reasonably decent reading experience. But org can also export to various formats, and exporting to HTML will give you readable API documentation for free!

By default, org-babel will only export source blocks — not the results. This is good default behavior since sometimes API results involve secrets (e.g. access tokens). To override this, annotate your code block with :exports both:

  #+NAME: auth
  #+begin_src shell :cache no :results verbatim :exports both
    curl -X POST \
      --data "grant_type=password&username=nikola&password=hunter22&scope=all" \
      http://localhost:8080/v1/auth
  #+end_src

  #+RESULTS[...]: auth
  : {"access_token": "f744cb5c55d2946c4jzb3d7d61fb4850967.dae42e4e4d40", "expires": "..."}
  #+end_src

Problems

When exporting, org will export your code blocks literally — any variables will be exported verbatim and not replaced by its respective value. This may or may not be desired — I think it's acceptable, but it can be confusing. There may be a snippet of Elisp somewhere that makes this go away.

Feedback

Feedback, corrections, comments & co. are welcome: you can write me an email or contact me on Matrix.

Gustaf Erikson (gerikson)

7,000 dead in Sweden December 03, 2020 02:31 PM

6,000 dead in Sweden December 03, 2020 02:30 PM

Jeremy Morgan (JeremyMorgan)

Easy IoT with Adafruit IO and a Raspberry Pi December 03, 2020 12:30 AM

There are a ton of IoT solutions in the cloud. Azure, AWS, and Google Cloud all have great and complex IoT platforms out there. They’re fantastic for doing enterprise IoT. But what about folks who just want to dip their toes into IoT? What if you just want to do a small hobby project to learn IoT before jumping into the big stuff?? Adafruit has your solution. They have a fully functional, easy to use IoT platform for hobbyists who just like to mess around and learn things.

December 01, 2020

Gustaf Erikson (gerikson)

November December 01, 2020 10:27 AM

November 30, 2020

Sevan Janiyan (sevan)

LFS, round #3 November 30, 2020 09:18 PM

With the OS image that I wrote about in the previous post I was able to build a new distro with the various substitution I’d made. There were 3 things that I wanted to mention in this post. First, turns out Linux has a bc(1) build dependency which I found when I omitted building it …

November 29, 2020

Derek Jones (derek-jones)

Software research is 200 years behind biology research November 29, 2020 10:29 PM

Evidence-based software research requires access to data, and Github has become the primary source of raw material for many (most?) researchers.

Parallels are starting to emerge between today’s researchers exploring Github and biologists exploring nature centuries ago.

Centuries ago scientific expeditions undertook difficult and hazardous journeys to various parts of the world, collecting and returning with many specimens which were housed and displayed in museums and botanical gardens. Researchers could then visit the museums and botanical gardens to study these specimens, without leaving the comforts of their home country. What is missing from these studies of collected specimens is information on the habitat in which they lived.

Github is a living museum of specimens that today’s researchers can study without leaving the comforts of their research environment. What is missing from these studies of collected specimens is information on the habitat in which the software was created.

Github researchers are starting the process of identifying and classifying specimens into species types, based on their defining characteristics, much like the botanist Carl_Linnaeus identified stamens as one of the defining characteristics of flowering plants. Some of the published work reads like the authors did some measurements, spotted some differences, and then invented a plausible story around what they had found. As a sometime inhabitant of this glasshouse I will refrain from throwing stones.

Zoologists study the animal kingdom, and entomologists specialize in the insect world, e.g., studying Butterflys. What name might be given to researchers who study software source code, and will there be specialists, e.g., those who study cryptocurrency projects?

The ecological definition of a biome, as the community of plants and animals that have common characteristics for the environment they exist in, maps to the end-user use of software systems. There does not appear to be a generic name for people who study the growth of plants and animals (or at least I cannot think of one).

There is only so much useful information that can be learned from studying specimens in museums, no matter how up to date the specimens are.

Studying the development and maintenance of software systems in the wild (i.e., dealing with the people who do it), requires researchers to forsake their creature comforts and undertake difficult and hazardous journeys into industry. While they are unlikely to experience any physical harm, there is a real risk that their egos will be seriously bruised.

I want to do what I can to prevent evidence-based software engineering from just being about mining Github. So I have a new policy for dealing with PhD/MSc student email requests for data (previously I did my best to point them at the data they sought). From now on, I will tell students that they need to behave like real researchers (e.g., Charles Darwin) who study software development in the wild. Charles Darwin is a great role model who should appeal to their sense of adventure (alternative suggestions welcome).

Mark J. Nelson (mjn)

Internships for credit at American University November 29, 2020 12:00 PM

This post is for American University students who are interested in me serving as faculty supervisor for an internship-for-credit.

Like many universities, AU allows students in some cases to receive academic credits for undertaking internships with an external organization (company or nonprofit). This can be in addition to being paid for the internship. To receive credits, there must be an AU faculty supervisor, and an agreement between the student and the faculty member about what the student expects to learn during the internship, and how that will be assessed and graded.

In general I'm not a very bureaucratic person, and would like to support our students doing anything that advances their intellectual interests and/or academic careers. But there are some requirements for giving academic credit for internships, detailed at this link.

Please check that link for the full details. The most important parts for someone asking me to be faculty supervisor are:

  1. Up to 1 credit-hour can be given for every 5 hours/week of internship, averaged over the semester. So for example a 20 hour/week internship could count for up to 4 credits.
  2. At least 85% of the internship must be devoted to "academic" rather than administrative work. The intent of this requirement is that we don't give course credit if an internship is primarily something like staffing a front desk in a building, setting up chairs for an event, sorting mail, etc. It should be doing something educational in the same field of study as the department that is giving the academic credits.
  3. Is there a supervisor at the internship who would be willing to participate in assigning a grade? If yes, the supervisor would need to agree to send me: a grade on the A+ to F scale, and a short (a paragraph is fine) summary of what work the internship involved, and how strong they thought the student's performance in the role was. The student and I would also need to agree ahead of time on what percentage of the final grade the workplace supervisor's assessment should account for, up to a maximum of 50%.
  4. The academic supervisor (me) needs to assign a grade for the academic side, which counts for the remaining percentage of the overall grade. For this portion, we need to agree ahead of time on one or more learning outcomes, and how they should be assessed. The assessment could be simply showing me things that were done in the internship anyway (e.g. a project), or it could be a write-up summarizing them, or something else depending on the topic of the internship. See this page from AU's Center for Teaching, Research & Learning for an overview of how to phrase learning outcomes, and how to choose assessments for them.

    An example: "By the end of this internship, the student will be able to implement basic iOS apps using the Swift programming language" is a possible learning outcome, and "The student will give a short presentation/demo of the app produced in the internship, explaining their contribution to it" is a possible means of assessment.

Once we've agreed on this, you'll need to fill out the Internship Registration Form [PDF].

November 28, 2020

Andrew Montalenti (amontalenti)

Managing software teams: the definitive reading list November 28, 2020 07:57 PM

.entry-content { hyphens: manual !important; }

Frederick Brooks once wrote:

The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds castles in the air, from air, creating by exertion of the imagination.

In his classic essay, “No Silver Bullet”, he also wrote about software’s “essential” complexity:

The complexity of software is an essential property […] Hence, descriptions of a software entity that abstract away its complexity often abstract away its essence. For three centuries, mathematics and the physical sciences made great strides by constructing simplified models of complex phenomena, deriving properties from the models, and verifying those properties by experiment. This paradigm worked because the complexities ignored in the models were not the essential properties of the phenomena. It does not work when the complexities are the essence.

It’s therefore no surprise that once a team develops expertise in a technical domain and starts to evolve a product from that foundation, there is still a huge heap of complexity left over.

This is made worse on bigger problems where more people are involved, thanks to Brooks’s Law. In addition to the software’s essential complexity, you also need to navigate the built-in complexity of human communication, and the associated combinatorial explosion of information pathways on growing teams.

Modern software management practices have thus evolved two distinct skillsets for navigating the problems of team coordination around complex software delivery. These two areas are product management and software engineering management.

Product management centers around areas like market positioning, product strategy, and customer-focused iterative development. The product manager, working closely with the engineers, determines the “what”, aligns the team around the “why”, and acts as a liaison to users and the wider market.

Software engineering management focuses on the practice of coordinating teams of programmers in the delivery of high-quality shipped software, often leveraging programming languages, tools, frameworks, cloud services, and lots of open source technology. Pithily, this is “the how”, as well as the art of making informed engineering tradeoffs, e.g. as relates to time-to-market, performance, quality, and cost.

This post will cover some of my favorite reading materials spanning product management and software engineering management, as disciplines. A preview of some of the titles is here:

I have curated these resources carefully, and I have recommended them at various times to my own product managers and my own software engineering managers over the years. I hope you find this a useful starting point for your own study.

It’s organized into a few sections, each featuring a few relevant books:

Here’s the full listing of book reviews, as well, for easy navigation. If you want to skip over this listing and jump right to the first review, click here.

Alright, let’s dive in!

Books about management as a high-leverage activity

The first thing to recognize about management activities of any kind is that they are “high leverage” activities. What I mean by this is that a management decision can ripple through an organization and have long-lasting effects on the day-to-day work of many members of your staff. This is very much unlike the decision of a single programmer, which is often (though not always) isolated in scope or impact. There is a nice short summary of the first book on my list available on Medium, “Top Takeaways from Andy Grove’s High Output Management”. This breezes through the concept of managerial leverage, and can help set your expectations for the rest of the readings on this list. It’s recommended. Once you get that high-level overview, these books can further solidify effective techniques at delegation and leverage in your organization.

High Output Management: the leverage of delegation

High Output Management, A. Grove, 1995.

For all intents and purposes, this is the “canonical” text on Silicon Valley company management, written by an engineer who worked his way up the ranks to become the CEO of Intel. This is the book most commonly recommended to startup founders by “in-the-know” Silicon Valley VCs. I recently discovered some nice notes on High-Output Management based on a presentation given by the VC, Keith Rabois. It’s entitled “How to be an effective executive”, and there is a corresponding slide presentation that is worth clicking through after you read the notes.

You might think between the summary/takeaways linked above and the notes linked in the last paragraph, you might have enough to be able to skip this book. But, that would be a mistake. Instead, I suggest you buy the book and use it as a starting point in your management journey. I recommend this to managers regardless of department (even spanning support, sales, finance). I find the book resonates particularly well with company founders and product/engineering managers.

Beyond Entrepreneurship: the leverage of culture and mission

Beyond Entrepreneurship, J. Collins, 1995.

One of my favorite business books overall, it covers businesses growing from 1-2 people to 10 people to 50 to 100, and it sort of “ends” there. It’s cool because it studies the early scaling of companies whose names you will recognize, but specifically centers around the mission/vision/strategy of founder-led companies, and how that emanates out from the founder to the rest of the employees via culture. I believe that managers of all kinds can learn lessons from this, as well-run teams are really “little” startups in disguise.

I first heard about the book when, in a video interview, Netflix CEO Reed Hastings was asked to give advice to new entrepreneurs, and he said, “Memorize the first 86 pages of Beyond Entrepreneurship by Jim Collins.” He went on to say, “Using the book’s lessons, you can fight the idea that as you get bigger, the culture gets worse.” Note: a second edition of this book will be available via Kindle and Audible, as well as print, in December 2020; it’s called BE 2.0 and is available here.

Managing Humans: the leverage of human connection

Managing Humans, M. Lopp, 2016.

This book is focused on people management in software projects. It is written by an engineering manager who is known online as “Rands” (with a great blog, randsinrepose.com), whose career spanned a number of classic Silicon Valley companies, including Borland, Netscape, Palantir, and Pinterest. After this book’s publication, Rands went on to leadership roles at Slack and Apple.

The book provides the engineer’s perspective on good vs bad managers. I find that for programmers-turned-managers this book provides a relateable explanation for why basic management practices — like running regular and effective 1:1’s, moderating team meetings, and listening deeply to problems reported by individual contributors — is key to team performance. I also find that for product managers, and managers of other teams, it provides an interesting window into the programmer-turned-manager, and the way they approach problems. It also provides some empathy for what it feels like to be managed well or poorly, as a programmer. Note: The author has a new book published in 2020 called The Art of Leadership, but I haven’t checked it out yet.

Books about product marketing and product management

Once you understand that management is a high-leverage activity, you might become more curious about how, exactly, to make the right high-leverage decisions in your management role. These books cover product marketing and product management, two areas that center around unlocking the value that your programming team can deliver through software. One way to think about this is that, if you have a talented programming team, they can build pretty much anything. So, as Brooks once said:

The hardest single part of building a software system is deciding precisely what to build.

Understanding product marketing is thus your first stop on this journey. You first need to understand how customers perceive software markets. This sometimes goes by the name “market positioning”, and involves identifying an ideal customer profile, or ICP. Once you have identified the category and customer persona, you can more effectively guide your team toward building for them. You’ll then need to learn about product management — which is about empowering your engineering team to build for that customer, while also acting as a liaison to the market. This is a role typically done by product managers, but is also often done by founders or engineering managers when teams are so small as not to be able to afford a full-time product manager. Finally, the last couple of recommendations in this category will center around the iterative product development process: how to actually hit your releases and get your product to market — and how to avoid a death march along the way.

Obviously Awesome: how to identify your ideal customer profile

Obviously Awesome, A. Dunford, 2019.

The (only) good book on market positioning, and it’s a very awesome book. Very helpful for thinking through the proper marketing and positioning of new features, which aids in ensuring you build stuff people actually want. Also important to understand how customers perceive your market, and how that perception often fuels your product’s adoption and growth.

The Startup Owner’s Manual: how to ship with limited runway

The Startup Owner’s Manual, S. Blank, 2012.

When I was getting going in the world of startups around 2008-2009, Steve Blank’s book, The Four Steps to the Epiphany, was passed around quietly among startup founders and their investors, as a “secret manual” of sorts for building successful software products. However, The Startup Owner’s Manual feels, to me, like a second edition of that book, rewritten to make sure it excluded dated examples and had a clearer narrative style. This might not seem like an obvious title in a list of books about product marketing and positioning. For example, should you read this book if you are working at an established company, startup or otherwise? I think so. I generally disagree with the idea that a product manager is a “CEO of product”, but I do agree with the idea that, like a startup CEO, the product manager has to worry much more than other members of the team about runway, time-to-market, and avoiding time sink activities. If you come from a big company to a much smaller one (e.g. if you used to work at a 500+ or 1,000+ employee company, and are now joining a sub-100 employee startup), this book will also be effective “deprogramming” on big company thinking.

Inspired: the one book for technology PMs

Inspired, M. Cagan, 2017.

If there is a single book to recommend to new product managers on software teams, this is it. You can get a quick sense of the philosophy in this book in the author’s widely-cited 2014 essay, “Good Product Team / Bad Product Team”. Indeed, the entire svpg blog has a lot of good essays available for free, but the book is very much recommended. This is a book 100% focused on product managers and the role they play at software companies of varying shapes and sizes. Also spends a good amount of time diagnosing one of the greatest sins of Silicon Valley: building something technically cool that no one wants to use or buy.

Shape Up: the one book on lightweight process

Shape Up, R. Singer, 2019.

Basecamp’s modern and practical (free!) book on product and project management. Takes a very calm approach to iterating on product, by being designer-led, timeboxed (6-week iterations), and light on process. If there’s one book you read on software delivery process, this is a good choice. In my opinion, there are way too many books on software delivery process (the ‘agile’ and ‘scrum’ crazes of the early 2000s are mostly to blame here), and too much process can often suck the soul out of the creative craft of building software. But, Shape Up strikes the right balance.

Managing the Design Factory: a dense read on cost of delay

Managing the Design Factory, D. Reinertsen, 1997.

Terrible title, but excellent and unique book on management of creative and design-driven projects. The key concept explored in the book is “the cost of delay”, which is to say, reconciling the real pressure you’ll get from your business to “ship whatever we possibly can, and ship it yesterday” with the internal pressure you’ll feel from programmers who will want to “ship the perfect thing, tomorrow, or whenever it’s ‘ready'”. Written in the late 90s, and takes a lot of lessons from hardware/software engineering. His second book, Principles of Product Development Flow, is much more famous. But, IMO, this is mostly due to its much better title. This book has a worse title, but is a much simpler read, and expresses many of the same key concepts. Both are already a bit dense for a business book, so I’d definitely recommend starting with this one. (Reinertsen himself has described Product Development Flow as the “double black diamond ski slope” of his principles, whereas Design Factory is the “green circle”.)

Making Things Happen: a light read on project coordination

Making Things Happen, S. Berkun, 2008.

The original title of this book was “The Art of Project Management”, but it was re-titled when “Project Management” became a bit of an anti-pattern in software teams, what with the rise of PMP certifications and similar. The lessons of the book were drawn from the author’s time acting as a product manager for Microsoft on the Internet Explorer team during the browser wars of the late 1990s, but, at that time, the term “product manager” was not in widespread usage. If I were to select a new title for this book, it’d be “The Art of Product Management.” The more abstract “Making Things Happen” does make some sense, however. Fundamentally, this is a book about spurring action on your engineering team — about acting as a catalyst, and a release coordinator, for teams that have their own essential complexity to deal with in the engineering problems and time pressures at hand. This is very much a “nuts and bolts of product management” book, and so will be especially helpful to inexperienced or young managers.

Making Work Visible: the best bits of ‘kanban’, summarized

Making Work Visible, D. DeGrandis, 2017.

Do you have a team where it seems like no matter what you agree to work on, they always end up working on something else, instead? If so, you might have a work visibility problem. I find this is especially common on teams where the engineers are expected to do “double-duty” in other more reactive roles, like DevOps, SysAdmin, Customer Integration, and/or Customer Support. It can also happen on teams where the engineers are a bit too focused on “new shiny” technologies, or where technical complexity has spiraled out of control due to poor engineering management. The kanban principle of creating visibility into all work and tracking work-in-progress can help you debug and diagnose these problems, by ensuring that rather than a “product” backlog of intentional work competing with a “shadow” backlog of reactive work, you have unified visibility into a single global backlog and global team-wide work-in-progress.

Books about debugging dysfunctional product cultures

Even the best product team cultures have a little dysfunction. We’re all human, after all. But, if you’re hired into an existing team, your first task might be to figure out just how much dysfunction, and whether you can do anything about it. These books cover various “failure scenarios” of product teams, and how to resolve them. This spans many areas: having an environment that’s not conducive to creative work; having uncollaborative programmers and/or uninclusive programmer practices; having the wrong business strategy set for a product team, such that no amount of ingenuity can lead to success; having overconfident engineers who succumb to programmer hubris; as well as, a well-intentioned and common need to ship features without purpose or customer impact, which ultimately leads to organizational waste and customer dissatisfaction.

Peopleware: shipping in spite of management

Peopleware, T. DeMarco, T. Lister, 2013.

Management of creatives in tech, distilled. Written for the 90s and aughts era of “IT”, but equally applicable to modern software teams. A bit of bias for colocation vs distributed teams in this one, however. The key insight of this book is that, often, “management” gets in the way of creatives shipping toward the company’s (and customer’s) goals, so the key task of management is to create an environment where the creatives can actually get their work done. Some parts of this book focus a lot on the physical spaces where programmers work, which might read as dated in an era of fully distributed collaboration, but we have a whole section on fully distributed teams later on.

Team Geek: moving beyond the hero programmer

Team Geek, B. Fitzpatrick, B. Collins-Sussman, 2012.

This is the software engineering management book that came out of Google’s Chicago engineering office in the early-to-mid aughts. The target audience is engineers who want to get out of the mindset that “the only thing that matters is code”, and refocus attention on the value of communication and teams. The author considers the book to be a “modern Peopleware, but from the individual contributor perspective, rather than the manager perspective.” I enjoy this book because it effectively shatters several “programmer myths”, including the genius/hero programmer, or the idea that the best programmer team size is a team of one. These myths resonate a lot with programmers, lined up with several mythologies behind open source projects like the Linux kernel (Linus Torvalds) or the Python interpreter (Guido van Rossum). But this book makes it clear that the best software is built by teams, and that though single programmers can be very productive, it’s really effective teams of programmers motivated by common purpose that are the most productive. This is also a good book for managers to recommend to the, ahem, “ornery” programmer contributors they may commonly come across on teams.

The Goal: identifying the critical constraint

The Goal, E. Goldratt, 2014.

This is probably the strangest business book on this list, but it’s fascinating. This is a “business novel” used to illustrate various concepts from a now very-well-understood business framework called “The Theory of Constraints”, or ToC, which has its main applications in lean manufacturing. The book is written in a super-unconventional style. In my view, the 30th anniversary audiobook on Audible is the way to go for this title, as I think it’s more fun as a dramatized audio book than as a sit-down-and-read “business novel”. The key learning you’ll take away from this one is that usually business performance is dependent on one or two critical goals/metrics (or “constraints”), and so no amount of individual ingenuity or team performance will overcome those if the strategy is set up in opposition to them. For example, given my background in SaaS, at a certain moment in my company’s history, it became clear to me that number of new leads / prospects / users / use cases of our software, not the engagement of our existing happy users / customers in a core use case that was already working, was the key limiter on the company’s growth. So if the product team only focused on building new features for existing customers in their core use case (which can seem like an obvious and “safe” decision), we would be missing out on growth in new use cases (and in new markets). This analysis led me to push our organization to hire a CRO, and this hire worked to accelerate our software sales velocity. But it also led me to focus a product manager on an emerging ideal customer profile for our product, and to reboot our product positioning and marketing, and both of these things also worked to accelerate our product’s adoption in new markets. Sometimes, you become the victim of your own success, and The Goal helps you realize how to make a high-leverage decision to get back on track.

Escaping the Build Trap: leaving behind the feature factory

Escaping the Build Trap, M. Perri, 2018.

Sometimes your team gets good at shipping, but, it turns out, merely shipping isn’t enough. It matters a whole lot what you ship. This book is focused on the perils of “build new features forever and ever” death march projects, especially in enterprise SaaS software. It has a bunch of practical tips and tricks for breaking the cycle of feature development to focus on real customer problems. Similar to “The Goal”, it’ll help you identify what the real business and customer constraints are, especially in the context of high-tech and enterprise SaaS.

Dreaming in Code: recovering from engineer hubris

Dreaming in Code, S. Rosenberg, 2008.

This is a wonderful journalistic foray into a well-funded software project — Chandler, an “Outlook-killer” funded by Mitch Kapor, the Lotus creator, in the early aughts — and everything that goes wrong up until the project’s failure. Probably the best modern “software project postmortem” ever written, focused on all the pathologies of software engineers and their managers when a project is off-track. If you’re interested in this genre, the film “General Magic” is the best Silicon Valley software/hardware postmortem ever captured in documentary form.

Books on the psychology of deep work

If you’re a programmer-turned-manager, it’ll be obvious to you that deep work is necessary for programmers to do their best work. That’s because programming is, very much, a flow activity. But, if you don’t come from a programming background, it may not be obvious to you the degree to which productivity is linked to deep concentration, and the many (many, many!) ways that modern businesses break the concentration of their creative workers. I discussed this a bit in my CTO Summit presentation, “Fully distributed and asynchronous: building eventually-coordinated teams that ship”.

These books will help you build empathy for deep work and flow.

The surefire way to earn the respect of programmers you work with in a management context is to protect them from organizational distraction and to encourage them to tackle their problems deeply and thoughtfully, with the time and space to drive their code toward a shippable state. What’s more, these books will also help you understand how good work feels to the best programmers, and why sometimes, what seems like a very small ask — a recurring status meeting, a request for regular reporting on a metric, a direct ping on Slack for a single customer’s problem — can have a much higher cost than you might initially think, and might ultimately be self-defeating toward your own goals as a manager.

Flow: the phenomenology of work

Flow, M. Csikszentmihalyi, 2008.

This is the “one book to read on productivity” that was recommended to me by a manager, when I was a young programmer, and leaning in to becoming a technical lead on a fully distributed team. It’s a thick book and is really more a work of psychology. But this book introduces two interlocking ideas: (1) that the way work feels to the creative worker is very important; and (2) that ‘flow state’ is a real physical change in the body/brain, which influences productivity and creativity directly. This is a precursor to other books on this list, like Drive and Deep Work. I actually re-read this book very recently and found it to be inspiring in a very different way, now that I am a full-time manager. It made me realize that creativity and flow is also important to me personally, and thus inspired me to mix in some “flow time” to my more reactive “manager schedule”. But ultimately, this book will help you understand what makes creative and technical people tick, from a psychological standpoint.

Drive: autonomy, mastery, and purpose

Drive, D. Pink, 2011.

If Flow strikes you as a bit long-winded or a little too academic, then this book, Drive, is the one book on engineer motivation to read. It’s sort of a like a shorter/condensed version of Flow, and more readable. Funny enough, this author also wrote an excellent book on the psychology of sales people, entitled To Sell is Human. In an enterprise SaaS context, I recommend Drive to salespeople, so that they can understand engineers. And then I recommend To Sell is Human to engineers, so they can understand salespeople. It’s an effective way to build cross-department empathy.

Deep Work: creating time and space to think

Deep Work, C. Newport, 2016.

A good friend of mine, with whom I often trade management book recommendations, once described this book as follows: “has a great title, and the title describes the key concept of the book — but! the key concept of the book didn’t need an entire book to describe it!”

I would tend to agree with this assessment. That said, if you haven’t thought much about deep work and programmers, this book will certainly get you up to speed. In it, an academic computer scientist and programmer changes his life to get back the focus required to do the computer science deep work that he loves. He documents the journey, but also uses historical examples and academic studies to showcase just why deep work (focus, concentration, long blocks of uninterrupted time) are so essential.

The book is also a nice antidote to our post-2016 era, where it feels like social media tools (like Twitter) and enterprise communication tools (like Slack) are using every trick in the book to interrupt our attention and steal our focus. If this doesn’t feel like a book you can devote to reading in its entirety, some of the key concepts that Newport describes here are humorously summarized by proxy in a talk about the “conditions of creativity” by the comedian John Cleese. The 1991 talk is linked/embedded on BrainPickings here and summarized by Quartz here.

Books about fully distributed teams

The Year Without Pants: grokking distributed teams

The Year Without Pants, S. Berkun, 2013.

A non-technical project manager (really, product manager) from the Microsoft era joins Automattic, a fully distributed team with open source DNA, and he shares what he learned in that cultural setting. Yes, this is the same author as Making Things Happen, from earlier in the list, but it’s written when he’s at a very different stage in his career. One of the few books written about the way fully distributed teams “feel” and the dynamic between in-person get-togethers vs online-first (or online-only) collaboration.

Remote: turning distributed into an advantage

Remote, J. Fried, D. H. Hansson, 2013.

The subtitle of this book is “Office Not Required”. This is the book on remote work written by the Basecamp (formerly: 37 Signals) folks. Though the team had a Chicago office, they operate that office on “library rules”. (On my team at Parse.ly, we’d refer to this as, “office as internet cafe, not office as headquarters.”) This is a short and easy read which can be used to onboard new hires to a distributed team, or to change the mind of office/colocation holdouts.

It Doesn’t Have to Be Crazy at Work: the calm company

It Doesn’t Have to Be Crazy at Work, J. Fried, D. H. Hansson, 2018.

The original working title of this book was “The Calm Company”. This is a book on “calm” work written, again, by the Basecamp founders. Though not entirely about distributed teams, this is a book that makes it clear how one can set up an “interruption-free” work culture atop digital tools. Unlike Remote, which is written from an individual contributor’s perspective, this book is written from a manager’s perspective, and emphasizes how to create the feeling of calm on your team, that allows the remote creative work to thrive.

Books on programmer mindset and philosophy

An Elegant Puzzle: product teams as systems

An Elegant Puzzle, W. Larson, 2019.

A book by an engineering manager at Stripe, focused on growing technology companies and their teams while viewing the whole company as a “system” that can be tamed using engineering modeling and analysis techniques. Note: This is a nice book to hold in your hand with an attractive hardcover printing, which is rare for books in this category. I once used this book, and its table of contents, as the “syllabus” for a multi-month software manager book club.

A Manager’s Path: programmer management as a career

A Manager’s Path, C. Fournier.

Written by a programmer-turned-manager, this book documents her path from individual contributor to CTO, and what she learned along the way about programming and management careers. Good for understanding how software engineers think about “leveling” in their careers. It covers topics like the split manager vs individual contributor paths at large tech companies; what different titles and roles mean on product teams; how to interact with the board; how to create a long-term roadmap; how to tackle key technology choice decisions; and how to manage “difficult” people on your team.

The Mythical Man-Month: team and system complexity as a liability

The Mythical Man-Month, F. Brooks, 1995.

Quoted earlier on in this post, this is one of the timeless and classic texts about software engineering management, dating all the way back to 1975 and the programmer culture at IBM, at that time. Though it talks of IBM mainframes and low-level operating system development, the lessons learned for large engineering teams has resonated through the ages. It’s the closest thing software engineering management has to a “founding text” for the discipline. Note: This book has some dated language, not just about technology, but even about business in general. For example, some don’t enjoy the term in the title — “Man-Month” — and Brooks has a tendency to reference “man”, “God”, and so forth. So you have to read it while doing a “modern dialect” translation of sorts. If you’re curious about my other thoughts on “old” programmer books, you can take a look at this lobste.rs thread.

Hackers and Painters: lateral thinking as a way of life

Hackers and Painters, P. Graham, 2004.

To me, this book served as the starter pistol shot of the race for a new world order of tech companies established 2002-2010 and beyond. This was the beginning of the era where programmers went from being a cost center for “IT organizations”, toward being the key talent (and profit center) for all modern tech-infused businesses. The author is Paul Graham, writer of many fine essays, the founder of Y Combinator, and an early founder and engineer at Viaweb, which eventually became Yahoo! Stores.

He successful predicts the programmer-centered rearrangement of the business world. In this new world, programmers (and their ideas, their creations) run most of the world around us, as is widely accepted now. Harnessing the power of automation becomes the key task of modern knowledge work. As a manager, this book will help you understand why productive programmers are not interchangeable “coding widgets”, but are, instead, unique creative artists with deep motivations and boundless curiosity. It’ll also help explain why many of the most product-minded engineers end up creating companies of their own.

The Cathedral and the Bazaar: productivity through open source

The Cathedral and the Bazaar, E. Raymond, 2001.

Although some will consider this book a historical relic, it is also one of the most influential books in programmer circles, whether directly or indirectly. This book is about the power of open source software, and the thinking behind its communities. In my view, open source is not given the credit it deserves for completely altering the fabric of our lives as knowledge workers. This book digs in on the motivations of open source developers, and how that contrasts from typical “top-down” R&D organizations. Given the timing of this book in the 1990s, the key target of ire is Microsoft (“the cathedral”), which was then seen as the monopolist and centralized approach to software management and user control. The key counter-example is the Linux kernel and its GNU utilities (“the bazaar”), then seen as a free-wheeling and decentralized approach to software evolution and user empowerment. Even if programmers have not read and internalized this book, they have probably been influenced by its thinking.

A Philosophy of Software Design: code as design

A Philosophy of Software Design, J. Ousterhout, 2018.

This is the only book on the list that contains code examples! But, it’s short and the writing around the code examples is comprehensible and eloquent. This book can help you understand the everyday trade-offs that programmers make, not in the product or the user experience, but in the code itself.

This is often the hardest part of the job for non-programmers to empathize with — after all, customers usually can’t see your code, and usually they don’t care how the code looks. But, as craftspeople and artists, programmers care deeply about the code and how it looks — and how it enables, or disables, future code evolution.

(I am personally no different; I even wrote and published a popular style guide for Python programming!)

Unlike a lot of other books on “code design”, this book doesn’t just give examples, but also explains the business reasons and product impact of code design decisions, and thus can serve as a helpful guide to build empathy with programmers while they, as Brooks put it, “build castles in the air, from air”.

Great management leads to great businesses, and happy employees

One of the things I’ve learned through my years of software team management is that even the best team will fall down with bad management.

Employee happiness isn’t linked so much to a great product, great customers, or a great mission. Those things matter. But management matters more. Interactions with managers are what really affects how the job feels to every employee. Thus, I believe one of the most useful things we can do, as an industry, is improve the quality of our management. If management is a high-leverage activity, then teaching good management to others is the ultimate high-leverage activity.

It’s often said that employees don’t leave the job — they leave the manager. And this is true. So, if you’ve got a early/voluntary attrition problem, you may just have a management problem. And that’s the first problem you need to fix before you can reliably get any work done.

But, here’s the good news. Management can be learned. And this reading list is a start. If you have other recommendations, or questions, feel free to reach out to me at @amontalenti on Twitter.


Yes, I really did read every single book in this list. I read a lot. For other similar round-ups, but more in the technical direction, you can check out my popular post on rapid web app development, which includes long-form reading, videos, and technical books covering Python and JavaScript. You can also take a look at my round-up of the 3 best Python programming books for your team.

Whenever book image art or links to book product pages were used, I provided images and links from Amazon.com, which might include affiliate link codes.

If you are interested in working in a programmer or product manager role on a team with thoughtful software engineering management and product management, you may want to take a look at the Parse.ly careers page.

November 27, 2020

Robin Schroer (sulami)

Lightning Introduction to Nix for Developers November 27, 2020 12:00 AM

Motivation

This is the part where Patrick Winston says I need to promise something you will gain by reading this. My promise is the following: This is a very quick introduction to setup Nix on a macOS (or Linux) machine, which will get you reproducible, sandboxed environments to do your development in, as well as atomic rollbacks.

There are many introductions to Nix, but this one aims for speed. I will be skipping over a lot of the fundamentals and only tell you what you absolutely necessary to get up and running. Nix is a large and complex system, but you can get some returns on your time investment within 15 minutes, and decide on delving in deeper later.

This guide is aimed at macOS, but most of it can be applied to Linux as well.

Homebrew?

The de-facto standard package manager for macOS is Homebrew. While it is a passable solution for installing Mac apps, it has a few shortcomings, some of which can be especially problematic for developers.

Sandboxing
Homebrew packages are for the most part installed into /usr/local/bin, which means they are always available to everyone. This can lead to conflicts which can require manual modification of $PATH to resolve. Especially programming languages tend to hit this, as some software only runs on specific versions of their language.
Freezing
Even though Homebrew is based on git, is does not support explicitly installing specific versions of a package, or pinning the version in a lockfile. This by extension also means that whenever you install a homebrew on a new system, you will not be able to reproduce the exact versions installed on a known good system.
Patching
If you want to modify a package, you have to do it manually after installing, and potentially after every update. Homebrew can apply patches during the build process (via brew edit), but again, there is no declarative way of doing so.

Both the freezing problem and the patching problem can be circumvented by maintaining your own tap, but this comes with a significant maintenance burden, and I would not recommend it.

I would like to note that you likely cannot replace Homebrew entirely by Nix, as a lot of macOS-exclusive apps are not packaged in Nixpkgs. You could probably package them yourself if you really wanted to, but this has the same problems as maintaining your own Homebrew tap.

Installing Nix

Before we can use it, of course we have to install Nix. I am using macOS, so I will also install nix-darwin. If you are using Linux, you can install home-manager instead for a declarative system setup.

Nix

To get started, first we install the Nix package manager and language itself.

curl -L https://nixos.org/nix/install > /tmp/install-nix.sh
less /tmp/install-nix.sh  # inspect the script
sh /tmp/install-nix.sh --darwin-use-unencrypted-nix-store-volume

The extra argument is specific to newer Macs with a T2 chip. Refer to the manual for more details.

Nix-Darwin

Next we install nix-darwin, which is essentially a framework written in the Nix language. It establishes a declarative configuration for the whole system, which packages are installed, all the way to defaults. One of my personal selling points is management of Launch Agents in Nix, which is much nicer to manage than writing XML and working with launchctl.

nix-build https://github.com/LnL7/nix-darwin/archive/master.tar.gz -A installer
./result/bin/darwin-installer

The installer will prompt us with a few questions along the way, which do not seem to be well documented. Generally we want to respond with y throughout (the first one is optional).

Would you like edit the default configuration.nix before starting? [y/n]
Would you like to manage <darwin> with nix-channel? [y/n]
Would you like to load darwin configuration in /etc/bashrc? [y/n]
Would you like to create /run? [y/n]

At this point we are all set. We might need to start up a new shell to load the newly installed commands. If everything worked, we should now have darwin-rebuild in our $PATH.

Declaring the System

The first use case we will be looking at is using Nix to setup our system as a whole.

How Nix Works

I will interrupt here for a brief (and simplified) explanation of how Nix works in the first place. Essentially Nix works by building and installing software according to a set of recipes (Nix expressions) in what is called the Nix store, which is just a directory at /nix. To actually make the software available, it creates symbolic links to into the store in a profile, which is just another directory. This profile can then be added to $PATH, so that we can just use the software installed. The beauty of the symbolic links is that we can create many profiles which link to different sets and/or different versions of software in the store.

This also allows us to version profiles, and switch atomically between them, because every time we run darwin-rebuild switch, a new profile is created and activated. Should anything break, we can just switch back to the old profile. In practice this means running darwin-rebuild --rollback. We can also switch to a specific version, using --list-generations and --switch-generation if we want to rollback more than one change.

Installing a Package

Before we can install a package, we need to find it first. Finding a package is as simple as running

nix search some-package

Let us modify $HOME/.nixpkgs/darwin-configuration.nix now. If we open that file, we should find a section similar to this:

environment.systemPackages =
  [ pkgs.vim
  ];

This is where nix-darwin declares the packages installed on the system. Go ahead and add a package to that list. Nix does not use commas to separate list items, just whitespace. The canonical package to add is pkgs.ripgrep, but any will do. Rebuild the system:

darwin-rebuild switch

We should now have rg in our $PATH, without having to open a new shell, as $PATH did not actually change. The nix-darwin manual has a big list of configuration options that might also be interesting, but are not required now.

Fetching Updates

As mentioned above, anything we build and install is controlled by our local Nix expressions in the Nix store. These are just build recipes in the Nix language, similar to Makefiles. The expressions usually pin a specific version of the software they build, and they themselves are also versioned. This means to update our packages, we need to update the expressions, which we do like so:

nix-channel --update

This fetches the latest versions of all channels we follow and updates our local Nix expressions accordingly. If a software definition got updated upstream, we can now rebuild it to get the updated version. Because channels are also versioned, we can even rollback channel updates if an upstream update broke for us.

To actually rebuild the packages according to the new definitions, we have to build a new version of our profile:

darwin-rebuild switch

Using nix-shell

There is another way of using Nix than installing all packages system-wide. If we just want to try out a package without having to rebuild our system (and reverting afterwards), we can simply run

nix-shell -p some-package

Nix will build the package in the Nix store and drop us into a shell that has access to the package. Add --pure, and we get a completely clean environment except for anything that we explicitly add to the shell. This can be useful if the mere existence of a system-wide piece of software is problematic.

If we use this method to setup a Nix environment for a specific project, we can use a shell.nix file to declaratively express the environment like so:

{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell {
  buildInputs = with pkgs; [
    python-2.6
  ];
  PYTHONDONTWRITEBYTECODE = "1";
}

This looks complicated, but this does two simple things: anything in buildInputs is made available to the shell, and anything else is injected as an environment variable. Just calling nix-shell in the same directory will automatically pick up this file and execute it.

In this case we are simply getting an older version of Python, and also setting a related environment variable. Anyone using this configuration will have the same environment, which mirrors some of the benefits of Docker, but without the overhead of running containers.

How to Debug Problems

This is the hard bit about Nix, the documentation is almost infamously sparse, and common recommendations are to either find an existing solution for your problem, or to read the Nix code involved. Because Nix includes a whole programming language, it allows users to build their own abstractions, which means that many packages have their own way of doing things.

I wish I could provide a sure way to solving all your Nix-related problems, but a lot of it comes back to pasting error messages into search engines and asking people online. Nix is not without its rough edges, and sooner or later you will run into one of them. I consider them learning opportunities, but they can be very frustrating.

Where to Go From Here

This is just the beginning, there are many more parts of Nix to discover. It is probably advisable to read through the Nix Pills to get a better understanding of the language and system.

home-manager is a project which manages a per-user Nix environment in a declarative way. If you are using nix-darwin it is somewhat optional, but can still be useful to build a more portable configuration. It can be installed as a nix-darwin plugin as well.

If you are looking for better project environment management with Nix, there are a few very useful tools. Niv allows you to declare and pin dependencies for a project. Lorri is a daemon that automates a lot of the nix-shell setup we have been doing by hand above, such as automatically loading and reloading an environment when you enter a project directory. Direnv and shadowenv are alternatives to lorri.

You might also want to try packaging some of your own software in Nix, or software that is not in Nixpkgs (yet). It is good exercise to gain a deeper understanding of the system, and as a bonus you get a more reproducible setup. Nix is a great fit to distribute internal developer tooling as well. I might write something on how to do this in the future.

Last but not least, if you really enjoy using Nix, you might want to try running NixOS, a whole Linux distribution which is configured using Nix.

November 26, 2020

Jan van den Berg (j11g)

Cruddiy: table relationship support via foreign keys November 26, 2020 05:47 PM

Read here what Cruddiy is and what it can do for you: here is the code.

TLDR: Cruddiy is no-code Bootstrap 4 PHP form builder for MySQL tables.

I started Cruddiy when the Covid-19 lockdowns happened this spring, to keep me busy. And I released it on GitHub. After 25 stars 🤩 and 13 forks on GitHub and a couple of really encouraging messages on the original Cruddiy post, I thought it was time for an update.

🥳 Cruddiy now supports table relationships via foreign keys. 🥳

This means:

  • You can add new or delete existing table relations by picking columns that have a foreign key relation.
  • You can add specific actions for each table relation, like:
    • ON UPDATE : CASCADE / SET NULL / RESTRICT
    • ON DELETE: CASCADE / SET NULL / RESTRICT
  • Picking specific actions will result in different behavior. Please read up what these mean (mostly you want ON UPDATE CASCADE and ON DELETE CASCADE).

Having table relations in place wil have the following results for Cruddiy:

  • On the Create form the field will be populated with the correct key information.
  • On the Update form: the correct stored value will be preselected but you can still choose (update) different values from the select list.

Note 1: the table relationship builder is the first step in the Cruddiy CRUD creation process. However it is completely safe to ignore this step and move to the next step! I would even strongly advise doing so if you are not sure what you need to do, because it might break things. Also if you just want one or a couple of simple unrelated forms it is perfectly safe to skip this step.

Note 2: the table relationship builder is of course just a GUI for something you can also do in PHPMyAdmin or MySQL from the commandline. However, Cruddiy makes it visible and easier to work with table relations.

Some screenshots:

Table relation builder. One existing relation. And a couple of dropdown lists to create new ones. Feel free to skip this step by hitting ‘Continue CRUD Creation Process’. In the column form columns with foreign keys are indicated by the blue icon. A simple two field table. As you can see the second item is preselected (as it should be). However it is still possible to select a different value.

The post Cruddiy: table relationship support via foreign keys appeared first on Jan van den Berg.

November 25, 2020

Gustaf Erikson (gerikson)

Return of a King: The Battle for Afghanistan by William Dalrymple November 25, 2020 05:46 AM

A good overview of the First Anglo-Afghan War. The parallels to today’s situation are presented but not in a polemical way. Dalrymple presents “both sides”, avoiding the all too common trope of only focusing on the British defeat and hardships.

November 24, 2020

Carlos Fenollosa (carlesfe)

Damn Small Linux on a Libretto 50CT November 24, 2020 06:30 PM

In about 2004 a friend from college got into his hands a series of Toshiba Libretto 50CT. They all came with Windows 95 pre installed, and we wiped them out and installed Debian with a 2.2 kernel from floppies, with much pain because of the unsupported disk drive. I remember it being so difficult that I don't even want to write about it. But it booted and was able to run Vim, Links, and connect to the internet. Enough for a network-enabled typewriter.

I got Richard Stallman to sign mine while it was running GNU/Linux, but when I bought a regular laptop I stopped using it because, well, it is too old, and its keyboard is too tiny to type comfortably with it.

The Libretto, closed

The Libretto, closed. It is a tiny machine, a wonderful piece of engineering.

Around 2012 I found it inside a box at home, working perfectly, with a battery life of a bit less than an hour—which is incredible for a 12+ year old machine— and DR-DOS installed. I can't remember why I did that, but Golden Axe was also installed, so there's a hint.

I decided to install a modern Linux and, at least, store it on a working condition to give some justice to Stallman's signature.

There are some tutorials available on the net, but none of them covered 100% of the hardware support for my machine. Most of them are also outdated, and refer to distributions that don't exist, don't have any support or are about unusable nowadays. However, I took many hints from those tutorials, and I will reference them accordingly.

Hardware

This laptop has a 75 MHz —actually, mine reports 74 MHz— with 24 MB of RAM, which is an upgrade from the original 16 MB which are usually bundled. The screen is 640x480, and if you choose a higher resolution, instead of scaling the image, it only displays the upper-leftmost 640x480 pixels, leaving the bottom and rightmost part of the area out of sight.

The mouse device is emulated as a PS/2, and physically is a "clitoris"-like pointing device. You know what I mean. Working with the X windows is a pain in the neck because the location of the mouse and buttons isn't very ergonomic, and clicking on a button makes the whole screen move on its hinges.

Next to the mouse there is a speaker which is similar in quality to those of a modern smartphone.

This device doesn't have any extension port other than a dock connector for a dock I don't have, and a 16-bit PCMCIA slot, which you will need for the network card. It doesn't have a COM port or anything like that, which is understandable, given the size of the case.

It does have an Infrared device, which is quite slow and useless, but for its time it was as good as wireless could get. The other holes correspond to the power adapter and the reset button next to the PCMCIA, big enough to be able to reset the laptop with a regular pen.

For the full specifications, please refer to the official leaflet.

The Libretto

The setup I will be using: the Libretto, and a 3Com 16-bit PCMCIA Ethernet card

Choosing a Linux distribution

I wanted to find some modern, low-demanding software, not unsupported versions of Debian or RedHat. As you might have expected by this page's title, I chose Damn Small Linux (DSL).

I was very lucky to find that my machine had been upgraded to 24 MB of RAM. Apparently, even low-end distros have difficulties booting a regular kernel with 16 MB. I didn't want to tune or recompile the kernel on a 75-MHz machine, so I had to do some tricks.

In order to decide on a distro, I tried to set some goals up:

  1. Discard modern distributions which require at least 256 MB of RAM. In fact, discard anything that doesn't work with 24 MB of RAM
  2. Try to avoid old versions of current distros (i.e. Debian Woody) because the ISOs and the packages might not be mirrored anymore and are difficult to be found.
  3. Use a distro which self-configures kernel modules on boot, because I will be installing from a Virtual Machine and the hardware will change between reboots. Recompiling the kernel is totally out of the picture.
  4. Kernel 2.4 if possible, to make both the audio and the Ethernet work
  5. As easy to configure as possible. I want to finish this in a few hours... [Narrator: he didn't]

I found myself with these contenders, Damn Small Linux, Puppy Linux and Tiny Core Linux.

DSL v4 was the chosen one for many reasons. First, the default software choice is a good compromise and finely tuned for low performing machines. The installation seemed the easiest of the three, and—very important—worked flawlessly inside VirtualBox. The documentation is very extensive and, as a slightly old distro, there are lots of manuals and forum posts with solutions to common problems.

There is also the fact that DSL is based on Knoppix, so it detected my hardware perfectly, didn't have to tweak the PCMCIA, and only had to configure the audio manually because the I/O ports were not the standard ones. This was a huge aid for me. PCMCIA Internet working out of the box is something I hadn't even imagined to have.

However, the decision also came with a few drawbacks. DSL has its own "package manager", which only works from X and can't uninstall packages.

apt-get can be enabled, but it might break packages already installed with MyDSL. Furthermore, those packages tend to disappear on a reboot for some reason. I'm still unsure on whether to use apt-get with MyDSL. We will not be using it.

The ACPI doesn't work, but I don't know whether it's the kernel or the Libretto's fault.

My biggest fear, however, is that most of the packages are old and might have security issues. However, as this will not be my main machine, and it won't run a browser with JavaScript enabled, I'm not very worried.

Why didn't I choose Tiny Core? because it didn't boot on a VirtualBox machine with 24 MB of RAM. It would have been my first choice, because it is better maintained than DSL. A real pity.

And what about Puppy? The LiveCD is great but the installation instructions were too complicated for me. I really didn't want to spend that much time configuring everything. It is maybe too modern (based on Slackware 13.37 with Kernel 3.1.10) and I doubt the Libretto could have handled its kernel.

Installation

Please note: I will assume that you have some experience with Linux, partitioning, and installing stuff from a console.

Strategy

There are two alternatives: use floppy disks or physically remove the drive and set up a VM. Years ago, I went the first path, because I had the floppy disk drive. Since I don't have it anymore, I found an awesome tutorial which suggested to physically remove the drive from the Libretto, attach it to a 2.5" IDE to USB adaptor, and install the system from another computer. Check out his pictures for details on how to remove the drive. My machine is in a bad condition (broken hinges, cracks all over the case, stuck screws) and I had to break some plastics and metal parts to access the drive.

So, we will use another Linux computer, which you probably already have, and set up a virtual machine inside VirtualBox. Then, we will remove the Libretto's physical HDD and attach it via USB to your computer, using an adapter. The DSL CD image and the new /dev/sd will be mapped inside the VM.

This way we can boot and install from a CD, instead of doing netinsts with the Debian Woody diskettes, as you will read on many other websites. It is the fastest and painless way, and if you don't have the floppy drive, it is the only way.

If you have the floppy drive and are wondering if it is worth to buy the adapter, go ahead! Walk the difficult path, install DOS, start a Linux setup from DOS, try to make the floppy disk work, then install from diskettes with a crappy kernel, fight with the PCMCIA driver until you are able to use the network, and install from the net. And, should the installation fail, start OVER AGAIN! When this happens, please send me an email so that I can pretend that I sympathize with you but actually laugh at your misery.

Talking seriously, I am just trying to warn you. I tried that, I failed, then I succeeded, and not even in my success I want even the worst of my enemies going that path. Buy it, then come back and follow these instructions.

Removing the hard drive

You already have the adapter? Great! I bought this one which worked great and allowed me to manipulate the drive from my main computer.

The 2.5inch IDE to USB adapter

This is the adapter in its box. It comes with an enclosure that I didn't use to avoid overheating, and a handy screwdriver.

The 2.5inch IDE to USB adapter, close up

A close up of the IDE adapter. Don't buy a SATA one by mistake!

Using the drive in VirtualBox

As stated before, we will use VirtualBox to make DSL think it is running on a real machine, and that the—now USB— hard drive is the main drive of the VM. Turns out that using a physical disk from /dev on VirtualBox isn't easy to find, but the actual command is simmple.

Please make sure that your Linux has detected the USB drive as /dev/sdb before proceeding or you might lose data on the wrong disk! If in case, use Disk Utility or check dmesg.

VBoxManage internalcommands createrawvmdk -filename disk.vmdk -rawdisk /dev/sdb
                                                                       ^^^^^^^^  -- check this

The command above will create a file named disk.vmdk, which is a short plaintext file which references to /dev/sdb. You can now add it to your VM using the normal VirtualBox Appliance Manager

Partitioning

Use your main Linux box to partition the hard drive. Disk Utility works well, but I used cfdisk.

The tutorial then notes that the last 32MB of the disk space are used for the Libretto's hardware Hibernate feature. I followed his partition table suggestions completely. Just in case his page is down, do this:

  • /dev/hda1 738.0 MB, ext2 (ext3 is slower, but more secure), mounted as /
  • /dev/hda2 40.3 MB, swap
  • A remaining free space of 37.2 MB. Don't worry if the figure is slightly higher or lower due to rounding.

Installing DSL

Now go ahead, and download the ISO image. I used the Release Candidate 4.11.rc1 and it didn't give me any problems

Set up a new VM with 24 MB of RAM, use the ISO as the CD drive, and the disk.vmdk as the hard drive. Then boot.

If everything goes well, you will be shown the desktop. Now, in order to install, I have adapted the official instructions

sudo -s
swapoff -a
mkswap /dev/hda2 ## Considering that you followed the partition scheme in the tutorial
swapon /dev/hda2
dsl-hdinstall

Follow the setup assistant from there. I chose Grub instead of LILO for the bootloader, and it worked. The network also works out of the box, so I didn't need to apply any modifications in /etc/default/pcmcia as stated in David's tutorial.

Now disconnect the USB adapter, remove the disk, put it back in the Libretto, and boot. You should be prompted with either the console login or a X desktop, depending on your setup.

Network

I have a PCMCIA 16-bit 5V 3Com Ethernet adapter and just recently acquired a wireless Orinoco Gold card, 16b 5V too, one of the few known to work with this Libretto model, albeit only in WEP-mode.

This Libretto only accepts Type II PCMCIA, so it is very difficult to find a Linux 2.4 compatible, WPA-capable wifi card. Please let me know if you managed to get WPA wifi working!

Here are some pictures, as a reference.

The 3Com card

The 3Com card, front

The 3Com card

The 3Com card, back. Note the "PC card" icon with the technical specs.

The Knoppix core of DSL detected my Ethernet card, configured it with DHCP, and it talked instantly to my home router. Woohoo, it's on the Internet! I actually didn't need to do anything, compared to the hell I suffered with the Debian setup some years ago.

Wireless

The Orinoco card

The Orinoco card

Again, thanks to the Knoppix core, the Libretto automatically detected the PCMCIA card, loaded the orinoco kernel module, and the card was ready to use.

First, prepare your wireless router to work with WEP. It is highly discouraged to do so, because it is a big security hole. Fortunately I had a spare router that I can use only for the Libretto and will turn it off after playing with it.

My recommended setup for the router is:

  • Hide the ESSID
  • Filter by MAC address
  • Use 802.11b with auto channel
  • Use a 128-bit ASCII WEP key

I tuned /opt/eth0.sh to run the iwconfig commands. Add this just below the #!/bin/bash line:

iwconfig eth0 essid ESSID_NAME
iwconfig eth0 key open s:WEPKEY
iwconfig eth0 mode managed
sleep 1

If you WEP key is hex and not ASCII, omit the s: part before it.

Wait for a few seconds, and when iwconfig reports a correct Access Point, you're on the internet. Congratulations!

Since the 50CT has very low specs, Firefox starts swapping like crazy. The best commandline browser is links2 and I recommend dillo if you run X.

Look ma, no cables!

Look ma, no cables!

Sound

This other tutorial points out some tricks to use all of the Libretto's capabilities. I didn't try most of them, but since I couldn't play any music, I went ahead and probed the opl3sa2 driver. At first, it didn't work, because the I/O parameters on my card weren't the same than on that page.

The BIOS of the Libretto

This is my main BIOS configuration

The BIOS of the Libretto, audio section

A detail of the audio section. From top to bottom, the values correspond to the following module parameters

  • mss_io
  • not used
  • not used
  • irq
  • dma
  • dma2
  • io
  • mpu_io

This means that we will load the module with the following parameters. Remember to check your BIOS and use the correct ones, or modprobe will fail

modprobe opl3sa2 io=0x370 mss_io=0x530 mpu_io=0x330 irq=7 dma=1 dma2=0

Note: to access the Libretto BIOS, reboot or reset it, and press <ESC> during or just after the memory check

Finally, the Pentium 75 CPU is able to play most mp3 files, but you will need to compile your own mpg123. DSL comes with mpg321, but the audio isn't fluid and for some reason only mpg123 is able to decode mp3 in realtime. Running it from a console instead of an X session also helps, though the main bottleneck is the CPU, not the RAM

ACPI/APM/Battery

I only managed to get APM working. Playing with the Grub boot options there is no way to enable ACPI.

This blog post has some pointers on how to install the Toshiba experimental ACPI driver, but as I didn't want to recompile the kernel, I couldn't use it. If you feel strong enough, use the same Virtual Machine that you used for the DSL install and recompile it there, with the power of a current computer.

The toshiba kernel module loads correctly (/proc/toshiba), but not toshiba_acpi (/dev/toshiba). Not a big deal for me, but if you managed to get it working without recompiling the Knoppix kernel, please let me know.

The Libretto does some power management by hardware (screen blanking, hibernation), and this is enough for me. However, to get the system to actually power off after a shutdown, edit the /boot/grub/menu.lst and change the parameter noapm to apm=on apm=power-off

torsmo, DSL's dashboard, usually manages to get my battery status, but I didn't investigate further.

Performance tricks

Here are some generic tips on how to save some RAM and CPU cycles

  • Enable DMA - For some reason, DSL disables DMA by default. To enable it, edit the Grub config file /boot/grub/menu.lst and change the boot parameter nodma for dma. You will then see a boot message saying that DMA has been enabled for /dev/hda
  • Disable ttys - Edit /etc/inittab and disable all consoles but one. Instead, run a GNU screen session to get terminal multiplexing.
  • No X - Disable the automatic X session that is launched on login. You might need to edit the .bashrc or the .bash_profile files. Comment out the startx command.
  • GNU tools - With those bytes we saved, use the DSL menu option "Upgrade to GNU tools" to replace the very basic BusyBox shell with the regular GNU tools.
  • Fix the date - Use MyDSL to install ntpdate and run it when coming back from hibernation, since the date will probably be incorrect: ntpdate ntp.apple.com

Currently my setup takes the following resources:

  • Used memory with X running: 10 MB
  • Used memory without X running: 3 MB
  • Used disk space: 290 MB

Not bad, right? 3 MB of RAM on boot and a full functioning X taking only 7 MB more. That leaves a whooping 14 MB for applications!

As the pointing device is not that great, I usually run a single tty with a screen session for terminal multiplexing, and do most of my work on the terminal. X is only needed for viewing PDFs or images, and that's a perfectly suitable task for that computer.

Final words

I find it amazing that a laptop from early 2001 can still hold about an hour of battery, its drive is still spinning, and that it overall works. DSL gave it a new life, and though it is tedious to use a cable or WEP to connect to the internet, it is a functioning UNIX system, with audio and a decent mobile typewriter. Yes, the keyboard is small and uncomfortable, but this thing fits in any bag. Why, by 1990s standards, it would "fit in your pocket"!

There is plenty of information out there on installing Linux on a Libretto, but at the time of writing this article, most of articles are about 7-10 years old. I hope that it can be useful for somebody who, like me, found this machine at the bottom of a drawer and might want to play with it a little, install a current Linux and maybe give it to your kids or use it as a second laptop.

I wouldn't use it as a server, since it has little memory to run a server daemon, the disk and fan are noisy, and keeping it on 24/7 would burn the machine. If you want a cheap server, go for an old Mac Mini and install the latest Debian there. The Libretto is a ultra portable laptop and, if yours still holds some battery charge, is a nice toy to write stuff on or browse the simple internet.

DSL is highly customizable, and there is plenty of documentation out there. The default software is great, and searching the net you will find current software which is suitable for low memory devices. You will find yourself with a machine capable of reading and writing emails, displaying images, playing music, and more.

The only sites it can't browse are those which use Flash or are heavy on JavaScript. Well, the modern web, Gmail, Facebook, Twitter... but if you try to use the mobile versions you might get a nice surprise. You can try to use the Firefox version bundled in DSL but I wouldn't recommend that, it's too slow.

Feel free to contact me if there is any mistake on the tutorial or if you have some contribution, for example, a command to make it run with WPA, or if you managed to make the ACPI work.

The Libretto running X

The Libretto running an X session

Originally published on Github in 2013

Tags: retro, hardware, unix

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

November 23, 2020

Ponylang (SeanTAllen)

Last Week in Pony - November 22, 2020 November 23, 2020 03:21 AM

The ponylang/glob package has upgraded its ponylang/regex dependency to improve the installation exprience for Windows users.

November 22, 2020

Derek Jones (derek-jones)

What software engineering data have I collected on subject X? November 22, 2020 10:32 PM

While it’s great that so much data was uncovered during the writing of the Evidence-based software engineering book, trying to locate data on a particular topic can be convoluted (not least because there might not be any). There are three sources of information about the data:

  • the paper(s) written by the researchers who collected the data,
  • my analysis and/or discussion of the data (which is frequently different from the original researchers),
  • the column names in the csv file, i.e., data is often available which neither the researchers nor I discuss.

At the beginning I expected there to be at most a few hundred datasets; easy enough to remember what they are about. While searching for some data, one day, I realised that relying on memory was not a good idea (it was never a good idea), and started including data identification tags in every R file (of which there are currently 980+). This week has been spent improving tag consistency and generally tidying them up.

How might data identification information be extracted from the paper that was the original source of the data (other than reading the paper)?

Named-entity recognition, NER, is a possible starting point; after all, the data has names associated with it.

Tools are available for extracting text from pdf file, and 10-lines of Python later we have a list of named entities:

import spacy

# Load English tokenizer, tagger, parser, NER and word vectors
nlp = spacy.load("en_core_web_sm")

file_name = 'eseur.txt'
soft_eng_text = open(file_name).read()
soft_eng_doc = nlp(soft_eng_text)

for ent in soft_eng_doc.ents:
     print(ent.text, ent.start_char, ent.end_char,
           ent.label_, spacy.explain(ent.label_))

The catch is that en_core_web_sm is a general model for English, and is not software engineering specific, i.e., the returned named entities are not that good (from a software perspective).

An application domain language model is likely to perform much better than a general English model. While there are some application domain models available for spaCy (e.g., biochemistry), and application datasets, I could not find any spaCy models for software engineering (I did find an interesting word2vec model trained on Stackoverflow posts, which would be great for comparing documents, but not what I was after).

While it’s easy to train a spaCy NER model, the time-consuming bit is collecting and cleaning the text needed. I have plenty of other things to keep me busy. But this would be a great project for somebody wanting to learn spaCy and natural language processing :-)

What information is contained in the undiscussed data columns? Or, from the practical point of view, what information can be extracted from these columns without too much effort?

The number of columns in a csv file is an indicator of the number of different kinds of information that might be present. If a csv is used in the analysis of X, and it contains lots of columns (say more than half-a-dozen), then it might be assumed that it contains more data relating to X.

Column names are not always suggestive of the information they contain, but might be of some use.

Many of the csv files contain just a few rows/columns. A list of csv files that contain lots of data would narrow down the search, at least for those looking for lots of data.

Another possibility is to group csv files by potential use of data, e.g., estimating, benchmarking, testing, etc.

More data is going to become available, and grouping by potential use has the advantage that it is easier to track the availability of new data that may supersede older data (that may contain few entries or apply to circumstances that no longer exist)

My current techniques for locating data on a given subject is either remembering the shape of a particular plot (and trying to find it), or using the pdf reader’s search function to locate likely words and phrases (and then look at the plots and citations).

Suggestions for searching or labelling the data, that don’t require lots of effort, welcome.

November 17, 2020

Bogdan Popa (bogdan)

Racketeering Gophers November 17, 2020 10:30 AM

rocketeering gopher
Close enough.

I’ve been working on a Wasm implementation in Racket for the past couple of weeks and have recently reached a neat milestone.

November 16, 2020

Andreas Zwinkau (qznc)

Use decision records already! November 16, 2020 12:00 AM

To develop more systematically, decision records are a great first step.

Read full article!

November 15, 2020

Derek Jones (derek-jones)

Researching programming languages November 15, 2020 10:12 PM

What useful things might be learned from evidence-based research into programming languages?

A common answer is researching how to design a programming language having a collection of desirable characteristics; with desirable characteristics including one or more of: supporting the creation of reliable, maintainable, readable, code, or being easy to learn, or easy to understand, etc.

Building a theory of, say, code readability is an iterative process. A theory is proposed, experiments are run, results are analysed; rinse and repeat until a theory having a good enough match to human behavior is found. One iteration will take many years: once a theory is proposed, an implementation has to be built, developers have to learn it, and spend lots of time using it to enable longer term readability data to be obtained. This iterative process is likely to take many decades.

Running one iteration will require 100+ developers using the language over several years. Why 100+? Lots of subjects are needed to obtain statistically meaningful results, people differ in their characteristics and previous software experience, and some will drop out of the experiment. Just one iteration is going to cost a lot of money.

If researchers do succeed in being funded and eventually discovering some good enough theories, will there be a mass migration of developers to using languages based on the results of the research findings? The huge investment in existing languages (both in terms of existing code and developer know-how) means that to stand any chance of being widely adopted these new language(s) are going to have to deliver a substantial benefit.

I don’t see a high cost multi-decade research project being funded, and based on the performance improvements seen in studies of programming constructs I don’t see the benefits being that great (benefits in use of particular constructs may be large, but I don’t see an overall factor of two improvement).

I think that creating new programming languages will continue to be a popular activity (it is vanity research), and I’m sure that the creators of these languages will continue to claim that their language has some collection of desirable characteristics without any evidence.

What programming research might be useful and practical to do?

One potentially practical and useful question is the lifecycle of programming languages. Where the components of the lifecycle includes developers who can code in the language, source code written in the language, and companies dependent on programs written in the language (who are therefore interested in hiring people fluent in the language).

Many languages come and go without many people noticing, a few become popular for a few years, and a handful continue to be widely used over decades. What are the stages of life for a programming language, what factors have the largest influence on how widely a language is used, and for how long it continues to be used?

Sixty years worth of data is waiting to be collected and collated; enough to keep researchers busy for many years.

The uses of a lifecycle model, that I can thinkk of, all involve the future of a language, e.g., how much of a future does it have and how might it be extended.

Some recent work looking at the rate of adoption of new language features includes: On the adoption, usage and evolution of Kotlin Features on Android development, and Understanding the use of lambda expressions in Java; also see section 7.3.1 of Evidence-based software engineering.

Andreas Zwinkau (qznc)

OKRs are about change November 15, 2020 12:00 AM

Objectives and Key Results is better understood by Change principles

Read full article!

November 14, 2020

Gokberk Yaltirakli (gkbrk)

Status update, November 2020 November 14, 2020 09:00 PM

I started working on a web front-end project. It is a compiler for web components. The compiler reads single-file web components similar to the Vue.js ones, and emits vanilla JavaScript.

I found that it works well for a small library of reusable components, and I prefer the minimal design over something like Vue or React which might be overkill for most components.

I put the code, along with a few examples, at element-compiler-python.


The second project I worked on is a homemade Version Control System (VCS). While it doesn’t have too much fancy functionality, it has basic support for branches and commits.

The project is following the Unix philosophy of small, composable tools that each do one thing well. The internal repo format is just simple plain text files. My goal with the project is to make it easy to write code that operates on the repo. Ideally, someone will be able to replace a sub-command like log or shortlog relatively easily.

All the commits are stored in diff form starting from an empty repo. In order to publish projects developed using the VCS, I also created a sub-command that pushes the commits into a Git mirror. This makes it possible to collaborate and share the projects even before the VCS has a non-zero traction.

While a lot of it currently works, I am planning to do a few ergonomics-related fixes before releasing the tool. An old version, which will be replaced by the final one after the release, can be found at dum. Oh yeah, I called it dum, because it is a short word that has a similar meaning to Git.


I started getting more familiar with the Gemini protocol and document format. I had previously written about these in Gemini. My goal is to write my own client and server, as per Gemini tradition.

I wanted something slightly easier to complete before I made a full client, so I decided to make yet another Gemini-to-HTTP proxy and host it as a serverless script. It is not feature-complete yet, but it can render most simple pages.

Here are a few examples:


I wrote a small utility, called httptime, that can synchronize your system time based on the HTTP Date header. If you know about the header, you will know that it has a 1-second resolution. This is usually not sufficient for accurate timekeeping. The script makes multiple HEAD requests at strategic timestamps in order to synchronize itself with the server time and increase the timing accuracy.

I found that this approach can get you to between 0.001 to 0.004 seconds of accuracy, which is good enough for most use cases.

In most cases, you should just use NTP or SNTP. But if you don’t want to get an NTP daemon, or you want to use TLS or plain HTTP for some reason, this approach works well is very minimal. Most importantly, the code can be understood by mere mortals in a few minutes.


I played around with neural networks and text generation. Instead of going with something fancy and using an attention mechanism, I just fed the last N characters to the network to predict the next one. The results were as expected, the rough format of words and sentences look okay but it lacks the context that makes the fancier text generation models more comprehensible.

I also started to collect data using the Twitch.tv GraphQL endpoint about when certain channels are online or offline. I might do a project in the future where I try to predict the future schedule of channels based on past data using machine learning.

That’s all for this month, thanks for reading!

Andreas Zwinkau (qznc)

Switch (review) November 14, 2020 12:00 AM

How to change things when change is hard

Read full article!

November 11, 2020

Grzegorz Antoniak (dark_grimoire)

Rant about Apple's keyboard hotkey system November 11, 2020 06:00 AM

Nothing could be easier to remember and understand -- I don't understand why Windows hasn't adopted it yet.

Command is generally used for command execution. But also it's used as a modification for mouse clicks. So, when you click with Command button active, you can sometimes get a different outcome.

Shift …

November 08, 2020

Derek Jones (derek-jones)

Evidence-based software engineering: book released November 08, 2020 11:30 PM

My book, Evidence-based software engineering, is now available; the pdf can be downloaded here, here and here, plus all the code+data. Report any issues here. I’m investigating the possibility of a printed version. Mobile friendly pdf (layout shaky in places).

The original goals of the book, from 10-years ago, have been met, i.e., discuss what is currently known about software engineering based on an analysis of all the publicly available software engineering data, and having the pdf+data+code freely available for download. The definition of “all the public data” started out as being “all”, but as larger and higher quality data was discovered the corresponding were ignored.

The intended audience has always been software developers and their managers. Some experience of building software systems is assumed.

How much data is there? The data directory contains 1,142 csv files and 985 R files, the book cites 895 papers that have data available of which 556 are cited in figure captions; there are 628 figures. I am currently quoting the figure of 600+ for the ‘amount of data’.


Cover image of book Evidence-based software engineering.

Things that might be learned from the analysis has been discussed in previous posts on the chapters: Human cognition, Cognitive capitalism, Ecosystems, Projects and Reliability.

The analysis of the available data is like a join-the-dots puzzle, except that the 600+ dots are not numbered, some of them are actually specs of dust, and many dots are likely to be missing. The future of software engineering research is joining the dots to build an understanding of the processes involved in building and maintaining software systems; work is also needed to replicate some of the dots to confirm that they are not specs of dust, and to discover missing dots.

Some missing dots are very important. For instance, there is almost no data on software use, but there can be lots of data on fault experiences. Without software usage data it is not possible to estimate whether the software is very reliable (i.e., few faults experienced per amount of use), or very unreliable (i.e., many faults experienced per amount of use).

The book treats the creation of software systems as an economically motivated cognitive activity occurring within one or more ecosystems. Algorithms are now commodities and are not discussed. The labour of the cognitariate is the means of production of software systems, and this is the focus of the discussion.

Existing books treat the creation of software as a craft activity, with developers applying the skills and know-how acquired through personal practical experience. The craft approach has survived because building software systems has been a sellers market, customers have paid what it takes because the potential benefits have been so much greater than the costs.

Is software development shifting from being a sellers market to a buyers market? In a competitive market for development work and staff, paying people to learn from mistakes that have already been made by many others is an unaffordable luxury; an engineering approach, derived from evidence, is a lot more cost-effective than craft development.

As always, if you know of any interesting software engineering data, please let me know.

Sevan Janiyan (sevan)

LFS, round #2, 3rd try November 08, 2020 02:33 AM

In my previous post I ended with the binutils test suite not being happy after steering off the guide and making some changes to which components were installed. I decided to start again but cut back on the changes and see just how much I could omit from installing to get to the point of …

Robin Schroer (sulami)

Writing for Reasons November 08, 2020 12:00 AM

This year, I have been writing more than even before over. In this article, I would like to discuss some of the reasons for writing and provide some thoughts on each.

Writing to Remember

This is probably the most obvious reason to write for a lot of people. Having written down a piece of information, you can come back later and recall it. Historical context can be invaluable for decision making, and often covers information that is not readily available anymore.

The key here is being able to find notes later on. Paper-based ones can be sorted by topic or chronologically, digital ones can be searched for. Formats can be useful here too, for example by supporting embedded code blocks or graphics.

Writing to Solve Problems

Early this year, before the pandemic hit Europe, I saw Paulus Esterhazy’s talk Angels Singing: Writing for Programmers at clojureD. It contained this great quote of Milton Friedman:

If you cannot state a proposition clearly and unambiguously, you do not understand it.

In another talk, Rich Hickey explained his notion of using notes as an extension of his working memory:

So we have a problem, in general, because we’re just being asked to write software that’s more and more complex as time goes by. And we know there’s a 7 +/- 2 sort of working memory limit and as smart as any of us are, we all suffer from the same limit but the problems that we are called upon to solve are much bigger than that. So what do we do if we can’t think the whole thing in our head at the same time? How can we work on a problem with more than nine components. What I’m going to recommend is that you write all the bits down.

[…]

But if we look at the 7 +/- 2 thing, we could say we can juggle seven to nine balls but if you can imagine having an assistant who every now and then can take one of those out and put a different color in and you can juggle balls of 20 different colors at the same time as long as there are only nine in the air at any one point in time. And that’s what you’re doing, you’re going to sort of look around at all these pieces and shift arbitrary shapes of seven into your head at different points in time.

Writing everything down allows digging deep into details and going off on tangents, and then returning to other aspects. As an added bonus, these notes can be useful in the future as well, if archived properly. I found org-mode outlines incredibly powerful for this purpose, with their foldable, tree-like structure that allows nesting sub-problems.

Writing to Make Decisions

Writing is invaluable for decision making. Not only does it aid the decision process (see above), it also allows returning to a decision later and reviewing it.

Architecture decision records (ADRs) are a tool established just for this purpose. The exact formats vary, and the details do not matter too much, but here are a few key points I consider essential:

  • The motivation for the decision
  • The constraints involved
  • The alternatives to consider and their respective tradeoffs

All of these are useful in several ways: they force you to acknowledge the components of the decision, make it simple to get an opinion on the matter from someone else, and also allow you to review the (potentially bad) decision later on.

There is one more point: the conclusion. This is easy to forget, because once a conclusion is reached, no one wants to spend time writing it down. But if you do not write it down, the document does not tell the whole story if reviewed in the future.

Writing to Develop Ideas

This year I have seen a lot of people writing about Sönke Ahrens’ How to Take Smart Notes, which is about taking notes as a means to develop long form writing. It popularised the idea of the Zettelkasten, a physical or virtual box of notes which reference each other to build an information network.

While I found the book quite interesting, I would not recommend it to everyone due to the significant organisation overhead involved.

That being said, I believe that if you have a digital system which can provide automatic back-links to avoid the exponentially growing amount of manual maintenance required, there is little harm in linking notes. At the very least it will make it easier to find a note, and maybe it can aid the thinking process by exposing previously unseen connections between concepts.

Writing to Communicate

This very article was written expressively to communicate information, and as such required some extra work for it to be effective.

The most important factor when writing for communication is the target audience. It dictates the format to use, and which prior knowledge can be assumed. Maximising information density by being as concise as possible is important to avoid wasting the reader’s time.

As an added difficulty, when writing something to be published you need to get it right the first time, there is no channel for discussing follow-up questions. The old adage in writing is “writing is rewriting”, and I very much believe that to be true in this case. Write an outline, then a first draft, then keep reading and revising it until it is just right. Maybe show it to someone you trust for feedback.

I personally also like to leave a draft and come back a few weeks later. This way I always have a few drafts for new articles ready for revision, until I feel that one is ready for publishing.

November 03, 2020

Wesley Moore (wezm)

Turning One Hundred Tweets Into a Blog Post November 03, 2020 12:40 AM

Near the conclusion of my #100binaries Twitter series I started working on the blog post that contained all the tweets. It ended up posing a number of interesting challenges and design decisions, as well as a couple of Rust binaries. Whilst I don't think the process was necessary optimal I thought I'd share the process to show my approach to solving the problem. Perhaps the tools used and approach taken is interesting to others.

My initial plan was to use Twitter embeds. Given a tweet URL it's relatively easy to turn it into some HTML markup. By including Twitter's embed JavaScript on the page the markup turns into rich Twitter embed. However there were a few things I didn't like about this option:

  • The page was going to end up massive, even split across a couple of pages because the Twitter JS was loading all the images for each tweet up front.
  • I didn't like relying on JavaScript for the page to render media.
  • I didn't really want to include Twitter's JavaScript (it's likely it would be blocked by visitors with an ad blocker anyway).

So I decided I'd render the content myself. I also decided that I'd host the original screenshots and videos instead of saving them from the tweets. This was relatively time consuming as they were across a couple of computers and not named well but I found them all in the end.

To ensure the page wasn't enormous I used the loading="lazy" attribute on images. This is a relatively new attribute that tells the browser to delay loading of images until they're within some threshold of the view port. It currently works in Firefox and Chrome.

I used preload="none" on videos to ensure video data was only loaded if the visitor attempted to play it.

To prevent the blog post from being too long/heavy I split it across two pages.

Collecting All the Tweet URLs

With the plan in mind the first step was getting the full list of tweets. For better or worse I decided to avoid using any of Twitter's APIs that require authentication. Instead I turned to nitter (an alternative Twitter front-end) for its simple markup and JS free rendering.

For each page of search results for '#100binaries from:@wezm' I ran the following in the JS Console in Firefox:

tweets = []
document.querySelectorAll('.tweet-date a').forEach(a => tweets.push(a.href))
copy(tweets.join("\n"))

and pasted the result into tweets.txt in Neovim.

When all pages had be processed I turned the nitter.net URLs in to twitter.com URLs: :%s/nitter\.net/twitter.com/.

This tells Neovim: for every line (%) substitute (s) nitter.net with twitter.com.

Turning Tweet URLs Into Tweet Content

Now I needed to turn the tweet URLs into tweet content. In hindsight it may have been better to use Twitter's GET statuses/show/:id API to do this (possibly via twurl) but that is not what I did. Onwards!

I used the unauthenticated oEmbed API to get some markup for each tweet. xargs was used to take a line from tweets.txt and make the API (HTTP) request with curl]

xargs -I '{url}' -a tweets.txt -n 1 curl https://api.twitter.com/1/statuses/oembed.json\?omit_script\=true\&dnt\=true\&lang\=en\&url\=\{url\} > tweets.json

This tells xargs to replace occurrences of {url} in the command with a line (-n 1) read from tweets.txt (-a tweets.txt).

The result of one of these API requests is JSON like this (formatted with jq for readability):

{
  "url": "https://twitter.com/wezm/status/1322855912076386304",
  "author_name": "Wesley Moore",
  "author_url": "https://twitter.com/wezm",
  "html": "<blockquote class=\"twitter-tweet\" data-lang=\"en\" data-dnt=\"true\"><p lang=\"en\" dir=\"ltr\">Day 100 of <a href=\"https://twitter.com/hashtag/100binaries?src=hash&amp;ref_src=twsrc%5Etfw\">#100binaries</a><br><br>Today I&#39;m featuring the Rust compiler — the binary that made the previous 99 fast, efficient, user-friendly, easy-to-build, and reliable binaries possible.<br><br>Thanks to all the people that have worked on it past, present, and future. <a href=\"https://t.co/aBEdLE87eq\">https://t.co/aBEdLE87eq</a> <a href=\"https://t.co/jzyJtIMGn1\">pic.twitter.com/jzyJtIMGn1</a></p>&mdash; Wesley Moore (@wezm) <a href=\"https://twitter.com/wezm/status/1322855912076386304?ref_src=twsrc%5Etfw\">November 1, 2020</a></blockquote>\n",
  "width": 550,
  "height": null,
  "type": "rich",
  "cache_age": "3153600000",
  "provider_name": "Twitter",
  "provider_url": "https://twitter.com",
  "version": "1.0"
}

The output from xargs is lots of these JSON objects all concatenated together. I needed to turn tweets.json into an array of objects to make it valid JSON. I opened up the file in Neovim and:

  • Added commas between the JSON objects: %s/}{/},\r{/g.
    • This is, substitute }{ with },{ and a newline (\r), multiple times (/g).
  • Added [ and ] to start and end of the file.

I then reversed the order of the objects and formatted the document with jq (from within Neovim): %!jq '.|reverse' -.

This filters the whole file though a command (%!). The command is jq and it filters the entire document ., read from stdin (-), through the reverse filter to reverse the order of the array. jq automatically pretty prints.

It would have been better to have reversed tweets.txt but I didn't realise they were in reverse chronological ordering until this point and doing it this way avoided making another 100 HTTP requests.

Rendering tweets.json

I created a custom Zola shortcode, tweet_list that reads tweets.json and renders each item in an ordered list. It evolved over time as I kept adding more information to the JSON file. It allowed me to see how the blog post looked as I implemented the following improvements.

💡
You used Rust for this!?

This is the sort of thing that would be well suited to a scripting language too. These days I tend to reach for Rust, even for little tasks like this. It's what I'm most familiar with nowadays and I can mostly write a "script" like this off the cuff with little need to refer to API docs.

The markup Twitter returns is full of t.co redirect links. I wanted to avoid sending my visitors through the Twitter redirect so I needed to expand these links to their target. I whipped up a little Rust program to do this: expand-t-co. It finds all t.co links with a regex (https://t\.co/[a-zA-Z0-9]+) and replaces each occurrence with the target of the link.

The target URL is determined by making making a HTTP HEAD request for the t.co URL and noting the value of the Location header. The tool caches the result in a HashMap to avoid repeating a request for the same t.co URL if it's encountered again.

I used the ureq crate to make the HTTP requests. Arguably it would have been better to use an async client so that more requests were made in parallel but that was added complexity I didn't want to deal with for a mostly one-off program.

Adding the Media

At this point I did a lot of manual work to find all the screenshots and videos that I shared in the tweets and added them to my blog. I also renamed them after the tool they depicted. As part of this process I noted the source of media files that I didn't create in a "media_source" key in tweets.json so that I could attribute them. I also added a "media" key with the name of the media file for each binary.

Some of the externally sourced images were animated GIFs, which lack playback controls and are very inefficient file size wise. Whenever I encountered an animated GIF I converted it to an MP4 with ffmpeg, resulting in large space savings:

ffmpeg -i ~/Downloads/so.gif -movflags faststart -pix_fmt yuv420p -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" so.mp4

This converts so.gif to so.mp4 and ensures the dimensions are a divisible by 2, which is apparently a requirement of H.264 streams encapsulated in MP4. I worked out how to do this from: https://unix.stackexchange.com/a/294892/5444

I also wanted to know the media dimensions for each file so that I could have them scaled properly on the page — most images are HiDPI and need to be presented at half their pixel width to appear the right size.

For this I used ffprobe, which is part of ffmpeg. I originally planned to use another tool to handle images (as opposed to videos) but it turns out ffprobe handles them too.

Since I wanted to update the values of JSON objects in tweets.json I opted to parse the JSON this time. Again I whipped up a little Rust "script": add-media-dimensions. It parses tweets.json and for each object in the array runs ffprobe on the media file, like this:

ffprobe -v quiet -print_format json -show_format -show_streams file.mp4

I learned how to do this from: https://stackoverflow.com/a/11236144/38820

With this invocation ffprobe produces JSON so add-media-dimensions also parses that and adds the width and height values to tweets.json. At the end the updated JSON document is printed to stdout. This turned out to be a handy sanity check as it detected a couple of copy/paste errors and typos in the manually added "media" values.

The oEmbed markup that Twitter returns includes links for each piece of media. Now that I'm handling that myself these can be deleted. Neovim is used for this:

:%s/ <a href=\\"https:\/\/twitter\.com[^"]\+\(photo\|video\)[^"]\+">pic.twitter.com[^<]\+<\/a>//

For each line of the file (%) substitute (s) matches with nothing. And that took care of them. Yes I'm matching HTML with a regex, no you shouldn't do this for something that's part of a program. For one-off text editing it's fine though, especially since you can eyeball the differences with git diff, or in my case tig status.

Adding a HiDPI Flag

I initially tried using a heuristic in tweet_list to determine if a media file was HiDPI or not but there were a few exceptions to the rule. I decided to add a "hidpi" value to the JSON to indicate if it was HiDPI media or not. A bit of trial and error with jq led to this:

jq 'map(. + if .width > 776 then {hidpi: true} else {hidpi:false} end)' tweets.json > tweets-hidpi.json

If the image is greater then 776 pixels wide then set the hidpi property to true, otherwise false. 776 was picked via visual inspection of the rendered page. Once satisfied with the result I examined the rendered result and flipped the hidpi value on some items where the heuristic was wrong.

Adding alt Text

Di, ever my good conscience when it comes to such things enquired at one point if I'd added alt text to the images. I was on the fence since the images were mostly there to show what the tools looked like — I didn't think they were really essential content — but she made a good argument for including some alt text even if it was fairly simplistic.

I turned to jq again to add a basic "media_description" to the JSON, which tweet_list would include as alt text:

jq 'map(. + {media_description: ("Screenshot of " + (.media // "????" | sub(".(png|gif|mp4|jpg)$"; "")) + " running in a terminal.")})' tweets.json > tweets-alt.json

For each object in the JSON array it adds a media_description key with a value derived from the media key (the file name with the extension removed). If the object doesn't have a media value then it is defaulted to "????" (.media // "????").

After these initial descriptions were added I went though the rendered page and updated the text of items where the description was incorrect or inadequate.

Video Poster Images

As it stood all the videos were just white boxes with playback controls since I has used preload="none" to limit the data usage of the page. I decided to pay the cost of the larger page weight and add poster images to each of the videos. I used ffmpeg to extract the first frame of each video as a PNG:

for m in *.mp4; do ffmpeg -i $m -vf "select=1" -vframes 1 $m.png; done

I learned how to do this from: https://superuser.com/a/1010108

I then converted the PNGs to JPEGs for smaller files. I could have generated JPEGs directly from ffmpeg but I didn't know how to control the quality — I wanted a relatively low quality for smaller files.

for f in *.mp4.png; do convert "$f" -quality 60 $f.jpg ; done

This produced files named filename.mp4.png.jpg. I'm yet to memorise how to manipulate file extensions in zsh, despite having been told how to do it, so I did a follow up step to rename them:

for f in *.mp4; do mv $f.png.jpg $f.jpg ; done

Wrapping Up

Lastly I ran pngcrush on all of the PNGs. It reliably reduces the file size in a lossless manner:

for f in *.png; do pngcrush -reduce -ow $f; done

With that I did some styling tweaks, added a little commentary and published the page.

If you made it this far, thanks for sticking with it to the end. I'm not sure how interesting or useful this post is but if you liked it let me know and I might do more like it in the future.

November 02, 2020

Wesley Moore (wezm)

One Hundred Rust Binaries - Page 2 November 02, 2020 02:00 AM

This is page two of my #100binaries list containing binaries 51–100. See the first page for the introduction and binaries 1–50.

  1. Screenshot of the set of images generated by color_blinder when applied the Rust home page.
  2. Source: https://github.com/Szymongib/bookmark/blob/f46e5361878de972b7f0d11565fbecdb6a66bad9/assets/bookmark-demo.gif
  3. Screenshot of Artichoke running a small Ruby program in a terminal.
  4. Screenshot of csview rendering a sample CSV file in a terminal using default, reinforced, and rounded styles. Source: https://github.com/wfxr/i/blob/e04314806087faf8715a753e70f1a77f10b189d2/csview-screenshot.png
  5. Source: https://github.com/marcusbuffett/pipe-rename/blob/b734616bab4b4ca4f31de0902479202f33bda545/renamer.gif
  6. Screenshot of Cogsy running in a terminal. Source: https://github.com/cartoon-raccoon/cogsy/blob/8111b15243398cfe9cec990b88ed19f6155f8b37/images/screenshots/cogsy_main.png
  7. Screenshot of tiny connected to several IRC channels on chat.freenode.net in a terminal.
  8. Source: https://github.com/orf/ptail/blob/b26b089816cf3f495dae26ae0316c91f724667ce/images/readme.gif
  9. Screenshot of procs running in a terminal.
  10. Screenshot of vopono running in a terminal and two different browsers, one showing the VPN applied, the other not. Source: https://github.com/jamesmcm/vopono/blob/ef9653b80aea5f1695f9ca02b06e2ff340f1fae0/screenshot.png
  11. Source: https://github.com/tarkah/tickrs/blob/a5bc18a470999b5c18c98a7188a477c8e305652b/assets/demo.gif
  12. Source: https://github.com/orf/git-workspace/blob/8403c57edd172e925b682ee6220653db37dd616c/images/readme-example.gif
  13. Source: https://github.com/wfxr/i/blob/e04314806087faf8715a753e70f1a77f10b189d2/minimap-vim.gif
  14. Screenshot of kmon running in a terminal.
  15. Source: https://github.com/samtay/so/blob/93c13cdbf3fecaf23f21237ecee42d62f62905e0/assets/demo.gif
  16. Screenshot of lipl plotting the CPU temperature of my computer in a terminal.
  17. Screenshot of Cicero running in a terminal, displaying the graphemes of the text 'Rust Café 🦀' and rendering the R glyph in PragmataPro.
  18. Screenshot of battop running in a terminal.
  19. Screenshot of xxv running in a terminal. Source: https://chrisvest.github.io/xxv/screenshot.png
  20. Screenshot of indexa running in a terminal.
  21. Screenshot of shy running in a terminal. Source: https://github.com/xvxx/shy/blob/21555eb5259fd498d1d8fb4a4c39cf90a502f443/img/screen1.jpeg
  22. Screenshot of frawk running in a terminal.
  23. Screenshot of serial-monitor running in a terminal.
  24. Screenshot of gfold running in a terminal.
  25. Screenshot of fselect running in a terminal.
  26. Screenshot of lfs running in a terminal.
  27. Screenshot of dotenv-linter running in a terminal.
  28. Screenshot of bottom running in a terminal.
  29. Screenshot of the output of huniq -h in a terminal.
  30. Screenshot of cargo-wipe being run on my Projects directory in a terminal.
  31. Screenshot of terminal-typeracer running in a terminal.
  32. Screenshot of Audiobench. Source: https://joshua-maros.github.io/audiobench/book/images/default_patch.png
  33. Animated GIF of rust-sloth rendering a 3D model of Pikachu in a terminal.
  34. Screenshot of fhc running in a terminal.
  35. Screenshot of desed running in a terminal.
  36. Screenshot of silver running in a terminal.
  37. Screenshot of fnm running in a terminal.
  38. Screenshot of the waitfor documentation showing the various condition flags it accepts.
  39. Screenshot of rusty-tags running in a terminal.
  40. Screenshot of the SongRec GUI after recognising a few songs. There is album art on the left and a history of recognised songs on the right.
  41. Screenshot of ddh running in a terminal.
  42. Source: https://github.com/Nukesor/images/blob/72c983b374ea32b64e5997477693030001bdd7a6/pueue.gif
  43. The Rust logo. Source: https://upload.wikimedia.org/wikipedia/commons/thumb/d/d5/Rust_programming_language_black_logo.svg/1200px-Rust_programming_language_black_logo.svg.png

« Back to page 1

One Hundred Rust Binaries November 02, 2020 02:00 AM

I recently completed a #100binaries series on Twitter wherein I shared one open-source Rust tool or application each day, for one hundred days (Jul—Nov 2020). This post lists binaries 1–50. See page 2 for binaries 51–100.

All images and videos without an explicit source were created by me for the series. Most picture the Alacritty terminal emulator running on Linux. I use the PragmataPro font and my prompt is generated by vim-promptline. The colour scheme is Base16 Default Dark.

I also wrote a follow-up post about how this page was built and the considerations that went into making it as lightweight as possible: Turning One Hundred Tweets Into a Blog Post.

  1. Screenshot of hexyl running in a terminal.
  2. Screenshot of exa running in a terminal.
  3. Screenshot of Alacritty disiplaying the Alacritty logo.
  4. Screenshot of Amp editing Rust source code in a terminal.
  5. Screenshot of the output of running Tokei on the Allsorts repository.
  6. Output generated by Silicon for a small Rust program.
  7. Screenshot of broot running in a terminal.
  8. Screenshot of viu rendering Ferris the Rustacean in a terminal.
  9. Screenshot of Emulsion displaying an image of Ferris the Rustacean.
  10. Screenshot of rusty-man rendering the Allsorts docs in a terminal.
  11. Source: https://github.com/imsnif/diskonaut/blob/2cf5c7bd061f42443288e538ae75fedf7a846d76/demo.gif
  12. Source: https://user-images.githubusercontent.com/12150276/75177190-91d4ab00-572d-11ea-80bd-c5e28c7b17ad.gif
  13. Screenshot of dijo running in a terminal.
  14. Screenshot of pastel running in a terminal.
  15. Screenshot of DWFV running in a terminal.
  16. Screenshot of Zenith running in a terminal displaying CPU, memory, network, disk, and process information.
  17. Screenshot of the output of dtool --help in a terminal.
  18. Screenshot of Castor displaying the Gemini home page.
  19. Screenshot of the output of watchexec --help in a terminal.
  20. Screenshot of meli running in a terminal. Source: https://meli.delivery/images/screenshots/threads.webp
  21. Screenshot of delta running in a terminal.
  22. Screenshot of sharewifi running in a terminal.
  23. Screenshot of eva running in a terminal.
  24. Screenshot of bat showing some Rust code in a terminal.
  25. Screenshot of dust running in a terminal.
  26. Screenshot taken by shotgun of mdcat rendering the shotgun README in a terminal.
  27. Screenshot of ripgrep running in a terminal.
  28. Screenshot of mdcat rendering a sampel Markdown document in a terminal.
  29. Source: https://github.com/hatoo/oha/blob/10b1dc0103c11e8144f3a61cbb481092d24a2062/demo.gif
  30. Source: https://starship.rs/demo.webm
  31. Source: https://raw.githubusercontent.com/foriequal0/git-trim/master/screencast.png
  32. Source: https://github.com/imsnif/bandwhich/blob/fde53ddb3bcb769bc3474ba3d739d268619bf138/demo.gif
  33. Screenshot of xsv running in a terminal.
  34. Screenshot of Shellcaster running in a terminal. Source: https://github.com/jeff-hughes/shellcaster/blob/f6cb4c55c4a6765483d7810a2b6d08a928e799e1/img/screenshot.png
  35. Screenshot of yj transformating a small YAML document into JSON in a terminal.
  36. Screenshot of tealdeer showing the tldr page for ls in a terminal.

Continue to page 2 »

November 01, 2020

Derek Jones (derek-jones)

The Weirdest people in the world November 01, 2020 11:13 PM

Western, Educated, Industrialized, Rich and Democratic: WEIRD people are the subject of Joseph Henrich’s latest book “The Weirdest People in the World: How the West Became Psychologically Peculiar and Particularly Prosperous”.

This book is in the mold of Jared Diamond’s Guns, Germs, and Steel: The Fates of Human Societies, but comes at the topic from a psychological/sociological angle.

This very readable book is essential reading for anyone wanting to understand how very different WEIRD people are, along with the societies they have created, compared to people and societies in the rest of the world today and the entire world up until around 500 years ago.

The analysis of WEIRD people/societies has three components: why we are different (I’m assuming that most of this blog’s readers are WEIRD), the important differences that are known about, and the cultural/societal consequences (the particularly prosperous in the subtitle is a big clue).

Henrich cites data to back up his theories.

Starting around 1,500 years ago the Catholic church started enforcing a ban on cousin marriage, which was an almost universal practice at the time and is still widely practiced in non-WEIRD societies. Over time the rules got stricter, until by the 11th century people were not allowed to marry anyone related out to their sixth cousin. The rules were not always strictly enforced, as Henrich documents, but the effect was to change the organization of society from being kin-based to being institution-based (in particular institutions such as the Church and state). Finding a wife/husband required people to interact with others outside their extended family.

Effects claimed, operating over centuries, of the shift from extended families to nuclear families are that people learned what Henrich calls “impersonal prosociality”, e.g., feeling comfortable dealing with strangers. People became more altruistic, the impartial rule of law spread (including democracy and human rights), plus other behaviors needed for the smooth running of large social units (such as towns, cities and countries).

The overall impact was that social units of WEIRD people could grow to include tens of thousands, even millions, or people, and successfully operate at this scale. Information about beneficial inventions could diffuse rapidly and people were free(ish) to try out new things (i.e., they were not held back by family customs), and operating in a society with free movement of people there were lots of efficiencies, e.g., companies were not obligated to hire family members, and could hire the best person they could find.

Consequently, the West got to take full advantage of scientific progress, invent and mass produce stuff. Outcompeting the non-WEIRD world.

The big ideas kind of hang together. Some of the details seem like a bit of a stretch, but I’m no expert.

My WEIRD story occurred about five years ago, when I was looking for a publisher for the book I was working on. One interested editor sent out an early draft for review. One of the chapters discusses human cognition, and I pointed out that it did not matter that most psychology experiments had been done using WEIRD subjects, because software developers were WEIRD (citing Henrich’s 2010 WEIRD paper). This discussion of WEIRD people was just too much for one of the reviewers, who sounded like he was foaming at the mouth when reviewing my draft (I also said a few things about academic researchers that upset him).

Gustaf Erikson (gerikson)

October 31, 2020

Patrick Louis (venam)

What Does It Take To Resolve A Hostname October 31, 2020 10:00 PM

slide1

Can also be found in presentation format here

Resolving A Name Is Complex

slide2

Resolving a domain name is complex. It’s not limited to the DNS, the Domain Name System — A decentralized and hierarchical system to associate names and other information to IP addresses.
It’s not something we, as users, usually pay attention to. We notice it only when we’re facing an issue. It normally works out of the box but really nobody get the crux.
You search online for clarifications but they barely help and add more confusion.

Here are some schemas trying to decipher the mystery that domain name resolution came to be.

slide3

One, two, and three, I think you get me, it is not easy. It’s never as simple as taking a hostname as a string, getting the DNS address in the /etc/resolv.conf config, then sending a request to the DNS on port 53 to be greeted back with the IP.
Behind the scene there are ton of files and libraries involved, all of this to get a domain name solved.

So in this talk we’ll try to create some order to try to understand thing as an end-user. Let’s make sense and reason behind this mess even if I have to say, I don’t get it much myself.
I can’t assess I haven’t made mistakes but if I did, please correct me, that would be great!

NIH

slide4

Let’s start with the misfits, the ones that don’t follow the rules, the not-invented-here syndrome found within our tools.
When it comes to DNS resolution, there’s no one-size fit all solution. Obviously, many of us don’t want to deal with all the complexity, so we say, “let’s pack these bytes ourselves, and forget the hassle”.
That’s pure heresy though. We’d prefer everything to work the same way, so that it’s easier to follow. It would be preferable that they all use the same lib, to all have the same behavior. That is, in our case to rely on the C standard lib, or the POSIX API our savior.

In all cases, let’s note some software that don’t rely on it, as we said, all the misfits.

  • The ISC/BSD BIND tools: from host, to dig, to drill, to nslookup, and more, used for debugging chores.
  • Firefox/Chrome/Chromium: There are the browsers, because they are one of a kind, bypassing libc and POSIX mechanism, implementing their own DNS API for performance reasons and perfectionism.
  • Any applications needing advanced DNS features, other than simple name to IP.
  • Language that don’t wrap around a libc: The Go programming language comes to mind. It implements it’s own resolver API.

Fortunately, I can ease your mind by letting you know that all of these will at least respect /etc/resolv.conf and /etc/hosts configurations. Files that we’ll see in the next sections.

Historic

slide5

I’ve taken a look at over a dozen different technologies and I think the best way to understand them is through their archaeologies. There’s a lot that can be explained about DNS resolution simply based on all the historic reasons.
The main thing you need to understand, is that there’s not a single clean library call to resolve a hostname. Standards and new specs have pilled up over the years, with some software that haven’t followed, but risking to disappear.

Overall, libc and POSIX provide multiple resolution APIs:

  • There’s the historic, low level one provided by ISC/BSD BIND resolver implementation within libc. Accessed though libresolv/resolv.h incantation.
  • The gethostbyname(3) and related functions, implementing an obsolete POSIX C specification.
  • The getaddrinfo(3), that is the modern POSIX C API for name resolution.

All these combinations, ladies and gentlemen, are the standard ways to resolve a name.
Newer applications will use getaddrinfo while older ones will use gethostbyname. Both of these 2 will often rely on something called NSS and another part to manage resolv.conf access.

Now let’s dive into each of these and you’ll get them like a breeze.

resolver(3)

slide6

The resolver layer is the oldest and most stable in our quest. It originates from 1983, today almost 37 years ago, at Berkeley university.

It comes from a project called BIND, Berkely Internet Name Domain, which was sponsored by a DARPA grants. And like the Berkeley socket that gave rise to the internet, it has now turned into much much pain.
It was the very first implementation of the DNS specifications. It got released in BSD4.3 and today the BIND project is maintained by the Internet Systems Consortium, aka ISC.

It not only offers servers and clients, and the debug tools which we mentioned earlier, but also offers a library called “libbind”. This library is the defacto implementation, the standard resolver, the one of a kind. It is initially based on all the original RFC discussions, namely RFC 881, 882, and 883.
The BSD people wrote technical papers assessing its feasibility, and went on recommending and implementing it within BSD.

At that point BIND wasn’t a standard yet, it was an optionally-compiled code for those who wanted to get their feet wet, those who wanted to try DNS.
Then it got part of the C standard library interface through resolver, libresolv, -lresolv, resolv.h, and closed the case

If you take a look at most Unix-like systems today, from MacOS, to OpenBSD, to Linux, and company, you’ll see clearly in resolv.h, the copyright going back to 1983, to that very date. But obviously, it depends on the choice of the implementer, a case by case

So then the code diverged, there’s the libresolv provided by the C standardization and the libbind provided by the BIND implementation. However, most Unix only add small specific changes to their needs. For example, resolver in glibc is baselined off libbind from BIND version 8.2.3.

This layer is normally used for low level DNS interactions because it’s missing the goodies we’ll see later in this presentation.

Now let’s talk about environments and configurations.

The resolver configuration file

The resolver configuration files were mentioned in BIND first release, in section 4.2.2.2 of “The Design and Implementation of ‘Domain Name Resolver’” by Mark Painter based on RFC883, part of the DNS RFC series.

This particular file being /etc/resolv.conf, you’ll see it hardcoded in resolv.h and if that file is missing, it’ll fall back to the localhost as the DNS, just to be safe.
Additionally, there’s /etc/host.conf, according to the manpage also “the resolver configuration file”, it’s so appropriately named. It’s a conf that dictates the working of /etc/hosts, the “static table lookup for hostsnames”.

So what’s in these files.
resolv.conf takes care of how to resolve names and which nameserver to use for that, while hosts simply has a list of known host aliases, ip + name, as simple as that.

Within resolv.conf you can also have a search list for domains. That’s if a name you’re searching for doesn’t have the minimum number of dots in it then it’ll add one of these TLD to it, top-level-domains, and keep searching until it finds something that fits.
This can also be manipulated in an environment variable LOCALDOMAIN.

$ echo 'example www.example.com' > ./host_aliases
$ HOSTALIASES="./host_aliases" getent hosts example
93.184.216.34   www.example.com

There can also be a sortlist IP netmask, for when there’s many results to match but you don’t want to give priority to the cloud VPS that lives only for cash.

Finally, there’s the option field, also overriden on the command line by the RES_OPTIONS environment variable. It manipulates the minimum number of dots we mentioned and also if you want can set debug as enabled.

Meanwhile, the hosts file is but a key-value db, simply made of domain names and IPs.

Its config also lets you change the order of results and for the rest you have host.conf to consult.

So remember, that all of these are mostly used everywhere because it’s the lowest layer. So it’s used by libbind and libresolv but also the custom NIH syndrome

Alright, so far that’s all classic clean stuff. Let’s move on to the next sections, you’ll scratch your head until there’s no dandruff.

gethostbyname(3) and getaddrinfo(3)

slide7

The C library POSIX specs create a superset over the C standard library. They add a few simpler calls to resolve hostnames and make it easy. These focus on returning A and AAAA records only, ipV4 and ipV6 respsectively.
There’s gethostbyname(3) which is deprecated, and there’s the newer getaddrinfo(3) defined in IEEE Std 1003.1g-2000, which mainly adds RFC3493 aka ipV6 is now supported. So applications are recommended to use this updated version unless they want to divert from mainland.

There are functions to resolve IP addresses to host names, but let’s focus only on name to ip for today, I know it’s lame.

Apart from ipV6 support being added, some internal structures have been updated as they weren’t so safe between subsequent calls and thus could be your demise and your fall.

Obviously they both return different structures.

hostent struct is returned to gethostbyname function caller. while getaddrinfo returns an addrinfo structure. Both being defined in the netdb.h header.

struct hostent {
	char  *h_name;            /* official name of host */
	char **h_aliases;         /* alias list */
	int    h_addrtype;        /* host address type */
	int    h_length;          /* length of address */
	char **h_addr_list;       /* list of addresses */
}
struct addrinfo {
	int              ai_flags;
	int              ai_family;
	int              ai_socktype;
	int              ai_protocol;
	socklen_t        ai_addrlen;
	struct sockaddr *ai_addr;
	char            *ai_canonname;
	struct addrinfo *ai_next;
};

Some libc implementations will get fancy and add their own modified versions of gethostbyname. For instance in glibc they add support for ipV6 in their modified gethostbyname2 for backward compatibility.

Regarding configuration files, getaddrinfo will consult /etc/gai.conf which takes care of the precedence of the addresses returned in the results. And now, you’re going to brandish your torch yelling at me “but resolver(3) already does that by default”. But I’ll let you know that resolver(3) is only interested in DNS calls only while these two POSIX functions in their egocentrism are more interested in all the ways, files, and mechanism that a name can be converted to an IP.
That is, they often rely on something called NSS which is what we’ll see in our next analysis.

nss(5)

slide8

Both gethostbyname(3) and getaddrinfo(3) will most likely rely on the NSS service, but what is NSS, aka Name Service Switch.
First of all it is not to be confuse with “Network Security Services”, which has the same accronym but has a lib called -libnss. In our case it’s -lnetdb, with the netdb.h header, so keep this in mind for later.

To understand what’s NSS is, we, again, have to go back in time, back when the tech was still in its prime.
There always has been the idea of sharing configurations between machines, however back in the days it was all hardcoded, with the exception of Ultrix.
Hardcoded in files like aliases for emails, /etc/hosts for local domains, the finger database and all that it entails. This idea dates back for so long that netdb.h header was almost always there, but was looking in these files we mentioned earlier

There are also a bunch of POSIX functions to get these values getservbyname, gethostent, gethostbyname, getservbyport, etc.. I think you can continue.

From that point on we needed something more flexible, and so Solaris OS said let’s not have it hardcoded, that’s not-acceptable. Let’s create something called the Yellow Page, a sort of phone book for configurations brokerage. But the name Yellow Page had legal issues so let’s go with NIS, for the Network Information Service.
Other Unices liked what they were doing in their business so they reproduced it in something called NSS. Though NSS, Network Service Switch is much simpler than NIS.

Let’s have a side note about OpenBSD OS which doesn’t implement NSS but has a pseudo-N.I.S., something called the ypserv(8), the Yellow Pages written by Theo de Raadt from scratch, but he doesn’t care about the legal name wrath.

On OpenBSD you can also find the nsdispatch(3) function The name-service switch dispatcher, something similar to NSS But I’m not sure, I’ll recheck my citations.

So let’s summarize, NSS is a client-server directory service protocol that has as role to distribute system config between different computers, to keep them harmozined. It is more flexible than the fixed files in libc and POSIX, and is arguably like LDAP, or zookeeper, if you know it. Or actually, like any modern way to share configs between containers and microservices.

“But what does it have to do with domain names”, you may ask, well, a map of name with ip is a config like any others, so it’s the same task. That also includes things from hosts, password, port, aliases, and groups. Yep, it’s quite the big soup.

Apart from the functions in POSIX there is command line utilities that goes by the name of getent that lets you access NSS facilities to do simple queries for its entries.

So for example you can get a service port based on the name of that service Yes, simple the name suffice.

> getent  services domain   
domain                53/tcp

This particular module will read the /etc/services file NSS is quite versatile.

We can obviously query for a hostname which is our main game.

And note that you can disable the IDN encoding too Remember all that domain name we did on the forums, all that voodoo

getent -i hosts 𝕟𝕚𝕩𝕖𝕣𝕤.𝕟𝕖𝕥 
getent  hosts 𝕟𝕚𝕩𝕖𝕣𝕤.𝕟𝕖𝕥 
#  178.62.236.80   STREAM nixers.net
#  178.62.236.80   DGRAM  
#  178.62.236.80   RAW  

So how is NSS actually working, how does it also do the resolving. The NSS library consults the /etc/nsswitch.conf and /etc/default/nss files and depending on the entries it will sequentially attempt until it’s satisfied, until it find what it wants until it got the demand.

You’ll find the “hosts” entry in this file, along with a list of string on its right.

hosts: files mymachines myhostname resolve [!UNAVAIL=return] dns

These strings are the modules which will dynamically be loaded and sequentially executed, the format even allowing to have appended conditional rules.
Like here I’m skipping resolve plugin if it’s not available on my machine.

To get a list of all modules, you can look in your lib directory mess for anything that starts with libnss_.

 /usr/lib > ls libnss_*
libnss_compat-2.32.so  libnss_dns-2.32.so    libnss_hesiod-2.32.so   libnss_systemd.so.2
libnss_compat.so       libnss_dns.so         libnss_hesiod.so        libnss_winbind.so
libnss_compat.so.2     libnss_dns.so.2       libnss_hesiod.so.2      libnss_winbind.so.2
libnss_db-2.32.so      libnss_files-2.32.so  libnss_myhostname.so.2  libnss_wins.so
libnss_db.so           libnss_files.so       libnss_mymachines.so.2  libnss_wins.so.2
libnss_db.so.2         libnss_files.so.2     libnss_resolve.so.2

The most common modules are the following: files, dns, nis, myhostname, and resolve (for systemd-resolved).

  • files: Reads a local file in our case /etc/resolv.conf or /etc/hosts, no polling or anything
  • dns: will try to resolve the name remotely, in this case yes, it’s pulling it.
  • nis: To use solaris YP/NIS
  • myhostname: which reads local files such as /etc/hosts and /etc/hostname similar to the files plugin in case you missed.
  • resolve: the resolve plugin is the systemd-resolved, yes don’t put me on a crucifix.

And theres a bunch of others In case you’re in a mood to be a crusader.

Let’s open a parenthesis on the resolve plugin, before you throw it quickly in the dustbin. It’s quite advanced having multiple features like caching, to DNSSEC validation, to resolveconf, as well as being an NSS plugin. And when used as an NSS plugin, you communicate with systemd-resolved via dbus sockets, otherwise it always listens on port 53 for fallback in case you didn’t use NSS.

You can consult its ResolveHostname() method/interface part of the org.freedesktop.resolve1.Manager dbus object.

Not let’s move to something else, something you haven’t thought of yet.

resolvconf(8)

slide9

As we said, resolv.conf is used by all these components, but not only them, also all network agents. They are also in charge of setting or changing the DNS address, each of them, from dhcp client, ppp daemon, vpn manager, network manager, they all want access. And what about having 2 network connections concurrently, each requiring their own separate DNS, obviously.

So everyone wants to use the resolv.conf file, thus we need a manager to handle it. We want to avoid an inconsistent state, it’s vital not let everyone mess with it, and that is what resolvconf(8) role is.
Anyone wanting to change the resolv.conf should instead pass through resolveconf to avoid the hassle. It does that by using it’s resolvconf command line executable. Similarly to the resolv.conf configuration, you can pass anything to it like domain, search, and options.

resolvconf -a eth0.dhclient << EOF
nameserver 10.0.0.42
nameserver 10.0.1.42
EOF

Now resolv.conf is rarely a plain normal file itself because the manager finds it easier to create a symbolic link and avoid the abusiveness. The default implementation has it in /run/resolvconf/name-interface/resolv.conf.

Accordingly, like any other tooling, resolvconf has configuration files in /etc/resolvconf.conf, and a directory with hooks in /etc/resolvconf/. Within these files you can mention if you want the symlink to be at another location.

resolv_conf=/var/adsuck/resolv.conf

I’m saying default implementation because like anything else on a system you can replace it with your own concoction. Two popular alternatives solution to this problem: openresolv, systemd-resolved, which we mentioned earlier.

So resolv.conf is rarely a file it’s more of a symlink, check all of these for example, you’ll be surprised I think.

/run/resolv.conf/resolv.conf
/run/systemd/resolve/stub-resolv.conf
/run/systemd/resolve/resolv.conf
/var/run/NetworkManager/resolv.conf
/var/run/NetworkManager/no-stub-resolv.conf

Caching

slide10

In computers you can make anything faster with another level of indirection. That’s what all cache mechanism try to offer and domain name resolving is no exception.
There are two places where caching is available, either through a local dns proxy or through something called nscd. Just remember that this last one isn’t very stable.

Let’s start with nscd which is an NSS proxy, so it not only caches the DNS queries but also anything related to getting an NSS entry.

The other caching method is to run your own local dns server, be it bind9, djbdns, dnscache, lwresd, dnscrypt-proxy or any other resolver.
These can either be full featured, bells and whistles or only provide lightweight cache proxy if you’re not feeling like you want the details.

Another reason to run such service would be to block ads and all their malice.

Also, just beware of flushing the cache, otherwise you’ll get surprises that will make you crash.

How To Debug

slide11

So now you sort of know that it depends on what everthing uses Once you got that you can now start an analysis.

You can use a BIND tool To debug if DNS is the fool Or simply do a wireshark trace if you don’t want to bother or these are not under your grace

You can also check which NSS pluging is loaded And make sure they’re not aborted

ltrace -e "*gethostbyname*@libnss*" getent hosts www.example.com

Remember that each tool can have their own configurations So it adds complexity to the equation.

Big Picture

slide12

Let’s conclude here.
You should now be comfortable with anything in the domain name resolution sphere. It’s all about shared config management, like zookeeper, ldap, and these other arrangements.

I hope you’ve learned a thing or two and that domain name resolution is less of a taboo.
Thanks for listening and have a nice evening.

slide13

References

Gokberk Yaltirakli (gkbrk)

How I keep track of what I’ve been working on October 31, 2020 09:00 PM

Especially on busy times, it is possible to forget the projects I’ve been working on. While I tend to remember the big ones, some small projects slip away from memory. This is troubling when someone asks if I’ve been working on anything interesting recently, or if I feel like I haven’t been productive. Seeing how many thing I managed to work on can be a good morale-booster.

This problem became more apparent recently when I started to publish “Status Update” blog posts, in which I write short notes about the projects I’ve been working on. Instead of looking through used to-do lists or diaries, I found a more effective solution using the POSIX tool find, specifically the -mtime flag.

When you call find with -mtime, it searches for files based on their modification dates. Here is the snippet I use to find the files I’ve worked on in the last month.

find ~/projects -mtime -30

The parameter -30 stands for the last 30 days, it can be modified as you wish. For example -7 would filter for the last week.

October 29, 2020

Maxwell Bernstein (tekknolagi)

Compiling a Lisp: Labelled procedure calls October 29, 2020 08:00 AM

firstprevious

Welcome back to the Compiling a Lisp series. Last time, we learned about Intel instruction encoding. This time, we’re going to use that knowledge to compile procedure calls.

The usual function expression in Lisp is a lambda — an anonymous function that can take arguments and close over variables. Procedure calls are not this. They are simpler constructs that just take arguments and return values.

We’re adding procedure calls first as a stepping stone to full closure support. This will help us get some kind of internal calling convention established and stack manipulation figured out before things get too complicated.

After this post, we will be able to support programs like the following:

(labels ((add (code (x y) (+ x y)))
         (sub (code (x y) (- x y))))
    (labelcall sub 4 (labelcall add 1 2)))
; => 1

and even this snazzy factorial function:

(labels ((factorial (code (x) 
            (if (< x 2) 1 (* x (labelcall factorial (- x 1)))))))
    (labelcall factorial 5))
; => 120

These are fairly pedestrian snippets of code but they demonstrate some new features we are adding, like:

  • A new labels form that all programs will now have to look like
  • A new code form for describing procedures and their parameters
  • A new labelcall expression for calling procedures

Ghuloum does not explain why he does this, but I imagine that the labels form was chosen over allowing multiple separate top-level bindings because it is easier to parse and traverse.

Big ideas

In order to compile a program, we are going to traverse every binding in the labels. For each binding, we will generate code for each code object.

Compiling code objects requires making an environment for their parameters. We’ll establish a calling convention later so that our compiler knows where to find the parameters.

Then, once we’ve emitted all the code for the bindings, we will compile the body. The body may, but is not required to, contain a labelcall expression.

In order to compile a labelcall expression, we will compile all of the arguments provided, save them in consecutive locations on the stack, and then emit a call instruction.

When all of these pieces come together, the resulting machine code will look something like this:

mov rsi, rdi  # prologue
label0:
  label0_code
label1:
  label1_code
main:
  main_code

You can see that all of the code objects will be compiled in sequence, followed by the body of the labels form.

Because I have not yet figured out how to start executing at somewhere other than the beginning of the generated code, and because I don't store generated code in any intermediate buffers, and because we don't know the sizes of any code in advance, I do this funky thing where I emit a `jmp` to the body code. If you, dear reader, have a better solution, please let me know.

Edit: jsmith45 gave me the encouragement I needed to work on this again. It turns out that storing the code offset of the beginning of the main_code (the labels body) adding that to the buf->address works just fine. I’ll explain more below.

A calling convention

We’re not going to use the System V AMD64 ABI. That calling convention requires that parameters are passed first in certain registers, and then on the stack. Instead, we will pass all parameters on the stack.

This makes our code simpler, but it also means that at some point later on, we will have to add a different kind of calling convention so that we can call foreign functions (like printf, or exit, or something). Those functions expect their parameters in registers. We’ll worry about that later.

If we borrow and adapt the excellent diagrams from the Ghuloum tutorial, this means that right before we make a procedure call, our stack will look like this:

               Low address

           |   ...            |
           +------------------+
           |   ...            |
           +------------------+
      +->  |   arg3           | rsp-56
  out |    +------------------+
  args|    |   arg2           | rsp-48
      |    +------------------+
      +->  |   arg1           | rsp-40
           +------------------+
           |                  | rsp-32
           +------------------+
      +->  |   local3         | rsp-24
      |    +------------------+
locals|    |   local2         | rsp-16
      |    +------------------+
      +->  |   local1         | rsp-8
           +------------------+
  base     |   return point   | rsp

               High address

Stack illustration courtesy of Leonard.

You can see the first return point at [rsp]. This is the return point placed by the caller of the current function.

Above that are whatever local variables we have declared with let or perhaps are intermediate values from some computation.

Above that is a blank space reserved for the second return point. This is the return point for the about-to-be-called function. The call instruction will fill in after evaluating all the arguments.

Above the return point are all the outgoing arguments. They will appear as locals for the procedure being called.

Finally, above the arguments, is untouched free stack space.

The call instruction decrements rsp and then writes to [rsp]. This means that if we just emitted a call, the first local would be overwritten. No good. Worse, the way the stack would be laid out would mean that the locals would look like arguments.

In order to solve this problem, we need to first adjust rsp to point to the last local. That way the decrement will move it below the local and the return address will go between the locals and the arguments.

After the call instruction, the stack will look different. Nothing will have actually changed, except for rsp. This change to rsp means that the callee has a different view:

               Low address

           |   ...            |
           +------------------+
           |   ...            |
           +------------------+
      +->  |   arg3           | rsp-24
  in  |    +------------------+
  args|    |   arg2           | rsp-16
      |    +------------------+
      +->  |   arg1           | rsp-8
           +------------------+
  base     |   return point   | rsp
           +------------------+
           |   ~~~~~~~~~~~~   |
           +------------------+
           |   ~~~~~~~~~~~~   |
           +------------------+
           |   ~~~~~~~~~~~~   |
           +------------------+
           |   ~~~~~~~~~~~~   |

               High address

Stack illustration courtesy of Leonard.

The empty colored in spaces below the return point indicate that the values on the stack are “hidden” from view, since they are above (higher addresses than) [rsp]. The called function will not be able to access those values.

If the called function wants to use one of its arguments, it can pull it off the stack from its designated location.

One unfortunate consequence of this calling convention is that Valgrind does not understand it. Valgrind cannot understand that the caller has placed data on the stack specifically for the callee to read it, and thinks this is a move/jump of an uninitialized value. This means that we get some errors now on these labelcall tests.

Eventually, when the function returns, the ret instruction will pop the return point off the stack and jump to it. This will bring us back to the previous call frame.

That’s that! I have yet to find a good tool that will let me visualize the stack as a program is executing. GDB probably has a mode hidden away somewhere undocumented that does exactly this. Cutter sort of does, but it’s finicky in ways I don’t really understand. Maybe one day Kartik’s x86-64 Mu fork will be able to do this.

Building procedure calls in small pieces

In order for this set of changes to make sense, I am going to explain all of the pieces one at a time, top-down.

First, we’ll look at the new-and-improved Compile_entry, which has been updated to handle the labels form. This will do the usual Lisp entrypoint setup and some checks about the structure of the AST.

Then, we’ll actually look at compiling the labels. This means going through the bindings one-by-one and compiling their code objects.

Then, we’ll look at what it means to compile a code object. Hint: it’s very much like let.

Last, we’ll tie it all together when compiling the body of the labels form.

Compiling the entrypoint

Most of this code is checking. What used to just compile an expression now validates that what we’ve passed in at least vaguely looks like a well-formed labels form before picking it into its component parts: the bindings and the body.

int Compile_entry(Buffer *buf, ASTNode *node) {
  assert(AST_is_pair(node) && "program must have labels");
  // Assume it's (labels ...)
  ASTNode *labels_sym = AST_pair_car(node);
  assert(AST_is_symbol(labels_sym) && "program must have labels");
  assert(AST_symbol_matches(labels_sym, "labels") &&
         "program must have labels");
  ASTNode *args = AST_pair_cdr(node);
  ASTNode *bindings = operand1(args);
  assert(AST_is_pair(bindings) || AST_is_nil(bindings));
  ASTNode *body = operand2(args);
  return Compile_labels(buf, bindings, body, /*labels=*/NULL);
}

Compile_entry dispatches to Compile_labels for iterating over all of the labels. Compile_labels is a recursive function that keeps track of all the labels so far in its arguments, so we start it off with an empty labels environment.

Compiling labels

In Compile_labels, we have first a base case: if there are no labels we should just emit the body.

int Compile_labels(Buffer *buf, ASTNode *bindings, ASTNode *body,
                   Env *labels) {
  if (AST_is_nil(bindings)) {
    buf->entrypoint = Buffer_len(buf);
    // Base case: no bindings. Compile the body
    Buffer_write_arr(buf, kEntryPrologue, sizeof kEntryPrologue);
    _(Compile_expr(buf, body, /*stack_index=*/-kWordSize, /*varenv=*/NULL,
                   labels));
    Buffer_write_arr(buf, kFunctionEpilogue, sizeof kFunctionEpilogue);
    return 0;
  }
  // ...
}

We also set the buffer entrypoint location to the position where we’re going to emit the body of the labels. We’ll use this later when executing, or later in the series when we emit ELF binaries. You’ll have to add a field word entrypoint to your Buffer struct.

We pass in an empty varenv, since we are not accumulating any locals along the way; only labels. For the same reason, we give a stack_index of -kWordSize — the first slot.

If we do have labels, on the other hand, we should deal with the first label. This means:

  • pulling out the name and the code object
  • binding the name to the code location (the current location)
  • compiling the code

And then from there we deal with the others recursively.

int Compile_labels(Buffer *buf, ASTNode *bindings, ASTNode *body,
                   Env *labels) {
  // ....
  assert(AST_is_pair(bindings));
  // Get the next binding
  ASTNode *binding = AST_pair_car(bindings);
  ASTNode *name = AST_pair_car(binding);
  assert(AST_is_symbol(name));
  ASTNode *binding_code = AST_pair_car(AST_pair_cdr(binding));
  word function_location = Buffer_len(buf);
  // Bind the name to the location in the instruction stream
  Env entry = Env_bind(AST_symbol_cstr(name), function_location, labels);
  // Compile the binding function
  _(Compile_code(buf, binding_code, &entry));
  return Compile_labels(buf, AST_pair_cdr(bindings), body, &entry);
}

It’s important to note that we are binding before we compile the code object and we are making the code location available before it is compiled! This means that code objects can reference themselves and even recursively call themselves.

Since we then pass that binding into labels for the recursive call, it also means that labels can access all labels defined before them, too.

Now let’s figure out what it means to compile a code object.

Compiling code

I split this into two functions: one helper that pulls apart code objects (I didn’t want to do that in labels because I thought it would clutter the meaning), and one recursive function that does the work of putting the parameters in the environment.

So Compile_code just pulls apart the (code (x y z ...) body) into the formal parameters and the body. Since Compile_code_impl will need to recursively build up information about the stack_index and varenv, we supply those.

int Compile_code(Buffer *buf, ASTNode *code, Env *labels) {
  assert(AST_is_pair(code));
  ASTNode *code_sym = AST_pair_car(code);
  assert(AST_is_symbol(code_sym));
  assert(AST_symbol_matches(code_sym, "code"));
  ASTNode *args = AST_pair_cdr(code);
  ASTNode *formals = operand1(args);
  ASTNode *code_body = operand2(args);
  return Compile_code_impl(buf, formals, code_body, /*stack_index=*/-kWordSize,
                           /*varenv=*/NULL, labels);
}

I said this would be like let. What I meant by that was that, like let bodies, code objects have “locals” — the formal parameters. We have to bind the names of the parameters to successive stack locations, as per our calling convention.

In the base case, we do not have any formals, so we compile the body:

int Compile_code_impl(Buffer *buf, ASTNode *formals, ASTNode *body,
                      word stack_index, Env *varenv, Env *labels) {
  if (AST_is_nil(formals)) {
    _(Compile_expr(buf, body, stack_index, varenv, labels));
    Buffer_write_arr(buf, kFunctionEpilogue, sizeof kFunctionEpilogue);
    return 0;
  }
  // ...
}

We also emit this function epilogue, which right now is just ret. I got rid of the push rbp/mov rbp, rsp/pop rbp dance because we switched to using rsp only instead. I alluded to this in the previous instruction encoding interlude post.

In the case where we have at least one formals, we bind the name to the stack location and go on our merry way.

int Compile_code_impl(Buffer *buf, ASTNode *formals, ASTNode *body,
                      word stack_index, Env *varenv, Env *labels) {
  // ...
  assert(AST_is_pair(formals));
  ASTNode *name = AST_pair_car(formals);
  assert(AST_is_symbol(name));
  Env entry = Env_bind(AST_symbol_cstr(name), stack_index, varenv);
  return Compile_code_impl(buf, AST_pair_cdr(formals), body,
                           stack_index - kWordSize, &entry, labels);
}

That’s it! That’s how you compile procedures.

Compiling labelcalls

What use are procedures if we can’t call them? Let’s figure out how to compile procedure calls.

Code for calling a procedure must put the arguments and return address on the stack precisely how the called procedure expects them.

Getting this contract right can be tricky. I spent several frustrated hours getting this to not crash. Then, even though it didn’t crash, it returned bad data. It turns out that I was overwriting the return address by accident and returning to someplace strange instead.

Making handmade diagrams that track the changes to rsp and the stack really helps with understanding calling convention bugs.

We’ll start off by dumping yet more code into Compile_call. This code will look for something of the form (labelcall name ...).

Before calling into a helper function Compile_labelcall, we get two bits of information ready:

  • arg_stack_index, which is the first place on the stack where args are supposed to go. Since we’re skipping a space for the return address, this is one more than the current (available) slot index.
  • rsp_adjust, which is the amount that we’re going to have to, well, adjust rsp. Without locals from let or incoming arguments from a procedure call, this will be 0. With locals and/or arguments, this will be the total amount of space taken up by those.

Then we call Compile_labelcall.

int Compile_call(Buffer *buf, ASTNode *callable, ASTNode *args,
                 word stack_index, Env *varenv, Env *labels) {
    // ...
    if (AST_symbol_matches(callable, "labelcall")) {
      ASTNode *label = operand1(args);
      assert(AST_is_symbol(label));
      ASTNode *call_args = AST_pair_cdr(args);
      // Skip a space on the stack to put the return address
      word arg_stack_index = stack_index - kWordSize;
      // We enter Compile_call with a stack_index pointing to the next
      // available spot on the stack. Add kWordSize (stack_index is negative)
      // so that it is only a multiple of the number of locals N, not N+1.
      word rsp_adjust = stack_index + kWordSize;
      return Compile_labelcall(buf, label, call_args, arg_stack_index, varenv,
                               labels, rsp_adjust);
    }
    // ...
}

Compile_labelcall is one of those fun recursive functions we write so frequently. Its job is to compile all of the arguments and store their results in successive stack locations.

In the base case, it has no arguments to compile. It should just adjust the stack pointer, call the procedure, adjust the stack pointer back, and return.

void Emit_rsp_adjust(Buffer *buf, word adjust) {
  if (adjust < 0) {
    Emit_sub_reg_imm32(buf, kRsp, -adjust);
  } else if (adjust > 0) {
    Emit_add_reg_imm32(buf, kRsp, adjust);
  }
}

int Compile_labelcall(Buffer *buf, ASTNode *callable, ASTNode *args,
                      word stack_index, Env *varenv, Env *labels,
                      word rsp_adjust) {
  if (AST_is_nil(args)) {
    word code_address;
    if (!Env_find(labels, AST_symbol_cstr(callable), &code_address)) {
      return -1;
    }
    // Save the locals
    Emit_rsp_adjust(buf, rsp_adjust);
    Emit_call_imm32(buf, code_address);
    // Unsave the locals
    Emit_rsp_adjust(buf, -rsp_adjust);
    return 0;
  }
  // ...
}

Emit_rsp_adjust is a convenience function that takes some stack adjustment delta. If it’s negative, it will issue a sub instruction. If it’s positive, an add. Otherwise, it’ll do nothing.

In the case with arguments, we should compile them one at a time:

int Compile_labelcall(Buffer *buf, ASTNode *callable, ASTNode *args,
                      word stack_index, Env *varenv, Env *labels,
                      word rsp_adjust) {
  // ...
  assert(AST_is_pair(args));
  ASTNode *arg = AST_pair_car(args);
  _(Compile_expr(buf, arg, stack_index, varenv, labels));
  Emit_store_reg_indirect(buf, Ind(kRsp, stack_index), kRax);
  return Compile_labelcall(buf, callable, AST_pair_cdr(args),
                           stack_index - kWordSize, varenv, labels, rsp_adjust);
}

There, that wasn’t so bad, was it? I mean, if you manage to get it right the first time. I certainly did not. In fact, I gave up on the first version of this compiler many months ago because I could not get procedure calls right. With this post, I have now made it past that particular thorny milestone!

One last thing: we’ll need to update the code that converts buf->address into a function pointer. We have to use the buf->entrypoint we set earlier.

uword Testing_execute_entry(Buffer *buf, uword *heap) {
  assert(buf != NULL);
  assert(buf->address != NULL);
  assert(buf->state == kExecutable);
  // The pointer-pointer cast is allowed but the underlying
  // data-to-function-pointer back-and-forth is only guaranteed to work on
  // POSIX systems (because of eg dlsym).
  byte *start_address = buf->address + buf->entrypoint;
  JitFunction function = *(JitFunction *)(&start_address);
  return function(heap);
}

Let’s test our implementation. Maybe these tests will help you.

Testing

I won’t include all the tests in this post, but a full battery of tests is available in compile-procedures.c. Here are some of them.

First, we should check that compiling code objects works:

TEST compile_code_with_two_params(Buffer *buf) {
  ASTNode *node = Reader_read("(code (x y) (+ x y))");
  int compile_result = Compile_code(buf, node, /*labels=*/NULL);
  ASSERT_EQ(compile_result, 0);
  // clang-format off
  byte expected[] = {
      // mov rax, [rsp-16]
      0x48, 0x8b, 0x44, 0x24, 0xf0,
      // mov [rsp-24], rax
      0x48, 0x89, 0x44, 0x24, 0xe8,
      // mov rax, [rsp-8]
      0x48, 0x8b, 0x44, 0x24, 0xf8,
      // add rax, [rsp-24]
      0x48, 0x03, 0x44, 0x24, 0xe8,
      // ret
      0xc3,
  };
  // clang-format on
  EXPECT_EQUALS_BYTES(buf, expected);
  AST_heap_free(node);
  PASS();
}

As expected, this takes the first argument in [rsp-8] and second in [rsp-16], storing a temporary in [rsp-24]. This test does not test execution because I did not want to write the testing infrastructure for manually setting up procedure calls.

Second, we should check that defining labels works:

TEST compile_labels_with_one_label(Buffer *buf) {
  ASTNode *node = Reader_read("(labels ((const (code () 5))) 1)");
  int compile_result = Compile_entry(buf, node);
  ASSERT_EQ(compile_result, 0);
  // clang-format off
  byte expected[] = {
      // mov rax, compile(5)
      0x48, 0xc7, 0xc0, 0x14, 0x00, 0x00, 0x00,
      // ret
      0xc3,
      // mov rsi, rdi
      0x48, 0x89, 0xfe,
      // mov rax, 0x2
      0x48, 0xc7, 0xc0, 0x04, 0x00, 0x00, 0x00,
      // ret
      0xc3,
  };
  // clang-format on
  EXPECT_EQUALS_BYTES(buf, expected);
  Buffer_make_executable(buf);
  uword result = Testing_execute_entry(buf, /*heap=*/NULL);
  ASSERT_EQ_FMT(Object_encode_integer(1), result, "0x%lx");
  AST_heap_free(node);
  PASS();
}

This tests for a jump over the compiled procedure bodies (CHECK!), emitting compiled procedure bodies (CHECK!), and emitting the body of the labels form (CHECK!). This one we can execute.

Third, we should check that passing arguments to procedures works:

TEST compile_labelcall_with_one_param(Buffer *buf) {
  ASTNode *node = Reader_read("(labels ((id (code (x) x))) (labelcall id 5))");
  int compile_result = Compile_entry(buf, node);
  ASSERT_EQ(compile_result, 0);
  // clang-format off
  byte expected[] = {
      // mov rax, [rsp-8]
      0x48, 0x8b, 0x44, 0x24, 0xf8,
      // ret
      0xc3,
      // mov rsi, rdi
      0x48, 0x89, 0xfe,
      // mov rax, compile(5)
      0x48, 0xc7, 0xc0, 0x14, 0x00, 0x00, 0x00,
      // mov [rsp-16], rax
      0x48, 0x89, 0x44, 0x24, 0xf0,
      // call `id`
      0xe8, 0xe6, 0xff, 0xff, 0xff,
      // ret
      0xc3,
  };
  // clang-format on
  EXPECT_EQUALS_BYTES(buf, expected);
  Buffer_make_executable(buf);
  uword result = Testing_execute_entry(buf, /*heap=*/NULL);
  ASSERT_EQ_FMT(Object_encode_integer(5), result, "0x%lx");
  AST_heap_free(node);
  PASS();
}

This tests that we put the arguments in the right stack locations (skipping a space for the return address), emit a call to the right relative address, and that the call returns successfully. All check!!

Fourth, we should check that we adjust the stack when we have locals:

TEST compile_labelcall_with_one_param_and_locals(Buffer *buf) {
  ASTNode *node = Reader_read(
      "(labels ((id (code (x) x))) (let ((a 1)) (labelcall id 5)))");
  int compile_result = Compile_entry(buf, node);
  ASSERT_EQ(compile_result, 0);
  // clang-format off
  byte expected[] = {
      // mov rax, [rsp-8]
      0x48, 0x8b, 0x44, 0x24, 0xf8,
      // ret
      0xc3,
      // mov rsi, rdi
      0x48, 0x89, 0xfe,
      // mov rax, compile(1)
      0x48, 0xc7, 0xc0, 0x04, 0x00, 0x00, 0x00,
      // mov [rsp-8], rax
      0x48, 0x89, 0x44, 0x24, 0xf8,
      // mov rax, compile(5)
      0x48, 0xc7, 0xc0, 0x14, 0x00, 0x00, 0x00,
      // mov [rsp-24], rax
      0x48, 0x89, 0x44, 0x24, 0xe8,
      // sub rsp, 8
      0x48, 0x81, 0xec, 0x08, 0x00, 0x00, 0x00,
      // call `id`
      0xe8, 0xd3, 0xff, 0xff, 0xff,
      // add rsp, 8
      0x48, 0x81, 0xc4, 0x08, 0x00, 0x00, 0x00,
      // ret
      0xc3,
  };
  // clang-format on
  EXPECT_EQUALS_BYTES(buf, expected);
  Buffer_make_executable(buf);
  uword result = Testing_execute_entry(buf, /*heap=*/NULL);
  ASSERT_EQ_FMT(Object_encode_integer(5), result, "0x%lx");
  AST_heap_free(node);
  PASS();
}

This tests the presence of sub and add instructions for adjusting rsp. It also tests that that did not mess up our stack frame for returning to the caller of the Lisp entrypoint — the test harness.

Fifth, we should check that procedures can refer to procedures defined before them:

TEST compile_multilevel_labelcall(Buffer *buf) {
  ASTNode *node =
      Reader_read("(labels ((add (code (x y) (+ x y)))"
                  "         (add2 (code (x y) (labelcall add x y))))"
                  "    (labelcall add2 1 2))");
  int compile_result = Compile_entry(buf, node);
  ASSERT_EQ(compile_result, 0);
  Buffer_make_executable(buf);
  uword result = Testing_execute_entry(buf, /*heap=*/NULL);
  ASSERT_EQ_FMT(Object_encode_integer(3), result, "0x%lx");
  AST_heap_free(node);
  PASS();
}

And last, but definitely not least, we should check that procedures can refer to themselves:

TEST compile_factorial_labelcall(Buffer *buf) {
  ASTNode *node = Reader_read(
      "(labels ((factorial (code (x) "
      "            (if (< x 2) 1 (* x (labelcall factorial (- x 1)))))))"
      "    (labelcall factorial 5))");
  int compile_result = Compile_entry(buf, node);
  ASSERT_EQ(compile_result, 0);
  Buffer_make_executable(buf);
  uword result = Testing_execute_entry(buf, /*heap=*/NULL);
  ASSERT_EQ_FMT(Object_encode_integer(120), result, "0x%lx");
  AST_heap_free(node);
  PASS();
}

Ugh, beautiful. Recursion works. Factorial works. I’m so happy.

What’s next?

The logical next step in our journey is to compile lambda expressions. This has some difficulty, notably that lambdas can capture variables from outside the lambda. This means that next time, we will implement closures.

For now, revel in your newfound procedural freedom.

Mini Table of Contents

October 25, 2020

Derek Jones (derek-jones)

Benchmarking desktop PCs circa 1990 October 25, 2020 11:05 PM

Before buying a computer customers want to be confident of choosing the best they can get for the money, and performance has often been a major consideration. Computer benchmark performance results were once widely discussed.

Knight’s analysis of early mainframe performance was widely cited for many years.

Performance on the Byte benchmarks was widely cited before Intel started spending billions on advertising, clock frequency has not always had the brand recognition it has today.

The Byte benchmark was originally designed for Intel x86 processors running Microsoft DOS; The benchmark was introduced in the June 1985 issue, and was written in the still relatively new C language (earlier microprocessor benchmarks were often written in BASIC, because early micros often came with a free BASIC interpreter), it was updated in the 1990s to be Windows based, and implemented for Unix.

Benchmarking computers using essentially the same cpu architecture and operating system removes many complications that have to be addressed when these differ. Before Wintel wiped them out, computers from different manufacturers (and often the same manufacturer) contained completely different cpu architectures, ran different operating systems, and compilers were usually created in-house by the manufacturer (or some university who got a large discount on their computer purchase).

The Fall 1990 issue of Byte contains tables of benchmark results from 1988-90. What can we learn from these results?

The most important takeaway from the tables is that those performing the benchmarks appreciated the importance of measuring hardware performance using the applications that customers are likely to be running on their computer, e.g., word processors, spreadsheets, databases, scientific calculations (computers were still sufficiently niche back then that scientific users were a non-trivial percentage of the market), and compiling (hackers were a large percentage of Byte’s readership).

The C benchmarks attempted to measure CPU, FPU (built-in hardware support for floating-point arrived with the 486 in April 1989, prior to that it was an add-on chip that required spending more money), Disk and Video (at the time support for color was becoming mainstream, but bundled hardware graphics support still tended to be minimal).

Running the application benchmarks takes a lot of time, plus the necessary software (which takes time to install from floppies, the distribution technology of the day). Running the C benchmarks is much quicker and simpler.

Ideally the C benchmarks are a reliable stand-in for the application benchmarks (meaning that only the C benchmarks need be run).

Let’s fit some regression models to the measurements of the 61 systems benchmarked, all supporting hardware floating-point (code+data). Surprisingly there is no mention of such an exercise being done by the Byte staff, even though one of the scientific benchmarks included regression fitting.

The following fitted equations explain around 90% of the variance of the data, i.e., they are good fits.

Wordprocessing=0.66+0.56*CPU+0.24*Disk

For wordprocessing, the CPU benchmark explains around twice as much as the Disk benchmark.

Spreedsheet=-0.46+0.8*CPU+1*Disk-0.16*CPU*Disk

For spreadsheets, CPU and Disk contribute about the same.

Database=0.6+0.01*CPU*FPU+0.53*Disk

Database is nearly all Disk.

ScientificEngineering=0.27+FPU*(0.59-0.17*Disk-0.03*CPU)+0.45*CPU*Disk

Scientific/Engineering is FPU, plus interactions with other components.

Compiling=-0.33+CPU*(1.1-0.09*Disk-0.16*Video)+0.33*Disk*Video

Compiling is CPU, plus interactions with other components.

Byte’s benchmark reports were great eye candy, and readers probably took away a rough feel for the performance of various systems. Perhaps somebody at the time also fitted regression models to the data. The magazine contained plenty of adverts for software to do this.

Gokberk Yaltirakli (gkbrk)

Dynamic DNS with AWS Route 53 October 25, 2020 09:00 PM

I occasionally need to SSH into my laptop, or use other services hosted on it, but my ISP gives me a dynamic IP. While it is stable most of the time, it does occasionally change.

To work around this, I had previously set up a cron job that curl’s a specific URL on my website, and I could get the IP by grep-ing through my server logs. But this is both time-consuming and requires me to update the IP address on different applications every time it changes.

I wanted to have a subdomain that always pointed to my laptop, so I used the AWS CLI to create a DIY dynamic DNS. It fetches your IP from the AWS endpoint, but any source can be used. You can also host this on a server or a serverless function to get the client IP and require less dependencies.

Here’s the shell script that runs every X minutes on my laptop in order to update the domain record.

#!/bin/sh

IP="$(curl -s http://checkip.amazonaws.com/)"

DNS="$(mktemp)"

cat > "${DNS}" <<EOF
{
  "Comment": "DDNS update",
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "ResourceRecords": [
          {"Value": "${IP}"}
        ],
        "Name": "subdomain123.gkbrk.com",
        "Type": "A",
        "TTL": 300
      }
    }
  ]
}
EOF

aws route53 change-resource-record-sets \
    --hosted-zone-id "/hostedzone/ZONEID" \
    --change-batch "file://${DNS}"

rm "${DNS}"

October 24, 2020

Andrew Montalenti (amontalenti)

New essay: The case for a vote to restore jobs and health October 24, 2020 04:19 PM

It’s not a “left” or a “right” thing. It’s about jobs and health. If you study the data, you’ll learn this is what’s at stake in 2020.

We can restore our country’s health. We can rebuild our economy. We can do both of these things.

There is a precedent for this.

To learn more behind my thinking — digging into the data, steering clear of the partisan bickering — check out this essay, a few thousand words, supported by real data.

2020 is a very important year for our country, and for the world. Whether you’re a Democrat, or a Republican, or an Independent, your vote matters.

Click through to the essay here:

https://amontalenti.com/pub/jobs-health-2020/

And if you want to share this essay on Twitter, here’s a place to start:

Mark J. Nelson (mjn)

Years since Republican control of major cities October 24, 2020 12:00 PM

A large urban–rural divide is a current feature of American politics. Republicans rarely win cities, and Democrats rarely win rural areas. This goes beyond regional concepts like "the heartland" and "coastal elites", and is replicated within just about every state.

There are many ways to slice and dice that divide. This post takes one of them. Republicans can rarely win in a major city. But how rarely? How long has it been since a Republican candidate has won the mayorship of major American cities?

From longest to shortest, here is the list of Republican shutouts in American municipal politics, among the top 20 largest cities:

City No R mayor since Notes
Chicago 1931 Officially nonpartisan since 1999
Philadelphia 1952
Indianapolis 1956
Denver 1963 Officially nonpartisan
San Francisco 1964 Officially nonpartisan
San Jose 1967/∞ Officially nonpartisan; elected office created in 1967
Seattle 1969 Officially nonpartisan
Washington, DC 1975/∞ Elected office created in 1975
Houston 1982 Officially nonpartisan
Austin 1991 Officially nonpartisan
Columbus 2000
Los Angeles 2001 Officially nonpartisan
San Antonio 2001 Officially nonpartisan
Phoenix 2004 Officially nonpartisan
New York City 2007 Bloomberg changed from R to I in 2007
Charlotte 2009
Dallas 2011 Officially nonpartisan
San Diego 2020 Current R officeholder; offically nonpartisan
Jacksonville 2020 Current R officeholder
Fort Worth 2020 Current R officeholder; offically nonpartisan

October 23, 2020

Gustaf Erikson (gerikson)

Fastnet, Force 10 by John Rousmaniere October 23, 2020 11:48 AM

Written shortly after the tragedy, this is a very 1970s book. The author describes himself unapologetically as a “WASP”, for example, which would probably not fly these days.

It’s long on descriptions but short on analysis. The descriptions however are pretty horrifying. If you ever feel like taking up ocean racing maybe read this first.

October 19, 2020

Marc Brooker (mjb)

Getting Big Things Done October 19, 2020 12:00 AM

Getting Big Things Done

In one particular context.

A while back, a colleague wanted to make a major change in the design of a system, the sort of change that was going to take a year or more, and many tens of person-years of effort. They asked me how to justify the project. This post is part of the email reply I sent. The advice is in context of technical leadership work at a big company, but perhaps it may apply elsewhere.

Is it the right solution?

I like to pay attention to ways I can easily fool myself. One of those ways is an availability heuristic applied to big problems. I see a big problem that needs a big solution, and am strongly biased to believe that the first big solution that presents itself is the right one. It takes intentional effort to figure out whether the big solution is, indeed, a solution to the big problem. Bold action, after all, isn't a solution itself.

Sometimes, in one of his more exuberant or desperate moods, Pa would go out in the veld and sprinkle brandy on the daisies to make them drunk so that they wouldn't feel the pain of shriveling up and dying. (André Brink)

Because I am so easily fooled in this way, I like to write my reasoning down. Two pages of prose normally does it, building an argument as to why this is the right solution to the problem. Almost every time, this exposes flaws in my reasoning, opportunities to find more data, or other solutions to explore. Thinking in my head doesn't have this effect for me, but writing does. Or, rather, the exercise of writing and reading does.

The first step is to write a succinct description of the problem, and what it means for the problem to be solved. Sometimes those are quantitative goals. Speeds and feeds. Sometimes, they are concrete goals. A product launch, or a document. Sometimes, it's something more qualitative and harder to explain. Thinking about the problem bears a great deal of fruit.

Then, the solution. The usual questions apply here, including cost, viability, scope and complexity. Most important is engaging with the problem statement. It's easy to make the exercise useless if you disconnect the problem statement from the solution.

It is important you feel comfortable with the outcome of this exercise, because losing faith in your own work is a sure way to have it fail. Confidence is one valuable outcome. Another one is a simpler solution.

Is it the right problem?

An elegant solution to the wrong problem is worse than no solution at all, at least in that it might fool people into thinking that the true problem has been solved, and to stop trying. You need to deeply understand the problem you are solving. Rarely, this will be an exercise only in technology or engineering. More commonly, large problems will span business, finance, engineering, management and more. You probably don't understand all of these things. Be sure to seek the help of people who do.

“Would you tell me, please, which way I ought to go from here?” “That depends a good deal on where you want to get to,” said the Cat.

Once I think I understand multiple perspectives on a problem, I like to write them down and run them by the people who explained the problem to me. They'll be able to point out where you're still wrong. Perhaps you're confusing your net and operational margins, or your hiring targets make no sense, or your customers see a different problem from you. This requires that the people you consult trust you, and you trust them. Fortunately, non-engineers in engineering organizations are always looking out for allies and friends. Most are, like engineers, only too excited to explain their work.

Engage with the doubters, but don't let them get you down

Be prepared! And be careful not to do Your good deeds when there's no one watching you (Tom Lehrer)

You will never convince everybody of your point of view. By now, you have two powerful tools to help convince people: A clear statement of the problem that considers multiple points-of-view, and a clear statement of the solution. Some people will read those and be convinced. Others would never be convinced, because their objections lie beyond the scope of your thinking. A third group will have real, honest, feedback about the problem and proposed solution. That feedback is gold that should be mined. Unfortunately, separating the golden nuggets of feedback from the pyrite nuggets of doubt isn't easy.

The doubters will get you down. Perhaps they think the problem doesn't exist, or that the solution is impractical. Perhaps they think you aren't the person to do it. Perhaps they think the same resources should be spent on a different problem, or a different solution. You'll repeat, repeat, and repeat. Get used to it. I'm still not used to it, but you should be.

Again, writing is a tool I reach for. "Today, I'm doubting my solution because..." Sometimes that doubt will be something more about you than the project. That's OK. Sometime it'll be about the project, and will identify a real problem. Often, it'll just point to one of those unknown unknowns that all projects have.

Meet the stakeholders where they are

Most likely, you're going to need to convince somebody to let you do the work. That's good, because doing big things requires time, people and money. You don't want to be working somewhere that's willing to waste time, people or money. If they're willing to waste time on your ill-conceived schemes, they'll be willing to waste your time on somebody else's ill-conceived schemes.

May your wisdom grace us until the stars rain down from the heavens. (C.S. Lewis)

The best advice I've received about convincing stakeholders is to write for them, not you. Try to predict what questions they are going to ask, what concerns they will have, and what objections they will bring up and have answers for those in the text. That doesn't mean you should be defensive. Don't aim only to flatter. Instead, tailor your approach. It can help to have the advice of people who've been through this journey before.

The previous paragraph may seem to you like politics, you may have a distaste for politics, or believe you can escape it by moving to a different business. It is. You may. You can't.

Leadership willing to engage with your ideas and challenge you on them is a blessing.

Build a team

You need to build two teams. Your local team is the team of engineers who are going to help you write, review, test, deploy, operate, and so on. These people are critical to your success, not because they are the fingers to your brain, but because the details are at least as important as the big picture. Get into some details yourself. Don't get into every detail. You can't.

Your extended team is a group of experts, managers, customer-facing folks, product managers, lawyers, designers and so on. Some of these people won't be engaged day-to-day. You need to find them, get them involved, and draw on them when you need help. You're not an expert in everything, but expertise in everything will be needed. Getting these people excited about your project, and bought into its success, is important.

Finally, find yourself a small group of people you trust, and ask them to keep you honest. Check in with them, to make sure your ideas still make sense. Share the documents you wrote with them.

Be willing to adapt

You will learn, at some point into doing your big project, that your solution is bullshit. You completely misunderstood the problem. You may feel like this leaves you back at the beginning, where you started. It doesn't. Instead, you've stepped up your level of expertise. Most likely, you can adapt that carefully-considered solution to the new problem, but you might need to throw it out entirely. Again, write it down. Be specific. What have you learned, and what did it teach you? Look for things you can recover, and don't throw things out prematurely.

Maxwell Bernstein (tekknolagi)

Compiling a Lisp: Instruction encoding interlude October 19, 2020 12:00 AM

firstprevious

Welcome back to the Compiling a Lisp series. In this thrilling new update, we will learn a little bit more about x86-64 instruction encoding instead of allocating more interesting things on the heap or adding procedure calls.

I am writing this interlude because I changed one register in my compiler code (kRbp to kRsp) and all hell broke loose — the resulting program was crashing, rasm2/Cutter were decoding wacky instructions when fed my binary, etc. Over the span of two very interesting but very frustrating hours, I learned why I had these problems and how to resolve them. You should learn, too.

State of the instruction encoder

Recall that I introduced at least 10 functions that looked vaguely like this:

void Emit_mov_reg_imm32(Buffer *buf, Register dst, int32_t src) {
  Buffer_write8(buf, kRexPrefix);
  Buffer_write8(buf, 0xc7);
  Buffer_write8(buf, 0xc0 + dst);
  Buffer_write32(buf, src);
}

These functions all purport to encode x86-64 instructions. They do, most of the time, but they do not tell the whole story. This function is supposed to encode an instruction of the form mov reg64, imm32. How does it do it? I don’t know!

They have all these magic numbers in them! What is a kRexPrefix? Well, it’s 0x48. Does that mean anything to us? No! It gets worse. What are 0xc7 and 0xc0 doing there? Why are we adding dst to 0xc0? Before this debugging and reading extravaganza, I could not have told you. Remember how somewhere in a previous post I mentioned I was getting these hex bytes from reading the compiled output on the Compiler Explorer? Yeah.

As it turns out, this is not a robust development strategy, at least with x86-64. It might be okay for some more regular or predictable instruction sets, but not this one.

Big scary documentation

So where do we go from here? How do we find out how to take these mystical hexes and incantations to something that better maps to the hardware? Well, we once again drag Tom1 into a debugging session and pull out the big ol’ Intel Software Developer Manual.

This is an enormous 26MB, 5000 page manual comprised of four volumes. It’s very intimidating. This is exactly why I didn’t want to pull it out earlier and do this properly from the beginning… but here we are, eventually needing to do it properly.

I will not pretend to understand all of this manual, nor will this post be a guide to the manual. I will just explain what sections and diagrams I found useful in understanding how this stuff works.

I only ever opened Volume 2, the instruction set reference. In that honking 2300 page volume are descriptions of every Intel x86-64 instruction and how they are encoded. The instructions are listed alphabetically and split into sections based on the first letter of each instruction name.

Let’s take a look at Chapter 3, specifically at the MOV instruction on page   1209. For those following along who do not want to download a massive PDF, this website has a bunch of the same data in HTML form. Here’s the page for MOV.

This page has every variant of MOV instruction. There are other instructions begin with MOV, like MOVAPD, MOVAPS, etc, but they are different enough that they are different instructions.

It has six columns:

  • Opcode, which describes the layout of the bytes in the instruction stream. This describes how we’ll encode instructions.
  • Instruction, which gives a text-assembly-esque representation of the instruction. This is useful for figuring out which one we actually want to encode.
  • Op/En, which stands for “Operand Encoding” and as far as I can tell describes the operand order with a symbol that is explained further in the “Instruction Operand Encoding” table on the following page.
  • 64-Bit Mode, which tells you if the instruction can be used in 64-bit mode (“Valid”) or not (something else, I guess).
  • Compat/Leg Mode, which tells you if the instruction can be used in some other mode, which I imagine is 32-bit mode or 16-bit mode. I don’t know. But it’s not relevant for us.
  • Description, which provides a “plain English” description of the opcode, for some definition of the words “plain” and “English”.

Other instructions have slightly different table layouts, so you’ll have to work out what the other columns mean.

Here’s a preview of some rows from the table, with HTML courtesy of Felix Cloutier’s aforementioned web docs:

Opcode Instruction Op/En 64-Bit Mode Compat/Leg Mode Description
88 /r MOV r/m8,r8 MR Valid Valid Move r8 to r/m8.
REX + 88 /r MOV r/m8***,r8*** MR Valid N.E. Move r8 to r/m8.
89 /r MOV r/m16,r16 MR Valid Valid Move r16 to r/m16.
89 /r MOV r/m32,r32 MR Valid Valid Move r32 to r/m32.
..................
C7 /0 id MOV r/m32, imm32 MI Valid Valid Move imm32 to r/m32.
REX.W + C7 /0 id MOV r/m64, imm32 MI Valid N.E. Move imm32 sign extended to 64-bits to r/m64.

If you take a look at the last entry in the table, you’ll see REX.W + C7 /0 id. Does that look familiar? Maybe, if you squint a little?

It turns out, that’s the description for encoding the instruction we originally wanted, and had a bad encoder for. Let’s try and figure out how to use this to make our encoder better. In order to do that, we’ll need to first understand a general layout for Intel instructions.

Instruction encoding, big picture

All Intel x86-64 instructions follow this general format:

  • optional instruction prefix
  • opcode (1, 2, or 3 bytes)
  • if required, Mod-Reg/Opcode-R/M, also known as ModR/M (1 byte)
  • if required, Scale-Index-Base, also known as SIB (1 byte)
  • displacement (1, 2, or 4 bytes, or none)
  • immediate data (1, 2, or 4 bytes, or none)

I found this information at the very beginning of Volume 2, Chapter 2 (page 527) in a section called “Instruction format for protected mode, real-address mode, and virtual-8086 mode”.

You, like me, may be wondering about the difference between “optional”, “if required”, and “…, or none”. I have no explanation, sorry.

I’m going to briefly explain each component here, followed up with a piece-by-piece dissection of the particular MOV instruction we want, so we get some hands-on practice.

Instruction prefixes

There are a couple kind of instruction prefixes, like REX (Section 2.2.1) and VEX (Section 2.3). We’re going to focus on REX prefixes, since they are needed for many (most?) x86-64 instructions, and we’re not emitting vector instructions.

The REX prefixes are used to indicate that an instruction, which might normally refer to a 32-bit register, should instead refer to a 64-bit register. Also some other things but we’re mostly concerned with register sizes.

Opcode

Take a look at Section 2.1.2 (page 529) for a brief explanation of opcodes. The gist is that the opcode is the meat of the instruction. It’s what makes a MOV a MOV and not a HALT. The other fields all modify the meaning given by this field.

ModR/M and SIB

Take a look at Section 2.1.3 (page 529) for a brief explanation of ModR/M and SIB bytes. The gist is that they encode what register sources and destinations to use.

Displacement and immediates

Take a look at Section 2.1.4 (page 529) for a brief explanation of displacement and immediate bytes. The gist is that they encode literal numbers used in the instructions that don’t encode registers or anything.

If you’re confused, that’s okay. It should maybe get clearer once we get our hands dirty. Reading all of this information in a vacuum is moderately useless if it’s your first time dealing with assembly like this, but I included this section first to help explain how to use the reference.

Encoding, piece by piece

Got all that? Maybe? No? Yeah, me neither. But let’s forge ahead anyway. Here’s the instruction we’re going to encode: REX.W + C7 /0 id.

REX.W

First, let’s figure out REX.W. According to Section 2.2.1, which explains REX prefixes in some detail, there are a couple of different prefixes. There’s a helpful table (Table 2-4, page 535) documenting them. Here’s a bit diagram with the same information:

@font-face { font-family: "Virgil"; src: url("https://excalidraw.com/FG_Virgil.woff2"); } @font-face { font-family: "Cascadia"; src: url("https://excalidraw.com/Cascadia.woff2"); } 0100WRXBHigh bitLow bitREX

In English, and zero-indexed:

  • Bits 7-4 are always 0b0100.
  • Bit 3 is the W prefix. If it’s 1, it means the operands are 64 bits. If it’s 0, “operand size [is] determined by CS.D”. Not sure what that means.
  • Bits 2, 1, and 0 are other types of REX prefixes that we may not end up using, so I am omitting them here. Please read further in the manual if you are curious!

This MOV instruction calls for REX.W, which means this byte will look like 0b01001000, also known as our friend 0x48. Mystery number one, solved!

C7

This is a hexadecimal literal 0xc7. It is the opcode. There are a couple of other entries with the opcode C7, modified by other bytes in the instruction (ModR/M, SIB, REX, …). Write it to the instruction stream. Mystery number two, solved!

/0

There’s a snippet in Section 2.1.5 that explains this notation:

If the instruction does not require a second operand, then the Reg/Opcode field may be used as an opcode extension. This use is represented by the sixth row in the tables (labeled “/digit (Opcode)”). Note that values in row six are represented in decimal form.

This is a little confusing because this operation clearly does have a second operand, denoted by the “MI” in the table, which shows Operand 1 being ModRM:r/m (w) and Operand 2 being imm8/16/32/64. I think it’s because it doesn’t have a second register operand that this space is free — the immediate is in a different place in the instruction.

In any case, this means that we have to make sure to put decimal 0 in the reg part of the ModR/M byte. We’ll see what the ModR/M byte looks like in greater detail shortly.

id

id refers to an immediate double word (32 bits). It’s called a double word because, a word (iw) is 16 bits. In increasing order of size, we have:

  • ib, byte (1 byte)
  • iw, word (2 bytes)
  • id, double word (4 bytes)
  • io, quad word (8 bytes)

This means we have to write our 32-bit value out to the instruction stream. These notations and encodings are explained further in Section 3.1.1.1 (page 596).

Overall, that means that this instruction will have the following form:

@font-face { font-family: "Virgil"; src: url("https://excalidraw.com/FG_Virgil.woff2"); } @font-face { font-family: "Cascadia"; src: url("https://excalidraw.com/Cascadia.woff2"); } REXOpModR/MImmediate01237

If we were to try and encode the particular instruction mov rax, 100, it would look like this:

@font-face { font-family: "Virgil"; src: url("https://excalidraw.com/FG_Virgil.woff2"); } @font-face { font-family: "Cascadia"; src: url("https://excalidraw.com/Cascadia.woff2"); } REXOpModR/MImmediate012370x480xc70xc00x64 0x00 0x00 0x00

This is how you read the table! Slowly, piece by piece, and with a nice cup of tea to help you in trying times. Now that we’ve read the table, let’s go on and write some code.

Encoding, programatically

While writing code, you will often need to reference two more tables than the ones we have looked at so far. These tables are Table 2-2 “32-Bit Addressing Forms with the ModR/M Byte” (page 532) and Table 2-3 “32-Bit Addressing Forms with the SIB Byte” (page 533). Although the tables describe 32-bit quantities, with the REX prefix all the Es get replaced with Rs and all of a sudden they can describe 64-bit quantities.

These tables are super helpful when figuring out how to put together ModR/M and SIB bytes.

Let’s start the encoding process by revisiting Emit_mov_reg_imm32/REX.W + C7 /0 id:

void Emit_mov_reg_imm32(Buffer *buf, Register dst, int32_t src) {
  // ...
}

Given a register dst and an immediate 32-bit integer src, we’re going to encode this instruction. Let’s do all the steps in order.

REX prefix

Since the instruction calls for REX.W, we can keep the first line the same as before:

void Emit_mov_reg_imm32(Buffer *buf, Register dst, int32_t src) {
  Buffer_write8(buf, kRexPrefix);
  // ...
}

Nice.

Opcode

This opcode is 0xc7, so we’ll write that directly:

void Emit_mov_reg_imm32(Buffer *buf, Register dst, int32_t src) {
  Buffer_write8(buf, kRexPrefix);
  Buffer_write8(buf, 0xc7);
  // ...
}

Also the same as before. Nice.

ModR/M byte

ModR/M bytes are where the code gets a little different. We want an abstraction to build them for us, instead of manually slinging integers like some kind of animal.

To do that, we should know how they are put together. ModR/M bytes are comprised of:

  • mod (high 2 bits), which describes what big row to use in the ModR/M table
  • reg (middle 3 bits), which either describes the second register operand or an opcode extension (like /0 above)
  • rm (low 3 bits), which describes the first operand

This means we can write a function modrm that puts these values together for us:

byte modrm(byte mod, byte rm, byte reg) {
  return ((mod & 0x3) << 6) | ((reg & 0x7) << 3) | (rm & 0x7);
}

The order of the parameters is a little different than the order of the bits. I did this because it looks a little more natural when calling the function from its callers. Maybe I’ll change it later because it’s too confusing.

For this instruction, we’re going to:

  • pass 0b11 (3) as mod, because we want to move directly into a 64-bit register, as opposed to [reg], which means that we want to dereference the value in the pointer
  • pass the destination register dst as rm, since it’s the first operand
  • pass 0b000 (0) as reg, since the /0 above told us to

That ends up looking like this:

void Emit_mov_reg_imm32(Buffer *buf, Register dst, int32_t src) {
  Buffer_write8(buf, kRexPrefix);
  Buffer_write8(buf, 0xc7);
  Buffer_write8(buf, modrm(/*direct*/ 3, dst, 0));
  // ...
}

Which for the above instruction mov rax, 100, produces a modrm byte that has this layout:

@font-face { font-family: "Virgil"; src: url("https://excalidraw.com/FG_Virgil.woff2"); } @font-face { font-family: "Cascadia"; src: url("https://excalidraw.com/Cascadia.woff2"); } ModR/Mmodregrm11000direct/0RAX000

I haven’t put a datatype for mods together because I don’t know if I’d be able to express it well. So for now I just added a comment.

Immediate value

Last, we have the immediate value. As I said above, all this entails is writing out a 32-bit quantity as we have always done:

void Emit_mov_reg_imm32(Buffer *buf, Register dst, int32_t src) {
  Buffer_write8(buf, kRexPrefix);
  Buffer_write8(buf, 0xc7);
  Buffer_write8(buf, modrm(/*direct*/ 3, dst, 0));
  Buffer_write32(buf, src);
}

And there you have it! It took us 2500 words to get us to these measly four bytes. The real success is the friends we made along the way.

Further instructions

“But Max,” you say, “this produces literally the same output as before with all cases! Why go to all this trouble? What gives?”

Well, dear reader, having a mod of 3 (direct) means that there is no special-case escape hatch when dst is RSP. This is unlike the other mods, where there’s this [--][--] in the table where RSP should be. That funky symbol indicates that there must be a Scale-Index-Base (SIB) byte following the ModR/M byte. This means that the overall format for this instruction should have the following layout:

@font-face { font-family: "Virgil"; src: url("https://excalidraw.com/FG_Virgil.woff2"); } @font-face { font-family: "Cascadia"; src: url("https://excalidraw.com/Cascadia.woff2"); } REXOpModR/M0123SIB4Disp5

If you’re trying to encode mov [rsp-8], rax, for example, the values should look like this:

@font-face { font-family: "Virgil"; src: url("https://excalidraw.com/FG_Virgil.woff2"); } @font-face { font-family: "Cascadia"; src: url("https://excalidraw.com/Cascadia.woff2"); } REXOpModR/M01230x480x890x44SIB4Disp50x240xf8

This is where an instruction like Emit_store_reg_indirect (mov [REG+disp], src) goes horribly awry with the homebrew encoding scheme I cooked up. When the dst in that instruction is RSP, it’s expected that the next byte is the SIB. And when you output other data instead (say, an immediate 8-bit displacement), you get really funky addressing modes. Like what the heck is this?

mov qword [rsp + rax*2 - 8], rax

This is actual disassembled assembly that I got from running my binary code through rasm2. Our compiler definitely does not emit anything that complicated, which is how I found out things were wrong.

Okay, so it’s wrong. We can’t just blindly multiply and add things. So what do we do?

The SIB byte

Take a look at Table 2-2 (page 532) again. See that trying to use RSP with any sort of displacement requires the SIB.

Now take a look at Table 2-3 (page 533) again. We’ll use this to put together the SIB.

We know from Section 2.1.3 that the SIB, like the ModR/M, is comprised of three fields:

  • scale (high 2 bits), specifies the scale factor
  • index (middle 3 bits), specifies the register number of the index register
  • base (low 3 bits), specifies the register number of the base register

Intel’s language is not so clear and is kind of circular. Let’s take a look at sample instruction to clear things up:

mov [base + index*scale + disp], src

Note that while index and base refer to registers, scale refers to one of 1, 2, 4, or 8, and disp is some immediate value.

This is a compact way of specifying a memory offset. It’s convenient for reading from and writing to arrays and structs. It’s also going to be necessary for us if we want to write to and read from random offsets from the stack pointer, RSP.

So let’s try and encode that Emit_store_reg_indirect.

Encoding the indirect mov

Let’s start by going back to the table enumerating all the kinds of MOV instructions (page 1209). The specific opcode we’re looking for is REX.W + 89 /r, or MOV r/m64, r64.

We already know what REX.W means:

void Emit_store_reg_indirect(Buffer *buf, Indirect dst, Register src) {
  Buffer_write8(buf, kRexPrefix);
  // ...
}

And next up is the literal 0x89, so we can write that straight out:

void Emit_store_reg_indirect(Buffer *buf, Indirect dst, Register src) {
  Buffer_write8(buf, kRexPrefix);
  Buffer_write8(buf, 0x89);
  // ...
}

So far, so good. Looking familiar. Now that we have both the instruction prefix and the opcode, it’s time to write the ModR/M byte. Our ModR/M will contain the following information:

  • mod of 1, since we want an 8-bit displacement
  • reg of whatever register the second operand is, since we have two register operands (the opcode field says /r)
  • rm of whatever register the first operand is

Alright, let’s put that together with our handy-dandy ModR/M function.

void Emit_store_reg_indirect(Buffer *buf, Indirect dst, Register src) {
  Buffer_write8(buf, kRexPrefix);
  Buffer_write8(buf, 0x89);
  // Wrong!
  Buffer_write8(buf, modrm(/*disp8*/ 1, dst.reg, src));
  // ...
}

But no, this is wrong. As it turns out, you still have do this special thing when dst.reg is RSP, as I keep mentioning. In that case, rm must be the special none value (as specified by the table). Then you also have to write a SIB byte.

void Emit_store_reg_indirect(Buffer *buf, Indirect dst, Register src) {
  Buffer_write8(buf, kRexPrefix);
  Buffer_write8(buf, 0x89);
  if (dst.reg == kRsp) {
    Buffer_write8(buf, modrm(/*disp8*/ 1, kIndexNone, src));
    // ...
  } else {
    Buffer_write8(buf, modrm(/*disp8*/ 1, dst.reg, src));
  }
  // ...
}

Astute readers will know that kRsp and kIndexNone have the same integral value of 4. I don’t know if this was intentional on the part of the Intel designers. Maybe it’s supposed to be like that so encoding is easier and doesn’t require a special case for both ModR/M and SIB. Maybe it’s coincidental. Either way, I found it very subtle and wanted to call it out explicitly.

For an instruction like mov [rsp-8], rax, our modrm byte will look like this:

@font-face { font-family: "Virgil"; src: url("https://excalidraw.com/FG_Virgil.woff2"); } @font-face { font-family: "Cascadia"; src: url("https://excalidraw.com/Cascadia.woff2"); } ModR/Mmodregrm11100disp8RAXnone000

Let’s go ahead and write that SIB byte. I made a sib helper function like modrm, with two small differences: the parameters are in order of low to high bit, and the parameters have their own special types instead of just being bytes.

typedef enum {
  Scale1 = 0,
  Scale2,
  Scale4,
  Scale8,
} Scale;

typedef enum {
  kIndexRax = 0,
  kIndexRcx,
  kIndexRdx,
  kIndexRbx,
  kIndexNone,
  kIndexRbp,
  kIndexRsi,
  kIndexRdi
} Index;

byte sib(Register base, Index index, Scale scale) {
  return ((scale & 0x3) << 6) | ((index & 0x7) << 3) | (base & 0x7);
}

I made all these datatypes to help readability, but you don’t have to use them if you don’t want to. The Index one is the only one that has a small gotcha: where kIndexRsp should be is kIndexNone because you can’t use RSP as an index register.

Let’s use this function to write a SIB byte in Emit_store_reg_indirect:

void Emit_store_reg_indirect(Buffer *buf, Indirect dst, Register src) {
  Buffer_write8(buf, kRexPrefix);
  Buffer_write8(buf, 0x89);
  if (dst.reg == kRsp) {
    Buffer_write8(buf, modrm(/*disp8*/ 1, kIndexNone, src));
    Buffer_write8(buf, sib(kRsp, kIndexNone, Scale1));
  } else {
    Buffer_write8(buf, modrm(/*disp8*/ 1, dst.reg, src));
  }
  // ...
}

If you get it right, the SIB byte will have the following layout:

@font-face { font-family: "Virgil"; src: url("https://excalidraw.com/FG_Virgil.woff2"); } @font-face { font-family: "Cascadia"; src: url("https://excalidraw.com/Cascadia.woff2"); } SIBscaleindexbase001000noneRSP100

This is a very verbose way of saying [rsp+DISP], but it’ll do. All that’s left now is to encode that displacement. To do that, we’ll just write it out:

void Emit_store_reg_indirect(Buffer *buf, Indirect dst, Register src) {
  Buffer_write8(buf, kRexPrefix);
  Buffer_write8(buf, 0x89);
  if (dst.reg == kRsp) {
    Buffer_write8(buf, modrm(/*disp8*/ 1, kIndexNone, src));
    Buffer_write8(buf, sib(kRsp, kIndexNone, Scale1));
  } else {
    Buffer_write8(buf, modrm(/*disp8*/ 1, dst.reg, src));
  }
  Buffer_write8(buf, disp8(indirect.disp));
}

Very nice. Now it’s your turn to go forth and convert the rest of the assembly functions in your compiler! I found it very helpful to extract the modrm/sib/disp8 calls into a helper function, because they’re mostly the same and very repetitive.

What did we learn?

This was a very long post. The longest post in the whole series so far, even. We should probably have some concrete takeaways.

If you read this post through, you should have gleaned some facts and lessons about:

  • Intel x86-64 instruction encoding terminology and details, and
  • how to read dense tables in the Intel Developers Manual
  • maybe some third thing, too, I dunno — this post was kind of a lot

Hopefully you enjoyed it. I’m going to go try and get a good night’s sleep. Until next time, when we’ll implement procedure calls!

Here’s a fun composite diagram for the road:

This is a composite of all the instruction encoding diagrams present in the post. If you're seeing this text, it means your browser cannot render SVG.

Mini Table of Contents



  1. If you are an avid reader of this blog (Do those people exist? Please reach out to me. I would love to chat.), you may notice that Tom gets pulled into shenanigans a lot. This is because Tom is the best debugger I have ever encountered, he’s good at reverse engineering, and he knows a lot about low-level things. I think right now he’s working on improving open-source tooling for a RISC-V board for fun. But also he’s very kind and helpful and generally interested in whatever ridiculous situation I’ve gotten myself into. Maybe I should add a list of the Tom Chronicles somewhere on this website. Anyway, everyone needs a Tom. 

October 18, 2020

Derek Jones (derek-jones)

Learning useful stuff from the Human cognition chapter of my book October 18, 2020 09:37 PM

What useful, practical things might professional software developers learn from the Human cognition chapter in my evidence-based software engineering book (an updated beta was release this week)?

Last week I checked the human cognition chapter; what useful things did I learn (combined with everything I learned during all the other weeks spent working on this chapter)?

I had spent a lot of time of learning about cognition when writing my C book; for this chapter I was catching up on what had happened in the last 10 years, which included: building executable models has become more popular, sample size has gotten larger (mostly thanks to Mechanical Turk), more researchers are making their data available on the web, and a few new theories (but mostly refinements of existing ideas).

Software is created by people, and it always seemed obvious to me that human cognition was a major topic in software engineering. But most researchers in computing departments joined the field because of their interest in maths, computers or software. The lack of interested in the human element means that the topic is rarely a research topic. There is a psychology of programming interest group, but most of those involved don’t appear to have read any psychology text books (I went to a couple of their annual workshops, and while writing the C book I was active on their mailing list for a few years).

What might readers learn from the chapter?

Visual processing: the rationale given for many code layout recommendations is plain daft; people need to learn something about how the brain processes images.

Models of reading. Existing readability claims are a joke (or bad marketing, take your pick). Researchers have been using eye trackers, since the 1960s, to figure out what actually happens when people read text, and various models have been built. Market researchers have been using eye trackers for decades to work out where best to place products on shelves, to maximise sales. In the last 10 years software researchers have started using eye trackers to study how people read code; next they need to learn about some of the existing models of how people read text. This chapter contains some handy discussion and references.

Learning and forgetting: it takes time to become proficient; going on a course is the start of the learning process, not the end.

One practical take away for readers of this chapter is being able to give good reasons how other people’s proposals, that are claimed to be based on how the brain operates, won’t work as claimed because that is not how the brain works. Actually, most of the time it is not possible to figure out whether something will work as advertised (this is why user interface testing is such a prolonged, and expensive, process), but the speaker with the most convincing techno-babble often wins the argument :-)

Readers might have a completely different learning experience from reading the human cognition chapter. What useful things did you learn from the human cognition chapter?

Bogdan Popa (bogdan)

Racket Web Development with Koyo October 18, 2020 06:00 PM

Inspired by Brian Adkins' RacketCon talk from yesterday, I decided to record a screencast on what it’s like to write a little web application using my not-quite-a-web-framework, koyo. You can watch it over on YouTube and you can find the resulting code on GitHub. It’s unscripted and I don’t go too deep on how everything works, but hopefully it’s easy enough to follow and I’ve left the various mistakes I’ve made in since it’s usually helpful to watch someone get out of a tricky situation so look forward to those if you watch it!

Carlos Fenollosa (carlesfe)

You may be using Mastodon wrong October 18, 2020 05:13 PM

I'm sure you have already heard about Mastodon, typically marketed as a Twitter alternative.

I will try to convince you that the word alternative doesn't mean here what you think it means, and why you may be using Mastodon wrong if you find it boring.

An alternative community

You should not expect to "migrate from Twitter to Mastodon."

Forget about the privacy angle for now. Mastodon is an alternative community, where people behave differently.

It's your chance to make new internet friends.

There may be some people for whom Mastodon is a safe haven. Yes, some users really do migrate there to avoid censorship or bullying but, for most of us, that will not be the case.

Let's put it this way: Mastodon is to Twitter what Linux is to Windows.

Linux is libre software. But that's not why most people use it. Linux users mostly want to get their work done, and Linux is an excellent platform. There is no Microsoft Word, no Adobe Photoshop, no Starcraft. If you need to use these tools, honestly, you'd better stick with Windows. You can use emulation, in the same way that there are utilities to post to Twitter from Mastodon, but that would miss the point.

The bottom line is, you can perform the same tasks, but the process will be different. You can post toots on Mastodon, upload gifs, send DMs... but it's not Twitter, and that is fine.

The Local Timeline is Mastodon's greatest invention

The problem most people have with Mastodon is that they "get bored" with it quickly. I've seen it a lot, and it means one thing: the person created their account on the wrong server.

"But," they say, "isn't Mastodon federated? Can't I chat with everybody, regardless of their server?" Yes, of course. But discoverability works differently on Mastodon.

Twitter has only two discoverability layers: your network and the whole world. Either a small group of contacts, or everybody in the whole world. That's crazy.

They try very hard to show you tweets from outside your network so you can discover new people. And, at the same time, they show your tweets to third parties, so you can get new followers. This is the way that they try to keep you engaged once your network is more or less stable and starts getting stale.

Mastodon, instead, has an extra layer between your network and the whole world: messages from people on your server. This is called the local timeline.

The local timeline is the key to enjoying Mastodon.

How long it's been since you made a new internet friend?

If you're of a certain age you may remember BBSs, Usenet, the IRC, or early internet forums. Do you recall how exciting it was to log into the unknown and realize that there were people all around the world who shared your interests?

It was an amazing feeling which got lost on the modern internet. Now you have a chance to relive it.

The local timeline dynamics are very different. There is a lot of respectful interactions among total strangers, because there is this feeling of community, of being in a neighborhood. Twitter is just the opposite, strangers shouting at each other.

Furthermore, since the local timeline is more or less limited in the amount of users, you have the chance to recognize usernames, and being recognized. You start interacting with strangers, mentioning them, sending them links they may like. You discover new websites, rabbit holes, new approaches to your hobbies.

I've made quite a few new internet friends on my Mastodon server, and I don't mean followers or contacts. I'm talking about human beings who I have never met in person but feel close to.

People are humble and respectful. And, for less nice users, admins enforce codes of conduct and, on extreme cases, users may get kicked off a server. But they are not being banned by a faceless corporation due to mass reports, everybody is given a chance.

How to choose the right server

The problem with "generalist" Mastodon servers like mastodon.social is that users have just too diverse interests and backgrounds. Therefore, there is no community feeling. For some people, that may be exactly what they're looking for. But, for most of us, there is more value on the smaller servers.

So, how can you choose the right server? Fortunately, you can do a bit of research. There is an official directory of Mastodon servers categorized by interests and regions.

Since you're reading my blog, start by taking a look at these:

And the regionals

There are many more. Simply search online for "mastodon server MY_FAVORITE_HOBBY." And believe me, servers between 500 and 5,000 people are the best.

Final tips

Before clicking on "sign up", always browse the local timeline, the about page, and the most active users list. You will get a pretty good idea of the kind of people who chat there. Once you feel right at home you can continue your adventure and start following users from other servers.

Mastodon has an option to only display toots in specific languages. It can be very useful to avoid being flooded by toots that you just have no chance of understanding or even getting what they're about.

You can also filter your notifications by types: replies, mentions, favorites, reposts, and more. This makes catching up much more manageable than on Twitter.

Finally, Mastodon has a built-in "Content Warning" feature. It allows you to hide text behind a short explanation, in case you want to talk about sensible topics or just about spoiling a recent movie.

Good luck with your search, and see you on the Fediverse! I'm at @cfenollosa@mastodon.sdf.org

Tags: internet

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

Sevan Janiyan (sevan)

LFS, round #1 October 18, 2020 02:07 AM

Following on from the previous blog post, I started on the path of build a Linux From Scratch distribution. The project offers two paths, one using traditional Sys V init and systemd for the other. I opted for systemd route and followed the guide, it was all very straight forward. Essentially you fetch a bunch …

October 16, 2020

Patrick Louis (venam)

October 2020 Projects October 16, 2020 09:00 PM

Conveyor belt

Seven long and perilous months have gone by since my previous article, what feels like an eternity, and yet feels like a day — Nothing and everything has happened.
All I can add to the situation in my country, that I’ve already drawn countless times, is that my expectations weren’t fulfilled. Indeed, after a governmental void and a horrific explosion engulfing a tremendous part of the capital, I’m not sure any words can express the conflicting feelings and anger I have. Today marks exactly 1 year since the people started revolting.
Sentimentalities aside, let’s get to what I’ve been up to.

Psychology, Philosophy & Books

zadig cover

Language: brains
Explanation: My reading routine has been focused one part on heavy technical books, and one part on leisure books.
On the leisure side, I’ve finished the following books:

  • The Better Angels of Our Nature: Why violence declined — Stephen Pinker
  • The Book of M — Peng Shepherd
  • The Gene — Siddhartha mukherjee
  • The second sex — Simone de Beauvoir
  • Aphorism on Love and Hate — Nietzsche
  • How To Use Your Enemies — Baltasar Gracian
  • Zadig ou La Destinée — Voltaire
  • Great Expectations — Charles Dickens [ongoing]

While on the technical side, I’ve finished these bricks:

  • Computer Architecture: A Quantitative approach - Hennesy Patterson
  • Beyond Software Architecture - Creating and sustaining winning solutions — Luke Hohmann
  • Compilers/dragon book - Aho Lam Seth Ullman
  • Operating system concepts - Silberschatz [ongoing]

Obviously, I still want to build a bookshelf, however the current situation has postponed this project.

As far as podcasts go, I’ve toned down on them and only listen when exercising; which doesn’t amount to much compared to when I was commuting to work.

Life Refresh & Growth

sunrise

Language: growth
Explanation: As you might have already noticed, I’ve redesigned my blog. I tried to give it more personality and to be more reflective of who I am as a person.
That involved reviewing the typography, adding meta-tags and previews, adding relevant pictures for every articles, including general and particular descriptions for sections of the blog, and more.

Additionally, I’ve polished my online presence on StackOverflow and LinkedIn. It is especially important these days, when in need of new opportunities.

LinkedIn Profile

As far as software architecture goes, I’m still on the learning path which consists of reading articles, watching videos, and trying to apply the topics to real scenarios. Recently, I’ve started following Mark Richards’ and Neil Ford’s Foundations Friday Forum, which is a monthly webinar on software architecture.

When it comes to articles, that’s where I’ve put the most energy. Here’s the list of new ones.

  • The Self, Metaperceptions, and Self-Transformation: One of my favorite article about the self and growth. It has been influenced by theories from Carl Jung and Nietzsche.
  • Software Distributions And Their Roles Today: This is an article I had in mind for a long while but didn’t get to write. It was initially supposed to be a group discussion as a podcast but I ended up writing it as an article, and then recording a podcast too.
  • Time on Unix: My biggest and most complete article to date. I consider it an achievement, and it has been well received by readers. It’s now the goto article when it comes to time.
  • Domain Driven Design Presentation: The transcript of a talk I’ve given for the MENA-Devs community.
  • Evolutionary Software Architecture: An article where I apply my knowledge of software architecture to explain a trending topic.
  • D-Bus and Polkit, No More Mysticism and Confusion: There’s a lot of confusion and hate about dbus and everything around it. I personally had no idea what these tech implied so I wrote an article and found out for myself if the hate was justified.
  • Computer Architecture Takeaways: An article reviewing my knowledge on computer architecture after reading a book on it.
  • Notes About Compilers: Another article reviewing my knowledge on compilers after finishing the dragon book and other related content.
  • Did You Know Fonts Could Do All This?: Fonts is a topic that is very deep and complex, you can talk endlessly about it. In this particular article, I had a go at different settings and how they affect the rendering of fonts.
  • Corruption Is Attractive!: I’m fascinated by glitch art, and so I wrote an article about it, trying to sum up different techniques and give my personal view of what it consists of.

Recently, I’ve also had an Interview with my friend Oday on his YouTube channel. We had an interesting talk.

Now on the programming language side, I’m hoping on the bandwagon and learning Rust. I’m still doing baby steps.

When it comes to personal fun, I’ve stopped my Elevate subscription because of the spending restriction my country has implemented. But instead, I’ve started with a word of the day app.

Mushrooms

mushroooomz

Language: mycelium
Explanation: Finally, I’ve pushed my research about mushrooms in Lebanon online. It’s here and is composed of a map with information about each specimen.
I hope to soon go hike and discover new ones.

Unix, Nixers, 2bwm

nixers workflow compilation

Language: Unix
Explanation: There were a lot of ups and downs in the nixers community the past few months. We had to detach ourselves from the previous people that managed IRC because of their unprofessional and unacceptable behavior. Soon after, I’ve created our own room on freenode, and since then things have gone smoothly. That was until hell happened around me and I decided to close the forums. However, the community made it clear that they wanted to help and keep it alive and well. Thus, I retracted my decision and started implementing mechanisms to make the forums more active such as: A thread of the week, a gopher server, a screenshots display, fixed dmarc for emails, and fix the forums mobile view.
I’ve also uploaded all the previous year’s video compilation on YouTube.

When it comes to 2bwm, we finally added support for separate workpaces per monitor.

CTF

CTF Arab Win

Language: Security
Explanation: A friend of mine recently invited me to be part of his CTF team for the national competition. Lo and behold a couple of months later we win the national competition, get 2nd place in the b01lers CTF, and get 1st place in the Arab & Africa CybertTalents regional CTF.
In the coming months I’ll train on topics I haven’t dealt with before.

Ascii Art & Art

Nature ASCII Art

Language: ASCII
Explanation: I haven’t drawn too many pieces recently, however I’d rather emphasize quality over quantity. You can check my pieces here.

Additionally, I’ve tried to make my RedBubble Shop more attractive, maybe it’ll help in these hard economic times. Otherwise, it’s always nice to have it around.
Other than that, I’ve joined the small tilde.town community, which also has a love for ascii art.

Life and other hobbies

Goat farm

Language: life
Explanation: Life has been a bit harsh recently but I’ve tried to make the most of it.

We started gardening my father and I, we planted everything from sunflowers, cucumbers, tomatoes, rocca, parsley, cilantro, zucchinis, eggplants, bell peppers, hot peppers, ginger, green beans, garlic, onions, and more.
After the tomato harvest, we made our own tomato sauce and pasteurized it — It was heaven.

Recently, I’ve visited a local farm called Gout Blanc, it was a fun experience, but marked in time by the reality of the economic crisis we are in. The owners were wonderful and friendly.

Like anyone in this lock down, I’m going through a bread making phase. I quite enjoy ciabatta bread with halloum:

Homemade Ciabatta Bread

When the initial lock down started, I ordered some joysticks to play retro games with my brother, little did I know that I would only get them 5 months later. My brother left Lebanon to continue his study in France by that time, but I still got a retro-gaming setup.
Like many people close to me, he left for better pastures…

An anecdote, I’ve started to be hassled by Google and YouTube. I simply cannot open a video these days without being asked to fill captchas, so I’ve gotten quite good at finding cars, traffic light, and other trifles in random pictures. More than that, the kind of ads I’ve been getting are of the weirdest kind. Just take a look.

ad 1 ad 2 ad 3 ad 4 ad 5

Now

What’s in store for tomorrow… I’m not sure anymore. There hasn’t been more of a need for change.

This is it!
As usual… If you want something done, no one’s gonna do it for you, use your own hands, even if it’s not much.
And let’s go for a beer together sometime, or just chill.





Attributions:

  • Internet Archive Book Images, No restrictions, via Wikimedia Commons
  • Claud Field, Public domain, via Wikimedia Commons

October 15, 2020

Caius Durling (caius)

Let's Peek: A tale of finding "Waypoint" October 15, 2020 07:00 PM

Following a product launch at work earlier this year, I theorised if someone was watching the published lists of SSL Certificates they could potentially sneak a peak at things before they were publicised. Probably far too much noise to monitor continuously, but as a potential hint towards naming of things with a more targeted search it might be useful. Sites like https://crt.sh/ and https://censys.io/certificates make these logs searchable and queryable.

Fast forward to this week, where at HashiConf Digital HashiCorp are announcing two new products, which they’ve been teasing for a month or so. Watching Boundary get announced in the HashiConf opening keynote I then wondered what the second project might be called.

I’ve spent a chunk of the last month looking at various HashiCorp documentation for their projects, and I noticed they have a pattern recently of using <name>project.io as the product websites. The newly announced Boundary also fits this pattern.

🤔 Could I figure out the second product name 24 hours before public release? Amazingly, yes! 🎉

Searching at random for all certificates issued for *project.io was probably going to be a bit futile, so to narrow the search space slightly I started by looking at when boundaryproject.io had its certificate issued, and who by. The list of things I spotted were:

  • Common name is “boundaryproject.io”
  • Issued by LetsEncrypt (no real surprise there)
  • Issued on 2020-09-23
  • Leaf certificate
  • Not yet expired (still trusted)
  • No alternate names in the certificate

Loading up https://censys.io/certificates and building a query for this, resulted in a regexp lookup against the common name, and an issued at date range of 10 days, just before and a week after the boundary certificate issued date.

parsed.subject.common_name:/[a-z]+project\.io/ AND
parsed.issuer.organization.raw:"Let's Encrypt" AND
parsed.validity.start:["2020-09-20" TO "2020-09-30"] AND
tags.raw:"leaf" AND
tags.raw:"trusted"

(Run the search yourself)

Searching brought back a couple of pages of results, I scanned them by eye and copied out the ones that only had the single name in the certificate which resulted in the following shortlist:

  • boundaryproject.io
  • essenceproject.io
  • lumiereproject.io
  • techproject.io
  • udproject.io
  • vesselproject.io
  • waypointproject.io

We already know about Boundary, so the fact I found it in our list suggests the query might have captured the new product site too. Loading all these sites in a web browser showed some had password protection on them (ooh!) and some just plain didn’t load (ooh!), and some others were blatently other things (boo!). Removing the latter ones left us with a much shorter list:

  • essenceproject.io
  • udproject.io1
  • waypointproject.io

All domains on the internet have to point somewhere, using DNS records. On a hunch I looked up a couple of the existing HashiCorp websites to see if they happened to all point at the same IP Address(es).

$ host boundaryproject.io
boundaryproject.io has address 76.76.21.21
$ host nomadproject.io
nomadproject.io has address 76.76.21.21
$ host hashicorp.com | head -1
hashicorp.com has address 76.76.21.21

Ah ha, now I wonder if any of the shortlist also points to 76.76.21.21 🤔2

$ host essenceproject.io | head -1
essenceproject.io has address 198.185.159.145
$ host udproject.io | head -1
udproject.io has address 137.74.116.3
$ host waypointproject.io
waypointproject.io has address 76.76.21.21

🎉 Excellent, https://waypointproject.io was a password protected site pointed at HashiCorp’s IP address 🎉

I then wondered if I could verify this somehow ahead of waiting for the second keynote. I firstly tweeted about it but didn’t name Waypoint explicitly, just hid “way” and “point” in the tweet. I got a reply from @ksatirli which suggested it was correct (and then later @mitchellh confirmed it.3)

HashiCorp also does a lot in public, and all the source code and related materials are on GitHub so perhaps some of their commit messages or marketing sites will contain reference to Waypoint. One github search later across their organisation: https://github.com/search?q=org%3Ahashicorp+waypoint&type=issues and I’d discovered a commit in the newly-public hashicorp/boundary-ui repo which references Waypoint: 346f76404

chore: tweak colors to match waypoint and for a11y

Good enough for me, now to wait and see what the project is for. Given it’s now all announced and live, you can just visit https://waypointproject.io to find out! (It’s so much cooler/useful than I’d hoped for.)


  1. I so hope whoever registered this was going for UDP in the name, rather than UD Project. ↩︎

  2. I’m a massive fan of IP address related quirks. Facebook’s IPv6 address contains face:b00c for example. A nice repeating 76.76.21.21 is almost IPv4 art somehow. ↩︎

  3. Secrets are more fun when they are kept secret. 🥳 ↩︎

Jeremy Morgan (JeremyMorgan)

Building a Go Web API with the Digital Ocean App Platform October 15, 2020 05:03 AM

Recently, Digital Ocean announced they’re entering the PaaS market with their new application platform. They’ve hosted virtual machines (droplets) and Kubernetes based services for years, but now they’re creating a platform that’s a simple point and click to get an application up and running. So I decided to try it. In this tutorial, we’re going to build an application on that platform. I will use Go to make a small web API and have it backed by a SQLite database.

October 14, 2020

Gokberk Yaltirakli (gkbrk)

Status update, October 2020 October 14, 2020 09:00 PM

To nobody’s surprise, the consistency of status updates have been less than perfect. But still, here I am with another catch-up post. Since the last update was a while back, this one might end up slightly longer.


First off, let me start with a career update. I have received my undergraduate degree and I am now officially a Software Engineer. Recently I’ve started working with a company that does mobile network optimization. I’m now a part of their Integration team, and I get to work with a lot of internals of mobile networks. This is exciting for me because of my interest in radio communications, as I get to work on non-toy problems now.

I migrated my personal finances from basic CSV files to double-entry bookkeeping. I decided to go with a homebrew solution, so I published ledger.py. It has a syntax that roughly resembles ledger-cli and beancount, but is currently not compatible with either.

I have also written a few throw-away scripts that can read both my previous budget CSV and exports from my previous bank, so I get to backfill a lot of historical data.

I started working on a networking stack, along with a custom packet routing algorithm. There is no name for the project yet, and it is not quite ready for a fancy public release, but I am occasionally publishing code dumps on gkbrk/network01. I am testing this network in a sort of closed-alpha with a small group of friends.

The network is intended to work with a topology where nodes don’t have direct links to other nodes. This is different from the so-called overlay networks. While most links between nodes go through the internet via our ISPs right now, we are intending to add radio links between some nodes in order reduce our reliance on ISPs. There is nothing in the network design that prevents different kinds of links from being used.

As of now, the network can find paths between nodes, can recover and discover new paths in case some links fail, and can route packets between all nodes. We have done some trivial tests including private messaging and a few extremely choppy voice calls.

I am intending to work more on this project and even write some blog posts about it if I manage to stay interested.

As I have moved countries, I have a lot of paperwork to do. And some of this paperwork involves grabbing difficult-to-get appointments. I had the joy of automating this work and keeping me up to date using selenium and the SMS API from AWS.

I initially thought I would go with Twilio, but to my disappointment things weren’t too smooth with them. Everything went smoothly and I started to integrate their APIs, and it was time to put some credits in my account. While I looked completely normal to their automated systems, they decided to block me seconds after charging my card. Apparently paying for services is suspicious these days. And of course, no reply to support tickets and currently no refund in sight.

That’s all for this month, thanks for reading!