Planet Crustaceans

This is a Planet instance for lobste.rs community feeds. To add/update an entry or otherwise improve things, fork this repo.

March 29, 2020

Derek Jones (derek-jones)

Influential programming languages: some of the considerations March 29, 2020 10:38 PM

Which programming languages have been the most influential?

Let’s define an influential language as one that has had an impact on lots of developers. What impact might a programming language have on developers?

To have an impact a language needs to be used by lots of people, or at least have a big impact on a language that is used by lots of people.

Figuring out the possible impacts a language might have had is very difficult, requiring knowledge of different application domains, software history, and implementation techniques. The following discussion of specific languages illustrate some of the issues.

Simula is an example of a language used by a handful of people, but a few of the people under its influence went on to create Smalltalk and C++. Some people reacted against the complexity of Algol 68, creating much simpler languages (e.g., Pascal), while others thought some of its feature were neat and reused them (e.g., Bourne shell).

Cobol has been very influential, at least within business computing (those who have not worked in business computing complain about constructs handling uses that it was not really designed to handle, rather than appreciating its strengths in doing what it was designed to do, e.g., reading/writing and converting a wide range of different numeric data formats). RPG may have been even more influential in this usage domain (all businesses have to specific requirements on formatting reports).

I suspect that most people could not point to the major influence C has had on almost every language since. No, not the use of { and }; if a single character is going to be used as a compound statement bracketing token, this pair are the only available choice. Almost every language now essentially uses C’s operator precedence (rather than Fortran‘s, which is slightly different; R follows Fortran).

Algol 60 has been very influential: until C came along it was the base template for many languages.

Fortran is still widely used in scientific and engineering software. Its impact on other languages may be unknown to those involved. The intricacies of floating-point arithmetic are difficult to get right, and WG5 (the ISO language committee, although the original work was done by the ANSI committee, J3). Fortran code is often computationally intensive, and many optimization techniques started out optimizing Fortran (see “Optimizing Compilers for Modern Architectures” by Allen and Kennedy).

BASIC showed how it was possible to create a usable interactive language system. The compactness of its many, and varied, implementations were successful because they did not take up much storage and were immediately usable.

Forth has been influential in the embedded systems domain, and also people fall in love with threaded code as an implementation technique (see “Threaded Interpretive Languages” by Loeliger).

During the mid-1990s the growth of the Internet enabled a few new languages to become widely used, e.g., PHP and Javascript. It’s difficult to say whether these were more influenced by what their creators ate the night before or earlier languages. PHP and Javascript are widely used, and they have influenced the creation of many languages designed to fix their myriad of issues.

March 28, 2020

Patrick Louis (venam)

Software Distributions And Their Roles Today March 28, 2020 10:00 PM

Mailroom

NB: This is a repost on this blog of a post made on nixers.net

What is a distribution

What are software distributions? You may think you know everything there is to know about the term software distribution, but take a moment to think about it, take a step back and try to see the big picture.

We often have in mind the thousands of Linux distributions when we hear it, however, this is far from limited to Linux, BSD, Berkeley Software Distribution, has software distribution right in the name. Android, and iOS are software distributions too.

Actually, it’s so prevalent, we may have stopped paying attention to the concept. We find it hard to put a definition together.
There’s definitely the part about distributing software in it. Software that may be commercial or not, open source or not.
To understand it better maybe investigating what problems software distributions address would clear things up.

Let’s imagine a world before software distributions, does that world exist? A world where software stays within boundaries, not shared with anyone outside of it.
Once we break these boundaries and we want to share it, we’ll find that we have to package all the software together in a meaningful way, configure them so that they work well together, adding some glue in between when necessary, find the appropriate medium to distribute the bundle, get it all from one end to another safely, make sure it installs properly, and follow up on it.

Thus, software distribution is about the mechanism and the community that takes the burden and decisions to build an assemblage of coherent software that can be shipped.

The operating system, or kernel if you like, could be, and is often, part of the collage offered, a software just like others.

The people behind it are called distribution maintainers, or package maintainers. Their role vary widely, they could write the software that stores all the packages called the repository, maintain a package manager with its format, maintain a full operating system installer, package and upload software they built or that someone else built on a specific time frame/life cycle, make sure there aren’t any malicious code uploaded on the repository, follow up on the latest security issues and bug reports, fix third party software to fit the distribution philosophical choices and configurations, and most importantly test, plan, and make sure everything holds up together.
These maintainers are the source of trust of the distribution, they take responsibility for it. In fact, I think it’s more accurate to call them distributors.

Different ways to approach it

There’s so many distributions it can make your head spin. The software world is booming, especially the open source one. For instance, we can find bifurcations of distributions that get copied by new maintainers and divert. This creates a tree like aspect, a genealogy of both common ancestors and/or influences in technical and philosophical choices.
Overall, we now have a vibrant ecosystem where a thing learned on a branch can help a completely unrelated leaf on another tree. There’s something for everyone.

Target and speciality

So what could be so different between all those software distributions, why not have a single platform that everyone can build on.

One thing is specialization and differentiation. Each distro caters to a different audience and is built by a community with its philosophy.

Let’s go over some of them:

  • A distribution can support specific sets and combinations of hardware: from CPU ISA to peripherals drivers
  • A distribution may be specifically optimized for a type of environment: Be it desktop, portable mobile device, servers, warehouse size computers, embedded devices, virtualised environment, etc..
  • A distribution can be commercially backed or not
  • A distribution can be designed for different levels of knowledge in a domain, professional or not. For instance, security research, scientific computing, music production, multimedia box, HUD in cars, mobile device interface, etc..
  • A distribution might have been certified to follow certain standards that need to be adhere to in professional settings, for example security standards and hardening
  • A distribution may have a single purpose in a commodity machine, specific machine functionalities such as firewall, a computer cluster, a router, etc..

That all comes to the raison d’être, the philosophy of the distribution, it guides every decision the maintainers have to make. It guides how they configure every software, how they think about security, portability, comprehensiveness.

For example, if a distribution cares about free software, it’s going to be strict about what software it includes and what licenses it allows in its repository, having software to check the consistency of licenses in the core.
Another example is if their goal is to target a desktop audience then internationalization, ease of use, user friendliness, having a large number of packages, is going to be prioritized. While, again, if the target is a real time embedded device, the size of the kernel is going to be small, configured and optimized for this purpose, and limiting and choosing the appropriate packages that work in this environment. Or if it’s targeted at advanced users that love having control of their machine, the maintainers will choose to let the users make most of the decisions, providing as many packages as possible with the latest version possible, with a loosely way to install the distribution, having a lot of libraries and software development tools.

What this means is that a distribution does anything it can to provide sane defaults that fit its mindset. It composes and configures a layer of components, a stack of software.

The layering

Distribution maintainers often have at their disposition different blocks and the ability to choose them, stacking them to create a unit we call a software distribution. There’s a range of approaches to this, they could choose to have more, or less, included in what they consider the core of the distribution and what is externally less important to it.
Moreover, sometimes they might even leave the core very small and loose, instead providing the glue software that makes it easy for the users to choose and swap the blocks at specific stages in time: installation, run time, maintenance mode, etc..

So what are those blocks of interdependent components.

The first part is the method of installation, this is what everything hinges on, the starting point.

The second part is the kernel, the real core of all operating systems today. But that doesn’t mean that the distribution has to enforce it. Some distributions may go as far as to provide multiple kernels specialised in different things or none at all.

The third part is the filesystem and file hierarchy, the component that manages where and how files are spread out on the physical or virtual hardware. This could be a mix and match where sections of the file system tree are stored on separate filesystems.

The fourth part is the init system, PID 1. This choice has generated a lot of contention these days. PID 1 being the mother process of all other processes on the system. What role it has and what functionalities it should include is a subject of debate.

The fifth part is composed of the shell utilities, what we sometimes refer to as the userland or user space, as its the first layer the user can directly interface with to have control of the operating system, the place where processes run. The userland implementations on Unix-based systems usually tries to follow the POSIX standard. There are many such implementations, also subject of contention.

The sixth part is made up of services and their management. The daemons, long running processes that keep the system in order. Many argue if the management functionality should be part of the init system or not.

The seventh part is documentation. Often it is forgotten but it is still very important.

The last part is about everything else, all the user interfaces and utilities a user can have and ways to manage them on the system.

Stable releases vs Rolling

There exists a spectrum on which distributions place themselves when it comes to keeping up to date with the versions of the software they provide. This most often applies to external third party open source software.
The spectrum is the following: Do we allow the users to always have the latest version of every software while running the risk of accidentally breaking their system, what we call bleeding edge or rolling distro, or do we take a more conservative approach and take the time to test every software properly before allowing it in the repository, while not having all the latest updates, features, and optimizations of those software, what we call release based distro.

The extreme of the first scenario would be to let users directly download from the software vendor/creator source code repository, or the opposite, let the software vendor/creator push directly to the distribution repository. Which could easily break or conflict with the user’s system or lead to security vulnerability. We’ll come back to this later, as this could be avoided if the software runs in a containerized environment.

When it comes to release distributions, it usually involves having a long term support stable version that keeps receiving and syncing with the necessary security updates and bug fixes on the long run while having another version running a bit ahead testing the future changes. On specific time frames, users can jump to the latest release of the distribution, which may involve a lot of changes in both configuration and software.
Some distributions decide they may want to break ABI or API of the kernel upon major releases, that means that everything in the system needs to be rebuilt and reinstalled.

The release cycle, and the rate of updates is really a spectrum.

When it comes to updates, in both cases, the distribution maintainers have to decide how to communicate and handle them. How to let the users know what changes. If a user configuration was swapped for a new one or merged with the new one, or copied aside.
Communication is essential, be it through official channels, logging, mails, etc.. Communication needs to be bi-directional, users report bugs and maintainers posts what their decisions are and if users need to be involved in them. This creates the community around the distribution.

Rolling releases require intensive efforts from package maintainers as they constantly have to keep up with software developers. Especially when it comes to the thousands of newest libraries that are part of recent programming languages and that keep on increasing.

Various users will want precise things out of a system. Enterprise environments and mission critical tasks will prefer stable releases, and software developers or normal end users may prefer to have the ability to use the latest current software.

Interdistribution standard

With all this, can’t there be an interdistribution standard that creates order, and would we want such standard.

At the user level, the differences are not always noticeable, most of the time everything seems to work as Unix systems are expected to work.
There’s no real standard between distributions other than that they are more or less following the POSIX standards.

Within the Linux ecosystem, the Free Standards Group tries to improve interoperability of software by fixing a common Linux ABI, file system hierarchy, naming conventions, and more. But that’s just the tip of the iceberg when it comes to having something that works interdistributions.

Furthermore, each part of the layering we’ve seen before could be said to have its own standards: There are desktop interoperability standards, filesystem standards, networking standards, security standards, etc..

The biggest player right now when it comes to this is systemd in association with the free desktop group, it tries to create (force) an interdistribution standard for Linux distribution.

But again, the big Question: Do we actually want such inter-distribution standards, can’t we be happy with the mix and match we currently have. Would we profit from such thing?

The package manager and packaging

Let’s now pay attention to the package themselves, how we store them, how we give secure access to them, how we are able to search amongst them, download them, install them, remove them, and anything related to their local management, versioning, and configuration.

Method of distribution

How do we distribute software, share them, what’s the front-end to this process.

First of all, where do we store this software.

Historically and still today, software can be shared via physical medium such as CD-ROM, DVD, USBs, etc.. This is common when it comes to proprietary vendors to have the distribution come with a piece of hardware they are selling, it’s also common for the procurement of the initial installation image.
However, with today’s hectic software growth, using a physical medium isn’t flexible. Sharing over the internet is more convenient, be it via FTP, HTTP, HTTPS, a publicly available svn or git repo, via central website hubs such as Github or appliation stores such the ones Apple and Google provide.

A requirement is that the storage and the communication to it should be secure, reliable against failures, and accessible from anywhere. Thus, replication is often done to avoid failures but also to have a sort of edge network speeding effect across the world, load balancing. Replication could be done in multiple ways, it could be a P2P distributed system for instance.

How we store it and in what format is up to the repository maintainers. Usually, this is a file system with a software API users can interact with over the wire. Two main format strategies exist: source based repositories and binary repositories.

Second of all, who can upload and manage the host of packages. Who has the right to replicate the repository.

As a source of truth for the users, it is important to make sure the packages have been verified and secured before being accepted on the repository.

Many distribution have the maintainers be the only ones that are able to do this. Giving them cryptographic keys to sign packages and validate them.

Others have their own users build the packages, send them to a central hub for automatic or manual verification and then uploaded to the repository. Each user having their own cryptographic key for signature verification.

This comes down to an issue of trust and stability. Having the users upload packages isn’t always feasible when using binary packages if the individual packages are not containerized properly.

There’s a third option, the road in between, having the two types, the core managed by the official distribution maintainers and the rest by its user community.

Finally, the packages reach the user.

How the user interact with the repository locally and remotely depends on the package management choices. Do users cache a version of the remote repository, like is common with the BSD port tree system.
How flexible can it be to track updates, locking versions of software, allowing downgrades. Can users download from different sources. Can users have multiple version of the same software on the their machine.

Format

As we’ve said there are two main philosophy of software sharing format: source code port-style and pre-built binary packages.

The software that manages those on the user side is called the package manager, it’s the link with the repository. Though, in source based repo I’m not sure we can call them this way, but regardless I’ll still refer to them as such.
Many distributions create their own or reuse a popular one. It does the search, download, install, update, and removal of local software. It’s not a small task.

The rule of the book is that if it isn’t installed by the package manager then it won’t be aware of its existence. Noting that distributions don’t have to be limited to a single package manager, there could be many.

Each package manager relies on a specific format and metadata to be able to manage software, be it source or binary formatted. This format can be composed of a group of files or a single binary file with specific information segments that together create recipes that help throughout its lifecycle. Some are easier to put together than others, incidentally allowing more user contributions.

Here’s a list of common information that the package manager needs:

  • The package name
  • The version
  • The description
  • The dependencies on other packages, along with their versions
  • The directory layout that needs to be created for the package
  • Along with the configuration files that it needs and if they should be overwritten or not
  • An integrity, or ECC, on all files, such as SHA256
  • Authenticity, to know that it comes from the trusted source, such as cryptographic signatures checked against a trusted store on the user’s machine
  • If this is a group of package, meta package, or a direct one
  • The actions to take on certain events: pre-installation, post-installation, pre-removal, and post removal
  • If there are specific configuration flags or parameter to pass to the package manager upon installation

So what’s the advantage of having pre-compiled binary packages instead of cloning the source code and compiling ourselves. Won’t that remove a burden from package maintainers.

One advantage is that pre-compiled packages are convenient, it’s easier to download them and run them instantly. It’s also hard, if not impossible, these days, and energy intensive, to compile huge software such as web browsers.
Another point, is that proprietary software are often already distributed as binary packages, which would creates a mix of source and binary packages.

Binary formats are also space efficient as the code is stored in a compressed archived format. For example: APK, Deb, Nix, ORB, PKG, RPM, Snap, pkg.tar.gz/xz, etc..
Some package managers may also choose to leave the choice of compression up to the user and dynamically discern from its configuration file how to decompress packages.

Let’s add that there exists tools, such as “Alien”, that facilitate the job of package maintainers by converting from one binary package format to another.

Conflict resolution & Dependencies management

Resolving dependencies

One of the hardest job of the package manager is to resolve dependencies.

A package manager has to keep a list of all the packages and their versions that are currently installed on the system and their dependencies.
When the user wants to install a package, it has to take as input the list of dependencies of that package, compare it against the one it already has and output a list of what needs to be installed in an order that satisfies all dependencies.

This is a problem that is commonly encountered in the software development world with build automation utilities such as make. The tool creates a directed acyclic graph (DAG), and using the power of graph theory and the acyclic dependencies principle (ADP) tries to find the right order. If no solution is found, or if there are conflicts or cycles in the graph, the action should be aborted.

The same applies in reverse, upon removal of the package. We have to make a decision, do we remove all the other packages that were installed as a dependency of that single one. What if newer packages depend on those dependencies, should we only allow the removal of the unused dependencies.

This is a hard problem, indeed.

Versioning

This problem increases when we add the factor of versioning to the mix, if we allow multiple versions of the same software to be installed on the system.

If we don’t, but allow switching from one version to another, do we also switch all other packages that depend on it too.

Versioning applies everywhere, not only to packages but to release versions of the distribution too. A lot of them attach certain version of packages to specific releases, and consequentially releases may have different repositories.

The choice of naming conventions also plays a role, it should convey to users what they are about and if any changes happened.

Should the package maintainer follow the naming convention of the software developer or should they use their own. What if the name of two software conflict with one another, this makes it impossible to have it in the repo, some extra information needs to be added.

Do we rely on semantic versioning, major, minor, patch, or do we rely on names like so many distributions releases do (toy story, deserts, etc..), or do we rely on the date it was released, or maybe simply an incremental number.

All those convey meaning to the user when they search and update packages from the repository.

Static vs dynamic linking

One thing that may not apply to source based distro, is the decision between building packages as statically linked to libraries or dynamically linked.

Dynamic linking is the process in which a program chooses not to include a library it depends upon in its executable but only a reference to it, which is then resolved at run-time by a dynamic linker that will load the shared object in memory upon usage. On the opposite, static linking means storing the libraries right inside the compiled executable program.

Dynamic linking is useful when many software rely on the same library, thus only a single instance of the library has to be in memory at a time. Executables sizes are also smaller, and when it is updated all programs relying on it get the benefit (as long as the interfaces are the same).

So what does this have to do with distributions and package management.

Package managers in dynamic linking environment have to take care of the versions of the libraries that are installed and which packages depend on them. This can create issues if different packages rely on different versions.

For this reason, some distro communities have chosen to get rid of dynamic linking altogether and rely on static linking, at least for things that are not related to the core system.

Another incidental advantage of static linking is that it doesn’t have to resolve dependencies with the dynamic linker, which makes it gain a small boost in speed.

So static builds simplify the package management process. There doesn’t need to be a complex DAG because everything is self contained. Additionally, this can allow to have multiple versions of the same software installed alongside one another without conflicts. Updates and rollbacks are not messy with static linking.

This gives rise to more containerised software, and continuing on this path leads to market platforms such as Android and iOS where distribution can be done by the individual software developers themselves, skipping the middle-man altogether and giving the ability for increasingly impatient users to always have the latest version that works for their current OS. Everything is self-packaged.
However, this relies heavily on the trust of the repository/marketplace. There needs to be many security mechanisms in place to not allow rogue software to be uploaded. We’ll talk more about this when we come back to containers

This is great for users and, from a certain perspective, software developers too as they can directly distribute pre-built packages, especially when there’s a stable ABI for the base system.

All this breaks the classic distribution scheme we’re accustomed to on the desktop.

Is it all roses and butterflies, though.

As we’ve said, packages take much more space with static linking, thus wasting resources (storage, memory, power).
Moreover, because it’s a model where software developers push directly to users, this removes the filtering that distribution maintainers have over the distro, and encourages licenses uncertainties. There’s no more overall philosophies that surrounds the distribution.
There’s also the issue of library updates, the weight is on the software developers to make sure they have no vulnerabilities or bugs in their code. This adds a veil on which software uses what, all we see is the end products.

From a software developer using this type of distribution perspective, this adds extra steps to download the source code of each libraries their software depends on, and build each one individually. Turning the system into a source based distro.

Reproducibility

Because package management is increasingly becoming messier the past few years, a new trend has emerged to put back a sense of order in all this, reproducibility.

It has been inspired by the world of functional programming and the world of containers. Package managers that respect reproducibility have each of their builds asserted to always produce the same output.
They allow for packages of different versions to be installed alongside one another, each living in its own tree, and it allows normal users to install packages only them can access. Thus, many users can have different packages.

They can be used as universal package managers, installed alongside any other package managers without conflict.

The most prominent example is Nix and Guix, that use a purely functional deployment model where software is installed into unique directories generated through cryptographic hashes. Dependencies from each software are included within each hash, solving the problem of dependency hell. This approach to package management promises to generate more reliable, reproducible, and portable packages.

Stateless and verifiable systems

The discussion about trust, portability, and reproducibility can also be applied to the whole system itself.

When we talked about repositories as marketplaces, where software developers push directly to it and the users have instant access to the latest version, we said it was mandatory to have additional measures for security.

One of them is to containerised, to sandbox every software. Having each software run in their own space not affecting the rest of the system resources. This removes the heavy burden of auditing and verifying each and every software. Many solutions exist to achieve this sandboxing, from docker, chroot, jails, firejail, selinux, cgroups, etc..

We could also distance the home directory of the users, making them self-contained, never installing or modifying the globally accessible places.

This could let us have the core of the system verifiable as it is not changed, as it stays pristine. Making sure it’s secure would be really easy.

The idea of having the user part of the distro as atomic, movable, containerized, and the rest reproducible is game changing. But again, do we want to move to a world where every distro is interchangeable?

Do Distros matter with containers, virtualisation, and specific and universal package managers

It remains to be asked if distributions still have a role today with all the containers, virtualisation, and specific and universal package managers.

When it comes to containers, they are still very important as they most often are the base of the stack the other components build upon.

The distribution is made up of people that work together to build and distribute the software and make sure it works fine. It isn’t the role of the person managing the container and much more convenient for them to rely on a distribution.

Another point, is that containers hide vulnerabilities, they aren’t checked after they are put together, while on the other hand, distribution maintainers, have as a role to communicate and follow up on security vulnerabilities and other bugs. Community is what solves daunting problems that everyone shares.
A system administrator building containers can’t possibly have the knowledge to manage and builds hundreds of software and libraries and ensure they work well together.

If packages are self-contained

Do distributions matter if packages are self-contained?

To an extent they do as they could be in this ecosystem the providers/distributors of such universal self-contained packages. And as we’ve said it is important to keep the philosophy of the distro and offer a tested toolbox that fits the use case.

What’s more probable is that we’ll move to a world with multiple package managers, each trusted for its specific space and purpose. Each with a different source of philosophical and technical truth.

Programming language package management specific

This phenomena is already exploding in the world of programming language package management.

The speed and granularity at which software is built today is almost impossible to follow using the old method of packaging. The old software release life cycle has been thrown out the window. Thus language-specific tools were developed, not limited to installing libraries but also software. We can now refer to the distribution offered package manager as system-level and others as application-level or specific package managers.

Consequentially, the complexity and conflicts within a system has exploded, and distribution package managers are finding it pointless to manage and maintain anything that can already be installed via those tools. Vice-versa, the specific tool makers are also not interested in having what they provide included in distribution system-level package managers.

Package managers that respect reproducibility, such as Nix, that we’ve mentioned, handle such cases more cleanly as they respect the idea of locality, everything residing withing a directory tree that isn’t maintained by the system-level package manager.

Again, same conclusion here, we’re stuck with multiple package managers that have different roles.

Going distro-less

A popular topic in the container world is “distro-less”.

It’s about replacing everything provided in a distribution, removing it’s customization, or building an image from scratch and maybe relying on universal package managers or none.

The advantage of such containers is that they are really small and targeted for a single purpose. This let the sysadmin have full control of what happens on that box.

However, remember that there’s a huge cost to controlling everything, just like we mentioned earlier. This moves the burden upon the sysadmin to manage and be responsible to keep up with bugs and security updates instead of the distribution maintainers

Conclusion

With everything we’ve presented about distributions, I hope we now have a clearer picture of what they are providing and their place in our current times.

What’s your opinion on this topic? Do you like the diversity? Which stack would you use to build a distribution? What’s your take on static builds, having users upload their own software to the repo? Do you have a solution to the trust issue? How do you see this evolve?

More discussion here: https://nixers.net/showthread.php?tid=2192


What is a distribution

Different ways to approach it

Package management

Conflict resolution & dependencies management

Do Distros matter with containers, virtualisation, and specific and universal package managers

Attributions:

  • Marjory Collins / Public domain

Gonçalo Valério (dethos)

CSP headers using Cloudflare Workers March 28, 2020 12:35 PM

Last January I made a small post about setting up a “Content-Security-Policy” header for this blog. On that post I described the steps I took to reach a final result, that I thought was good enough given the “threats” this website faces.

This process usually isn’t hard If you develop the website’s software and have an high level of control over the development decisions, the end result ends up being a simple yet very strict policy. However if you do not have that degree of control over the code (and do not want to break the functionality) the policy can end up more complex and lax than you were initially hoping for. That’s what happened in my case, since I currently use a standard installation of WordPress for the blog.

The end result was a different security policy for different routes and sections (this part was not included on the blog post), that made the web-server configuration quite messy.

(With this intro, you might have already noticed that I’m just making excuses to redo the initial and working implementation, in order to test some sort of new technology)

Given the blog is behind the Cloudflare CDN and they introduced their “serverless” product called “Workers” a while ago, I decided that I could try to manage the policy dynamically on their servers.

Browser <--> CF Edge Server <--> Web Server <--> App

The above line describes the current setup, so instead of adding the CSP header on the “App” or the “Web Server” stages, the header is now added to the response on the last stage before reaching the browser. Let me describe how I’ve done it.

Cloudflare Workers

First a very small introduction to Workers, later you can find more detailed information on Workers.dev.

So, first Cloudflare added the v8 engine to all edge servers that route the traffic of their clients, then more recently they started letting these users write small programs that can run on those servers inside the v8 sandbox.

The programs are built very similarly to how you would build a service worker (they use the same API), the main difference being where the code runs (browser vs edge server).

These “serverless” scripts can then be called directly through a specific endpoint provided by Cloudflare. In this case they should create and return a response to the requests.

Or you can instruct Cloudflare to execute them on specific routes of your website, this means that the worker can generate the response, execute any action before the request reaches your website or change the response that is returned.

This service is charged based on the number of requests handled by the “workers”.

The implementation

Going back to the original problem and based on the above description, we can dynamically introduce or modify the “Content-Security-Policy” for each request that goes through the worker which give us an high degree of flexibility.

So for my case a simple script like the one below, did the job just fine.

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

/**
 * Forward the request and swap the response's CSP header
 * @param {Request} request
 */
async function handleRequest(request) {
  let policy = "<your-custom-policy-here>"
  let originalResponse = await fetch(request)
  response = new Response(originalResponse.body, originalResponse)
  response.headers.set('Content-Security-Policy', policy)
  return response
}

The script just listens for the request, passes it to a handler function (lines 1-3), forwards to the origin server (line 12), grabs the response (line 13), replaced the CSP header with the defined policy (line 14) and then returns the response.

If I needed something more complex, like making slight changes to the policy depending on the User-Agent to make sure different browsers behave as expected given the different implementations or compatibility issues, it would also be easy. This is something that would be harder to achieve in the config file of a regular web server (nginx, apache, etc).

Enabling the worker

Now that the script is done and the worker deployed, in order to make it run on certain requests to my blog, I just had to go to the Cloudflare’s dashboard of my domain, click on the “workers” section and add the routes I want it to be executed:

cloudflare workers routes modalConfiguring the routes that will use the worker

The settings displayed on the above picture will run the worker on all requests to this blog, but is can be made more specific and I can even have multiple workers for different routes.

Some sort of conclusion

Despite the use-case described in this post being very simple, there is potential in this new “serverless” offering from Cloudflare. It definitely helped me solve the problem of having different policies for different sections of the website without much trouble.

In the future I might comeback to it, to explore other user-cases or implementation details.

Nikola Plejić (nikola)

Living and Coding In Times of Crises March 28, 2020 10:08 AM

Note: These are a few slightly melodramatic thoughts written in the aftermath of several somewhat traumatic weeks involving a pandemic, an earthquake, and a substantial change in the everyday. The few links that are scattered around I find interesting and important; the rest is here as a permanent reminder to myself. Hope everyone's well. Wash your hands.

I had hoped I would be able to write the word "crisis" in singular in the title of this post, but life tends to be a prescriptivist bastard. Writing this in the middle of a pandemic, after a non-catastrophic, yet devastating earthquake in Zagreb on Sunday, feels surreal. The apartment I just recently moved into roughed up, parts of it possibly significantly damaged, unable to socialize even in the most mundane of ways, exams canceled. Two events tearing into the fabric of what one considers the most social and the most personal, at the same time... and I'm lucky to be healthy, safe, and sound.

It's interesting how disruptive events quickly redefine the "normal". Just a few weeks ago, no one could imagine being quarantined indefinitely, yet here we are in week one of being unable to leave our place of residence. Redefining normal also tends to give way to malice, as we witness more and more calls to increase surveillance and invade people's privacy in order to (at least formally) keep track of the spread and unwanted public gatherings. Once again the political emerges as inseparable from the technological to the detriment of the techno-elite. Go figure.

Science, with all of its problems, gives us hope. Both of these events have been properly assessed: epidemiologists and infectious disease experts have been warning us of the possible consequences of SARS-CoV-2 weeks before the situation got out of control outside of China, and seismologists have been fairly clear that a strong earthquake around Zagreb is imminent. The problem with complex systems is that they're hard to grasp even for experts, and our intuition for statistics is slim to none. Even when we do manage to become aware of some aspects of the risks involved, this tends to fall apart as quickly as it materializes. Coupled with the fear of "collapsing markets" and their existential consequences, this often means we're more comfortable with the status quo than with taking action. Informed apathy can be as dangerous as uninformed sympathy.


As an aside, existential terror is a powerful weapon, and it was interesting to observe its effects on one's thoughts and approaches to life. I am well-aware that there are no reliable methods of predicting earthquakes to any precise extent, yet when on the day of the earthquake someone blurted out a hoax that "there's a stronger one coming at 8:45am", for a few moments it seemed like the most inevitable and obvious thing in the world. I think I've revisited that highly instructive moment more often than the earthquake itself.


Even though it's impossible not to be affected by this, people in IT, and primarily in software, seem to be in a unique position to handle this situation better than the average worker. There will be consequences, but the vast amount of money in tech means there's less chance of being fired, especially in times of enormous reliance on the tech infrastructure. The level of technological literacy gives us an enormous advantage in navigating the inevitable mess during the initial period of adjustment. A lot of us have been able to seamlessly switch to working from home, if we haven't already been doing so.

I believe this gives us a fair deal of responsibility, too: it's increasingly important to use our skills to help the ones in need. Large-scale tragedies take a psychological and organizational toll, and we have tools to help. This means being aware of the fact that the technology we produce is not as obvious as it might seem, and that people often need assistance and support which isn't a sign of weakness or lack of competence, but rather a natural consequence of using a new tool in an unfamiliar setting. It also means using the free time that isolation gives us to help people and organizations by providing them with necessary technological infrastructure and assistance.

More in-depth political analyses have been written by more eloquent and better-read comrades over at Pirate Care, focused on COVID-19 in particular, and by Tomislav, focused on the pandemic in combination with the earthquake. I can just echo that I find it important not to let our imagination atrophy and our empathy to subside, and that I hope the lessons we learn here will be the ones of solidarity and care. ☭

March 27, 2020

Jan van den Berg (j11g)

Dylan Thomas – Sidney Michaels March 27, 2020 09:23 AM

This book is a play from 1965, based on several accounts of the infamous travels Welsh poet Dylan Thomas made in the early 1950s to the US. If you know anything about Dylan Thomas you probably know he died young (39), and that he was a alcoholic.

Dylan Thomas – Sidney Michaels (1965) – 111 pages

This play captures the last two,three years of his life rather vividly. It’s an alcoholic mess and details an explosive marriage. His tumultuous life and and classic ‘poets-die-young’ death only deepened the already legend. So much so, that a young fellow named Robert Zimmerman, based his stage name on the famous poet a few years later. And that’s why I picked up this book, and learned a little bit more.

The post Dylan Thomas – Sidney Michaels appeared first on Jan van den Berg.

Pete Corey (petecorey)

Glorious Voice Leader Reborn March 27, 2020 12:00 AM

I recently finished a major visual overhaul of my current pet project, Glorious Voice Leader. The redesign was primarily intended to give musicians a (hopefully) more natural interface for working with the tool.

Originally, Glorious Voice Leader displayed its chord suggestions as a series of horizontal fretboards stacked vertically:

The old Glorious Voice Leader.

This gave the guitarist lots of control, but it wasn’t a natural way for them to interact with or consume a chord progression.

Instead, I decided to transition to a more familiar chart-based design where every chord in the progression is represented by a small chord chart that should be familiar to any guitarist. The chord currently being added or modified is also displayed on a full fretboard laid out vertically on the side of the page:

The newly redesigned Glorious Voice Leader.

The redesign was largely a cosmetic overhaul. A few new minor features were added in the process, such as progressive chord naming, and better keyboard controls, but the meat of Glorious Voice Leader is all still there. Your old saved URLs will still work!

Check out Glorious Voice Leader now! And while I’ve got you hooked, be sure to read more about the project and it’s roots in my other projects.

March 24, 2020

Patrick Louis (venam)

The Self, Metaperceptions, and Self-Transformation March 24, 2020 10:00 PM

infinite reflections

How would you describe yourself?
How do you usually talk about yourself?
Do you feel like you are the writer of your own narrative?
Who are you?

We all stand on a balance of being perceived and perceiving, of having a visible and owning an invisible part, and of having control over and being controlled by. It is amongst all this that we can find the nebulous definition of who we are, what Locke calls “the sameness of a rational being”.
In view of this, we are both passengers and conductors of our narrative. So how do we drive this narrative forward, is it possible to have more agency in it than we currently have. And if we are our narrative can we, as the narrative, choose another narrative without self-annihilation.
Metacognition can be dizzying.

I’ve previously discussed the topic of what we are and now I’d like to focus on the self, its formation, its transformation, and its actualization.

Who one is, over time, is created by the amalgamation of the historical events, physical aspects, and external and internal reflections, that get incorporated into one’s identity. The self is this element that sits in the middle, taking in and taking out, what makes sense to us and for us.

From an external point of view, we could define ourselves in reaction to the roles we play for others, the way we interact with them, eventually adjusting our selves to the labels we’ve been given or have chosen.
It is helpful to have others act as calibration to our internal system when we have nothing else to base our definition unto, especially when we are starting with our self exploration in teenage years. We aren’t brains in a vat, a self doesn’t exist without a world. Yet, if we overly emphasize on this sort of self definition, the other becomes our worse nightmare and our only way of finding meaning and salvation. It leads to interpreting the world with a heavy filter, and judge it the way we think it judges us, harshly, and frequently inaccurately, because it is shaped by our individual self-concept and personal biases. This is what we call metaperception, the idea we have of how other people’s view of us.

Metaperception can be destructive if not handled properly. For instance, someone who fixates on it may act in a self-centered way, imagining that everyone is watching and evaluating their every move, that they are the center of social interaction. They’ll shut themselves, limit their spontaneity, and have an increasingly fragile ego. All the while, not considering the unbridgeable gap that exists between selves.
Being overwhelmed by the other makes it difficult to accept criticism, to interpret someone else’s response; everything becomes emotionally charged in a frenzied uncontrollable internal state.

From an internal point of view, we could defined ourselves as the main character of our lives, the maker of the story. We could move in the world in relation to what we perceive we’re doing to it.
We are our own persons, with our own choices, so why not make the world what we want of it. Yet, if we overly emphasize on this sort of self definition, we become an actor, the protagonist, reading the main script, trying to get the center of the stage, while everyone around plays a minor role. It leads to clashes in narratives, cognitive dissonance, an illusion of superiority, and an egocentric bias. Just like over-metaperception creates lenses, the self-made-story does too. We cannot deny that, by analogical thinking, others exist and that they have their own selves.

Therefore, that’s where the balance lies: knowing that we can be the masters of our destinies, and knowing we are creatures living in a limited social and physical world.
How do we learn to be comfortable with the ambiguity of the self/other boundary and get a better life experience. Why and how do we change our selves.

Change is hard. It’s an arduous task we’d rather let happen by itself gradually, let it pass by, barely noticing it after it has happened. Unfortunately, life is riddled with issues and dissatisfactions.
At first, we may pretend they are benign, non-existent, trivial. We dismiss them and move on to do activities that take our focus away from them…
Until they are not trivial anymore, until our behavioral pattern becomes destructive, until they become unmanageable, until we become intimately aware of them.
Then a realization emerges: Those problems are created by our sense of self, they are the product of our definition, part of the narrative. Rejecting them would mean rejecting one self. And this is what we do, we build immunity to change, we protect our self consistence, we cocoon ourselves away from the unknown changes. This is what we are and we feel stuck with it.

The years pass by and nothing seems to change. - Henry David Thoreau

Gradually, we may build constant feelings of guilt, shame, anxiety, and regret. Desperate for change we see as unattainable, seeing everything as an unfulfilling experience. Are we to forever remain haunted by what might have been?

To cope with such emotions, we could rely on our old friend: self-suppressive escapism. Namely, anything that is numbing, numbing to the critical evaluation of the self, a cognitive narrowing, a cognitive detachment from the disturbing elements of the self. All of this being the easiest way to avoid the source of despair. Moreover, in themselves, these kinds of actions could be blamed for our current state. We can blame our inability to take productive actions to change on our anxiety, depression, fear, or lack of confidence in our abilities. Additionally, we may even believe that we have to first get rid of such feelings before moving on to change, we’ll try meditation and introspection. Or we may believe that we’ve wasted too much time, that it’s too late, and be overwhelmed by intense feelings of guilt and regret. However, the negative emotions are not the results of those but they are inherent to the way we define ourselves and our fear of change.

Any obstruction of the natural processes of development …or getting stuck on a level unsuited to one’s age, takes its revenge, if not immediately, then later at the onset of the second half of life, in the form of serious crises, nervous breakdowns, and all manner of physical and psychic sufferings. Mostly they are accompanied by vague feelings of guilt, by tormenting pangs of conscience, often not understood, in face of which the individual is helpless. He knows he is not guilty of any bad deed, he has not given way to an illicit impulse, and yet he is plagued by uncertainty, discontent, despair, and above all by anxiety – a constant, indefinable anxiety. And in truth he must usually be pronounced “guilty”. His guilt does not lie in the fact that he has a neurosis, but in the fact that, knowing he has one, he does nothing to set about curing it. - Jolande Jacobi, The Way of Individuation

We cannot change anything unless we accept it. - Carl Jung

Thus, we should find the courage to tackle personal growth. If we don’t accept what has been, we can’t move to what will be. The feelings of dissatisfaction should be the catalyst of change, they should be welcomed as stimuli in the struggle for the development of personality.

Neurotic symptoms such as these are a direct result of an inadequate approach to life and act as signals communicating the necessity of change. - Carl Jung

Small changes are great and cumulate but when we’ve reached a point where each step forward gets repelled by all our insecurities, we need more assurance, we need to know which sort of exact self-induced changes are the most useful.
And this is what we need to do, we need to break our immunity to change, we need to remove the shield of our self consistence, we need to face the unknown changes right on. This is the step where we need to take the courage to sacrifice our selves to be reborn.

Sacrifice always means the renunciation of a valuable part of oneself, and through it the sacrificer escapes being devoured. Difficult but necessary step to abandon an aspect of ourself in order to pave the way for the emergence of the new. The sacrifice is critical in the process of rebirth because what keeps us locked in our problem is the inability to recognize that ways of life that served us in our past may morph from promoters of our well being to the acute cause of our suffering. - Carl Jung

…The dying of one attitude or need may be the other side of the birth of something new. One an choose to kill a neurotic strategy, a dependency, a clinging, and then find that he can choose to live as a freer self… A “dying” of part of one’s self is often followed by a heightened awareness of self, a heightened sense of possibility. - Rollo May

Unsurprisingly, any sudden unnatural change involves risks, especially when already deep into the abyss. Such change may lead to disorder if, by removing part of ourselves, we have nothing else to fill it with. This may take us to the path of chaos and psychological breakdown.

[This] …is similar in principle to a psychotic disturbance; that is, it differs from the initial stage of mental illness only by the fact that it leads in the end to greater health, while the latter leads to yet greater destruction. - Carl Jung

Enters a labyrinth, and multiplies a thousandfold the dangers that life in itself brings with it - of which not the least is that nobody can see how and where he loses his way, becomes solitary, and is torn to pieces by some cave-Minotaur of conscience. - Nietzsche

So now that we’re aware of our situation and have the courage to leap and let ourselves go, how do we direct the change and get out of the loop we’re currently stuck in, how do we stop being an immovable pillar of the past.

Just like we’ve accepted that parts of the self can be sacrificed, we have to accept its ambiguity and its chaotic nature. Accept that there’s a world within us that we may not currently understand, that there’s a depth in each of us that remains to be discovered. We have to be aware of the reality of our psyche.
Indeed, a lot of the reasons why it remains hidden are related to the self-suppression we externally exert on elements of our personality that we think run counter to the moral system of our days. We are blinded by the unquestioned social beliefs and standards.

All psychology so far has got hung up on moral prejudices and fears: it has not dared to descend into the depths. - Nietzsche

Similarly, how can we know ourselves if we are actors in a play, wearer of the masks of society. How can we know ourselves if all we strive for is fake-perfectionism, a fetishism for perfection and a repulsion for anything that isn’t. Perfection hinders our development.
Consequentially, we have to accept that perfection is not a thing to aim for, that it is non-existent. Because how can one know what perfection is if it is not complete to begin with.

One should never think that man can reach perfection, he can only aim at completion - not to be perfect but to be complete. That would be the necessity and the indispensable condition if there were only question of perfection at all. For how can you perfect a thing if it is not complete?

Make it complete first and see what it is then. But to make it complete is already a mountain of a task, and by the time you arrive at absolute completion, you find that you are already dead, so you never read that preliminary condition for perfecting yourself. - Carl Jung

Completeness means grasping the wholeness of the self, the inner foundation of one’s mind, how to impose form and harmony on the chaos that is the totality of the self. We are chaos.
We are composed of a multitude, and we need to make those emerge, to bring the parts to light. As much as we dig, as much as the elements of consciousness become apparent. We should do all we can to promote this growth, we should be heroes and explorers of the chaos not passive observers that are controlled by its forces. The myriads of chunks are then seen from afar, for what they are, none taking authority on the whole.
Just like the body is coordinated, there can be a master to the chaos of the mind. We can have an organizing idea that drives the rest. But what is this drive, how do we cultivate it, how can we muster its power to train the body it’s composed of.

If we keep experimenting and spend time bringing forward the chaotic parts to understand them, it becomes clear. Our drives don’t come out of thin air. Analogically to the reframing of the definition of the self, we can reframe our connection to space and time.
We are part of our history, our dreams, our will of achievements, our unconscious thoughts, our past cultures, our institutions, our traditions, our physical predispositions, our limits, our sense of imminent death, and our primitive drives, all constitute our location in space-time. Those are also expressions of our personal and collective unconscious. Altogether, those create a common symbolic language, archetypes, universal elements, original forms that are part of the specification of human nature.
When one becomes conscious of their link with archetypes, they gain knowledge of the timeless “pattern of human life”, it provides a link with humanity. Additionally, this type of learning dissolves the feeling that everything is absurd and provides a sense of being rooted.
Considering this, we need to actively learn about basic prehistorical drives and how we can take power over them, we need to learn about history and how it repeats itself, we need to be familiar with the vestiges of the past, feel them flow within us instead of being ashamed and repressing them. The ruling passion should drive it all, sculpting our own heroic meaning to life.

For that, we can choose to study examples of archetypes, anyone and anything that we find excels in life, be it an imaginary being or not. Like an artist, we can be active makers, using the same method that gave birth to our self in the first place: imitation and emulation of the others. We can create a second self based on traits, characteristics, and how they handle adversity and challenges, of a role model while still respecting what we know we can’t change in ourselves, our innate strength and weaknesses.
This mechanism, of having an alter-ego archetype as a feedback mechanism that we slowly dissolve into, makes escapism easier and directed. Instead of reaching for the numbing actions we should reach for the second self we’ve created. Importantly keeping in mind that one of those numbing action is to divulge to others that we are simple “acting it out”, informing others would only be a sign of the anxiety we are having when facing this novel situation and that we are looking for the recalibration of our social norms via the others.
We should not pretend, we should act as if we already were and remind ourselves constantly, when we fall back to our old habits, that the past was the actual acting and that the present is the reality.

All of this until we engender the shift in our mindset. We’ll finally be a new self, looking above the previous one from a distance. It’s only by taking a distance that we can understand the whole.
Thus, we can continue on our self discovery, peeling more and more, revealing potential we didn’t know we had. At this point we can rediscover our internal and external worlds like never before. Introspection, retrospection, discussion, connection of the minds, these take on a different perspective: what we call psychological mindedness.

The field of developmental psychology has been enamored with such encapsulated form of growth, subject/object, the one where as you move along the previous self becomes the object of the current self or where you radically change your perspective on life. For instance, Abraham Maslow, Jean Piaget, Erik Erikson, Robert Kegan, are psychologists that used such methods to convey cognitive and personality development.

Maslow talks about self-actualization, “man’s tendency to actualize himself, to become his potentialities”, “the desire for self-fulfillment”. This can be achieve by having something to aim at, not for external rewards or achievement of the goal but rather because the transformation of the self forces us to do it. The type of behavior that requires self-discipline, skills, and that is constructive.
Piaget and Kegan focus on the subject-object and inter-relation when it comes to defining the self. Giving a sense to the self in a world that is nebulous and has roots in history.
Erikson prefers to emphasize what is important at every stage of life, the “ego identities”, from childhood to senior years, all together encompassing a stable self. Putting in perspective our common humanity.


Finally, maybe eventually, with all this, we can know who we are and how to describe ourselves.











Attributions:

  • Gianni Crestani / CC0

Jeremy Morgan (JeremyMorgan)

Stay Home and Learn JavaScript March 24, 2020 08:32 PM

If you’re quarantined at home and always wanted to learn to code, now’s your chance. I’ll be creating a series of tutorials designed to take you from “zero to hero” as a React Developer. Before you start with React you should know some JavaScript. Unlike many front end frameworks/libraries React is uses JavaScript patterns extensively. So we’ll cover some JavaScript basics. In this tutorial, you will learn: How to get your first webpage set up How to write text to the browser How to write text to the console We will cover some of the basics of JavaScript but won’t get too deep.

March 23, 2020

Joe Nelson (begriffs)

Concurrent programming, with examples March 23, 2020 12:00 AM

Mention concurrency and you’re bound to get two kinds of unsolicited advice: first that it’s a nightmarish problem which will melt your brain, and second that there’s a magical programming language or niche paradigm which will make all your problems disappear.

We won’t run to either extreme here. Instead we’ll cover the production workhorses for concurrent software – threading and locking – and learn about them through a series of interesting programs. By the end of this article you’ll know the terminology and patterns used by POSIX threads (pthreads).

This is an introduction rather than a reference. Plenty of reference material exists for pthreads – whole books in fact. I won’t dwell on all the options of the API, but will briskly give you the big picture. None of the examples contain error handling because it would merely clutter them.

Table of contents

Concurrency vs parallelism

First it’s important to distinguish concurrency vs parallelism. Concurrency is the ability of parts of a program to work correctly when executed out of order. For instance, imagine tasks A and B. One way to execute them is sequentially, meaning doing all steps for A, then all for B:

A B

Concurrent execution, on the other hand, alternates doing a little of each task until both are all complete:

Concurrency allows a program to make progress even when certain parts are blocked. For instance, when one task is waiting for user input, the system can switch to another task and do calculations.

When tasks don’t just interleave, but run at the same time, that’s called parallelism. Multiple CPU cores can run instructions simultaneously:

A B

When a program – even without hardware parallelism – switches rapidly enough from one task to another, it can feel to the user that tasks are executing at the same time. You could say it provides the “illusion of parallelism.” However, true parallelism has the potential for greater processor throughput for problems that can be broken into independent subtasks. Some ways of dealing with concurrency, such as multi-threaded programming, can exploit hardware parallelism automatically when available.

Some languages (or more accurately, some language implementations) are unable to achieve true multi-threaded parallelism. Ruby MRI and CPython for instance use a global interpreter lock (GIL) to simplify their implementation. The GIL prevents more than one thread from running at once. Programs in these interpreters can benefit from I/O concurrency, but not extra computational power.

Our first concurrent program

Languages and libraries offer different ways to add concurrency to a program. UNIX for instance has a bunch of disjointed mechanisms like signals, asynchronous I/O (AIO), select, poll, and setjmp/longjmp. Using these mechanisms can complicate program structure and make programs harder to read than sequential code.

Threads offer a cleaner and more consistent way to address these motivations. For I/O they’re usually clearer than polling or callbacks, and for processing they are more efficient than Unix processes.

Crazy bankers

Let’s get started by adding concurrency to a program to simulate a bunch of crazy bankers sending random amounts of money from one bank account to another. The bankers don’t communicate with one another, so this is a demonstration of concurrency without synchronization.

Adding concurrency is the easy part. The real work is in making threads wait for one another to ensure a correct result. We’ll see a number of mechanisms and patterns for synchronization later, but for now let’s see what goes wrong without synchronization.

/* banker.c */

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <time.h>

#define N_ACCOUNTS 10
#define N_THREADS  20
#define N_ROUNDS   10000

/* 10 accounts with $100 apiece means there's $1,000
   in the system. Let's hope it stays that way...  */
#define INIT_BALANCE 100

/* making a struct here for the benefit of future
   versions of this program */
struct account
{
	long balance;
} accts[N_ACCOUNTS];

/* Helper for bankers to choose an account and amount at
   random. It came from Steve Summit's excellent C FAQ
   http://c-faq.com/lib/randrange.html */
int rand_range(int N)
{
	return (int)((double)rand() / ((double)RAND_MAX + 1) * N);
}

/* each banker will run this function concurrently. The
   weird signature is required for a thread function */
void *disburse(void *arg)
{
	size_t i, from, to;
	long payment;

	/* idiom to tell compiler arg is unused */
	(void)arg;

	for (i = 0; i < N_ROUNDS; i++)
	{
		/* pick distinct 'from' and 'to' accounts */
		from = rand_range(N_ACCOUNTS);
		do {
			to = rand_range(N_ACCOUNTS);
		} while (to == from);

		/* go nuts sending money, try not to overdraft */
		if (accts[from].balance > 0)
		{
			payment = 1 + rand_range(accts[from].balance);
			accts[from].balance -= payment;
			accts[to].balance   += payment;
		}
	}
	return NULL;
}

int main(void)
{
	size_t i;
	long total;
	pthread_t ts[N_THREADS];

	srand(time(NULL));

	for (i = 0; i < N_ACCOUNTS; i++)
		accts[i].balance = INIT_BALANCE;

	printf("Initial money in system: %d\n",
		N_ACCOUNTS * INIT_BALANCE);

	/* start the threads, using whatever parallelism the
	   system happens to offer. Note that pthread_create
	   is the *only* function that creates concurrency */
	for (i = 0; i < N_THREADS; i++)
		pthread_create(&ts[i], NULL, disburse, NULL);

	/* wait for the threads to all finish, using the
	   pthread_t handles pthread_create gave us */
	for (i = 0; i < N_THREADS; i++)
		pthread_join(ts[i], NULL);

	for (total = 0, i = 0; i < N_ACCOUNTS; i++)
		total += accts[i].balance;

	printf("Final money in system: %ld\n", total);
}

The following simple Makefile can be used to compile all the programs in this article:

.POSIX:
CFLAGS = -std=c99 -pedantic -D_POSIX_C_SOURCE=200809L -Wall -Wextra
LDFLAGS = -lpthread

Make’s default suffix rules mean that if you have foo.c you can simply run make foo and it knows what to do without your needing to add any extra rules to the Makefile.

Data races

Try compiling and running banker.c. Notice anything strange?

Threads share memory directly. Each thread can read and write variables in shared memory without any overhead. However when threads simultaneously read and write the same data it’s called a data race and generally causes problems.

In particular, threads in banker.c have data races when they read and write account balances. The bankers program moves money between accounts, however the total amount of money in the system does not remain constant. The books don’t balance. Exactly how the program behaves depends on thread scheduling policies of the operating system. On OpenBSD the total money seldom stays at $1,000. Sometimes money gets duplicated, sometimes it vanishes. On macOS the result is generally that all the money disappears, or even becomes negative!

The property that money is neither created nor destroyed in a bank is an example of a program invariant, and it gets violated by data races. Note that parallelism is not required for a race, only concurrency.

Here’s the problematic code in the disburse() function:

payment = 1 + rand_range(accts[from].balance);
accts[from].balance -= payment;
accts[to].balance   += payment;

The threads running this code can be paused or interleaved at any time. Not just between any of the statements, but partway through arithmetic operations which may not execute atomically on the hardware. Never rely on “thread inertia,” which is the mistaken feeling that the thread will finish a group of statements without interference.

Let’s examine exactly how statements can interleave between banker threads, and the resulting problems. The columns of the table below are threads, and the rows are moments in time.

Here’s a timeline where two threads read the same account balance when planning how much money to transfer. It can cause an overdraft.

Overdrafting
Thread A Thread B
payment = 1 + rand_range(accts[from].balance);
payment = 1 + rand_range(accts[from].balance);
At this point, thread B’s payment-to-be may be in excess of the true balance because thread A has already earmarked some of the money unbeknownst to B.
accts[from].balance -= payment;
accts[from].balance -= payment;
Some of the same dollars could be transferred twice and the originating account could even go negative if the overlap of the payments is big enough.

Here’s a timeline where the debit made by one thread can be undone by that made by another.

Lost debit
Thread A Thread B
accts[from].balance -= payment; accts[from].balance -= payment;
If -= is not atomic, the threads might switch execution after reading the balance and after doing arithmetic, but before assignment. Thus one assignment would be overwritten by the other. The “lost update” creates extra money in the system.

Similar problems can occur when bankers have a data race in destination accounts. Races in the destination account would tend to decrease total money supply. (To learn more about concurrency problems, see my article Practical Guide to SQL Transaction Isolation).

Locks and deadlock

In the example above, we found that a certain section of code was vulnerable to data races. Such tricky parts of a program are called critical sections. We must ensure each thread gets all the way through the section before another thread is allowed to enter it.

To give threads mutually exclusive access to a critical section, pthreads provides the mutually exclusive lock (mutex for short). The pattern is:

pthread_mutex_lock(&some_mutex);

/* ... do things in the critical section ... */

pthread_mutex_unlock(&some_mutex);

Any thread calling pthread_mutex_lock on a previously locked mutex will go to sleep and not be scheduled until the mutex is unlocked (and any other threads already waiting on the mutex have gone first).

Another way to look at mutexes is that their job is to preserve program invariants. The critical section between locking and unlocking is a place where a certain invariant may be temporarily broken, as long as it is restored by the end. Some people recommend adding an assert() statement before unlocking, to help document the invariant. If an invariant is difficult to specify in an assertion, a comment can be useful instead.

A function is called thread-safe if multiple invocations can safely run concurrently. A cheap, but inefficient, way to make any function thread-safe is to give it its own mutex and lock it right away:

/* inefficient but effective way to protect a function */

pthread_mutex_t foo_mtx = PTHREAD_MUTEX_INITIALIZER;

void foo(/* some arguments */)
{
	pthread_mutex_lock(&foo_mtx);

	/* we're safe in here, but it's a bottleneck */

	pthread_mutex_unlock(&foo_mtx);
}

To see why this is inefficient, imagine if foo() was designed to output characters to a file specified in its arguments. Because the function takes a global lock, no two threads could run it at once, even if they wanted to write to different files. Writing to different files should be independent activities, and what we really want to protect against are two threads concurrently writing the same file.

The amount of data that a mutex protects is called its granularity, and smaller granularity can often be more efficient. In our foo() example, we could store a mutex for every file we write, and have the function choose and lock the appropriate mutex. Multi-threaded programs typically add a mutex as a member variable to data structures, to associate the lock with its data.

Let’s update the banker program to keep a mutex in each account and prevent data races.

/* banker_lock.c */

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <time.h>

#define N_ACCOUNTS 10
#define N_THREADS  100
#define N_ROUNDS   10000

struct account
{
	long balance;
	/* add a mutex to prevent races on balance */
	pthread_mutex_t mtx;
} accts[N_ACCOUNTS];

int rand_range(int N)
{
	return (int)((double)rand() / ((double)RAND_MAX + 1) * N);
}

void *disburse(void *arg)
{
	size_t i, from, to;
	long payment;

	(void)arg;

	for (i = 0; i < N_ROUNDS; i++)
	{
		from = rand_range(N_ACCOUNTS);
		do {
			to = rand_range(N_ACCOUNTS);
		} while (to == from);

		/* get an exclusive lock on both balances before
		   updating (there's a problem with this, see below) */
		pthread_mutex_lock(&accts[from].mtx);
		pthread_mutex_lock(&accts[to].mtx);
		if (accts[from].balance > 0)
		{
			payment = 1 + rand_range(accts[from].balance);
			accts[from].balance -= payment;
			accts[to].balance   += payment;
		}
		pthread_mutex_unlock(&accts[to].mtx);
		pthread_mutex_unlock(&accts[from].mtx);
	}
	return NULL;
}

int main(void)
{
	size_t i;
	long total;
	pthread_t ts[N_THREADS];

	srand(time(NULL));

	/* set the initial balance, but also create a
	   new mutex for each account */
	for (i = 0; i < N_ACCOUNTS; i++)
		accts[i] = (struct account)
			{100, PTHREAD_MUTEX_INITIALIZER};

	for (i = 0; i < N_THREADS; i++)
		pthread_create(&ts[i], NULL, disburse, NULL);

	for (i = 0; i < N_THREADS; i++)
		pthread_join(ts[i], NULL);

	for (total = 0, i = 0; i < N_ACCOUNTS; i++)
		total += accts[i].balance;

	printf("Total money in system: %ld\n", total);
}

Now everything should be safe. No money being created or destroyed, just perfect exchanges between the accounts. The invariant is that the total balance of the source and destination accounts is the same before we transfer the money as after. It’s broken only inside the critical section.

As a side note, at this point you might think it would be more efficient be to take a single lock at a time, like this:

  • lock the source account
  • withdraw money into a thread local variable
  • unlock the source account
  • (danger zone!)
  • lock the destination account
  • deposit the money
  • unlock the destination account

This would not be safe. During the time between unlocking the source account and locking the destination, the invariant does not hold, yet another thread could observe this state. For instance a report running in another thread just at that time could read the balance of both accounts and observe money missing from the system.

We do need to lock both accounts during the transfer. However the way we’re doing it causes a different problem. Try to run the program. It gets stuck forever and never prints the final balance! It’s threads are deadlocked.

Deadlock is the second villain of concurrent programming, and happens when threads wait on each others’ locks, but no thread unlocks for any other. The case of the bankers is a classic simple form called the deadly embrace. Here’s how it plays out:

Deadly embrace
Thread A Thread B
lock account 1
lock account 2
lock account 2
At this point thread A is blocked because thread B already holds a lock on account 2.
lock account 1
Now thread B is blocked because thread A holds a lock on account 1. However thread A will never unlock account 1 because thread A is blocked!

The problem happens because threads lock resources in different orders, and because they refuse to give locks up. We can solve the problem by addressing either of these causes.

The first approach to preventing deadlock is to enforce a locking hierarchy. This means the programmer comes up with an arbitrary order for locks, and always takes “earlier” locks before “later” ones. The terminology comes from locks in hierarchical data structures like trees, but it really amounts to using any kind of consistent locking order.

In our case of the banker program we store all the accounts in an array, so we can use the array index as the lock order. Let’s compare.

/* the original way to lock mutexes, which caused deadlock */

pthread_mutex_lock(&accts[from].mtx);
pthread_mutex_lock(&accts[to].mtx);
/* move money */
pthread_mutex_unlock(&accts[to].mtx);
pthread_mutex_unlock(&accts[from].mtx);

Here’s a safe way, enforcing a locking hierarchy:

/* lock mutexes in earlier accounts first */

#define MIN(a,b) ((a) < (b) ? (a) : (b))
#define MAX(a,b) ((a) < (b) ? (b) : (a))

pthread_mutex_lock(&accts[MIN(from, to)].mtx);
pthread_mutex_lock(&accts[MAX(from, to)].mtx);
/* move money */
pthread_mutex_unlock(&accts[MAX(from, to)].mtx);
pthread_mutex_unlock(&accts[MIN(from, to)].mtx);

/* notice we unlock in opposite order */

A locking hierarchy is the most efficient way to prevent deadlock, but it isn’t always easy to contrive. It’s also creates a potentially undocumented coupling between different parts of a program which need to collaborate in the convention.

Backoff is a different way to prevent deadlock which works for locks taken in any order. It takes a lock, but then checks whether the next is obtainable. If not, it unlocks the first to allow another thread to make progress, and tries again.

/* using pthread_mutex_trylock to dodge deadlock */

while (1)
{
	pthread_mutex_lock(&accts[from].mtx);
	
	if (pthread_mutex_trylock(&accts[to].mtx) == 0)
		break; /* got both locks */

	/* didn't get the second one, so unlock the first */
	pthread_mutex_unlock(&accts[from].mtx);
	/* force a sleep so another thread can try --
	   include <sched.h> for this function */
	sched_yield();
}
/* move money */
pthread_mutex_unlock(&accts[to].mtx);
pthread_mutex_unlock(&accts[from].mtx);

One tricky part is the call to sched_yield(). Without it the loop will immediately try to grab the lock again, competing as hard as it can with other threads who could make more productive use of the lock. This causes livelock, where threads fight for access to the locks. The sched_yield() puts the calling thread to sleep and at the back of the scheduler’s run queue.

Despite its flexibility, backoff is definitely less efficient than a locking hierarchy because it can make wasted calls to lock and unlock mutexes. Try modifying the banker program with these approaches and measure how fast they run.

Condition variables

After safely getting access to a shared variable with a mutex, a thread may discover that the value of the variable is not yet suitable for the thread to act upon. For instance, if the thread was looking for an item to process in a shared queue, but found the queue was empty. The thread could poll the value, but this is inefficient. Pthreads provides condition variables to allow threads to wait for events of interest or notify other threads when these events happen.

Condition variables are not themselves locks, nor do they hold any value of their own. They are merely events with a programmer-assigned meaning. For example, a structure representing a queue could have a mutex for safely accessing the data, plus some condition variables. One to represent the event of the queue becoming empty, and another to announce when a new item is added.

Before getting deeper into how condition variables work, let’s see one in action with our banker program. We’ll measure contention between the bankers. First we’ll increase the number of threads and accounts, and keep statistics about how many bankers manage to get inside the disburse() critical section at once. Any time the max score is broken, we’ll signal a condition variable. A dedicated thread will wait on it and update a scoreboard.

/* banker_stats.c */

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <time.h>

/* increase the accounts and threads, but make sure there are
 * "too many" threads so they tend to block each other */
#define N_ACCOUNTS 50
#define N_THREADS  100
#define N_ROUNDS   10000

#define MIN(a,b) ((a) < (b) ? (a) : (b))
#define MAX(a,b) ((a) < (b) ? (b) : (a))

struct account
{
	long balance;
	pthread_mutex_t mtx;
} accts[N_ACCOUNTS];

int rand_range(int N)
{
	return (int)((double)rand() / ((double)RAND_MAX + 1) * N);
}

/* keep a special mutex and condition variable
 * reserved for just the stats */
pthread_mutex_t stats_mtx = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t  stats_cnd = PTHREAD_COND_INITIALIZER;
int stats_curr = 0, stats_best = 0;

/* use this interface to modify the stats */
void stats_change(int delta)
{
	pthread_mutex_lock(&stats_mtx);
	stats_curr += delta;
	if (stats_curr > stats_best)
	{
		stats_best = stats_curr;
		/* signal new high score */
		pthread_cond_broadcast(&stats_cnd);
	}
	pthread_mutex_unlock(&stats_mtx);
}

/* a dedicated thread to update the scoreboard UI */
void *stats_print(void *arg)
{
	int prev_best;

	(void)arg;

	/* we never return, nobody needs to
	 * pthread_join() with us */
	pthread_detach(pthread_self());

	while (1)
	{
		pthread_mutex_lock(&stats_mtx);

		prev_best = stats_best;
		/* go to sleep until stats change, and always
		 * check that they actually have changed */
		while (prev_best == stats_best)
			pthread_cond_wait(
				&stats_cnd, &stats_mtx);

		/* overwrite current line with new score */
		printf("\r%2d", stats_best);
		pthread_mutex_unlock(&stats_mtx);

		fflush(stdout);
	}
}

void *disburse(void *arg)
{
	size_t i, from, to;
	long payment;

	(void)arg;

	for (i = 0; i < N_ROUNDS; i++)
	{
		from = rand_range(N_ACCOUNTS);
		do {
			to = rand_range(N_ACCOUNTS);
		} while (to == from);

		pthread_mutex_lock(&accts[MIN(from, to)].mtx);
		pthread_mutex_lock(&accts[MAX(from, to)].mtx);

		/* notice we still have a lock hierarchy, because
		 * we call stats_change() after locking all account
		 * mutexes (stats_mtx comes last) */
		stats_change(1); /* another banker in crit sec */
		if (accts[from].balance > 0)
		{
			payment = 1 + rand_range(accts[from].balance);
			accts[from].balance -= payment;
			accts[to].balance   += payment;
		}
		stats_change(-1); /* leaving crit sec */

		pthread_mutex_unlock(&accts[MAX(from, to)].mtx);
		pthread_mutex_unlock(&accts[MIN(from, to)].mtx);
	}
	return NULL;
}

int main(void)
{
	size_t i;
	long total;
	pthread_t ts[N_THREADS], stats;

	srand(time(NULL));

	for (i = 0; i < N_ACCOUNTS; i++)
		accts[i] = (struct account)
			{100, PTHREAD_MUTEX_INITIALIZER};

	for (i = 0; i < N_THREADS; i++)
		pthread_create(&ts[i], NULL, disburse, NULL);

	/* start thread to update the user on how many bankers
	 * are in the disburse() critical section at once */
	pthread_create(&stats, NULL, stats_print, NULL);

	for (i = 0; i < N_THREADS; i++)
		pthread_join(ts[i], NULL);

	/* not joining with the thread running stats_print,
	 * we'll let it disappar when main exits */

	for (total = 0, i = 0; i < N_ACCOUNTS; i++)
		total += accts[i].balance;

	printf("\nTotal money in system: %ld\n", total);
}

With fifty accounts and a hundred threads, not all threads will be able to be in the critical section of disburse() at once. It varies between runs. Run the program and see how well it does on your machine. (One complication is that making all threads synchronize on stats_mtx may throw off the measurement, because there are threads who could have executed independently but now must interact.)

Let’s look at how to properly use condition variables. We notified threads of a new event with pthread_cond_broadcast(&stats_cnd). This function marks all threads waiting on state_cnd as ready to run.

Sometimes multiple threads are waiting on a single cond var. A broadcast will wake them all, but sometimes the event source knows that only one thread will be able to do any work. For instance if only one item is added to a shared queue. In that case the pthread_cond_signal function is better than pthread_cond_broadcast. Unnecessarily waking multiple threads causes overhead. In our case we know that only one thread is waiting on the cond var, so it really makes no difference.

Remember that it’s never wrong to use a broadcast, whereas in some cases it might be wrong to use a signal. Signal is just an optimized broadcast.

The waiting side of a cond var ought always to have this pattern:

pthread_mutex_lock(&mutex);
while (!PREDICATE)
	pthread_cond_wait(&cond_var, &mutex);
pthread_mutex_unlock(&mutex);

Condition variables are always associated with a predicate, and the association is implicit in the programmer’s head. You shouldn’t reuse a condition variable for multiple predicates. The intention is that code will signal the cond var when the predicate becomes true.

Before testing the predicate we lock a mutex that covers the data being tested. That way no other thread can change the data immediately after we test it (also pthread_cond_wait() requires a locked mutex). If the predicate is already true we needn’t wait on the cond var, so the loop falls through, otherwise the thread begins to wait.

Condition variables allow you to make this series of events atomic: unlock a mutex, register our interest in the event, and block. Without that atomicity another thread might awaken to take our lock and broadcast before we’ve registered ourselves as interested. Without the atomicity we could be blocked forever.

When pthread_cond_wait() returns, the calling thread awakens and atomically gets its mutex back. It’s all set to check the predicate again in the loop. But why check the predicate? Wasn’t the cond var signaled because the predicate was true, and isn’t the relevant data protected by a mutex? There are three reasons to check:

  1. If the condition variable had been broadcast, other threads might have been listening, and another might have been scheduled first and might have done our job. The loop tests for that interception.
  2. On some multiprocessor systems, making condition variable wakeup completely predictable might substantially slow down all cond var operations. Such systems allow spurious wakeups, and threads need to be prepared to check if they were woken appropriately.
  3. It can be convenient to signal on a loose predicate. Threads can signal the variables when the event seems likely, or even mistakenly signal, and the program will still work. For instance, we signal when when stats_best gets a new high score, but we could have chosen to signal at every invocation of stats_change().

Given that we have to pass a locked mutex to pthread_cond_wait(), which we had to create, why don’t cond vars come with their own built-in mutex? The reason is flexibility. Although you should use only one mutex with a cond var, there can be multiple cond vars for the same mutex. Think of the example of the mutex protecting a queue, and the different events that can happen in the queue.

Other synchronization primitives

Barriers

It’s time to bid farewell to the banker programs, and turn to something more lively: Conway’s Game of Life! The game has a set of rules operating on a grid of cells that determines which cells live or die based on how many living neighbors each has.

The game can take advantage of multiple processors, using each processor to operate on a different part of the grid in parallel. It’s a so-called embarrassingly parallel problem because each section of the grid can be processed in isolation, without needing results from other sections.

Barriers ensure that all threads have reached a particular stage in a parallel computation before allowing any to proceed to the next stage. Each thread calls pthread_barrier_wait() to rendezvous with the others. One of the threads, chosen randomly, will see the PTHREAD_BARRIER_SERIAL_THREAD return value, which nominates that thread to do any cleanup or preparation between stages.

/* life.c */

#include <assert.h>
#include <pthread.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

/* mandatory in POSIX.1-2008, but check laggards like macOS */
#include <unistd.h>
#if !defined(_POSIX_BARRIERS) || _POSIX_BARRIERS < 0
#error your OS lacks POSIX barrier support
#endif

/* dimensions of board */
#define ROWS 32
#define COLS 78
/* how long to pause between rounds */
#define FRAME_MS 100
#define THREADS 4

/* proper modulus (in C, '%' is merely remainder) */
#define MOD(x,N) (((x) < 0) ? ((x) % (N) + (N)) : ((x) % (N)))

bool alive[ROWS][COLS], alive_next[ROWS][COLS];
pthread_barrier_t tick;

/* Should a cell live or die? Using ssize_t because we have
   to deal with signed arithmetic like row-1 when row=0 */
bool fate(ssize_t row, ssize_t col)
{
	ssize_t i, j;
	short neighbors = 0;

	assert(0 <= row && row < ROWS);
	assert(0 <= col && col < COLS);

	/* joined edges form a torus */
	for (i = row-1; i <= row+1; i++)
		for (j = col-1; j <= col+1; j++)
			neighbors += alive[MOD(i, ROWS)][MOD(j, COLS)];
	/* don't count self as a neighbor */
	neighbors -= alive[row][col];

	return neighbors == 3 ||
		(neighbors == 2 && alive[row][col]);
}

/* overwrite the board on screen */
void draw(void)
{
	ssize_t i, j;

	/* clear screen (non portable, requires ANSI terminal) */
	fputs("\033[2J\033[1;1H", stdout);

	flockfile(stdout);
	for (i = 0; i < ROWS; i++)
	{
		/* putchar_unlocked is thread safe when stdout is locked,
		   and it's as fast as single-threaded putchar */
		for (j = 0; j < COLS; j++)
			putchar_unlocked(alive[i][j] ? 'X' : ' ');
		putchar_unlocked('\n');
	}
	funlockfile(stdout);
	fflush(stdout);
}

void *update_strip(void *arg)
{
	ssize_t offset = *(ssize_t*)arg, i, j;
	struct timespec t;

	t.tv_sec = 0;
	t.tv_nsec = FRAME_MS * 1000000;

	while (1)
	{
		if (pthread_barrier_wait(&tick) ==
			PTHREAD_BARRIER_SERIAL_THREAD)
		{
			/* we drew the short straw, so we're on graphics duty */

			/* could have used pointers to multidimensional
			 * arrays and swapped them rather than memcpy'ing
			 * the array contents, but it makes the code a
			 * little more complicated with dereferences */
			memcpy(alive, alive_next, sizeof alive);
			draw();
			nanosleep(&t, NULL);
		}

		/* rejoin at another barrier to avoid data race on
		   the game board while it's copied and drawn */
		pthread_barrier_wait(&tick);
		for (i = offset; i < offset + (ROWS / THREADS); i++)
			for (j = 0; j < COLS; j++)
				alive_next[i][j] = fate(i, j);
	}

	return NULL;
}

int main(void)
{
	pthread_t *workers;
	ssize_t *offsets;
	size_t i, j;

	assert(ROWS % THREADS == 0);
	/* main counts as a thread, so need only THREADS-1 more */
	workers = malloc(sizeof(*workers) * (THREADS-1));
	offsets = malloc(sizeof(*offsets) * ROWS / THREADS);

	srand(time(NULL));
	for (i = 0; i < ROWS; i++)
		for (j = 0; j < COLS; j++)
			alive_next[i][j] = rand() < (int)((RAND_MAX+1u) / 3);

	pthread_barrier_init(&tick, NULL, THREADS);
	for (i = 0; i < THREADS-1; i++)
	{
		offsets[i] = i * ROWS / THREADS;
		pthread_create(&workers[i], NULL, update_strip, &offsets[i]);
	}

	/* use current thread as a worker too */
	offsets[i] = i * ROWS / THREADS;
	update_strip(&offsets[i]);

	/* shouldn't ever get here */
	pthread_barrier_destroy(&tick);
	free(offsets);
	free(workers);
	return EXIT_SUCCESS;
}

It’s a fun example although slightly contrived. We’re adding a sleep between rounds to slow down the animation, so it’s unnecessary to chase parallelism. Also there’s a memoized algorithm called hashlife we should be using if pure speed is the goal. However our code illustrates a natural use for barriers.

Notice how we wait at the barrier twice in rapid succession. After emerging from the first barrier, one of the threads (chosen at random) copies the new state to the board and draws it. The other threads run ahead to the next barrier and wait there so they don’t cause a data race writing to the board. Once the drawing thread arrives at the barrier with them, then all can proceed to calculate cells’ fate for the next round.

Barriers are guaranteed to be present in POSIX.1-2008, but are optional in earlier versions of the standard. Notably macOS is stuck at an old version of POSIX. Presumably they’re too busy “innovating” with their keyboard touchbar to invest in operating system fundamentals.

Spinlocks

Spinlocks are implementations of mutexes optimized for fine-grained locking. Often used in low level code like drivers or operating systems, spinlocks are designed to be the most primitive and fastest sync mechanism available. They’re generally not appropriate for application programming. They are only truly necessary for situations like interrupt handlers when a thread is not allowed to go to sleep for any reason.

Aside from that scenario, it’s better to just use a mutex, since mutexes are pretty efficient these days. Modern mutexes often try a short-lived internal spinlock and fall back to heavier techniques only as needed. Mutexes also sometimes use a wait queue called a futex, which can take a lock in user-space whenever there is no contention from another thread.

When attempting to lock a spinlock, a thread runs a tight loop repeatedly checking a value in shared memory for a sign it’s safe to proceed. Spinlock implementations use special atomic assembly language instructions to test that the value is unlocked and lock it. The particular instructions vary per architecture, and can be performed in user space to avoid the overhead of a system call.

The while waiting for a lock, the loop doesn’t block the thread, but instead continues running and burns CPU energy. The technique works only on true multi-processor systems or a uniprocessor system with preemption enabled. On a uniprocessor system with cooperative threading the loop could never be interrupted, and will livelock.

In POSIX.1-2008 spinlock support is mandatory. In previous versions the presence of this feature was indicated by the _POSIX_SPIN_LOCKS macro. Spinlock functions start with pthread_spin_.

Reader-writer locks

Whereas a mutex enforces mutual exclusion, a reader-writer lock allows concurrent read access. Multiple threads can read in parallel, but all block when a thread takes the lock for writing. The increased concurrency can improve application performance. However, blindly replacing mutexes with reader-writer locks “for performance” doesn’t work. Our earlier banker program, for instance, could suffer from duplicate withdrawals if it allowed multiple readers in an account at once.

Below is an rwlock example. It’s a password cracker I call 5dm (md5 backwards). It aims for maximum parallelism searching for a preimage of an MD5 hash. Worker threads periodically poll whether one among them has found an answer, and they use a reader-writer lock to avoid blocking on each other when doing so.

The example is slightly contrived, in that the difficulty of brute forcing passwords increases exponentially with their length. Using multiple threads reduces the time by only a constant factor – but 4x faster is still 4x faster on a four core computer!

The example below uses MD5() from OpenSSL. To build it, include pkg-config --cflags libcrypto in the CFLAGS and pkg-config --libs libcrypto in LDFLAGS. To run it, pass in an MD5 hash and max preimage search length. Note the -n in echo to suppress the newline, since newline is not in our search alphabet:

$ time ./5dm $(echo -n 'fun' | md5) 5
fun

real  0m0.067s
user  0m0.205s
sys	  0m0.007s

Notice how 0.2 seconds of CPU time elapsed in parallel, but the user got their answer in 0.067 seconds.

On to the code:

/* 5dm.c */

#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <openssl/md5.h>
#include <pthread.h>

/* build arbitrary words from the ascii between ' ' and '~' */
#define ASCII_FIRST ' '
#define ASCII_LAST  '~'
#define N_ALPHA (1 + ASCII_LAST - ASCII_FIRST)
/* refuse to search beyond this astronomical length */
#define LONGEST_PREIMAGE 128

#define MAX(x,y) ((x)<(y) ? (y) : (x))

/* a fast way to enumerate words, operating on an array in-place */
unsigned word_advance(char *word, unsigned delta)
{
	if (delta == 0)
		return 0;
	if (*word == '\0')
	{
		*word++ = ASCII_FIRST + delta - 1;
		*word = '\0';
	}
	else
	{
		char c = *word - ASCII_FIRST;
		*word = ASCII_FIRST + ((c + delta) % N_ALPHA);
		if (c + delta >= N_ALPHA)
			return 1 + word_advance(word+1, 1 /* not delta */);
	}
	return 1;
}

/* pack each pair of ASCII hex digits into single bytes */
bool hex2md5(const char *hex, unsigned char *b)
{
	int offset = 0;
	if(strlen(hex) != MD5_DIGEST_LENGTH*2)
		return false;
	while (offset < MD5_DIGEST_LENGTH*2)
	{
		if (sscanf(hex+offset, "%2hhx", b++) == 1)
			offset += 2;
		else
			return false;
	}
	return true;
}

/* random things a worker will need, since thread
 * functions receive only one argument */
struct goal
{
	/* input */
	pthread_t *workers;
	size_t n_workers;
	size_t max_len;
	unsigned char hash[MD5_DIGEST_LENGTH];

	/* output */
	pthread_rwlock_t lock;
	char preimage[LONGEST_PREIMAGE];
	bool success;
};

/* custom starting word for each worker, but shared goal */
struct task
{
	struct goal *goal;
	char initial_preimage[LONGEST_PREIMAGE];
};

void *crack_thread(void *arg)
{
	struct task *t = arg;
	unsigned len, changed;
	unsigned char hashed[MD5_DIGEST_LENGTH];
	char preimage[LONGEST_PREIMAGE];
	int iterations = 0;

	strcpy(preimage, t->initial_preimage);
	len = strlen(preimage);

	while (len <= t->goal->max_len)
	{
		MD5((const unsigned char*)preimage, len, hashed);
		if (memcmp(hashed, t->goal->hash, MD5_DIGEST_LENGTH) == 0)
		{
			/* success -- tell others to call it off */
			pthread_rwlock_wrlock(&t->goal->lock);

			t->goal->success = true;
			strcpy(t->goal->preimage, preimage);

			pthread_rwlock_unlock(&t->goal->lock);
			return NULL;
		}
		/* each worker jumps ahead n_workers words, and all workers
		   started at an offset, so all words are covered */
		changed = word_advance(preimage, t->goal->n_workers);
		len = MAX(len, changed);

		/* check if another worker has succeeded, but only every
		   thousandth iteration, since taking the lock adds overhead */
		if (iterations++ % 1000 == 0)
		{
			/* in the overwhelming majority of cases workers only read,
			   so an rwlock allows them to continue in parallel */
			pthread_rwlock_rdlock(&t->goal->lock);
			int success = t->goal->success;
			pthread_rwlock_unlock(&t->goal->lock);
			if (success)
				return NULL;
		}
	}
	return NULL;
}

/* launch a parallel search for an md5 preimage */
bool crack(const unsigned char *md5, size_t max_len,
           unsigned threads, char *result)
{
	struct goal g =
	{
		.workers   = malloc(threads * sizeof(pthread_t)),
		.n_workers = threads,
		.max_len   = max_len,
		.success   = false,
		.lock      = PTHREAD_RWLOCK_INITIALIZER
	};
	memcpy(g.hash, md5, MD5_DIGEST_LENGTH);

	struct task *tasks = malloc(threads * sizeof(struct task));

	for (size_t i = 0; i < threads; i++)
	{
		tasks[i].goal = &g;
		tasks[i].initial_preimage[0] = '\0';
		/* offset the starting word for each worker by i */
		word_advance(tasks[i].initial_preimage, i);
		pthread_create(g.workers+i, NULL, crack_thread, tasks+i);
	}

	/* if one worker finds the answer, others will abort */
	for (size_t i = 0; i < threads; i++)
		pthread_join(g.workers[i], NULL);

	if (g.success)
		strcpy(result, g.preimage);

	free(tasks);
	free(g.workers);
	return g.success;
}

int main(int argc, char **argv)
{
	char preimage[LONGEST_PREIMAGE];
	int max_len = 4;
	unsigned char md5[MD5_DIGEST_LENGTH];

	if (argc != 2 && argc != 3)
	{
		fprintf(stderr,
		        "Usage: %s md5-string [search-depth]\n",
		        argv[0]);
		return EXIT_FAILURE;
	}

	if (!hex2md5(argv[1], md5))
	{
		fprintf(stderr,
		       "Could not parse as md5: %s\n", argv[1]);
		return EXIT_FAILURE;
	}

	if (argc > 2 && strtol(argv[2], NULL, 10))
		if ((max_len = strtol(argv[2], NULL, 10)) > LONGEST_PREIMAGE)
		{
			fprintf(stderr,
					"Preimages limited to %d characters\n",
					LONGEST_PREIMAGE);
			return EXIT_FAILURE;
		}

	if (crack(md5, max_len, 4, preimage))
	{
		puts(preimage);
		return EXIT_SUCCESS;
	}
	else
	{
		fprintf(stderr,
				"Could not find result in strings up to length %d\n",
		        max_len);
		return EXIT_FAILURE;
	}
}

Although read-write locks can be implemented in terms of mutexes and condition variables, such implementations are significantly less efficient than is possible. Therefore, this synchronization primitive is included in POSIX.1-2008 for the purpose of allowing more efficient implementations in multi-processor systems.

The final thing to be aware of is that an rwlock implementation can choose either reader-preference or writer-preference. When readers and writers are contending for a lock, the preference determines who gets to skip the queue and go first. When there is a lot of reader activity with a reader-preference, then a writer will continually get moved to the end of the line and experience starvation, where it never gets to write. I noticed writer starvation on Linux (glibc) when running four threads on a little 1-core virtual machine. Glibc provides the nonportable pthread_rwlockattr_setkind_np() function to specify a preference.

You may have noticed that workers in our password cracker use polling to see whether the solution has been found, and whether they should give up. We’ll examine a more explicit method of cancellation in a later section.

Semaphores

Semaphores keep count of, in the abstract, an amount of resource “units” available. Threads can safely add or remove a unit without causing a data race. When a thread requests a unit but there are none, then the thread will block.

A semaphore is like a mix between a lock and a condition variable. Unlike mutexes, semaphores have no concept of an owner. Any thread may release threads blocked on a semaphore, whereas with a mutex the lock holder must unlock it. Unlike a condition variable, a semaphore operates independently of a predicate.

An example of a problem uniquely suited for semaphores would be to ensure that exactly two threads run at once on a task. You would initialize the semaphore to the value two, and allow a bunch of threads to wait on the semaphore. After two get past, the rest will block. When each thread is done, it posts one unit back to the semaphore, which allows another thread to take its place.

In reality, if you’ve got pthreads, you only need semaphores for asynchronous signal handlers. You can use them in other situations, but this is the only place they are needed. Mutexes aren’t async signal safe. Making them so would be much slower than an implementation that isn’t async signal safe, and would slow down ordinary mutex operation.

Here’s an example of posting a semaphore from a signal handler:

/* sem_tickler.c */

#include <semaphore.h>
#include <signal.h>
#include <stdio.h>

#include <unistd.h>
#if !defined(_POSIX_SEMAPHORES) || _POSIX_SEMAPHORES < 0
#error your OS lacks POSIX semaphore support
#endif

sem_t tickler;

void int_catch(int sig)
{
	(void) sig;

	signal(SIGINT, &int_catch);
	sem_post(&tickler); /* async signal safe: */
}

int main(void)
{
	sem_init(&tickler, 0, 0);
	signal(SIGINT, &int_catch);

	for (int i = 0; i < 3; i++)
	{
		sem_wait(&tickler);
		puts("That tickles!");
	}
	puts("(Died from overtickling)");
	return 0;
}

Semaphores aren’t even necessary for proper signal handling. It’s easier to have a thread simply sigwait() than it is to set up an asynchronous handler. In the example below, the main thread waits, but you can spawn a dedicated thread for this in a real application.

/* sigwait_tickler.c */

#include <signal.h>
#include <stdio.h>

int main(void)
{
	sigset_t set;
	int which;
	sigemptyset(&set);
	sigaddset(&set, SIGINT);

	for (int i = 0; i < 3; i++)
	{
		sigwait(&set, &which);
		puts("That tickles!");
	}
	puts("(Died from overtickling)");
	return 0;
}

So don’t feel dependent on semaphores. In fact your system may not have them. The POSIX semaphore API works with pthreads and is present in POSIX.1-2008, but is an optional part of POSIX.1b in earlier versions. Apple, for one, decided to punt, so the semaphore functions on macOS are stubbed to return error codes.

Cancellation

Thread cancellation is generally used when you have threads doing long-running tasks and there’s a way for a user to abort through the UI or console. Another common scenario is when multiple threads set off to explore a search space and one finds the answer first.

Our previous reader-writer lock example was the second scenario, where the threads explored a search space. It was an example of do-it-yourself cancellation through polling. However sometimes threads aren’t able to poll, such as when they are blocked on I/O or a lock. Pthreads offers an API to cancel threads even in those situations.

By default a cancelled thread isn’t immediately blown away, because it may have a mutex locked, be holding resources, or have a potentially broken invariant. The canceller wouldn’t know how to repair that invariant without some complicated logic. The thread to be canceled needs to be written to do cleanup and unlock mutexes.

For each thread, cancellation can be enabled or disabled, and if enabled, may be in deferred or asynchronous mode. The default is enabled and deferred, which allows a cancelled thread to survive until the next cancellation points, such as waiting on a condition variable or blocking on IO (see full list). In a purely computational section of code you can add your own cancellation points with pthread_testcancel().

Let’s see how to modify our previous MD5 cracking example using standard pthread cancellation. Three of the functions are the same as before: word_advance(), hex2md5(), and main(). But we now use a condition variable to alert crack() whenever a crack_thread() returns.

/* 5dm-testcancel.c */

#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <openssl/md5.h>
#include <pthread.h>

#define ASCII_FIRST ' '
#define ASCII_LAST  '~'
#define N_ALPHA (1 + ASCII_LAST - ASCII_FIRST)
#define LONGEST_PREIMAGE 128

#define MAX(x,y) ((x)<(y) ? (y) : (x))

unsigned word_advance(char *word, unsigned delta)
{
	if (delta == 0)
		return 0;
	if (*word == '\0')
	{
		*word++ = ASCII_FIRST + delta - 1;
		*word = '\0';
	}
	else
	{
		char c = *word - ASCII_FIRST;
		*word = ASCII_FIRST + ((c + delta) % N_ALPHA);
		if (c + delta >= N_ALPHA)
			return 1 + word_advance(word+1, 1 /* not delta */);
	}
	return 1;
}

bool hex2md5(const char *hex, unsigned char *b)
{
	int offset = 0;
	if(strlen(hex) != MD5_DIGEST_LENGTH*2)
		return false;
	while (offset < MD5_DIGEST_LENGTH*2)
	{
		if (sscanf(hex+offset, "%2hhx", b++) == 1)
			offset += 2;
		else
			return false;
	}
	return true;
}

struct goal
{
	/* input */
	pthread_t *workers;
	size_t n_workers;
	size_t max_len;
	unsigned char hash[MD5_DIGEST_LENGTH];

	/* output */
	pthread_mutex_t lock;
	pthread_cond_t returning;
	unsigned n_done;
	char preimage[LONGEST_PREIMAGE];
	bool success;
};

struct task
{
	struct goal *goal;
	char initial_preimage[LONGEST_PREIMAGE];
};

void *crack_thread(void *arg)
{
	struct task *t = arg;
	unsigned len, changed;
	unsigned char hashed[MD5_DIGEST_LENGTH];
	char preimage[LONGEST_PREIMAGE];
	int iterations = 0;

	strcpy(preimage, t->initial_preimage);
	len = strlen(preimage);

	while (len <= t->goal->max_len)
	{
		MD5((const unsigned char*)preimage, len, hashed);
		if (memcmp(hashed, t->goal->hash, MD5_DIGEST_LENGTH) == 0)
		{
			pthread_mutex_lock(&t->goal->lock);

			t->goal->success = true;
			strcpy(t->goal->preimage, preimage);
			t->goal->n_done++;

			/* alert the boss that another worker is done */
			pthread_cond_signal(&t->goal->returning);
			pthread_mutex_unlock(&t->goal->lock);
			return NULL;
		}
		changed = word_advance(preimage, t->goal->n_workers);
		len = MAX(len, changed);

		if (iterations++ % 1000 == 0)
			pthread_testcancel(); /* add a cancellation point */
	}

	pthread_mutex_lock(&t->goal->lock);
	t->goal->n_done++;
	/* alert the boss that another worker is done */
	pthread_cond_signal(&t->goal->returning);
	pthread_mutex_unlock(&t->goal->lock);
	return NULL;
}

/* cancellation cleanup function that we also call
 * during regular exit from the crack() function */
void crack_cleanup(void *arg)
{
	struct task *tasks = arg;
	struct goal *g = tasks[0].goal;

	/* this mutex unlock pairs with the lock in the crack() function */
	pthread_mutex_unlock(&g->lock);
	for (size_t i = 0; i < g->n_workers; i++)
	{
		pthread_cancel(g->workers[i]);
		/* must wait for each to terminate, so that freeing
		 * their shared memory is safe */
		pthread_join(g->workers[i], NULL);
	}
	/* now it's safe to free memory */
	free(g->workers);
	free(tasks);
}

bool crack(const unsigned char *md5, size_t max_len,
           unsigned threads, char *result)
{
	struct goal g =
	{
		.workers   = malloc(threads * sizeof(pthread_t)),
		.n_workers = threads,
		.max_len   = max_len,
		.success   = false,
		.n_done    = 0,
		.lock      = PTHREAD_MUTEX_INITIALIZER,
		.returning = PTHREAD_COND_INITIALIZER
	};
	memcpy(g.hash, md5, MD5_DIGEST_LENGTH);

	struct task *tasks = malloc(threads * sizeof(struct task));

	for (size_t i = 0; i < threads; i++)
	{
		tasks[i].goal = &g;
		tasks[i].initial_preimage[0] = '\0';
		word_advance(tasks[i].initial_preimage, i);
		pthread_create(g.workers+i, NULL, crack_thread, tasks+i);
	}

	/* coming up to cancellation points, so establish
	 * a cleanup handler */
	pthread_cleanup_push(crack_cleanup, tasks);

	pthread_mutex_lock(&g.lock);
	/* We can't join() on all the workers now because it's up to
	 * us to cancel them after one finds the answer. We have to
	 * remain responsive and not block on any particular worker */
	while (!g.success && g.n_done < threads)
		pthread_cond_wait(&g.returning, &g.lock);
	/* at this point either a thread succeeded or all have given up */
	if (g.success)
		strcpy(result, g.preimage);
	/* mutex unlocked in the cleanup handler */

	/* Use the same cleanup handler for normal exit too. The "1"
	 * argument says to execute the function we had previous pushed */
	pthread_cleanup_pop(1);
	return g.success;
}

int main(int argc, char **argv)
{
	char preimage[LONGEST_PREIMAGE];
	int max_len = 4;
	unsigned char md5[MD5_DIGEST_LENGTH];

	if (argc != 2 && argc != 3)
	{
		fprintf(stderr,
		        "Usage: %s md5-string [search-depth]\n",
		        argv[0]);
		return EXIT_FAILURE;
	}

	if (!hex2md5(argv[1], md5))
	{
		fprintf(stderr,
		       "Could not parse as md5: %s\n", argv[1]);
		return EXIT_FAILURE;
	}

	if (argc > 2 && strtol(argv[2], NULL, 10))
		if ((max_len = strtol(argv[2], NULL, 10)) > LONGEST_PREIMAGE)
		{
			fprintf(stderr,
					"Preimages limited to %d characters\n",
					LONGEST_PREIMAGE);
			return EXIT_FAILURE;
		}

	if (crack(md5, max_len, 4, preimage))
	{
		puts(preimage);
		return EXIT_SUCCESS;
	}
	else
	{
		fprintf(stderr,
				"Could not find result in strings up to length %d\n",
		        max_len);
		return EXIT_FAILURE;
	}
}

Using cancellation is actually a little more flexible than our rwlock implementation in 5dm. If the crack() function is running in its own thread, the whole thing can now be cancelled. The cancellation handler will “pass along” the cancellation to each of the worker threads.

Writing general purpose library code that works with threads requires some care. It should handle deferred cancellation gracefully, including disabling cancellation when appropriate and always using cleanup handlers.

For cleanup handlers, notice the pattern of how we pthread_cleanup_push() the cancellation handler, and later pthread_cleanup_pop() it for regular (non-cancel) cleanup too. Using the same cleanup procedure in all situations makes the code more reliable.

Also notice how the boss thread now cancels workers, rather than the winning worker cancelling the others. You can join a canceled thread, but you can’t cancel an already joined (or detached) thread. If you want to both cancel and join a thread it ought to be done in one place.

Let’s turn out attention to the new worker threads. They are still polling for cancellation, like they polled with the reader-writer locks, but in this case they do it with a new function:

if (iterations++ % 1000 == 0)
	pthread_testcancel();

Admittedly it adds a little overhead to poll every thousandth loop, both with the rwlock, and with the testcancel. It also adds latency to the time between the cancellation request and the thread quitting, since the loop could run up to 999 times in between. A more efficient but dangerous method is to enable asynchronous cancellation, meaning the thread immediately dies when cancelled.

Async cancellation is dangerous because code is seldom async-cancel-safe. Anything that uses locks or works with shared state even slightly can break badly. Async-cancel-safe code can call very few functions, since those functions may not be safe. This includes calling libraries that use something as innocent as malloc(), since stopping malloc part way through could corrupt the heap.

Our crack_thread() function should be async-cancel-safe, at least during its calculation and not when taking locks. The MD5() function from OpenSSL also appears to be safe. Here’s how we can rewrite our function (notice how we disable cancellation before taking a lock):

/* rewritten to use async cancellation */

void *crack_thread(void *arg)
{
	struct task *t = arg;
	unsigned len, changed;
	unsigned char hashed[MD5_DIGEST_LENGTH];
	char preimage[LONGEST_PREIMAGE];
	int cancel_type, cancel_state;

	strcpy(preimage, t->initial_preimage);
	len = strlen(preimage);

	/* async so we don't have to pthread_testcancel() */
	pthread_setcanceltype(
			PTHREAD_CANCEL_ASYNCHRONOUS, &cancel_type);

	while (len <= t->goal->max_len)
	{
		MD5((const unsigned char*)preimage, len, hashed);
		if (memcmp(hashed, t->goal->hash, MD5_DIGEST_LENGTH) == 0)
		{
			/* protect the mutex against async cancellation */
			pthread_setcancelstate(
					PTHREAD_CANCEL_DISABLE, &cancel_state);
			pthread_mutex_lock(&t->goal->lock);

			t->goal->success = true;
			strcpy(t->goal->preimage, preimage);
			t->goal->n_done++;

			pthread_cond_signal(&t->goal->returning);
			pthread_mutex_unlock(&t->goal->lock);
			return NULL;
		}
		changed = word_advance(preimage, t->goal->n_workers);
		len = MAX(len, changed);
	}

	/* restore original cancellation type */
	pthread_setcanceltype(cancel_type, &cancel_type);

	pthread_mutex_lock(&t->goal->lock);
	t->goal->n_done++;
	pthread_cond_signal(&t->goal->returning);
	pthread_mutex_unlock(&t->goal->lock);
	return NULL;
}

Asynchronous cancellation does not appear to work on macOS, but as we’ve seen that’s par for the course on that operating system.

Development tools

Valgrind DRD and helgrind

DRD and Helgrind are Valgrind tools for detecting errors in multithreaded C and C++ programs. The tools work for any program that uses the POSIX threading primitives or that uses threading concepts built on top of the POSIX threading primitives.

The tools have overlapping abilities like detecting data races and improper use of the pthreads API. Additionally, Helgrind can detect locking hierarchy violations, and DRD can alert when there is lock contention.

Both tools pinpoint the lines of code where problems arise. For example, we can run DRD on our first crazy bankers program:

valgrind --tool=drd ./banker

Here is a characteristic example of an error it emits:

==8524== Thread 3:
==8524== Conflicting load by thread 3 at 0x003090b0 size 8
==8524==    at 0x1088BD: disburse (banker.c:48)
==8524==    by 0x4C324F3: vgDrd_thread_wrapper (drd_pthread_intercepts.c:444)
==8524==    by 0x4E514A3: start_thread (pthread_create.c:456)
==8524== Allocation context: BSS section of /home/admin/banker
==8524== Other segment start (thread 2)
==8524==    at 0x514FD01: clone (clone.S:80)
==8524== Other segment end (thread 2)
==8524==    at 0x509D820: rand (rand.c:26)
==8524==    by 0x108857: rand_range (banker.c:26)
==8524==    by 0x1088A0: disburse (banker.c:42)
==8524==    by 0x4C324F3: vgDrd_thread_wrapper (drd_pthread_intercepts.c:444)
==8524==    by 0x4E514A3: start_thread (pthread_create.c:456)

It finds conflicting loads and stores from lines 48, 51, and 52.

48: if (accts[from].balance > 0)
49: {
50:		payment = 1 + rand_range(accts[from].balance);
51:		accts[from].balance -= payment;
52:		accts[to].balance   += payment;
53: }

Helgrind can identify the lock hierarchy violation in our example of deadlocking bankers:

valgrind --tool=helgrind ./banker_lock
==8989== Thread #4: lock order "0x3091F8 before 0x3090D8" violated
==8989==
==8989== Observed (incorrect) order is: acquisition of lock at 0x3090D8
==8989==    at 0x4C3010C: mutex_lock_WRK (hg_intercepts.c:904)
==8989==    by 0x1089B9: disburse (banker_lock.c:38)
==8989==    by 0x4C32D06: mythread_wrapper (hg_intercepts.c:389)
==8989==    by 0x4E454A3: start_thread (pthread_create.c:456)
==8989==
==8989==  followed by a later acquisition of lock at 0x3091F8
==8989==    at 0x4C3010C: mutex_lock_WRK (hg_intercepts.c:904)
==8989==    by 0x1089D1: disburse (banker_lock.c:39)
==8989==    by 0x4C32D06: mythread_wrapper (hg_intercepts.c:389)
==8989==    by 0x4E454A3: start_thread (pthread_create.c:456)

To identify when there is too much contention for a lock, we can ask DRD to alert us when a thread blocks for more than n milliseconds on a mutex:

valgrind --tool=drd --exclusive-threshold=2 ./banker_lock_hierarchy

Since we throw too many threads at a small number of accounts, we see wait times that cross the threshold, like this one that waited seven ms:

==7565== Acquired at:
==7565==    at 0x483F428: pthread_mutex_lock_intercept (drd_pthread_intercepts.c:888)
==7565==    by 0x483F428: pthread_mutex_lock (drd_pthread_intercepts.c:898)
==7565==    by 0x109280: disburse (banker_lock_hierarchy.c:40)
==7565==    by 0x483C114: vgDrd_thread_wrapper (drd_pthread_intercepts.c:444)
==7565==    by 0x4863FA2: start_thread (pthread_create.c:486)
==7565==    by 0x49764CE: clone (clone.S:95)
==7565== Lock on mutex 0x10c258 was held during 7 ms (threshold: 2 ms).
==7565==    at 0x4840478: pthread_mutex_unlock_intercept (drd_pthread_intercepts.c:978)
==7565==    by 0x4840478: pthread_mutex_unlock (drd_pthread_intercepts.c:991)
==7565==    by 0x109395: disburse (banker_lock_hierarchy.c:47)
==7565==    by 0x483C114: vgDrd_thread_wrapper (drd_pthread_intercepts.c:444)
==7565==    by 0x4863FA2: start_thread (pthread_create.c:486)
==7565==    by 0x49764CE: clone (clone.S:95)
==7565== mutex 0x10c258 was first observed at:
==7565==    at 0x483F368: pthread_mutex_lock_intercept (drd_pthread_intercepts.c:885)
==7565==    by 0x483F368: pthread_mutex_lock (drd_pthread_intercepts.c:898)
==7565==    by 0x109280: disburse (banker_lock_hierarchy.c:40)
==7565==    by 0x483C114: vgDrd_thread_wrapper (drd_pthread_intercepts.c:444)
==7565==    by 0x4863FA2: start_thread (pthread_create.c:486)
==7565==    by 0x49764CE: clone (clone.S:95)

Clang ThreadSanitizer (TSan)

ThreadSanitizer is a clang instrumentation module. To use it, choose CC = clang and add -fsanitize=thread to CFLAGS. Then when you build programs, they will be modified to detect data races and print statistics to stderr.

Here’s a portion of the output when running the bankers program:

WARNING: ThreadSanitizer: data race (pid=11312)
  Read of size 8 at 0x0000014aeeb0 by thread T2:
    #0 disburse /home/admin/banker.c:48 (banker+0x0000004a4372)

  Previous write of size 8 at 0x0000014aeeb0 by thread T1:
    #0 disburse /home/admin/banker.c:52 (banker+0x0000004a43ba)

TSan can also detect lock hierarchy violations, such as in banker_lock:

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=10095)
  Cycle in lock order graph: M1 (0x0000014aef78) => M2 (0x0000014aeeb8) => M1

  Mutex M2 acquired here while holding mutex M1 in thread T1:
    #0 pthread_mutex_lock <null> (banker_lock+0x000000439a10)
    #1 disburse /home/admin/banker_lock.c:39 (banker_lock+0x0000004a4398)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M1 acquired here while holding mutex M2 in thread T9:
    #0 pthread_mutex_lock <null> (banker_lock+0x000000439a10)
    #1 disburse /home/admin/banker_lock.c:39 (banker_lock+0x0000004a4398)

Mutrace

While Valgrind DRD can identify highly contended locks, it virtualizes the execution of the program under test, and skews the numbers. Other utilities can use software probes to get this information from a test running at full speed. In BSD land there is the plockstat provider for DTrace, and on Linux there is the specially-written mutrace. I had a lot of trouble trying to get plockstat to work on FreeBSD, so here’s an example of using mutrace to analyze our banker program.

mutrace ./banker_lock_hierarchy
mutrace: Showing 10 most contended mutexes:

 Mutex #   Locked  Changed    Cont. tot.Time[ms] avg.Time[ms] max.Time[ms]  Flags
       0   200211   153664    95985      991.349        0.005        0.267 M-.--.
       1   200552   142173    61902      641.963        0.003        0.170 M-.--.
       2   199657   140837    47723      476.737        0.002        0.125 M-.--.
       3   199566   140863    39268      371.451        0.002        0.108 M-.--.
       4   199936   141381    33243      295.909        0.001        0.090 M-.--.
       5   199548   141297    28193      232.647        0.001        0.084 M-.--.
       6   200329   142027    24230      183.301        0.001        0.066 M-.--.
       7   199951   142338    21018      142.494        0.001        0.057 M-.--.
       8   200145   142990    18201      107.692        0.001        0.052 M-.--.
       9   200105   143794    15713       76.231        0.000        0.028 M-.--.
                                                                           ||||||
                                                                           /|||||
          Object:                                     M = Mutex, W = RWLock /||||
           State:                                 x = dead, ! = inconsistent /|||
             Use:                                 R = used in realtime thread /||
      Mutex Type:                 r = RECURSIVE, e = ERRRORCHECK, a = ADAPTIVE /|
  Mutex Protocol:                                      i = INHERIT, p = PROTECT /
     RWLock Kind: r = PREFER_READER, w = PREFER_WRITER, W = PREFER_WRITER_NONREC

mutrace: Note that the flags column R is only valid in --track-rt mode!

mutrace: Total runtime is 1896.903 ms.

mutrace: Results for SMP with 4 processors.

Off-CPU profiling

Typical profilers measure the amount of CPU time spent in each function. However when a thread is blocked by I/O, a lock, or a condition variable, then it isn’t using CPU time. To determine where functions spend the most “wall clock time,” we need to sample the call stack for all threads at intervals, and count how frequently we see each entry. When a thread is off-CPU its call stack stays unchanged.

The pstack program is traditionally the way to get a snapshot of a running program’s stack. It exists on old Unices, and used to be on Linux until Linux made a breaking change. The most portable way to get stack snapshots is using gdb with an awk wrapper, as documented in the Poor Man’s Profiler.

Remember our early condition variable example that measured how many threads entered the critical section in disburse() at once? We asked whether synchronization on stats_mtx threw off the measurement. With off-CPU profiling we can look for clues.

Here’s a script based on the Poor Man’s Profiler:

./banker_stats &
pid=$!

while kill -0 $pid
  do
    gdb -ex "set pagination 0" -ex "thread apply all bt" -batch -p $pid
  done | \
awk '
  BEGIN { s = ""; }
  /^Thread/ { print s; s = ""; }
  /^\#/ { if (s != "" ) { s = s "," $4} else { s = $4 } }
  END { print s }' | \
sort | uniq -c | sort -r -n -k 1,1

It outputs limited information, but we can see that waiting for locks in disburse() takes the majority of program time, being present in 872 of our samples. By contrast, waiting for the stats_mtx lock in stats_update() doesn’t appear in our sample at all. It must have had very little affect on our parallelism.

    872 at,__GI___pthread_mutex_lock,disburse,start_thread,clone
     11 at,__random,rand,rand_range,disburse,start_thread,clone
      9 expected=0,,mutex=0x562533c3f0c0,<stats_cnd>,,stats_print,start_thread,clone
      9 __GI___pthread_timedjoin_ex,main
      5 at,__pthread_mutex_unlock_usercnt,disburse,start_thread,clone
      1 at,__pthread_mutex_unlock_usercnt,stats_change,disburse,start_thread,clone
      1 at,__GI___pthread_mutex_lock,stats_change,disburse,start_thread,clone
      1 __random,rand,rand_range,disburse,start_thread,clone

macOS Instruments

Although Mac’s POSIX thread support is pretty weak, its XCode tooling does include a nice profiler. From the Instruments application, choose the profiling template called “System Trace.” It adds a GUI on top of DTrace to display thread states (among other things). I modified our banker program to use only five threads and recorded its run. The Instruments app visualizes every event that happens, including threads blocking and being interrupted:

thread states

thread states

Within the program you can zoom into the history and hover over events for info.

perf c2c

Perf is a Linux tool to measure hardware performance counters during the execution of a program. Joe Mario created a Perf feature called c2c which detects false sharing of variables between CPUs.

In a NUMA multi-core computer, each CPU has its own set of caches, and all CPUs share main memory. Memory is divided into fixed size blocks (often 64 bytes) called cache lines. Any time a CPU reads or writes memory, it must fetch or store the entire cache line surrounding the desired address. If one CPU has already cached a line, and another CPU writes to that area in memory, the system has to perform an expensive operation to make the caches coherent.

When two unrelated variables in a program are stored close enough together in memory to be in the same cache line, it can cause a performance problem in multi-threaded programs. If threads running on separate CPUs access the unrelated variables, it can cause a tug of war between their underlying cache line, which is called false sharing.

For instance, our Game of Life simulator could potentially have false sharing at the edges of each section of board accessed by each thread. To verify this, I attempted to run perf c2c on an Amazon EC2 instance (since I lack a physical computer running Linux), but got an error that memory events are not supported on the virtual machine. I was running kernel 4.19.0 on Intel Xeon Platinum 8124M CPUs, so I assume this was a security restriction from Amazon.

If you are able to run c2c, and detect false sharing in a multi-threaded program, the solution is to align the variables more aggressively. POSIX provides the posix_memalign() function to allocate bytes aligned on a desired boundary. In our Life example, we could have used an array of pointers to dynamically allocated rows rather than a contiguous two-dimensional array.

Intel VTune Profiler

The VTune Profiler is available for free (with registration) on Linux, macOS, and Windows. It works on x86 hardware only of course. I haven’t used it, but their marketing page shows some nice pictures. The tool can visually identify the granularity of locks, present a prioritized list of synchronization objects that hurt performance, and visualize lock contention.

Further reading

March 22, 2020

Derek Jones (derek-jones)

Coronavirus: a silver lining for evidence-based software engineering? March 22, 2020 09:39 PM

People rarely measure things in software engineering, and when they do they rarely hang onto the measurements; this might also be true in many other work disciplines.

When I worked on optimizing compilers, I used to spend time comparing code size and performance. It surprised me that many others in the field did not, they seemed to think that if they implemented an optimization, things would get better and that was it. Customers would buy optimizers without knowing how long their programs took to do a task, they seemed to want things to go faster, and they had some money to spend buying stuff to make them feel that things had gotten faster. I quickly learned to stop asking too many questions, like “how fast does your code currently run”, or “how fast would you like it to run”. Sell them something to make them feel better, don’t spoil things by pointing out that their code might already be fast enough.

In one very embarrassing incident, the potential customer was interested in measuring performance, and my optimizer make their important program go slower! As best I could tell, the size of the existing code just fitted in memory, and optimizing for performance made it larger; the system started thrashing and went a lot slower.

What question did potential customers ask? They usually asked whether particular optimizations were implemented (because they had read about them someplace). Now some of these optimizations were likely to make very little difference to performance, but they were easy to understand and short enough to write articles about. And, yes. I always made sure to implement these ‘minor’ optimizations purely to keep customers happy (and increase the chances of making a sale).

Now I work on evidence-based software engineering, and developers rarely measure things, and when they do they rarely hang onto the measurements. So many people have said I could have their data, if they had it!

Will the Coronavirus change things? With everybody working from home, management won’t be able to walk up to developers and ask what they have been doing. Perhaps stuff will start getting recorded more often, and some of it might be kept.

A year from now it might be a lot easier to find information about what developers do. I will let you know next year.

Marc Brooker (mjb)

Two Years With Rust March 22, 2020 12:00 AM

Two Years With Rust

I like it. I hope it's going to be big.

It's been just over two years since I started learning Rust. Since then, I've used it heavily at my day job, including work in the Firecracker code base, and a number of other projects. Rust is a great fit for the systems-level work I've been doing over the last few years: often performance- and density-sensitive, always security-sensitive. I find the type system, object life cycle, and threading model both well-suited to this kind of work and fairly intuitive. Like most people, I still fight with the compiler from time-to-time, but we mostly get on now.

Rust has also mostly replaced Go as my go-to language for writing small performance-sensitive programs, like the numerical simulators I use a lot. Go replaced C in that role for me, and joined R and Python as my day-to-day go-to tools. I've found that I still spend more time writing a Rust program than I do Go, and more than C (except where C is held back by a lack of sane data structures and string handling). I've also found that programs seem more likely to work on their first run, but haven't made any effort to quantify that.

Over my career, I've done for-pay work in C, C++, Java, Python, Ruby, Go, Rust, Scheme, Basic, Perl, Bash, TLA+, Delphi, Matlab, ARM and x86 assembly, and R (probably forgetting a few). There's likely some of my code in each of those languages still running somewhere. I've also learned a bunch of other languages, because it's something I enjoy doing. Recently, for example, I've been loving playing with Frink. I don't tend to be highly opinionated about languages.

However, in some cases I steer colleagues and teams away from particular choices. C and C++, for example, seem to be difficult and expensive to use in a way that avoids dangerous memory-safety bugs, and users need to be willing to invest deeply in their code if these bugs matter to them. It's possible to write great safe C, but the path there requires a challenging blend of tools and humility. Rust isn't a panacea, but is a really nice alternative where they were fairly thin before. I find myself recommending and choosing it more and more often for small command-line programs, high-performance services, and system-level code.

Why I like Rust There are a lot of good programming languages in the world. There are even multiple that fit Rust's broad description, and place in the ecosystem. This is a very good place, with real problems to solve. I'm not convinced that Rust is necessarily technically superior to its nearest neighbors, but there are some things it seems to do particularly well.

I like how friendly and helpful the compiler's error messages are. The free book and standard library documentation are all very good. The type system is nice to work with. The built-in tooling (rustup, cargo and friends) are easy and powerful. A standard formatting tool goes a long way to keeping code-bases tidy and bikesheds unpainted. Static linking and cross-compiling are built-in. The smattering of functional idioms seem to add a good amount of power and expressiveness. Features that actively lead to obtuse code (like macros) are discouraged. Out-of-the-box performance is pretty great. Fearless Concurrency actually delivers.

There's a lot more, too.

What might make Rust unsuccessful? There are also some things I don't particularly like about Rust. Some of those are short-term. Learning how to write async networking code in Rust during the year or so before async and await were stabilized was a frustrating mess of inconsistent documentation and broken APIs. The compiler isn't as smart about optimizations like loop unrolling and autovectorization as C compilers tend to be (even where it does a great job eliding the safety checks, and other Rust-specific overhead). Some parts of the specification, like aliasing rules and the exact definitions of atomic memory orderings, are still a little fuzzier than I would like. Static analysis tooling has a way to go. Allocating aligned memory is tricky, especially if you still want to use some of the standard data structures. And so on.

In each of these cases, and more like them, the situation seems to have improved every time I look at it in detail. The community seems to be making great progress. async and await were particularly big wins.

The biggest long-term issue in my mind is unsafe. Rust makes what seems like a very reasonable decision to allow sections of code to be marked as unsafe, which allows one to color outside the lines of the memory and life cycle guarantees. As the name implies unsafe code tends to be unsafe. The big problem with unsafe code isn't that the code inside the block is unsafe, it's that it can break the safety properties of safe code in subtle and non-obvious ways. Even safe code that's thousands of lines away. This kind of action-at-a-distance can make it difficult to reason about the properties of any code-base that contains unsafe code. For low-level systems code, that's probably all of them.

This isn't a surprise to the community. The Rust community is very realistic about the costs and benefits of unsafe. Sometimes that debate goes too far (as Steve Klabnik has written about), but mostly the debate and culture seems healthy to me as a relative outsider.

The problem is that this spooky behavior of unsafe tends not to be obvious to new Rust programmers. The mental model I've seen nearly everybody start with, including myself, is that unsafe blocks can break things inside them and so care needs to be paid to writing that code well. Unfortunately, that's not sufficient.

Better static and dynamic analysis tooling could help here, as well as some better help from the compiler, and alternatives to some uses of unsafe. I suspect that the long-term success of Rust as a systems language is going to depend on how well the community and tools handle unsafe. A lot of the value of Rust lies in its safety, and it's still too easy to break that safety without knowing it.

Another long-term risk is the size of the language. It's been over 10 years since I last worked with C++ every day, and I'm nowhere near being a competent C++ programmer anymore. Part of that is because C++ has evolved, which is a very good thing. Part of it is because C++ is huge. From a decade away, it seems hard to be a competent part-time C++ programmer: you need to be fully immersed, or you'll never fit the whole thing in your head. Rust could go that way too, and it would be a pity.

Jeremy Morgan (JeremyMorgan)

How to Find Your Browser Developer Console March 22, 2020 12:00 AM

Debugging web applications can be tricky. Accessing your browser’s developer console is the easiest way to “pop the hood” and see what’s going on with your application. It’s like a secret backdoor that isn’t very secret. Google Chrome Google’s browser is one of the most popular out there, and it’s easy to access the developer console: In the upper right hand corner of the browser, click the 3 dots: Then select More Tools -> then Developer Tools

March 21, 2020

Jan van den Berg (j11g)

My Music Discoveries of 2019 March 21, 2020 09:24 PM

Here are my favorite music discoveries of 2019. Earlier editions are here: 20152016, 2017 and 2018 (part 1) and 2018 (part 2).

You know the drill. A few sentences and a YouTube video. All songs bear hearing. All songs I obsessed about at one point or another in 2019. Here we go — in no particular order.

The first and last video on the list are from the same band. I am relatively late to the Fontaines D.C. party. But I am here now. Hands down the best band of 2019 for me. Fresh, cocky, Irish and absolutely terrific. I haven’t felt this excited about a band since the Arctic Monkeys debut.

This next band probably has the best name of all the bands on the list. The Menzingers. I mean, come on that’s a pretty good name! This track is absolutely outstanding in urgency. And I’m gonna share the single and the acoustic, because they’re the same but completely different.

Same song. Different settings. Just great. 10/10 would listen on repeat.

This next song is messy and far from a perfect live performance, but I dig the taupe oversized shirts, keyboards and intensity.

And old favorite I rediscovered last year. Few bands can hold the candle to Wovenhand. They are not like any other band.

You know how you sometimes can forget about a song, but when you rediscover it, it’s like finding a severed finger? This Placebo song is like that. Needs to be played at absolutely no less than 11. The SG and piano part demand it.

I’ve posted VTr in a prrrrrevious list before, I really like this bands’ style. Running away doesn’t feel so bad. You don’t say.

There are MANY versions of the classic Irish rebel song Come Out Ye’ Black and Tans, but it so happens that I played this specific version the most.

S&G were probably my first love, and they will always hold a special place. Their songs are firmly locked in a time and place. I can gather all the news I need on the weather report.

SOAK. Soak it in. This song hit me in the teeth. I truly do not understand why she is not a superstar already. Tremendous song.

The Decemberists. You know exactly what you’ll get. Relentless and will wait for absolutely no one. No time to breathe, go go go go go!

I am aware there are only a few women on this list. But Sharon van Ettens’ Glastonbury performance, was one of the best performances I saw last year. Raw, intense, broody.

Echo and the Bunnymen. In an alternate universe, people talk about Echo and the Bunnymen the way we talk about the Beatles. I am sure of it.

Is this how the end begins? Well, who knows.

This song stopped me in my tracks. Where on earth did this guy come from? Are you kidding me! Fantastic song. Eres especial para mí. Si. Muy, muy bien.

Oh, you like synths and heavy basses? Well, are you on the Drab Majesty bandwagon yet? No!? Here you go.

The lyrics may be a tad too much. But nonetheless yes, this will work. This will work indeed. One Horse Town.

This is a very special song. It’s Dutch, but I would like to think it transcends language. Highly underappreciated. Absolutely glorious.

I tend to skip bands like this. They seem a dime a dozen. But this song really holds up. Really well actually. What are you waiting for Archie?

Let’s close this list how we started. Fontaines D.C. Man, this is the stuff.

The post My Music Discoveries of 2019 appeared first on Jan van den Berg.

Andrew Montalenti (amontalenti)

Best remote work equipment in 2020 March 21, 2020 09:07 PM

If you’re working on a fully distributed team, partially remote team, or even just working from home occasionally, this is a selection of low-cost equipment you can use to get your home office setup to a “professional” level.

All of this equipment has been tested extensively to work on:

  • Google Hangouts
  • Google Meet
  • Zoom Video Conferencing
  • Skype
  • FaceTime (on OSX)
  • Alternative browser-based video conference tools, like WhereBy

And, all of this equipment has further been tested on every operating system:

  • Mac OS X (including Catalina)
  • Linux (including Ubuntu 18.04 and beyond)
  • Windows 10

All of the recommendations here also work without special drivers or software.

The main reason for the compatibility is the use of standard USB 2.0 interfaces, which are really the best way to go for the equipment to work well.

The best remote work webcam: Logitech C925e

You should always prefer an external webcam & mic combo to your built-in webcam and mic on your laptop or desktop machine.

The Logitech C920+ models are extremely good and reliable on every operating system.

I recommend the Logitech C925e. This model has a great built-in stereo mic and 720p / 1080p video. It also supports hardware acceleration (aka h.264 encoding) on many platforms and with many video conference tools, which reduces the load on your computer’s CPU (and keeps your computer running cooler and using less battery). This particular model also features a simple built-in privacy shutter, excellent monitor mount, and a long USB cable.

logitech-c925e

If you can’t find the C925e, your best alternative is the Logitech C920. This is the consumer version of the same hardware — the webcam looks a little different but is also a 720p / 1080p HD cam with built-in mics. It doesn’t have the privacy shutter, however.


NOTE: As of March 28, 2020, I’ve been informed that likely due to everyone working from home across the country, Amazon has sold out of most Logitech webcams, including this model. You might see some Logitech 920/925 webcams on there for $200 or $300. Do not buy those. This is a $100 webcam. If it’s being sold for those prices, it’s price gouging due to the shortage. Instead, your best bet is to pick this camera up from a couple of alternative online retailers. I list a couple here:


The best high-quality speakermic for remote work: Jabra Speak 410

You may think you don’t need a speakermic for your work-from-home setup — that perhaps this only makes sense for conference rooms and offices. But trust me: this is the secret weapon of every well-equipped home office. This recommended model is very low-cost to boot!

My favorite model here is the tried-and-true JABRA Speak 410 USB conference mic + speaker combo. This should run you $90-$100.

jabra-410

I have three of these in my home office: one connected to my Mac Mini; one connected to my laptop docking station, which happens to have a Lenovo laptop running Ubuntu Linux that I use for programming; and one connected to my Windows 10 Media PC, that I use for Skype and phone calls that I take from the comfort of my couch.

Here’s what makes these conference-room-quality speakermics so great. First, assuming you have a quiet home office space, you can use them without earbuds, headphones, or any wires except for the wire that connects the speakermic to your computer. This also means that when you take video calls, you don’t have to worry about having that weird “telemarketer” look on the call.

The mic quality is excellent. The hardware controls for volume and mute are extremely intuitive. Surrounding the central speaker are touch controls for volume up, volume down, and mute. Your system volume is visualized with a ring of LED lights, and when you hit mute, the entire ring turns red. This is super easy to use during calls and helps when you frequently switch between Zoom, Google Meet, and other conference software, each of which has its own mute and volume controls.

Since it’s a speakerphone, you don’t need to worry about echo. Let me repeat that (unironically): you don’t need to worry about echo! This is because the speaker and the mic are coming from the same physical location — so, you get echo-free communication without the need for earbuds or headphones. If you have meetings all day, this will be much more comfortable than wearing pretty much any ear-covering hardware.

What’s more: this Jabra device has a built-in aux port so you can plug in your headphones when you need more privacy. So it pairs nicely with a recommendation I have further down about having a spare pair of wired earbuds available for calls.

Jabra has some other nice models in this series, too, if you have other preferences. If you’re willing to spend a couple hundred bucks on this problem, they have a premium version of this speakermic that has a nice built-in kickstand, built-in Bluetooth support, and a rechargeable battery (claimed at 15 hours, but many users say the true battery life is closer to 5-10 hours). It also still includes a plain USB 2 cable for direct connection and to charge the battery. This model is called the Jabra Speak 710. Unfortunately, this model is missing the aux port, but if you’re going all-in on Bluetooth, this might work well with your existing headphones.

Finally, if you have the space for it, Jabra also has a much higher-quality (and much bigger) model of this speakermic called the Jabra Speak 810. This model is nowhere near as portable as the 410, but the larger speaker is more powerful and the mic array has a lot more noise cancellation built in. Plus, it’s separately powered by an AC adapter, and has a built-in aux cable (not just an aux port) that you can use to plug in a phone or other device.

The best wired earbuds for remote work (and why you need them)

If you use the Jabra recommendation above as your daily driver for audio on calls, you’ll rarely need to use headphones. However, if you use pretty much any other hardware for your video calls, you’ll want a pair of wired earbuds handy. This is because operating systems and video conference software are simply no good at echo cancellation, so it’s best to fix the issue at the hardware level.

For example, let’s say you use your computer’s built-in webcam and mic for a video call. Typically the speakers for your laptop are located near the keyboard, the mic is located near the webcam atop your screen. Depending on a number of environmental factors, calls taken with this setup will introduce a bit of echo. But, you can kill the echo by plugging in some wired earbuds.

Likewise, if you purchase the Logitech C925e webcam discussed above, it has high-quality stereo mics built in to the webcam itself. But, depending on where your computer’s speakers are positioned relative to the webcam, you’ll get echo without buds if you use the webcam as your mic.

You may be wondering about wireless earbuds. Yes, right now Bluetooth and wireless headphones are all the rage, but note that Bluetooth is finicky with connection issues, and even introduces a perceptible 100ms of audio latency. Old school wired aux headphones work on every operating system, every time, and introduce no connection issues or latency. Since aux ports aren’t standard on every modern laptop, you may need to have an adapter available. However, remember that the Jabra speakermic mentioned above has an aux port built-in that translates the audio over your USB 2 cable. So, that can also help!

If you want to get fancy from a comfort/quality standpoint, the Bose Soundsport Wired Earbuds are a good choice, though make sure to get the “Audio Only” (no mic) option so that you don’t have to fiddle with the mic settings and can just use the built-in on your webcam or computer, which tend to be higher quality. There are also a plethora of other options available if the stock on these is low.


NOTE: As of March 28, 2020, a couple people wrote me to say that the above Bose Soundsport Wired Earbuds are discontinued and thus very hard to find. In a pinch, I’d say the best alternative is to just pick up the standard Apple White Wired Aux Earbuds, which you can find on the Apple Store here. If you don’t have an aux port on your laptop, USB-C and Lightning adapters are widely available for just a few bucks. Some folks also suggested the Sennheiser CX 100 in-ear earbuds, which also seem like a solid set of aux headphones that come in black or white styles, and Amazon seems to have a stock of those.


There are a couple of other options to have a “wired backup” to a pair of wireless headphones. These are the Bose Soundlink On-Ear Bluetooth Headphones and the Bose 700 Over-Ear Bluetooth Headphones.

Each of these headphones includes a wired aux cable that plugs into the wireless headphones and allows them to work without Bluetooth and even without any battery charge. This can combine nicely with the Jabra speakermic above. As for deciding between the on-ear Bose Soundlink and the over-ear Bose 700, here’s a quick pro/con based on my own personal experience with both:

  • The Bose Soundlink On-Ear is a super portable pair of headphones. They “collapse” into a tight bundle that can be tucked away in a drawer or in your backpack. Since they are on-ear models, they also tend to be more comfortable for extended wear. The volume and Bluetooth controls are hardware buttons that are easy-to-use. However, if you are using them for Bluetooth calls (that is, as a mic and as a speaker), the mic quality is not necessarily the best. And the noise cancellation is minimal.
  • The Bose 700 Over-Ear headphones are less portable, but they are a super-high-quality, both the audio quality and the fit-and-finish. They have a number of features specifically designed to work well with your smartphone — for example, integration with Google Assistant (on Android) or Siri (on iPhone). They also uses the latest Bluetooth standard (5.0) and a rapid Bluetooth pairing speed. If you choose to use these for Bluetooth calls, they have the best noise cancellation sound you can find, and even have an extensive ambient noise cancellation for the stereo mic array.

bose-700

Between these two, I think the Bose 700 is the better option in 2020, especially if portability isn’t a concern.

One more note on Bluetooth devices

I generally advise all remote workers to avoid Bluetooth devices altogether. If you insist on using them, you should follow these rules:

  • Keep the headphone or speakermic permanently paired with your computer, as the slow speed of pairing can be really annoying in conference calls.
  • Make sure your device, your computer, and your operating system are all at least Bluetooth 4.0 compatible. If your computer isn’t, buy a cheap USB adapter to add Bluetooth 4.0+ support. If all of your devices aren’t Bluetooth 4.0 compatible, your audio quality on calls will be terrible for the other parties on your call, even though the quality for you might be fine. If you’ve ever been on calls where the other person sounds really “tinny” and their mic quality sounds terrible, it’s probably because they were using Bluetooth 1.0 – 3.0 somewhere in their device chain. The Bose Soundlink On-Ear Headphones I recommend above have Bluetooth 4.0 support.
  • Ideally, opt for Bluetooth 5.0 devices. These are newer and have some further improvements. The Bose 700 Over-Ear Headphones I recommend above have Bluetooth 5.0 support.

But, given all the issues with Bluetooth — range, connectivity, pairing, different versions of the standard, and software issues on the operating systems themselves — I really recommend you treat these devices as “backup options” and go with higher-quality USB 2 devices recommended above as your daily drivers.

Another tip: if you’re low on ports, use a USB hub or dock

People who work from home regularly want to avoid the scramble typically involved to plug in all the devices needed for a good video call. A USB hub or dock makes this a snap — reducing the problem to a single cable. Here are some recommendations:

Overall, these are cheap, solid options that you’ll probably need given how many devices you’ll want to plug in at-once via a single cable.

Some software recommendations

Once you have all this great hardware, you also need to master the software on your operating system of choice. Here, I have some recommendations for Mac OS X software you can use for a number of common remote working problems. There’s probably equivalent software available for Linux and Windows, too.

  • Adjust your mic settings: On OS X, you can keep a shortcut to the “Systems Preferences” application on your dock, and you can also use the Volume Control in the top-right of your operating system to quickly switch output devices. You can also use the Skype call testing “echo” service to easily test your audio settings. Here’s how the Sound preferences panel looks. Make sure to check off the checkbox at the bottom:

mic-settings-osx

  • Adjust screensharing resolution: when screensharing, you’ll want to turn down your screen resolution, especially if you have a 4K monitor or retina display. Switching to 720p (e.g. 1366×768) or 1080p (1920×1080) tends to be the best to provide real estate while also making text readable for most screenshare watchers — salespeople, this also applies to when doing webinars or sales calls for prospects and customers! An excellent free app on Mac OS X for this is called BestRes. It lets you switch resolutions in a single click. Here’s how that looks:

bestres-osx

  • Turn off notifications: there’s nothing more embarassing than getting a text message on iMessage in the middle of an important work meeting. On OSX, the free app Muzzel will hide all of your notifications whenever it notices that a Zoom or Google Meet call is running. This is very important. If you’re the kind of person who gets a lot of “private” text messages throughout the day, you don’t want these being streamed over your video conference to your boss and work colleagues! Here’s a couple of “embarrassing message” examples from the Muzzle website:

notifications

  • Toggle your webcam’s video stream: The Logitech C920 webcams recommended above have two different “streams”, the “Standard Definition” (360p) stream and the “High Definition” (720p or 1080p) stream. Unfortunately this can’t be toggled at an operating system level, you have to check the settings for each individual app to adjust this. But, for the highest quality calls, you’ll want to make sure “High Definition” is selected. You can use “Standard Definition” when bandwidth usage is a concern. Here you can see the way these settings show up in Google Meet. To get to this panel, you need to click the “…” button and then goto “Settings”.

hd-webcam-settings-update

Final real-world tip: use a simple real-life backdrop

People tend to get needlessly distracted by what’s in the background of your video calls. Ideally use a plain white backdrop or tailor your backdrop with things like a neat bookshelf or wall art.

Here’s a screenshot I took of a recent team call, where I blurred out people and then gave “report card grades” to their backgrounds. Note that two of these folks, I had no clue what their actual background was, since one picked a NASA control room as their “virtual background” and the other picked the University of Virginia campus!

zoom-calls-backgrounds

If your room is a mess, pick a nice virtual background! This is only really supported in Zoom via its Virtual Background feature, but note that it increases the CPU load of calls and requires a computer with a modern GPU.

Toward distributed team zen

I hope you found all of these recommendations useful.

If you have other questions about how to manage your fully distributed team, you can reach out to me to schedule a quick 30-minute call and get some advice via this link:

https://calendly.com/amontalenti/distributed

Or, to ask me quick questions online, you ping me on Twitter at @amontalenti.

Pages From The Fire (kghose)

Buffering writes March 21, 2020 03:05 AM

Many scientific computing applications consist of processing data and then writing out to disk. Often the application will be I/O bound, which means that the computation time is shorter that the time it takes to read or write data to disk. Even though modern OSes do their best to write efficiently to the hardware, there… Read More Buffering writes

March 20, 2020

Wesley Moore (wezm)

Comparing Alternatives to top Written in Rust March 20, 2020 11:45 PM

Recently I aliased top to ytop. Then I became aware of bottom, and zenith. These are all terminal based system monitoring tools that you might use instead of top. In this post I set out to compare them.

Screenshot of ytop, bottom, and zenith while building some Rust code Left to right: ytop, bottom, and zenith.
💡
Why Rust?

The Rust programming language helps people write efficient, reliable software. I like the language and the tools people are building with it.

If you're interested in more Rust CLI tools check out my post: An Illustrated Guide to Some Useful Command Line Tools.

As the title states all three tools are written in Rust. They show similar information and are all open source under the MIT license. I tested each one on: Arch Linux, FreeBSD 12.1, macOS Mojave, and Windows 10. At the time of testing, all three failed to build on FreeBSD and Windows. Figures below are from the Arch Linux system, which is a 12 core AMD Ryzen desktop PC.

ytop and bottom use a layout that appears to be inspired by gotop. In fact, ytop is written by the same person as gotop. zenith uses a layout that's a bit more like traditional top with histograms above the process list.

I typically use top to:

  • Check on overall system load and free memory.
  • Find specific processes that are using a lot of CPU or memory.
  • Observe CPU and memory use over time.
  • Occasionally kill processes.

I find the zenith layout more information dense, with less space taken up with graphs. I also like the header row with info and help. The main feature that it is missing compared to the others is temperatures — but that's in the list of planned features. There is one issue with zenith: it doesn't show my ZFS pool. My system has an NVMe system disk and a ZFS pool of 3 SSDs that is mounted as /home, which is absent in the disk summary. I've raised an issue on GitHub.

Update 27 March 2020: Zenith 0.7.7 now shows ZFS pools.

The individual lines for each CPU in bottom makes the display quite noisy. I prefer the aggregated line that ytop shows. ytop has a handy -m option to only show CPU, memory, and process list.

So, after reviewing them all I'm going to start using zenith. It might not be quite as pretty in screenshots but it's the best for doing this things I want to do with a top like tool.

Read on for further information about each tool, including an additional honorable mention, bb.

bottom

Screnshot of bottom

Repository

Version tested: 0.2.2
Runtime dependencies: None
Lines of code: 4894
Cargo dependencies: 109
Stripped binary size: 3.4MiB

bottom has vi style key bindings for navigating the process list. The selection is not stable across updates but there is a key binding, f to freeze the display that allows you navigate the list without it changing. It supports killing processes with dd but does not prompt for the signal to send. It shows a confirmation before killing the process.

Usage:

bottom 0.2.2
Clement Tsang <cjhtsang@uwaterloo.ca>
A cross-platform graphical process/system monitor with a customizable interface and a multitude of features. Supports
Linux, macOS, and Windows.

USAGE:
    btm [FLAGS] [OPTIONS]

FLAGS:
    -a, --avg_cpu                Enables showing the average CPU usage.
    -S, --case_sensitive         Match case when searching by default.
    -c, --celsius                Sets the temperature type to Celsius.  This is the default option.
        --cpu_default            Selects the CPU widget to be selected by default.
        --disk_default           Selects the disk widget to be selected by default.
    -m, --dot_marker             Use a dot marker instead of the default braille marker.
    -f, --fahrenheit             Sets the temperature type to Fahrenheit.
    -g, --group                  Groups processes with the same name together on launch.
    -k, --kelvin                 Sets the temperature type to Kelvin.
    -l, --left_legend            Puts external chart legends on the left side rather than the default right side.
        --memory_default         Selects the memory widget to be selected by default.
        --network_default        Selects the network widget to be selected by default.
        --process_default        Selects the process widget to be selected by default.  This is the default if nothing
                                 is set.
    -R, --regex                  Use regex in searching by default.
    -s, --show_disabled_data     Show disabled data entries.
        --temperature_default    Selects the temp widget to be selected by default.
    -u, --current_usage          Within Linux, sets a process' CPU usage to be based on the total current CPU usage,
                                 rather than assuming 100% usage.
    -W, --whole_word             Match whole word when searching by default.
    -h, --help                   Prints help information
    -V, --version                Prints version information

OPTIONS:
    -C, --config <CONFIG_LOCATION>    Sets the location of the config file.  Expects a config file in the TOML format.
    -r, --rate <RATE_MILLIS>          Sets a refresh rate in milliseconds; the minimum is 250ms, defaults to 1000ms.
                                      Smaller values may take more resources.

ytop

Screnshot of ytop

Repository

Version tested: 0.5.1
Runtime dependencies: None
Lines of code: 1903
Cargo dependencies: 89
Stripped binary size: 2.1MiB

ytop has vi style key bindings for navigating the process list. The selection remains on the same process across updates. It supports killing processes with dd but does not prompt for the signal to send. There is no confirmation when typing dd.

Usage:

ytop 0.5.1

USAGE:
    ytop [FLAGS] [OPTIONS]

FLAGS:
    -a, --average-cpu    Show average CPU in the CPU widget
    -b, --battery        Show Battery widget (overridden by 'minimal' flag)
    -f, --fahrenheit     Show temperatures in fahrenheit
    -h, --help           Prints help information
    -m, --minimal        Only show the CPU, Mem, and Process widgets
    -p, --per-cpu        Show each CPU in the CPU widget
    -s, --statusbar      Show a statusbar with the time
    -V, --version        Prints version information

OPTIONS:
    -c, --colorscheme <colorscheme>    Set a colorscheme [default: default]
    -i, --interface <interface>        The name of the network interface to show in the Net widget. 'all' shows all
                                       interfaces [default: all]
    -I, --interval <interval>          Interval in seconds between updates of the CPU and Mem widgets. Can specify
                                       either a whole number or a fraction with a numerator of 1 [default: 1]

zenith

Screnshot of zenith

Repository

Version tested: 0.7.5
Runtime dependencies: None
Lines of code: 2006
Cargo dependencies: 105
Stripped binary size: 2.6MiB

zenith has a small number of key bindings for changing the panes. You can navigate the process list with the arrow keys. The selection is not stable across updates but the default update frequency is 2 seconds. Pressing Enter on process shows expanded information about it and allows you to send it various signals:

zenith process view

Usage:

zenith 0.7.5
Benjamin Vaisvil <ben@neuon.com>
Zenith, sort of like top but with histograms.
Up/down arrow keys move around the process table. Return (enter) will focus on a process.
Tab switches the active section. Active sections can be expanded (e) and minimized (m).
Using this you can create the layout you want.

USAGE:
    zenith [FLAGS] [OPTIONS]

FLAGS:
        --disable-history    Disables history when flag is present
    -h, --help               Prints help information
    -V, --version            Prints version information

OPTIONS:
    -c, --cpu-height <INT>        Height of CPU/Memory visualization. [default: 10]
        --db <STRING>             Database to use, if any. [default: /home/wmoore/.zenith]
    -d, --disk-height <INT>       Height of Disk visualization. [default: 10]
    -n, --net-height <INT>        Height of Network visualization. [default: 10]
    -p, --process-height <INT>    Min Height of Process Table. [default: 8]
    -r, --refresh-rate <INT>      Refresh rate in milliseconds. [default: 2000]

bb

Screnshot of bb

Repository

Version tested: git 35c3017
Runtime dependencies: None
Lines of code: 8450
Cargo dependencies: 27
Stripped binary size: 534KiB

bb is closer to regular top than the other tools. It shows a CPU histograms and a process list. It has the best process view though, allowing sending all named signals, filtering the list by name or pid, toggleable tree view and following a process group. It also has the fewest crate dependencies and smallest binary. The drawback is the author describes it as, "a "weekend" side-project made for fun", and it hasn't seen any updates since Nov 2019.

Usage:

bb does not have any command line arguments.

Test Notes

The dependency count was calculated using cargo-tree as follows:

cargo tree --no-dev-dependencies --no-indent -q | sed 's/ (\*)$//' | sort -u | wc -l

The lines of code values were calculated using tokei. The value of the Code column in the Total row from the output of tokei src was used. E.g. 2372 in the output below:

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 JSON                    5          105          105            0            0
 Markdown                2          231          231            0            0
 Rust                   18         2404         2001           87          316
 TOML                    2           39           35            2            2
-------------------------------------------------------------------------------
 Total                  27         2779         2372           89          318
-------------------------------------------------------------------------------

Andrew Montalenti (amontalenti)

Chat with me for 30 minutes about distributed team management March 20, 2020 10:32 PM

Have you been trying to figure out this new world order with regard to work-from-home (WFH), remote work, distributed teams, and the like? I’ve opened up my calendar for 30-minute chats. You can schedule them with me here:

https://calendly.com/amontalenti/distributed

If you want to read up on distributed teams, here are some past posts from my blog and the Parse.ly blog on the subject. Open up these links and skim to get a sense of my past writings.

I think you’ll find that some of the best advice on distributed teams come from people who have been doing it for years — rather than from folks that are scrambling to do it, under less-than-ideal circumstances, in the last few weeks. Again, if you need help, you can find me on my calendar here.

It’s more important than ever that you lead and inspire your teams in an effective way. Remember: not only are they operating on a fully remote model — possibly for the first time in their lives — but, also, there is a wider economic calamity around them. They are looking toward work for a sense of normalcy and purpose.

If you can give that normalcy and purpose to them, you can do them a great service. I am glad to help in any way I can to think through the leadership challenges in a distributed setting.

Stay safe out there!

Why might I have a good background to chat with you about distributed teams? I’ve been doing it for over ten years. I currently lead the engineering, product, and design team at Parse.ly, which is one of the earliest practitioners of the “fully distributed” model among modern software & SaaS companies. It scaled up to millions in revenue and a staff of 50+ amazing contributors. Parse.ly’s fully distributed team spans across sales, marketing, support, product, engineering, and design. Even prior to my time at Parse.ly, I worked on distributed teams in the open source community and in Fortune 500 firms. Feel free to get in touch.

March 16, 2020

Jan van den Berg (j11g)

Corona Links March 16, 2020 07:11 AM

This is a collection of COVID-19 related information links.

Stats

Background

The most important metric, that everyone should understand:

The post Corona Links appeared first on Jan van den Berg.

Pete Corey (petecorey)

Lissajous Curves March 16, 2020 12:00 AM

I recently watched this Standup Maths video on Lissajous curves and quickly fell into a wikipedia hole researching the topic. It turns out that Lissajous curves are a set of parametric equations defined in terms of a, b, and δ (delta):

{\displaystyle x=A\sin(at+\delta ),\quad y=B\sin(bt),}

When a and b are natural numbers and a/b is rational, the corresponding Lissajous curve is “closed” and resembles some kind of knot with a horizontal lobes and b vertical lobes. This relationship fascinated me, so I wrote a quick React application that renders an n by n grid of Lissajous curves to an HTML5 canvas:

Each cell in the grid represents an intersection of an a value, plotted on the vertical axis and ranging from 1 at the top to 20 at the bottom, and a b value, plotted on the horizontal axis and once again ranging from 1 on the left to 20 on the right.

The complete repository for this project can be found on Github. The Lissajous component renders a single Lissajous curve for a given a and b, while the MultiLissajous component efficiently renders the grid of n Lissajous curves.

March 15, 2020

Jeremy Morgan (JeremyMorgan)

Donate Your Unused CPU Cycles to Fight Coronavirus March 15, 2020 10:45 PM

The Folding at Home research project uses crowd sourced CPU power to help model simulations to develop treatments for diseases. You can help them by taking 5 minutes to download their client, and donate some CPU Cycles. For more info: After initial quality control and limited testing phases, Folding@home team has released an initial wave of projects simulating potentially druggable protein targets from SARS-CoV-2 (the virus that causes COVID-19) and the related SARS-CoV virus (for which more structural data is available) into full production on Folding@home.

Derek Jones (derek-jones)

Exercises in Programming Style: the python way March 15, 2020 10:37 PM

Exercises in Programming Style by Cristina Lopes is an interesting little book.

The books I have previously read on programming style pick a language, and then write various programs in that language using different styles, idioms, or just following quirky rules, e.g., no explicit loops, must use sets, etc. “Algorithms in Snobol 4” by James F. Gimpel is a fascinating read, but something of an acquired taste.

EPS does pick a language, Python, but the bulk of the book is really a series of example programs illustrating a language feature/concept that is central to a particular kind of language, e.g., continuation-passing style, publish-subscribe architecture, and reflection. All the programs implement the same problem: counting the number of occurrences of each word in a text file (Jane Austin’s Pride and Prejudice is used).

The 33 chapters are each about six or seven pages long, and contain a page or two or code. Everything is very succinct, and does a good job of illustrating one main idea.

While the first example does not ring true, things quickly pick up and there are lots of interesting insights to be had. The first example is based on limited storage (1,024 bytes), and just does not make efficient use of the available bits (e.g., upper case letters can be represented using 5-bits, leaving three unused bits or 37% of available storage; a developer limited to 1K would not waste such a large amount of storage).

Solving the same problem in each example removes the overhead of having to learn what is essentially housekeeping material. It also makes it easy to compare the solutions created using different ideas. The downside is that there is not always a good fit between the idea being illustrated and the problem being solved.

There is one major omission. Unstructured programming; back in the day it was just called programming, but then structured programming came along, and want went before was called unstructured. Structured programming allowed a conditional statement to apply to multiple statements, an obviously simple idea once somebody tells you.

When an if-statement can only be followed by a single statement, that statement has to be a goto; an if/else is implemented as (using Fortran, I wrote lots of code like this during my first few years of programming):

      IF (I .EQ. J)
      GOTO 100
      Z=1
      GOTO 200
100   Z=2
200

Based on the EPS code in chapter 3, Monolithic, an unstructured Python example might look like (if Python supported goto):

for line in open(sys.argv[1]):
    start_char = None
    i = 0
    for c in line:
        if start_char != None:
           goto L0100
        if not c.isalnum():
           goto L0300
        # We found the start of a word
        start_char = i
        goto L0300
        L0100:
        if c.isalnum():
           goto L0300
        # We found the end of a word. Process it
        found = False
        word = line[start_char:i].lower()
        # Ignore stop words
        if word in stop_words:
           goto L0280
        pair_index = 0
        # Let's see if it already exists
        for pair in word_freqs:
            if word != pair[0]:
               goto L0210
            pair[1] += 1
            found = True
            goto L0220
            L0210:
            pair_index += 1
        L0220:
        if found:
           goto L0230
        word_freqs.append([word, 1])
        goto L0300
        L0230:
        if len(word_freqs) = 1:
           goto L0300:
        # We may need to reorder
        for n in reversed(range(pair_index)):
            if word_freqs[pair_index][1] = word_freqs[n][1]:
               goto L0240
            # swap
            word_freqs[n], word_freqs[pair_index] = word_freqs[pair_index], word_freqs[n]
            pair_index = n
            L0240:
        goto L0300
        L0280:
        # Let's reset
        start_char = None
        L0300:
        i += 1

If you do feel a yearning for the good ol days, a goto package is available, enabling developers to write code such as:

from goto import with_goto

@with_goto
def range(start, stop):
    i = start
    result = []

    label .begin
    if i == stop:
        goto .end

    result.append(i)
    i += 1
    goto .begin

    label .end
    return result

Jeff Carpenter (jeffcarp)

The Most Commonly Used Chinese Words March 15, 2020 10:23 PM

I think the most effective way to learn a language is to prioritize learning the day-to-day most frequently used words. Picking words to study in order of frequency is the optimal way to maximally increase your marginal understanding of the language for each successive word you learn. To that end, I took a dataset of Weibo (China’s Twitter) posts and ranked all words that appeared in the dataset order of frequency.

Gokberk Yaltirakli (gkbrk)

Rolling your own encryption March 15, 2020 12:00 AM

Encryption is tricky to get right. Because some data and communications might be very sensitive or even life-critical for people, beginners are often - and quite rudely- shunned away from playing around with how it works. But if people don’t learn by making bad ciphers, they will have difficulty understanding why the good ones are good.

Software is usually one of the most welcoming fields for beginners, but encryption seems to be the one thing that can’t be experimented with.

Let’s ignore all that and have some fun, after all encryption is the perfect project for recreational programming. It might end up being very weak or decently strong; in either case I’ll learn something new after this is done through constructive criticism.

Here’s a description of my first-ever cipher, hope you have fun breaking it.

Meet XOR

XOR is the operation that can combine two bitstreams in a way that is reversible. By applying the same key through XOR, data can be encrypted and decrypted. Here’s a short example of how it works.

data = [1, 2, 3] # Secret data
key  = [7, 8, 4] # Completely arbitrary

# ^ is the Python operator for XOR
encrypted = [data[i] ^ key[i] for i in range(3)]

# Decode the encrypted bytes with the same key
decrypted = [encrypted[i] ^ key[i] for i in range(3)]

assert data == decrypted

This will be a useful building block of our cipher, so let’s turn it into a function. The function will be able to handle having different amounts of data and key material as well, in order to support different file sizes.

def xor_vectors(v1, v2):
    size = min(len(v1), len(v2))
    return [v1[x] ^ v2[x] for x in range(size)]

If you and the person you’re trying to communicate with shared a key that was the same size as the message you’re trying to send, and you only used that key once, you could achieve perfect encryption that would be impossible to decrypt. This is called a One-time Pad.

The problem is getting the large amount of keys to them securely. If we can generate the same random bitstream reliably, we can use it as our XOR key in order to encrypt and decrypt the file. We will address this in the next section.

Hash chains

In order to generate a random bitstream, we will use the SHA512 hash function. This takes any number of bytes, and turns them into a 64 byte output. By repeatedly feeding the hash output back into the hash function, we can get a stream of random bytes that can be used to encrypt our file.

This function implements the hashing for us.

def rng_bitstream(val):
    return hashlib.sha512(bytes(val)).digest()

Here’s a function that will read chunks from STDIN and yield 64-byte chunks of encrypted data.

def encrypt():
    last = get_entropy(64)
    yield last

    for chunk in stdin_chunks():
        xored = xor_vectors(chunk, rng_bitstream(last))
        last = xored
        yield xored

When we’re encrypting the files, we’re going to make the first 64 bytes completely random. This is what the line with get_entropy does. This will make it so that encrypting the same file with the same password twice will produce different results. It also covers for any lack of entropy from the file since the first chunk will always be completely random.

def get_entropy(length):
    """Read length bytes of randomness from /dev/random."""
    return os.getrandom(length, os.GRND_RANDOM)

os.getrandom basically does the equivalent of

with open('/dev/random', 'rb') as f:
    return f.read(length)

Decrypting the chunks

Decrypting the chunks is similar, but instead of feeding the XOR’ed chunk to the hash function, you use the original chunk.

def decrypt(chunks):
    last = next(chunks)
    for chunk in chunks:
        xored = xor_vectors(chunk, rng_bitstream(last))
        last = chunk
        yield xored

Encrypting with a password

So far; the cipher we wrote only encrypts our data with a random key prepended to the file. While that obfuscates the whole file and essentially turns it into random data, it is still possible to decrypt it easily. We’re basically creating a secret document with the key written on top of it.

In order to fix this, we will XOR all the chunks again with a hash chain. But this time instead of using entropy, we will begin the hash chain with a password.

def passkey_stream(key, stream):
    last = key.encode("utf-8")
    for chunk in stream:
        last = rng_bitstream(last)
        yield xor_vectors(chunk, last)

Regardless of the password length, the XOR key starts with 64 bytes. A weak password is easy to brute force, but the hashing operation at least makes the data look random. So without brute forcing there is no way to know anything about the password.

Putting it all together

The snippet below parses the command line arguments in order to encrypt/decrypt files.

if __name__ == "__main__":
    key = sys.argv[2]
    chunks = None
    if sys.argv[1] == 'encrypt':
        chunks = passkey_stream(key, encrypt())
    else:
        chunks = decrypt(passkey_stream(key, stdin_chunks()))
    for chunk in chunks:
        os.write(1, bytes(chunk))

The Challenge

Here’s a small challenge, I’m linking to an encrypted file. Try to get some information from this file, or break the cipher in any way. Comment, email and blog about how you went about it. Let’s have some fun discussions about homebrew DIY crypto and why it might be a good / bad idea.

Full code

You can find the full code here:

#!/usr/bin/env python3
import hashlib
import sys
import os


def get_entropy(length):
    """Read length bytes of randomness from /dev/random."""
    return os.getrandom(length, os.GRND_RANDOM)


def xor_vectors(v1, v2):
    return [v1[x] ^ v2[x] for x in range(min(len(v1), len(v2)))]


def rng_bitstream(val):
    return hashlib.sha512(bytes(val)).digest()


def stdin_chunks():
    with open("/dev/stdin", "rb") as f:
        while True:
            b = f.read(64)
            if b:
                yield b
            else:
                break


def encrypt():
    last = get_entropy(64)
    yield last

    for chunk in stdin_chunks():
        xored = xor_vectors(chunk, rng_bitstream(last))
        last = xored
        yield xored


def decrypt(chunks):
    last = next(chunks)
    for chunk in chunks:
        xored = xor_vectors(chunk, rng_bitstream(last))
        last = chunk
        yield xored


def passkey_stream(key, stream):
    last = key.encode("utf-8")
    for chunk in stream:
        last = rng_bitstream(last)
        yield xor_vectors(chunk, last)

if __name__ == "__main__":
    key = sys.argv[2]
    chunks = None
    if sys.argv[1] == 'encrypt':
        chunks = passkey_stream(key, encrypt())
    else:
        chunks = decrypt(passkey_stream(key, stdin_chunks()))
    for chunk in chunks:
        os.write(1, bytes(chunk))

March 14, 2020

Scott Sievert (stsievert)

Visualization of the COVID-19 infection rates March 14, 2020 05:00 AM

COVID-19/coronavirus is spreading through the world. The WHO classifies it as a pandemic (source), and the United States has declared a national emergency because of it (source).

Data on the infection rate is difficult to find. This post will present some data sources and show some visualizations of the infections. It’ll also provide some tips on how to avoid coronavirus.

This post will be updated frequently (hopefully daily).

First, let’s visualize the fraction of people that are infected in each country. The very useful data sources are mentioned in the appendix.

This is frightening on 2020-03-14. It looks like the US health care system will be overrun in early April. Hopefully the infection rate slows before that can happen. How can we aid the infection slowing down?

Slowing the rate of infection

Right now (2020-03-14) it seems that the only hope is social distancing. I think the timeline for a vaccine is too long (around 18 months), and cures/therapeutic drugs aren’t here yet (though they’re coming). Here are some resources to help with social distancing:

In short, avoid social contact as much as possible. The biggest tips are to work from home and avoid large crowds. Social distancing is why so many sporting events have been canceled and so many universities are canceling class.

The positive effect of immediate and extreme social distancing is best illustrated with the 1918 flu epidemic/the Spanish flu. The hardest hit city was Philadelphia. The infection hit the city very suddenly, meaning the health care system was quickly saturated and overrun. One effect of this: they had to stack bodies on the sidewalk because the morgues were overflowing (source).

However, St. Louis practiced immediate and extreme social distancing when the flu hit the city. Just 2 days after the flu hit the civilian population, Dr. Max Starkloff quickly ordered the closer of schools, movie theaters, sporting events and public gatherings.1

This meant the flu spread much more slowly throughout St. Louis than Philadelphia. Here’s a graph of the death rate for the two cities1

Functionally, the time taken to start social distancing is the only difference between these two curves. St. Louis responded about 14 days quicker than Philadelphia. That meant St. Louis had half the total death rate of Philadelphia, and they didn’t have to pile bodies on the streets because their morgues were too full.

The difference between these two cities is discussed more in these two articles:

How fast is the virus spreading?

The direct visualization of this is with the number of new cases each day:

This perhaps isn’t the best visualization; it’s pretty noisy, and there’s no information about the retired population or the quality/size of the health care system. Here’s a different visualization:

It appears the number of confirmed cases is doubling every 2.4 days before the infection gets under control2 (as with South Korea and Italy). Details on the back-of-the-envelope calculation are in the appendix; briefly, I take the number of cases required to Italy’s health care system and scale by the population that’s 65 years or older and number of hospital beds.

This looks to be rather scary for the U.K. The Prime Minister Boris Johnson has largely kept the UK open. Schools and universities have been kept open far too long, and many businesses and citizens didn’t take coronavirus seriously (source). For example, on March 12th the government said it didn’t want to “disrupt daily life before it was absolutely necessary”.3 It looks like the UK will have more deaths and a higher economic cost.4

How fast is the virus spreading in the US?

How many infections does the US have? When will the number of people who need hospitalization overwhelm the number of hospital beds?

If social distancing does not work and no medicine is developed, it looks like this virus will overwhelm the health care system sometime in late March.

How fast is the US spreading in each state?

These tests are confounded by testing. On 2020-03-22, it looks like New York is rapidly expanding testing, or the virus is spreading faster. I believe they’re expanding testing – they recently opened a drive-thru testing facility (source), and NY has about 8× the number of cases as WA but only 1.2× the number of deaths (on 2020-03-22).

In other states, it appears the number of confirmed coronavirus cases are growing slower! The number of days to double is slowing; Minnesota is doubling every 3.5 days instead of 2.4. If true, this is great news! Here’s a table of just how good that is:

State Cases on Mar. 16th Cases on Mar. 23rd Cases on Mar. 23rd if 2× every 2.4 days
MN 54 234 (5.4×) 407 (7.5×)
CA 557 2108 (3.7×) 4,356 (7.5×)
WA 904 2221 (2.5×) 6,826 (7.5×)

I interpret the increase in testing to be finding cases earlier in the infection cycle and/or discovering the infection in unconcerned populations (e.g., college students), especially because I doubt the virus can spread faster than doubling once every 2.4 days. The increased testing is likely masking the effect I’m concerned about, the load on the health care system.

It appears New York City is a hotspot, and confounded by testing. Regardless if that’s the case, let’s generate the same plot but without the data from New York5:

It looks like the infections in the US are starting to slow down, presuming that NY is doing more testing. If true, that’s great news! A couple points:

  • Social distancing is working! The effect we’re seeing is probably from 5–10 days previous, so March 11th–16th on 2020-03-21. That’s right when the US started social distancing.
  • If it continues, social distancing has bought the US at least 4 days, probably 5 before the health care system saturates. That’s 4× fewer total cases!

How serious are people about social distancing?

Let’s study a weak proxy for this question, the number of restaurant reservations bookings through OpenTable:

Here’s a visualization of the subway usage in New York City:

For an interactive version of this graph, visit the New York City Subway Usage Dashboard by Todd Schneider.

Appendix

U.K.’s approach

On March 12th, the U.K. goverment said that their approach…

include[d] asking people with even [a] very mild cough and flu symptoms to stay home, but stopped short of the measures seen in most other European countries and increasingly in North America, including closing schools and colleges.

The government said Thursday [March 12th] that it may have to impose these more stringent measures at some point in the future, but that doing so now would be premature, disrupting daily life before it was absolutely necessary, and risking the public would grow tired of complying with the restrictions and begin to ignore them just when the peak number of infections might be expected ….

—”Even while canceling mass gatherings, the U.K. is still aiming for deliberate ‘herd immunity’”. Fortune, March 14th.

They waited 6 days until March 18th before they “disrupted daily life”. It looks like they waited too long – 6 days means 5.6× more cases because the infection doubles every 2.4 days. That’d be bad even if they had a quality health care system; however, I estimate the U.K.’s health care system can handle 5× fewer patients than the US health care system.

Health care breakdown estimation

The number of infections I estimate each country can handle comes from Italy’s health care system being overrun, which happened when the had approximately 12,500 cases when their health care system reached capacity. I scale each countries infection density by the percent of the population over 65 and the number of “critical care beds”. This is a back-of-the-envelope calculation that only relies on all health care systems being equally capable.

To estimate the number of cases each country can handle, I use this equation:

# Measures how much greater concentration the given country can handle
ratio = (frac_over_65 / italy_frac_over_65) * (n_beds / italy_beds)

# Scale by population
limit = ratio * (pop / italy_pop) * italy_cases_breakdown

Data sources

My visualizations are geared towards to the US because I live there. If you’d like to customize this plot, I encourage you to download and customize the plots in the following notebooks:

Data sources include the following:

COVID-19 kills older people more often. 99% of the deaths are for people that are 50 or older. That’s not to say younger people who contract the virus does not have impacts: “some recovered patients may have reduced lung function and are left gasping for air while walking briskly” (source).

Here’s the fraction of each countries population that is over 65:

Here are the number on critical care beds per capita (source):

The Wikipedia page on “2020 coronavirus pandemic in the United States” also has this graph.

“Health care saturation” dates:

Social distancing dates:

  • Italy: Feb. 25th, chosen because on that day the following happening in Italy:
    • Morgan Stanley, Barclays, Mediobanca and UniCredit requested their employees work from home.
    • The University of Palermo suspended all activities.
    • The Italian Basketball Federation suspended all of its championship games.
    • The Italian stock index (FTSE MID) fell by 6%.
    • (Feb. 23rd) Last two days of the Carnival of Venice are cancelled
    • (Feb. 24th) The Basilicata instituted a 14-day mandatory quarantine for people arriving from Northern Italy
  • US: March 12th because the following happened in the US:
    • On March 11th or 12th, many universities announced canceled class would be canceled or held online when the students returned from Spring Break (including Cornell, Harvard, USC, UMN and UW–Madison).
    • On March 11th, Google, Facebook, Microsoft and E3 announced their technology conferences would be held online. Apple announced their technology conference would be moved online on March 13th.
    • On March 12th, The NHL, MLS and Women’s Soccer League announced they would cancel their season.
    • On March 12th, the NCAA announced it would cancel March Madness. 2020 is the first year in it’s 81-year history that it will not be held.
  • UK: March 19th because on March 18th, 19th and 20th the following happened in the UK:
    • On March 18th, McDonald’s and others “announced that they would not permit customers to sit and eat in stores”
    • On March 18th, many schools announced all university buildings would be closed to students and staff on March 20th (that Friday), including Welsh, Scottish and Northern Ireland K-12 schools and Cambridge University (detail).
    • On March 19th, a Sheffield light rail network and Yorkshire bus operators announced a reduced timetable

For the restaurant bookings chart, I used the data from OpenTable’s “The state of the restaurant industry

  1. Public health interventions and epidemic intensity during the 1918 influenza pandemic” by Richard J. Hatchett, Carter E. Mecher, and Marc Lipsitch. 2007. https://doi.org/10.1073/pnas.0610941104 (the chart I show is a modification of fig. 1).  2

  2. China first reported the virus doubled every 6.2 days (source). Either China is lying or the virus becoming more virulent as it ages. 

  3. From an article published March 14th in Fortune. More context for the quote is shown in the appendix 

  4. I can’t estimate the amount (on 2020-03-21). 

  5. And (naively) scaling the beds/ventilators. 

Robin Schroer (sulami)

How to Read a Book March 14, 2020 12:00 AM

A while ago I read How to Read a Book, the guide to extracting information from non-fiction books first published in 1940. Like many older books, there are parts of it that haven’t aged well, and in a particularly ironic fashion the book is quite verbose. I thought it might be useful to compile a short and simple step-by-step guide to efficiently reading non-fiction.

Imagine a book as an object with two dimensions: scope and detail. If you read the entire book cover to cover, you will learn about the full scope covered, in the level of detail of the book.Well, not really. Performing a single, cover to cover reading usually does not suffice to extract everything from a book.

But many books are several hundred pages long, and active reading demands attention and energy.

You can compromise on either of those dimensions (or both). If you just read the first half of the book, you are compromising on scope, as there is a whole second half of the subject you did not cover at all. If instead you want to compromise on detail, you should read “outside-in”, starting with strategic skimming and delving in deeper afterwards.

Start by getting a vague idea of everything covered in the book, going more and more into detail at each step. At any step you can stop and decide if the next level of details is worth the additional time required, but the knowledge you have extracted at that point is likely more useful than if you had compromised on scope.

The outside-in method is also useful for picking out books to read at a bookstore, as you can just start with the first few steps right there, and buy the book if you want to continue.

Simply follow these steps below, as long as you feel like gaining deeper understanding is worth the additional time. Ideally you want to have a way of taking notes, both to capture sections or topics to investigate further, but also to aid active reading.Active reading is one of these barely defined terms, but what I’m trying to express is the notion of thinking about the text as you read it, and actively trying to take in information. I might publish a piece on note-taking in the future.

  1. Read the back cover, and any other summaries and publisher blurbs.
  2. Read the table of contents, note the scope and structure. Identify the important and pivotal chapters.
  3. Read the preface.
  4. Scan the index and references for anything interesting to look at, both inside and outside this book.
  5. Read the opening and closing parts of the pivotal chapters, usually about a page each.
  6. Read the final part of the book containing content, such as a conclusion chapter or section.
  7. Pick a handful of random sections throughout the whole book and read a page or two each.

At this point you should have spent between 30 minutes and one hour, and have answers to these questions:

  • What is the book about, and which scope does it cover?
  • Are there any additional topics or works to investigate outside of its scope?
  • Which conclusions or opinions does it offer?

The last question is particularly important if you want to continue, as it sets the context of this book in comparison to others covering the same subject. Authors have different backgrounds and biases, which colour their works.A good example is David Allen’s Getting Things Done, which does hold some value for everyone, like the notion of the task inbox, but also contains a lot of absurd examples and use-cases which clearly show the kind of life he’s living, and the priorities he has.

The more thorough levels of reading take a lot more time, but can be useful if you want to get a deeper understanding of the subject. There are two more steps you can take to gain more insight:

  1. Continue reading the chapters, actually reading entire chapters now.
  2. Use other works covering the same subject and cross-reference the important points as you come across them. This allows you to get different viewpoints on the individual points.This is what Adler calls “syntopical” reading, which is a whole discipline on its own. I won’t cover it here, but you can find plenty of material online, and How to Read a Book also has a more detailed explanation.

If you get this deep into a subject, the first few steps will be very quick to perform, as you are already familiar with the subject, so you only need to find answers to the questions above before delving in deeper.

March 13, 2020

Bogdan Popa (bogdan)

Testing a Web API using rackcheck March 13, 2020 09:00 AM

Yesterday, I announced rackcheck, my new property-based testing library for Racket and I wanted to do a quick dive into one of the examples in the rackcheck repo where a simple web API is integration tested using PBT. You can find the full example here. The app being tested is a simple leaderboard HTTP API with 3 endpoints: GET /players: lists all the registered players in order from highest score to lowest.

Gonçalo Valério (dethos)

Django Friday Tips: Testing emails March 13, 2020 12:03 AM

I haven’t written one of these supposedly weekly posts with small Django tips for a while, but at least I always post them on Fridays.

This time I gonna address how we can test emails with the tools that Django provides and more precisely how to check the attachments of those emails.

The testing behavior of emails is very well documented (Django’s documentation is one of the best I’ve seen) and can be found here.

Summing it up, if you want to test some business logic that sends an email, Django replaces the EMAIL_BACKEND setting with a testing backend during the execution of your test suite and makes the outbox available through django.core.mail.outbox.

But what about attachments? Since each item on the testing outbox is an instance of the EmailMessage class, it contains an attribute named “attachments” (surprise!) that is list of tuples with all the relevant information:

("<filename>", "<contents>", "<mime type>")

Here is an example:

# utils.py
from django.core.mail import EmailMessage


def some_function_that_sends_emails():
    msg = EmailMessage(
        subject="Example email",
        body="This is the content of the email",
        from_email="some@email.address",
        to=["destination@email.address"],
    )
    msg.attach("sometext.txt", "The content of the file", "text/plain")
    msg.send()


# tests.py
from django.test import TestCase
from django.core import mail

from .utils import some_function_that_sends_emails


class ExampleEmailTest(TestCase):
    def test_example_function(self):
        some_function_that_sends_emails()

        self.assertEqual(len(mail.outbox), 1)

        email_message = mail.outbox[0]
        self.assertEqual(email_message.subject, "Example email")
        self.assertEqual(email_message.body, "This is the content of the email")
        self.assertEqual(len(email_message.attachments), 1)

        file_name, content, mimetype = email_message.attachments[0]
        self.assertEqual(file_name, "sometext.txt")
        self.assertEqual(content, "The content of the file")
        self.assertEqual(mimetype, "text/plain")

If you are using pytest-django the same can be achieved with the mailoutbox fixture:

import pytest

from .utils import some_function_that_sends_emails


def test_example_function(mailoutbox):
    some_function_that_sends_emails()

    assert len(mailoutbox) == 1

    email_message = mailoutbox[0]
    assert email_message.subject == "Example email"
    assert email_message.body == "This is the content of the email"
    assert len(email_message.attachments) == 1

    file_name, content, mimetype = email_message.attachments[0]
    assert file_name == "sometext.txt"
    assert content == "The content of the file"
    assert mimetype == "text/plain"

And this is it for today.

March 12, 2020

Jan van den Berg (j11g)

Remote – Jason Fried and David Heinemeier Hansson March 12, 2020 10:21 PM

I saw this tweet yesterday, and If you know me, you know I will not pass on an opportunity for a free book!

But more seriously, I have known Jason Fried and DHH for some time now. From their blog, their Twitter and multiple different podcasts. They have built their company around very clear and levelheaded thinking. So I wanted to read this anyway, and not just because it was free.

Enter COVID-19

Because of the Corona virus outbreak our 100 person company is on the verge of moving from a dedicated office to fully remote.

This Monday.

Yes.

So I am in dire need of some pointers. As a CTO and as rookie remote worker myself. And this book offers advice for both roles.

Remote – Jason Fried and David Heinemeier Hansson (2013) – 256 pages

To my surprise most of the ideas in the book were already familiar to me. I had heard them in one form or another in a podcast, blog or tweet. Nonetheless I love that it is all in one place!

The book is one of the quickest reads ever, and the structure and clear writing have everything to do with that. It is set up as a collection of themes around the topic of remote work. Every theme is 2 or 3 pages long and to the point! The chapters could be blog posts. And each one hammers one concept down.

Most of the value for me was near the end, the chapters about forward motion culture (good times), leveling the playing field and removing roadblocks (those are the chapter names) specifically stood out.

TLDR

Key takeaways — or reminders for myself — in order as they appear in the book. Some need extra context, but most speak for themselves I assume.

  • Trust: of course, this is absolutely key to almost anything. If you don’t trust your coworkers to do their work remote, then you probably don’t trust them when they are in the office I assume? And your job is not to babysit, but to manage. And management starts with trust!
  • Most people want to work as long as it’s stimulating and rewarding.
  • Culture is the spoken and unspoken values and actions of a company. Action speaks louder than words.
  • Remote work == work. Remote worker == worker. This is an important mindset. There is nothing less about it.
  • To instill a sense of company cohesion and to share forward motion everyone needs to feel that they’re in the loop.
  • Progress is a joy best shared with a coworker.
  • Meetings should be like salt — sprinkled carefully to enhance a dish, not poured recklessly over every forkful.
  • What a manager needs to establish is a culture of reasonable expectations.
  • Doing great work with great people is one of the most durable sources of happiness we humans can tap into.
  • The best ballast you can have is as many folks in your boat as possible with a thoroughly optimistic outlook.
  • Equal pay for equal work.
  • Great workers exhibit two great qualities: Smart and Get Things Done.
  • You should read, read and read some more.
  • Whatever it is, make it meaningful.
  • The job of a manager is not to herd cats, but to lead and verify the work.
  • Intrinsic motivation, open communication and meeting occasionally are the factors that make opensource projects successful and they apply just as much to successful remote work.
  • The goal is really just to keep a consistent, open line of communication.
  • The real dangers are the small (personal) things, that built up over time. Keep an eye out as for those as a manager (e.g. with one-on-one meetings).
  • It’s overwork, not underwork that is the real enemy in a successful remote-working environment.
  • Demarcate the difference between work and play. So, so important! Keep a separate room, separate computer, separate everything (as much as possible) for work.
  • The presence of other people (even if you don’t know them) can trick your mind into thinking that being productive is the only thing to do. So a coffeeshop might work for you!
  • The only reliable way to muster motivation is by encouraging people to work on the stuff they like, and care about, with people they like and care about.

Work == Remote work

Maybe you’re thinking that a lot of the items on the above list have little to do with remote work per se. Congratulations! You get it! This is sort of the point. There are of course differences, but the principles stay the same, it’s just that the needles moves a bit — as what to focus on or be mindful of — when it involves remote work. And this book provides clear and practical pointers for that.

This book is already from 2013, but Jason and DHH were clearly right when they saw remote work as the future of work. It makes a lot of sense. And not just because of a virus outbreak.

The post Remote – Jason Fried and David Heinemeier Hansson appeared first on Jan van den Berg.

Bogdan Popa (bogdan)

Announcing rackcheck March 12, 2020 12:00 PM

I’ve been playing around with property-based testing in Racket this past week. I started by forking the existing quickcheck library to try and add support for shrinking, but I quickly realized that I’d have to make a number of breaking changes to get it to work the way I wanted so, in the end, I decided to start a new library from scratch. The library is called rackcheck and you can grab it off of the package server.

March 11, 2020

Jeff Carpenter (jeffcarp)

Napa Half Race Report March 11, 2020 04:24 PM

The end of the pain cave. Note: the clock time in the picture is for a different race. 😆 I PR’d my half marathon at the Napa Half!! Yeahh!!!! Old PR: 1:31:37 at the 2019 Kaiser SF Half. New PR: 1:28:59 (link). Improvement: 2.9% (-2:38) In this post I want to look into the race and more closely analyze what went well, what could have gone better, and what was interesting about this race.

Dan Luu (dl)

How (some) good corporate engineering blogs are written March 11, 2020 12:00 AM

I've been comparing notes with people who run corporate engineering blogs and one thing that I think is curious is that it's pretty common for my personal blog to get more traffic than the entire corp eng blog for a company with a nine to ten figure valuation and it's not uncommon for my blog to get an order of magnitude more traffic.

I think this is odd because tech companies in that class often have hundreds to thousands of employees. They're overwhelmingly likely to be better equipped to write a compelling blog than I am and companies get a lot more value from having a compelling blog than I do.

With respect to the former, employees of the company will have done more interesting engineering work, have more fun stories, and have more in-depth knowledge than any one person who has a personal blog. On the latter, my blog helps me with job searching and it helps companies hire. But I only need one job, so more exposure, at best, gets me a slightly better job, whereas all but one tech company I've worked for is desperate to hire and loses candidates to other companies all the time. Moreover, I'm not really competing against other candidates when I interview (even if we interview for the same job, if the company likes more than one of us, it will usually just make more jobs). The high-order bit on this blog with respect to job searching is whether or not the process can take significant non-interview feedback or if I'll fail the interview because they do a conventional interview and the marginal value of an additional post is probably very low with respect to that. On the other hand, companies compete relatively directly when recruiting, so being more compelling relative to another company has value to them; replicating the playbook Cloudflare or Segment has used with their engineering "brands" would be a significant recruiting advantage. The playbook isn't secret: these companies broadcast their output to the world and are generally happy to talk about their blogging process.

Despite the seemingly obvious benefits of having a "good" corp eng blog, most corp eng blogs are full of stuff engineers don't want to read. Vague, high-level fluff about how amazing everything is, content marketing, handwave-y posts about the new hotness (today, that might be using deep learning for inappropriate applications; ten years ago, that might have been using "big data" for inappropriate applications), etc.

To try to understand what companies with good corporate engineering blog have in common, I interviewed folks at three different companies that have compelling corporate engineering blogs (Cloudflare, Heap, and Segment) as well as folks at three different companies that have lame corporate engineering blogs (which I'm not going to name).

At a high-level, the compelling engineering blogs had processes that shared the following properties:

  • Easy approval process, not many approvals necessary
  • Few or no non-engineering approvals required
  • Implicit or explicit fast SLO on approvals
  • Approval/editing process mainly makes posts more compelling to engineers
  • Direct, high-level (co-founder, C-level, or VP-level) support for keeping blog process lightweight

The less compelling engineering blogs had processes that shared the following properties:

  • Slow approval process
  • Many approvals necessary
  • Significant non-engineering approvals necessary
    • Non-engineering approvals suggest changes authors find frustrating
    • Back-and-forth can go on for months
  • Approval/editing process mainly de-risks posts, removes references to specifics, makes posts vaguer and less interesting to engineers
  • Effectively no high-level support for blogging
    • Leadership may agree that blogging is good in the abstract, but it's not a high enough priority to take concrete action
    • Reforming process to make blogging easier very difficult; previous efforts have failed
    • Changing process to reduce overhead requires all "stakeholders" to sign off (14 in one case)
      • Any single stakeholder can block
      • No single stakeholder can approve
    • Stakeholders wary of approving anything that reduces overhead
      • Approving involves taking on perceived risk (what if something bad happens) with no perceived benefit to them

One person at a company with a compelling blog noted that a downside of having only one approver and/or one primary approver is that if that person is busy, it can takes weeks to get posts approved. That's fair, that's a downside of having centralized approval. However, when we compare to the alternative processes, at one company, people noted that it's typical for approvals to take three to six months and tail cases can take a year.

While a few weeks can seem like a long time for someone used to a fast moving company, people at slower moving companies would be ecstatic to have an approval process that only takes twice that long.

Here are the processes, as described to me, for the three companies I interviewed (presented in sha512sum order, which is coincidentally ordered by increasing size of company, from a couple hundred employees to nearly one thousand employees):

Heap

  • Someone has an idea to write a post
  • Writer (who is an engineer) is paired with a "buddy", who edits and then approves the post
    • Buddy is an engineer who has a track record of producing reasonable writing
    • This may take a few rounds, may change thrust of the post
  • CTO reads and approves
    • Usually only minor feedback
    • May make suggestions like "a designer could make this graph look better"
  • Publish post

The first editing phase used to involve posting a draft to a slack channel where "everyone" would comment on the post. This was an unpleasant experience since "everyone" would make comments and a lot of revision would be required. This process was designed to avoid getting "too much" feedback.

Segment

  • Someone has an idea to write a post
    • Often comes from: internal docs, external talk, shipped project, open source tooling (built by Segment)
  • Writer (who is an engineer) writes a draft
    • May have a senior eng work with them to write the draft
  • Until recently, no one really owned the feedback process
    • Calvin French-Owen (co-founder) and Rick (engineering manager) would usually give most feedback
    • Maybe also get feedback from manager and eng leadership
    • Typically, 3rd draft is considered finished
    • Now, have a full-time editor who owns editing posts
  • Also socialize among eng team, get get feedback from 15-20 people
  • PR and legal will take a look, lightweight approval

Some changes that have been made include

  • At one point, when trying to establish an "engineering brand", making in-depth technical posts a top-level priority
  • had a "blogging retreat", one week spent on writing a post
  • added writing and speaking as explicit criteria to be rewarded in performance reviews and career ladders

Although there's legal and PR approval, Calvin noted "In general we try to keep it fairly lightweight. I see the bigger problem with blogging being a lack of posts or vague, high level content which isn't interesting rather than revealing too much."

Cloudflare

  • Someone has an idea to write a post
    • Internal blogging is part of the culture, some posts come from the internal blog
  • John Graham-Cumming (CTO) reads every post, other folks will read and comment
    • John is approver for posts
  • Matthew Prince (CEO) also generally supportive of blogging
  • "Very quick" legal approval process, SLO of 1 hour
    • This process is so lightweight that one person didn't really think of it as an approval, another person didn't mention it at all (a third person did mention this step)
    • Comms generally not involved

One thing to note is that this only applies to technical blog posts. Product announcements have a heavier process because they're tied to sales material, press releases, etc.

One thing I find interesting is that Marek interviewed at Cloudflare because of their blog (this 2013 blog post on their 4th generation servers caught his eye) and he's now both a key engineer for them as well as one of the main sources of compelling Cloudflare blog posts. At this point, the Cloudflare blog has generated at least a few more generations of folks who interviewed because they saw a blog post and now write compelling posts for the blog.

General comments

My opinion is that the natural state of a corp eng blog where people get a bit of feedback is a pretty interesting blog. There's a dearth of real, in-depth, technical writing, which makes any half decent, honest, public writing about technical work interesting.

In order to have a boring blog, the corporation has to actively stop engineers from putting interesting content out there. Unfortunately, it appears that the natural state of large corporations tends towards risk aversion and blocking people from writing, just in case it causes a legal or PR or other problem. Individual contributors (ICs) might have the opinion that it's ridiculous to block engineers from writing low-risk technical posts while, simultaneously, C-level execs and VPs regularly make public comments that turn into PR disasters, but ICs in large companies don't have the authority or don't feel like they have the authority to do something just because it makes sense. And none of the fourteen stakeholders who'd have to sign off on approving a streamlined process care about streamlining the process since that would be good for the company in a way that doesn't really impact them, not when that would mean seemingly taking responsibility for the risk a streamlined process would add, however small. An exec or a senior VP willing to take a risk can take responsibility for the fallout and, if they're interested in engineering recruiting or morale, they may see a reason to do so.

One comment I've often heard from people at more bureaucratic companies is something like "every company our size is like this", but that's not true. Cloudflare, a $6B company approaching 1k employees is in the same size class as many other companies with a much more onerous blogging process. The corp eng blog situation seems similar to situation on giving real interview feedback. interviewing.io claims that there's significant upside and very little downside to doing so. Some companies actually do give real feedback and the ones that do generally find that it gives them an easy advantage in recruiting with little downside, but the vast majority of companies don't do this and people at those companies will claim that it's impossible to do give feedback since you'll get sued or the company will be "cancelled" even though this generally doesn't happen to companies that give feedback and there are even entire industries where it's common to give interview feedback. It's easy to handwave that some risk exists and very few people have the authority to dismiss vague handwaving about risk when it's coming from multiple orgs.

Although this is a small sample size and it's dangerous to generalize too much from small samples, the idea that you need high-level support to blast through bureaucracy is consistent with what I've seen in other areas where most large companies have a hard time doing something easy that has obvious but diffuse value. While this post happens to be about blogging, I've heard stories that are the same shape on a wide variety of topics.

Appendix: examples of compelling blog posts

Here are some blog posts from the blogs mentioned with a short comment on why I thought the post was compelling. This time, in reverse sha512 hash order.

Cloudflare

Segment

Heap

One thing to note is that these blogs all have different styles. Personally, I prefer the style of Cloudflare's blog, which has a higher proportion of "deep dive" technical posts, but different people will prefer different styles. There are a lot of styles that can work.

Thanks to Marek Majkowski, Kamal Marhubi, Calvin French-Owen, John Graham-Cunning, Michael Malis, Matthew Prince, Yuri Vishnevsky, Julia Evans, Wesley Aptekar-Cassels, Nathan Reed, an anonymous commenter, plus sources from the companies I didn't name for comments/corrections/discussion; none of the people explicitly mentioned in the acknowledgements were sources for information on the less compelling blogs

March 10, 2020

Gustaf Erikson (gerikson)

Old Venus, George R.R. Martin & Gardner Dozois (editors) March 10, 2020 06:13 PM

A collection of newer SF stories set in a “retro-future” Venus - one where our neighbor planet isn’t a poisonous hellscape but where authors can indulge in imagining it before the Venera probes told us the sad truth.

A lot of stories riff off the Venera angle and include a sizable Soviet/Russian population on their version of the Morning Star.

Jan van den Berg (j11g)

The Quotable Walt Disney – Disney Book Group March 10, 2020 05:54 PM

I picked up this book on our honeymoon to Disney World. But I never read it, because; how do you read a book full of quotes? The answer is slowly!

Just grab it every now and then. And read a few pages.

The Quotable Walt Disney – Disney Book Group (2001) – 272 pages

Of course this is probably a heavily edited and well scrutinized book. Assembled under brand protection of the Disney Company, one of the biggest media companies in the world.

Nonetheless I learned quite a few things, and I have a much clearer understanding of the man who started it all. His choices, beliefs and character are still visible to this day in the fabric of the Disney company.

His ideas about what a park should be, what makes a good story and specifically what the role of entertainment should be. You get a pretty good idea about his thoughts on these matters by reading quotes.

The post The Quotable Walt Disney – Disney Book Group appeared first on Jan van den Berg.

Pete Corey (petecorey)

Querying for MongoDB Documents Where Every Sub-document Matches a Pattern March 10, 2020 12:00 AM

Imagine you’re implementing a feature to remove all “empty” documents from a MongoDB “events” collection. These documents aren’t completely empty. Each document will always have an array of checks, and each check will have an array of sources that satisfied that check.

For example, here are a few documents in our events collection:


{
    "checks": [
        {
            "sources": []
        },
        {
            "sources": []
        },
    ]
}

{
    "checks": [
        {
            "sources": [
                {
                    "_id": 1
                }
            ]
        },
        {
            "sources": []
        },
    ]
}

The first document is considered “empty”, because all of the sources arrays nested within each of the checks is empty ([]). The second document isn’t empty, because the sources array within the first check isn’t an empty array.

That’s Not How All Works!

That’s the situation I found myself in while working on a recent client project. Skimming (too quickly) through the MongoDB documentation, I decided that the $all query operator would be the perfect tool to solve my problem. After all, I was looking for documents where all of the sources arrays were empty.

My first stab at cleaning up these empty event documents looked something like this:


await Event.model.deleteMany({
    checks: {
        $all: [
            {
                $elemMatch: {
                    sources: { $eq: [] }
                }
            }
        ]
    }
});

My plan was to use the $all and $elemMatch MongoDB operators together to find and remove any event documents who’s checks sub-documents all contain empty ({ $eq: [] }) sources arrays.

Unfortunately, this is not how $all works.

The $all operator matches on any documents who’s sub-documents contain “all” of the elements listed within the $all operator. It does not match on documents where every sub-document matches the described element.

For example, a query like { foo: { $all: [ 1, 2, 3 ] } } will match on documents that have 1, 2, and 3 within their foo array. Their foo arrays can contain other elements, like 4, 5, or 6, but it must at least contain all of our specified values.

Applying this to our situation, we can see that our query will delete all documents that have at least one empty check, and not documents that have all empty checks.

This means we’re deleting event documents that aren’t empty!

We Don’t Even Need All

Armed with this new knowledge and guided by a failing test, I went to work trying to refactor my solution. After much tinkering, I came to the conclusion that the query I’m after can’t be directly expressed using the $all query operator. However, flipping the problem around, I came to a simpler solution that doesn’t make use of $all at all!

Instead of looking for documents where every sub-document in the checks array contains an empty sources array, let’s look for documents that have a non-empty sources array in any one of their checks. The result set we’re looking for is the inverse of this set of documents.

We can express this nice and neatly with MongoDB’s $not operator:


await Event.model.deleteMany({
    checks: {
        $not: {
            $elemMatch: {
                sources: { $ne: [] }
            }
        }
    }
});

As you can see, we’re removing any events who’s checks array does not contain any sub-documents with a non-empty sources array.

That’s it! With that, we’re able to clean up any truly “empty” events, while preserving any events with non-empty sources.

March 09, 2020

Patrick Louis (venam)

March 2020 Projects March 09, 2020 10:00 PM

Here comes another life update.
My biological clock seems to have chosen to remind me to post these updates once every 6 months, with seasonal changes.

Since the previous post, everything has been hectic. The country I’m living in, Lebanon, has plunged into an economic, financial, social, political, and more recently, health crisis. As I’m writing these words, it has defaulted on its debt payment.
While the focus of these posts is usually not my surroundings, the gravity of the situation is coloring whatever I’ve been up to recently. Thus, it’s imperative to start with a summary.

The past few months in Lebanon

Beirut 1913

Language: fire
Explanation:

My last post was on the 14th of September 2019, soon after publishing it a lot of events unfolded.
For already a year, the Lebanese government had been in a dire economical situation and clumsily tried to argue over which austerity measures they would implement as a remediation. One culprit was the currency itself. Having the Lebanese Lira as a pegged currency has advantages and disadvantages. At this point in time, it was apparent that keeping the peg was costly, it required the central bank to perform juggling of financial instruments, buying debts from the governments with its reserve, that would be paid back from the yearly revenue of the country. Debts that could be given in both Lebanese pounds and Eurobonds in US Dollars. However, the price of this maneuver grew quite a lot. The debt to gdp ratio had become ~150% and debt servicing made up 25 to 30 percent of the government expenditures (2019 budget, 2020 budget). Adding insult to injury, the majority of those debts used to be bought by local banks owned directly or indirectly by politicians.
The lack of growth in the economy, which relied heavily on tourism and a generous diaspora, can’t forever balance the peg. And the financial moves cover the fact that people are literally paying for it from their own pockets by giving up on basic public goods and instead investing in their currency keeping its value. Additionally, the almost non-existence of an industrial sector, or any export sectors for that matter, means that all products are imported and paid for in foreign currency, creating a vicious rampant inflation cycle. This lead to a rapid increase in the price of living in Lebanon, in power-distance, and in inequalities the past years.

In sum, there were a lot of talks about new taxes, fees, cutting wages, increasing the retirement age in the public sector and a freeze in hiring of government workers, but not a lot of talk related to the stimulation of the job market other than telling people to buy local goods and products regardless of their higher prices. Even though starting a business in Lebanon is a lot of a hassle and could be made easier. There has been a lot of sweet talk about this but rather delayed. Who would want to give up their current lifestyle, who would bear the burden. Should we play the blame game?

Consequentially, there were a lot of small protests but the first milestone, one that put the country on a standstill by having the highways arteries blocked, was by retired veterans of the security forces that disagreed and feared a bill that would reduce their retirement pensions and privileges. There was no actual decision at this point, only rumors, but the officials made it clear, to appease the crowd, that they wouldn’t touch retirees pensions.
The security sector is a sensible zone to wander in legislatively. There’s too much feelings involved as the army is the core of any country. Any form of disrespect would mean spitting on their service. This is the old adage from Ancient Rome.

An article on reuters quotes it best:

“This is our right, our sweat and blood,” former sergeant major Khodr Noureddine said. “Our salaries can’t even feed us for a day, there’s no school funding, no good healthcare.”

Yet, the collective memory forgets that this exact same scenario happened just two years earlier, in 2017, and that as a result many civil servants had gotten a raise of ~40%. What a conundrum for the dwindling budget.

The government was running out of options, options that wouldn’t put in peril their political future and their personal pockets.

We absolutely have to mention that the Lebanese army is heavily sponsored, not only by the Lebanese government where it makes around ~15% of the government expenditure, but by foreign benefactors such as the USA, Canada, Russia, the Netherlands, the UK, Italy, France, and more, which give billions in aid, donations, training, and equipment. Actually, the Lebanese government spends almost nothing on equipment and everything on personnel.

There was a time, before the civil war, where the military service used to be obligatory in Lebanon. The army it composed used to be the smallest one in the Middle East. After the civil war, in 1989, most of the militias merged their arms together, other than Hezbollah, forming a new army and officially taking over the country with the sectarian Taif agreement.
As a consequence, some politicians still use the army as their own militia, or to sway public opinions. They use it as a bargaining tool, offering jobs to the poorest while making their families pledge allegiance to the political movement. The army has become a sort of universal basic income in Lebanon, which means that everyone is related to someone with a link in the army. The army is engrained in everything.

Meanwhile, Hezbollah was playing tug of war with Israel, as it usually do, showing it doesn’t care for Lebanese issues but more about its own interests, its brand of resistance™. If there was nothing to resist against they would have no raison d’être.

While security is extremely important, political elites play the sentiment card to siphon foreign aids and bribe higher ups.
The entirety of the Lebanese political system works like this, factions taking care of their turfs, appropriating themselves positions in the parliaments that are related to projects where they can cook the books, redirect funds, or take gigantic commissions. Each party has a cartel of its own, even the non-official ones. It’s a system were hiring is done on “wasta”, greed, bribe, and nepotism. A system where someone do you favors so that you are forced to pay them back, mafia-style.
The ongoing internationally funded projects in Lebanon are countless, but the huge discrepancies between what’s on paper and what’s on the ground is astonishing. Lebanon receives an insane amount of funding from: the UN, the EU, the World Bank, the Islamic development bank, Commonwealth Bank, Arab Fund for Economic and Social Development, OPEC Fund for International Development, Italy, French development agency, Japan, Saudi Arabia, UAE, Kuwait, Qatar, Germany, USA/USAID, China, Turkey, Oman, Spain, etc.. Yet, money always seems to be a problem, and projects almost never get finished. This example from the European union funds shows how rotten the public sector is.
Clientelism reigns and companies that don’t accept the status-quo never get contracts.

There’s no real long term planning, no real urban planning. Why is that? No organization in Lebanon has the right to sign big contracts other than the CDR, the Commission for Development and Reconstruction. A defunct branch of government used after the civil war that was brought back from its grave by the late Rafik Hariri to bypass regulation and legislation, to make the projects he wanted to do quicker. The CDR is under direct supervision from the prime minister, which was still Saad Hariri until recently. The projects and their fundings are listed on the cdr.gov.lb site, here is the 2018 report, and as you can see, a big share of the funding doesn’t come from Lebanon, most of the infrastructure in Lebanon was built by foreign entities.

A scandal was reported at the start of October: The PM, Hariri, had given $16M to a South African bikini model named Candice Van Der Merwe as a “gift” and her government was suing her for not paying taxes on it. And while the public took it only from the shock value perspective, what was going on under the table was even more appalling. A nugget of information that was lost was that Candice is the daughter of a real estate tycoon and money launderer named Gary Van Der Merwe, that resides and stores his money in the Seychelles tax haven. The same Gary that appeared 350 times in court for shady businesses.
What a way to stir up a population fearing economic collapse and already sick of everything!

Soon after an environmental catastrophe shook the country, one where the inefficacy of the decision makers was apparent: The forest of Lebanon were on fire and it couldn’t be stopped. Everyone soon learned that the government had no equipment to deal with the situation other than using their anti-riot vehicles mounted with water cannons. The firefighter helicopters weren’t maintained and aid from the neighbors Cyprus, Turkey, Greece, and Jordan was required.
The population got even more stirred up, sentiments were at their highest, even though some politicians tried to move the discussion to their advantage by blaming and using sectarianism.

A week or so after the wildfires, rumors got out that part of the austerity measures would include new taxes on cigarettes and VoIP calls. People got the word and went to the streets. What used to be small protests until that point, soon erupted into humongous ones.
Disapprovement was on everyone’s tongue.

Folks still wonder to this day what could’ve went into the mind of the person that proposed the idea, in reality it was a proposal based on deprecated Lebanese telecommunication laws, old man’s thinking per say. To understand this decision we have to read the telecom sector history.
It may sound surprising to hear this in 2020, but VoIP is still technically illegal in Lebanon even though anyone with a computing device can open an app and call anyone anywhere else on the planet and only pay for the internet data consumption. The ban is only enforced on the telecom level to not loose profit from calls, this is why we don’t have VoIP with LTE.
Back in the days, before 2004-2005, two private companies used to manage and build the telecom sector, Cellis and Libancell. Each had very long term contracts with the government with a maximum quota of subscribers and a specific percentage it would pay the government from its profit (from 20-40%). Their pricing used to be exorbitant compared to anywhere else. However, those companies got greedy, they went above the quota of subscribers and hid their profits. So, the government sued them for breach of contract and required them to pay a fine. The contract was void, the issue was settled, and Lebanon had in its hands telecom equipment but nobody to run them.
A government agency called the TRA, Telecom Regulation Authority, got created to set the national standards and regulations, set the pricing of telecom related activities, meanwhile a bidding war got announced for who would manage the network and at which price. The contract would work like this: the government would pay for the telecom equipment and the expansion, the companies winning the bids would manage the network and get paid a fixed fee on a 3-4 year contract basis, and any profit from it would go in its entirety to the government. (for more info see 2007 and 2005)
This is the situation we have today and the reason why someone thought it was a novel idea to tax VoIP but in reality it was an archaic one.

While the price for mobile related activities has radically changed with the creation of the TRA, old perceptions are hard to move. The price on mobile is like a tax that cancels itself to pay back the energy sector debt. Plus, mobiles are an inherent part of Lebanese society and its business related activities, especially WhatsApp. Everyone is on their phones all the time.

Hence, the nation got on the streets and showed its discontentment, more so than when it did in 2015-2016 with the YouStink! movement that I indirectly hinted at in this post.
It was surprising to see the awakening of a population that had been trained to stay away from politics, to not have its own opinion and leave the field for the big players to enjoy. Lebanese have been numbed, apathetic, and sickened of this from years of civil war, they have been tortured by fear transmitted through sectarianism propaganda and tribalism. The majority being completely disconnected from their own country, acting as if they weren’t living in it, dreaming of fleeing and always thinking of the worst.
Political discussions in the country are still immature, based solely on emotions, personal benefits, turf and religious zeal. Moreover, the social contract is missing, there’s a lack of social responsibilities when it concerns the country as a whole. In sum, there’s no unity, everyone has to live in their own bubbles and that means a full on disrespect of anything that doesn’t profit single individuals.
Impatient opportunistic exploiters has been the way I used to describe the daily social interactions in Lebanon.

For these reasons, I thought the protests wouldn’t stick but they did, one after the other, day after day, it erupted on the street. Enormous groups gathered, dancing and chanting “revolution”, “everyone means everyone”, wanting the fall of the current political elites, not sparring anyone. Overall, the protests were non-violent.

That led the PM giving himself an ultimatum to adjust his austerity plan to please the populace, otherwise he’d resign. Although we all know that a crowd isn’t a beast that can be appeased with words and reasons. Thus, the PM Hariri resigned along with his cabinet, leaving the country in a chaotic state, on the 29th of October 2019.

This event engender yet another slip in the Lebanese bonds, one more dive for the economy to handle: The lack of a functioning government.

The banks, which had closed on and off since the protests started for security reasons, instituted restrictions on customers’ accounts. From now on, no exchange from Lebanese Lira to US Dollar would be allowed, which used to be the norm as Lebanon has a pegged currency. Additionally, customers would be rationed their amount of hard cash currency per week and per month, locally and internationally. The ration went from $400/week to now $200/week in cash. That had the aftereffect of causing a slow run on banks and giving rise to many parallel markets for dollars which the central bank has tried to thwart but couldn’t because it lacked the reserves to inject money into the system.
This created an increase in prices in any imported products as money exchangers were taking a higher conversion rate from LBP to USD. The country experienced protests from shortages in gas stations, bakeries, hospitals, etc.. Currently, Lebanese have lost around 40% of their purchasing power. The biggest loss is on people that don’t deal internationally, the ones that are paid exclusively in Lebanese Lira, the people working for the government for instance.

Then a new government was formed, so called politico-technocrat, led by Hassan Diab as PM, a PhD professor of Engineering at the American University of Beirut that published over 150 scientific articles and that previously ran as minister of education from 2011 till 2014.
He was torn apart between having to face both the anger of the street, the previous political cartel of sectarian civil war veterans, the elephant in the room, Hezbollah, and a monstrous economical and financial situation. Not an easy feat to tackle, a political suicide one might say.
His small cabinet of 20, defunct of Hezbollah opposition parties which boycotted the formation, had the difficult task to balance public trust, local and international, along with austerity measures, a crumbling money with a population losing their livelihood, and the financial sector on a locked down.

Naturally, the anger on the street couldn’t be rationalized with. Emotions create rise to hatred and blame. The train couldn’t be stopped. Indirectly, the people made a choice because they had no trust in the system, a system they were disconnected from. They refused austerity measures because they didn’t believe they would work. They felt disrespected, they felt a lack of true leadership.
Lebanese made a choice, a self-imposed reset, whatever the cost.

So on the 9th of March 2020, Lebanon’s first-ever debt default happened.

Amidst all this, the covid-19 cases in the country are increasing day by day, currently sitting at around 50 cases, the first patients coming from Iran and Egypt.

And.. The global stock market is in a whirlwind of a mess too.

Is it a collapse, I’ll let you judge.

This is what’s been happening the past few months. If you want to read more about this, check this wikipedia article.

Psychology, Philosophy & Books

Language: brains
Explanation:

Collapse

I’ve read quite a lot, on average 2 to 3 books a months.

Here’s a list of the technical books:

  • Software Architecture in Practice 3rd edition - Finished it since last time
  • Building evolutionary architectures - Neal Ford
  • The clean coder: A code of conduct for professional programmers - Robert Martin
  • Domain-Driven Design - Eric Evans - Currently reading, almost done

I’ve also embarked on the first technical podcast that I enjoyed:

The non-technical books are:

  • Artemis - Andy Weir
  • Station Eleven - St.John Mandel Emily
  • The institute - Stephen King
  • Maybe you should talk to someone - Lori Gottlieb
  • Howto - Randall Munroe
  • Whatif - Randall Munroe
  • The handmaid’s Tale - Margaret Atwood
  • The design of everyday things - Don Norman
  • The Art of loving - Erich Fromm
  • Thinking, Fast and slow - Kahneman
  • Collapse, how societies choose to fail or survice - Jared Diamond’s
  • Capitalism without capital: The rise of the intangible economy - Jonathan Haskel
  • Alchemy & Mysticism (Bibliotheca Universalis) - Taschen - Currently reading
  • Factfulness - Rossling - Currently devouring

Other podcasts I’ve added to my long subscription list are:

  • The motley fool money
  • Freakonomics radio
  • The Lebanese politics podcast
  • Mythical monsters

Usually I avoid getting books before I’m done reading what I have but considering the current situation I’ve added the following never-old classics to my collection:

  • The better angels of our nature: Why violence declined - Stephen Pinker
  • The book of M - Peng Shepherd
  • The gene - Siddhartha mukherjee
  • Beyond software architecture: Hohmann
  • Algorithms: Sedgewick
  • Computer Architecture: A Quantitative approach - Hennesy Patterson
  • Operating system concepts - Silberschatz
  • Compilers - Aho Lam Seth Ullman

My book collection is stacking up and I’m really thinking I should build a library.

Learning, Growth

Language:growth
Explanation:

Roman blacksmith

I’ve grown quite fond of the domain of software architecture. So far I’m on a roll, reading and watching videos every day on the topic.

In total:

  • 335 articles
  • 95 videos
  • 10 research papers
  • 5 books

My Anki decks are full of knowledge that I review to not let it spill out after I’m done with consuming the content.

I’ve attempted Kata programming exercises, which is something anyone should try once to get out of their routine coding techniques.

I’ve published my first blog post related to software engineering practices.

Eventually, I want to pass the Carnegie Mellon University software architecture certification program. But, as you’ve probably noticed, it’s not the best moment to spend money online with all the banking restrictions.
So instead I’ll continue writing articles whenever I can, sharing knowledge.

Ascii Art & Art

Language: ASCII
Explanation:

Lebanon 2020
Building a Universe

I haven’t published many ASCII art pieces. The one you see above and the ones in Impure 74 and Impure 75 packs.

Other Learning

Language: gray matter
Explanation:

Elevate App

It’s been almost two months since I’ve done my last singing session. Instead, I’ve learned to juggle, at least with 3 balls.

As far as the brainteaser, Elevate, goes, I’ve used it religiously and am on a 318 days streak.

Apart from those I’ve been reading on emotional intelligence on Emotional Granurality and been watching ton of documentaries on Curiosity Stream along with some movies about the 2008 crisis and the Enron company.

On the other side, food wise, I’ve been gifted a fantastic apron for my birthday and it has reinvigorated my love for cooking.

Finally, it’s noticeable that I’ve developed a huge interest in Urban planning, management, leadership view, decision making, and been digging into RFPs, big projects, history, sociology, deals, primarily I’ve been interested in politics.

Life and other hobbies

Language: life
Explanation:

brussels

I spent a week in Brussels at the beginning of February to attend FOSDEM and do some sightseeing.

At FOSDEM I’ve finally met face to face with some nice fellows from nixers.net, namely eyenx, joshua, and pyratebeard. The conference itself has so many talks that it was a hassle to plan properly which one to attend. In the end it was a success and I didn’t miss any of my favorite speakers or topics.

In Belgium I had a blast at the museum of art and history, I’m a big fan of museums.

Now

I would end with “Which all leads to what’s in store for tomorrow. More of the same but upgraded.” but today I need a bit of change.

I’ll still put all my efforts on software architecture but I’ll also try to open new opportunities. Freshness is really needed right now.
Plus whatever I’m already doing but more of it!

This is it!

As usual… If you want something done, no one’s gonna do it for you, use your own hands.
And let’s go for a beer together sometime, or just chill.

March 06, 2020

Jeff Carpenter (jeffcarp)

Stop Using Bookmarks March 06, 2020 05:42 PM

The bookmark. A small, flimsy piece of paper or plastic wedged into the latest book you’re reading (unless you deface your books by dog-earing them—but hear me out dog-earers: this essay is still for you). Bookmarks are a convenience I’m guessing you take for granted and to which you don’t give a second thought. In this essay I will attempt to pursuade you to stop using them altogether. Why?? You ask, would you ever want to eliminate such a simple convenience?

Tobias Pfeiffer (PragTob)

Slides: Stories in Open Source March 06, 2020 09:23 AM

Yesterday at RUG::B I tried something I’d never done before: a more personal, story driven talk. And in order to adequately illustrate it and how different Open Source feel to me I also opted paint some very special drawings. Open Source is something I’ve been doing for longer than people pay me to do programming. […]

March 03, 2020

Gustaf Erikson (gerikson)

Armageddon: The Battle for Germany, 1944-1945 by Max Hastings March 03, 2020 07:10 PM

A great book about the tragic and bloody end of World War 2. Hastings excels at switching from grand strategy to the viewpoints of individuals, soldiers, civilians, prisoners.

Dan Luu (dl)

The growth of command line options, 1979-Present March 03, 2020 12:00 AM

My hobby: opening up McIlroy’s UNIX philosophy on one monitor while reading manpages on the other.

The first of McIlroy's dicta is often paraphrased as "do one thing and do it well", which is shortened from "Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new 'features.'"

McIlroy's example of this dictum is:

Surprising to outsiders is the fact that UNIX compilers produce no listings: printing can be done better and more flexibly by a separate program.

If you open up a manpage for ls on mac, you’ll see that it starts with

ls [-ABCFGHLOPRSTUW@abcdefghiklmnopqrstuwx1] [file ...]

That is, the one-letter flags to ls include every lowercase letter except for {jvyz}, 14 uppercase letters, plus @ and 1. That’s 22 + 14 + 2 = 38 single-character options alone.

On ubuntu 17, if you read the manpage for coreutils ls, you don’t get a nice summary of options, but you’ll see that ls has 58 options (including --help and --version).

To see if ls is an aberration or if it's normal to have commands that do this much stuff, we can look at some common commands, sorted by frequency of use.

table {border-collapse:collapse;margin:0px auto;}table,th,td {border: 1px solid black;}td {text-align:center;}

command1979199620152017
ls11425858
rm371112
mkdir0467
mv091314
cp0183032
cat1121212
pwd0244
chmod0699
echo1455
man5163940
which011
sudo02325
tar1253134139
touch191111
clear000
find14578282
ln0111516
ps4228585
ping121229
kill1333
ifconfig162525
chown061515
grep11224545
tail171213
df0101718
top61214

This table has the number of command line options for various commands for v7 Unix (1979), slackware 3.1 (1996), ubuntu 12 (2015), and ubuntu 17 (2017). Cells are darker and blue-er when they have more options (log scale) and are greyed out if no command was found.

We can see that the number of command line options has dramatically increased over time; entries tend to get darker going to the right (more options) and there are no cases where entries get lighter (fewer options). 

McIlroy has long decried the increase in the number of options, size, and general functionality of commands1:

Everything was small and my heart sinks for Linux when I see the size [inaudible]. The same utilities that used to fit in eight k[ilobytes] are a meg now. And the manual page, which used to really fit on, which used to really be a manual page, is now a small volume with a thousand options... We used to sit around in the UNIX room saying "what can we throw out? Why is there this option?" It's usually, it's often because there's some deficiency in the basic design -- you didn't really hit the right design point. Instead of putting in an option, figure out why, what was forcing you to add that option. This viewpoint, which was imposed partly because there was very small hardware ... has been lost and we're not better off for it.

Ironically, one of the reasons for the rise in the number of command line options is another McIlroy dictum, "Write programs to handle text streams, because that is a universal interface" (see ls for one example of this).

If structured data or objects were passed around, formatting could be left to a final formatting pass. But, with plain text, the formatting and the content are intermingled; because formatting can only be done by parsing the content out, it's common for commands to add formatting options for convenience. Alternately, formatting can be done when the user leverages their knowledge of the structure of the data and encodes that knowledge into arguments to cut, awk, sed, etc. (also using their knowledge of how those programs handle formatting; it's different for different programs and the user is expected to, for example, know how cut -f4 is different from awk '{ print $4 }'2). That's a lot more hassle than passing in one or two arguments to the last command in a sequence and it pushes the complexity from the tool to the user.

People sometimes say that they don't want to support structured data because they'd have to support multiple formats to make a universal tool, but they already have to support multiple formats to make a universal tool. Some standard commands can't read output from other commands because they use different formats, wc -w doesn't handle Unicode correctly, etc. Saying that "text" is a universal format is like saying that "binary" is a universal format.

I've heard people say that there isn't really any alternative to this kind of complexity for command line tools, but people who say that have never really tried the alternative, something like PowerShell. I have plenty of complaints about PowerShell, but passing structured data around and easily being able to operate on structured data without having to hold metadata information in my head so that I can pass the appropriate metadata to the right command line tools at that right places the pipeline isn't among my complaints3.

The sleight of hand that's happening when someone says that we can keep software simple and compatible by making everything handle text is the pretense that text data doesn't have a structure that needs to be parsed4. In some cases, we can just think of everything as a single space separated line, or maybe a table with some row and column separators that we specify (with some behavior that isn't consistent across tools, of course). That adds some hassle when it works, and then there are the cases where serializing data to a flat text format adds considerable complexity since the structure of data means that simple flattening requires significant parsing work to re-ingest the data in a meaningful way.

Another reason commands now have more options is that people have added convenience flags for functionality that could have been done by cobbling together a series of commands. These go all the way back to v7 unix, where ls has an option to reverse the sort order (which could have been done by passing the output to something like tac had they written tac instead of adding a special-case reverse option).

Over time, more convenience options have been added. For example, to pick a command that originally has zero options, mv can move and create a backup (three options; two are different ways to specify a backup, one of which takes an argument and the other of which takes zero explicit arguments and reads an implicit argument from the VERSION_CONTROL environment variable; one option allows overriding the default backup suffix). mv now also has options to never overwrite and to only overwrite if the file is newer.

mkdir is another program that used to have no options where, excluding security things for SELinux or SMACK as well as help and version options, the added options are convenience flags: setting the permissions of the new directory and making parent directories if they don't exist.

If we look at tail, which originally had one option (-number, telling tail where to start), it's added both formatting and convenience options For formatting, it has -z, which makes the line delimiter null instead of a newline. Some examples of convenience options are -f to print when there are new changes, -s to set the sleep interval between checking for -f changes, --retry to retry if the file isn't accessible.

McIlroy says "we're not better off" for having added all of these options but I'm better off. I've never used some of the options we've discussed and only rarely use others, but that's the beauty of command line options -- unlike with a GUI, adding these options doesn't clutter up the interface. The manpage can get cluttered, but in the age of google and stackoverflow, I suspect many people just search for a solution to what they're trying to do without reading the manpage anyway.

This isn't to say there's no cost to adding options -- more options means more maintenance burden, but that's a cost that maintainers pay to benefit users, which isn't obviously unreasonable considering the ratio of maintainers to users. This is analogous to Gary Bernhardt's comment that it's reasonable to practice a talk fifty times since, if there's a three hundred person audience, the ratio of time spent watching to the talk to time spent practicing will still only be 1:6. In general, this ratio will be even more extreme with commonly used command line tools.

Someone might argue that all these extra options create a burden for users. That's not exactly wrong, but that complexity burden was always going to be there, it's just a question of where the burden was going to lie. If you think of the set of command line tools along with a shell as forming a language, a language where anyone can write a new method and it effectively gets added to the standard library if it becomes popular, where standards are defined by dicta like "write programs to handle text streams, because that is a universal interface", the language was always going to turn into a write-only incoherent mess when taken as a whole. At least with tools that bundle up more functionality and options than is UNIX-y users can replace a gigantic set of wildly inconsistent tools with a merely large set of tools that, while inconsistent with each other, may have some internal consistency.

McIlroy implies that the problem is that people didn't think hard enough, the old school UNIX mavens would have sat down in the same room and thought longer and harder until they came up with a set of consistent tools that has "unusual simplicity". But that was never going to scale, the philosophy made the mess we're in inevitable. It's not a matter of not thinking longer or harder; it's a matter of having a philosophy that cannot scale unless you have a relatively small team with a shared cultural understanding, able to to sit down in the same room.

If anyone can write a tool and the main instruction comes from "the unix philosophy", people will have different opinions about what "simplicity" or "doing one thing"5 means, what the right way to do things is, and inconsistency will bloom, resulting in the kind of complexity you get when dealing with a wildly inconsistent language, like PHP. People make fun of PHP and javascript for having all sorts of warts and weird inconsistencies, but as a language and a standard library, any commonly used shell plus the collection of widely used *nix tools taken together is much worse and contains much more accidental complexity due to inconsistency even within a single Linux distro and there's no other way it could have turned out. If you compare across Linux distros, BSDs, Solaris, AIX, etc., the amount of accidental complexity that users have to hold in their heads when switching systems dwarfs PHP or javascript's incoherence. The most widely mocked programming languages are paragons of great design by comparison.

To be clear, I'm not saying that I or anyone else could have done better with the knowledge available in the 70s in terms of making a system that was practically useful at the time that would be elegant today. It's easy to look back and find issues with the benefit of hindsight. What I disagree with are comments from Unix mavens speaking today; comments like McIlroy's, which imply that we just forgot or don't understand the value of simplicity, or Ken Thompson saying that C is as safe a language as any and if we don't want bugs we should just write bug-free code. These kinds of comments imply that there's not much to learn from hindsight; in the 70s, we were building systems as effectively as anyone can today; five decades of collective experience, tens of millions of person-years, have taught us nothing; if we just go back to building systems like the original Unix mavens did, all will be well. I respectfully disagree.

Appendix: memory

Although addressing McIlroy's complaints about binary size bloat is a bit out of scope for this, I will note that, in 2017, I bought a Chromebook that had 16GB of RAM for $300. A 1 meg binary might have been a serious problem in 1979, when a standard Apple II had 4KB. An Apple II cost $1298 in 1979 dollars, or $4612 in 2020 dollars. You can get a low end Chromebook that costs less than 1/15th as much which has four million times more memory. Complaining that memory usage grew by a factor of one thousand when a (portable!) machine that's more than an order of magnitude cheaper has four million times more memory seems a bit ridiculous.

I prefer slimmer software, which is why I optimized my home page down to two packets (it would be a single packet if my CDN served high-level brotli), but that's purely an aesthetic preference, something I do for fun. The bottleneck for command line tools isn't memory usage and spending time optimizing the memory footprint of a tool that takes one meg is like getting a homepage down to a single packet. Perhaps a fun hobby, but not something that anyone should prescribe.

Methodology for table

Command frequencies were sourced from public command history files on github, not necessarily representative of your personal usage. Only "simple" commands were kept, which ruled out things like curl, git, gcc (which has > 1000 options), and wget. What's considered simple is arbitrary. Shell builtins, like cd weren't included.

Repeated options aren't counted as separate options. For example, git blame -C, git blame -C -C, and git blame -C -C -C have different behavior, but these would all be counted as a single argument even though -C -C is effectively a different argument from -C.

The table counts sub-options as a single option. For example, ls has the following:

--format=WORD across -x, commas -m,  horizontal  -x,  long  -l,  single-column  -1,  verbose  -l, vertical -C

Even though there are seven format options, this is considered to be only one option.

Options that are explicitly listed as not doing anything are still counted as options, e.g., ls -g, which reads Ignored; for Unix compatibility. is counted as an option.

Multiple versions of the same option are also considered to be one option. For example, with ls, -A and --almost-all are counted as a single option.

In cases where the manpage says an option is supposed to exist, but doesn't, the option isn't counted. For example, the v7 mv manpage says

BUGS

If file1 and file2 lie on different file systems, mv must copy the file and delete the original. In this case the owner name becomes that of the copying process and any linking relationship with other files is lost.

Mv should take -f flag, like rm, to suppress the question if the target exists and is not writable.

-f isn't counted as a flag in the table because the option doesn't actually exist.

The latest year in the table is 2017 because I wrote the first draft for this post in 2017 and didn't get around to cleaning it up until 2020.

mjd on the Unix philosophy, with an aside into the mess of /usr/bin/time vs. built-in time.

mjd making a joke about the proliferation of command line options in 1991.

On HN:

p1mrx:

It's strange that ls has grown to 58 options, but still can't output \0-terminated filenames

As an exercise, try to sort a directory by size or date, and pass the result to xargs, while supporting any valid filename. I eventually just gave up and made my script ignore any filenames containing \n.

whelming_wave:

Here you go: sort all files in the current directory by modification time, whitespace-in-filenames-safe. The printf (od -> sed)' construction converts back out of null-separated characters into newline-separated, though feel free to replace that with anything accepting null-separated input. Granted,sort --zero-terminated' is a GNU extension and kinda cheating, but it's even available on macOS so it's probably fine.

      printf '%b' $(
        find . -maxdepth 1 -exec sh -c '
          printf '\''%s %s\0'\'' "$(stat -f '\''%m'\'' "$1")" "$1"
        ' sh {} \; | \
        sort --zero-terminated | \
        od -v -b | \
        sed 's/^[^ ]*//
      s/ *$//
      s/  */ \\/g
      s/\\000/\\012/g')

If you're running this under zsh, you'll need to prefix it with `command' to use the system executable: zsh's builtin printf doesn't support printing octal escape codes for normally printable characters, and you may have to assign the output to a variable and explicitly word-split it.

This is all POSIX as far as I know, except for the sort.

The Unix haters handbook.

Why create a new shell?

Thanks to Leah Hanson, Hillel Wayne, Wesley Aptekar-Cassels, Mark Jason Dominus, Travis Downs, and Yuri Vishnevsky for comments/corrections/discussion.


  1. This quote is slightly different than the version I've seen everywhere because I watched the source video. AFAICT, every copy of this quote that's on the internet (indexed by Bing, DuckDuckGo, or Google) is a copy of one person's transcription of the quote. There's some ambiguity because the audio is low quality and I hear something a bit different than whoever transcribed that quote heard. [return]
  2. Another example of something where the user absorbs the complexity because different commands handle formatting differently is time formatting -- the shell builtin time is, of course, inconsistent with /usr/bin/time and the user is expected to know this and know how to handle it. [return]
  3. Just for example, you can use ConvertTo-Json or ConvertTo-CSV on any object, you can use cmdlets to change how properties are displayed for objects, and you can write formatting configuration files that define how you prefer things to be formatted.

    Another way to look at this is through the lens of Conway's law. If we have a set of command line tools that are built by different people, often not organizationally connected, the tools are going to be wildly inconsistent unless someone can define a standard and get people to adopt it. This actually works relatively well on Windows, and not just in PowerShell.

    A common complaint about Microsoft is that they've created massive API churn, often for non-technical organizational reasons (e.g., a Sinofsky power play, like the one described in the replies to the now-deleted Tweet at https://twitter.com/stevesi/status/733654590034300929). It's true. Even so, from the standpoint of a naive user, off-the-shelf Windows software is generally a lot better at passing non-textual data around than *nix. One thing this falls out of is Windows's embracing of non-textual data, which goes back at least to COM in 1999 (and arguably OLE and DDE, released in 1990 and 1987, respectively).

    For example, if you copy from Foo, which supports binary formats A and B, into Bar, which supports formats B and C and you then copy from Bar into Baz, which supports C and D, this will work even though Foo and Baz have no commonly supported formats. 

    When you cut/copy something, the application basically "tells" the clipboard what formats it could provide data in. When you paste into the application, the destination application can request the data in any of the formats in which it's available. If the data is already in the clipboard, "Windows" provides it. If it isn't, Windows gets the data from the source application and then gives to the destination application and a copy is saved for some length of time in Windows. If you "cut" from Excel it will tell "you" that it has the data available in many tens of formats. This kind of system is pretty good for compatibility, although it definitely isn't simple or minimal.

    In addition to nicely supporting many different formats and doing so for long enough that a lot of software plays nicely with this, Windows also generally has nicer clipboard support out of the box.

    Let's say you copy and then paste a small amount of text. Most of the time, this will work like you'd expect on both Windows and Linux. But now let's say you copy some text, close the program you copied from, and then paste it. A mental model that a lot of people have is that when they copy, the data is stored in the clipboard, not in the program being copied from. On Windows, software is typically written to conform to this expectation (although, technically, users of the clipboard API don't have to do this). This is less common on Linux with X, where the correct mental model for most software is that copying stores a pointer to the data, which is still owned by the program the data was copied from, which means that paste won't work if the program is closed. When I've (informally) surveyed programmers, they're usually surprised by this if they haven't actually done copy+paste related work for an application. When I've surveyed non-programmers, they tend to find the behavior to be confusing as well as surprising.

    The downside of having the OS effectively own the contents of the clipboard is that it's expensive to copy large amounts of data. Let's say you copy a really large amount of text, many gigabytes, or some complex object and then never paste it. You don't really want to copy that data from your program into the OS so that it can be available. Windows also handles this reasonably: applications can provide data only on request when that's deemed advantageous. In the case mentioned above, when someone closes the program, the program can decide whether or not it should push that data into the clipboard or discard it. In that circumstance, a lot of software (e.g., Excel) will prompt to "keep" the data in the clipboard or discard it, which is pretty reasonable.

    It's not impossible to support some of this on Linux. For example, the ClipboardManager spec describes a persistence mechanism and GNOME applications generally kind of sort of support it (although there are some bugs) but the situation on *nix is really different from the more pervasive support Windows applications tend to have for nice clipboard behavior.

    [return]
  4. Another example of this are tools that are available on top of modern compilers. If we go back and look at McIlroy's canonical example, how proper UNIX compilers are so specialized that listings are a separate tool, we can see that this has changed even if there's still a separate tool you can use for listings. Some commonly used Linux compilers have literally thousands of options and do many things. For example, one of the many things clang now does is static analysis. As of this writing, there are 79 normal static analysis checks and 44 experimental checks. If these were separate commands (perhaps individual commands or perhaps a static_analysis command, they'd still rely on the same underlying compiler infrastructure and impose the same maintenance burden -- it's not really reasonable to have these static analysis tools operate on plain text and reimplement the entire compiler toolchain necessary to get the point where they can do static analysis. They could be separate commands instead of bundled into clang, but they'd still take a dependency on the same machinery that's used for the compiler and either impose a maintenance and complexity burden on the compiler (which has to support non-breaking interfaces for the tools built on top) or they'd break all the time.

    Just make everything text so that it's simple makes for a nice soundbite, but in reality the textual representation of the data is often not what you want if you want to do actually useful work.

    And on clang in particular, whether you make it a monolithic command or thousands of smaller commands, clang simply does more than any compiler that existed in 1979 or even all compilers that existed in 1979 combined. It's easy to say that things were simpler in 1979 and that us modern programmers have lost our way. It's harder to actually propose a design that's actually much simpler and could really get adopted. It's impossible that such a design could maintain all of the existing functionality and configurability and be as simple as something from 1979.

    [return]
  5. Since its inception, curl has gone from supporting 3 protocols to 40. Does that mean it does 40 things and it would be more "UNIX-y" to split it up into 40 separate commands? Depends on who you ask. If each protocol were its own command, created and maintained by a different person, we'd be in the same situation we are with other commands. Inconsistent command line options, inconsistent output formats despite it all being text streams, etc. Would that be closer to the simplicity McIlroy advocates for? Depends on who you ask. [return]

March 01, 2020

Pierre Chapuis (catwell)

Tools March 01, 2020 06:00 PM

A few days ago I read this article which made me want to list some of the tools used at the four startups I have worked at so far.

I have mostly listed SaaS tools here, but you can also check out the ~stack tag on my Pinboard for more.

I have also added a section at the end for some tools I use for my own projects.

Moodstocks (2010 - 2013)

Lima (2014 - 2019)

Chilli (eFounders, 2019)

Inch (2019 - 2020)

Personal

February 29, 2020

Derek Jones (derek-jones)

Source code chapter of ‘evidence-based software engineering’ reworked February 29, 2020 10:33 PM

The Source code chapter of my evidence-based software engineering book has been reworked (draft pdf).

When writing the first version of this chapter, I was not certain whether source code was a topic warranting a chapter to itself, in an evidence-based software engineering book. Now I am certain. Source code is the primary product delivery, for a software system, and it is takes up much of the available cognitive effort.

What are the desirable characteristics that source code should have, to minimise production costs per unit of functionality? This is what an evidence-based chapter on source code is all about.

The release of this chapter completes my second pass over the material. Readers will notice the text still contains ... and ?‘s. The third pass will either delete these, or say something interesting (I suspect mostly the former, because of lack of data).

Talking of data, February has been a bumper month for data (apologies if you responded to my email asking for data, and it has not appeared in this release; a higher than average number of people have replied with data).

The plan is to spend a few months getting a beta release ready. Have the beta release run over the summer, with the book in the shops for Christmas.

I’m looking at getting a few hundred printed, for those wanting paper.

The only publisher that did not mind me making the pdf freely available was MIT Press. Unfortunately one of the reviewers was foaming at the mouth about the things I had to say about software engineering researcher (it did not help that I had written a blog post containing a less than glowing commentary on academic researchers, the week of the review {mid-2017}); the second reviewer was mildly against, and the third recommended it.

If any readers knows the editors at MIT Press, do suggest they have another look at the book. I would rather a real publisher make paper available.

Next, getting the ‘statistics for software engineers’ second half of the book ready for a beta release.

Jan van den Berg (j11g)

Surely You’re Joking, Mr. Feynman! – Ralph Leighton and Richard Feynman February 29, 2020 02:09 PM

The next day I rolled up my picture, put it in the back of my station wagon, and my wife Gweneth wished me good luck as I set out to visit the brothels of Pasadena to sell my drawing.

Richard Feynman

Well, I definitely wasn’t expecting such sentences in this collection of anecdotes from famous popular physicist Feynman. But I can safely say, it certainly is in line with the rest of the book!

Surely You’re Joking, Mr. Feynman! – Ralph Leighton and Richard Feynman (1985) – 368 pages

Richard Feyman, the eccentric Nobel laureate physicist lived a remarkable life. And he liked to tell stories. This combination makes this collection of stories in this book widely appealing and entertaining. There is surprisingly very little mention of hardcore science, and absolutely no mention of his most famous and arguably important finding. It is though, about everything else.

So you can expect stories about stories about: picking locks, building the A-bomb, cracking codes, painting, playing the bongos, teaching in South America, learning languages and — to my surprise — unabashedly chasing girls, and visiting topless bars and Las Vegas. Feynman was not your typical scientist.

However what shines through in all these stories are two things: Feyman’s almost unmatched curiosity in almost anything (except the social studies) and his relentless pursuit for applying critical — scientific — thinking in everything (go to the source, can you explain it in an example?). The combination of these factors make Feyman an extraordinary and relentless teacher. He wants to understand and greatly feels the need to explain things.

The book is a lot of fun and surprising on some level (I won’t spoil the fun). Even though some parts make your eyebrows raise.

Though he passed away in 1988, Youtube is filled with videos of Feyman explaining things and they are a pure delight. Here he is explaining fire:

The post Surely You’re Joking, Mr. Feynman! – Ralph Leighton and Richard Feynman appeared first on Jan van den Berg.

February 26, 2020

Sevan Janiyan (sevan)

Full name of root account in BSD February 26, 2020 10:44 PM

NetBSD now has a users(7) and groups(7) manual. Looking into what entries existed in the passwdand groupfiles I wondered about root’s full name who we now know as Charlie Root in the BSDs. Root was called Ernie Co-vax in 3BSD. An Ernie Kovacs also shows up in adduser(8). The 4.4BSD System Manager’s Manual mentions “at Berkeley we have …

Átila on Code (atilaneves)

Seriously, just use D to call C from Python February 26, 2020 02:49 PM

Last week I suggested that if you want to call C code from Python that you should use D. I still think that 4 lines of code (two of which are header includes) and a build system is hard to beat if that indeed is your goal. The internet, of course, had different ideas. I […]

February 23, 2020

Derek Jones (derek-jones)

The wisdom of the ancients February 23, 2020 08:51 PM

The software engineering ancients are people like Halstead and McCabe, along with less well known ancients (because they don’t name anything after them) such as Boehm for cost estimation, Lehman for software evolution, and Brooks because of a book; these ancients date from before 1980.

Why is the wisdom of these ancients still venerated (i.e., people treat them as being true), despite the evidence that they are very inaccurate (apart from Brooks)?

People hate a belief vacuum, they want to believe things.

The correlation between Halstead’s and McCabe’s metrics, and various software characteristics is no better than counting lines of code, but using a fancy formula feels more sophisticated and everybody else uses them, and we don’t have anything more accurate.

That last point is the killer, in many (most?) cases we don’t have any metrics that perform better than counting lines of code (other than taking the log of the number of lines of code).

Something similar happened in astronomy. Placing the Earth at the center of the solar system results in inaccurate predictions of where the planets are going to be in the sky; adding epicycles to the model helps to reduce the error. Until Newton came along with a model that produced very accurate results, people stuck with what they knew.

The continued visibility of COCOMO is a good example of how academic advertising (i.e., publishing papers) can keep an idea alive. Despite being more sophisticated, the Putnam model is not nearly as well known; Putnam formed a consulting company to promote this model, and so advertised to a different market.

Both COCOMO and Putnam have lines of code as an integral component of their models, and there is huge variability in the number of lines written by different people to implement the same functionality.

People tend not to talk about Lehman’s quantitative work on software evolution (tiny data set, and the fitted equation is very different from what is seen today). However, Lehman stated enough laws, and changed them often enough, that it’s possible to find something in there that relates to today’s view of software evolution.

Brooks’ book “The Mythical Man-Month” deals with project progress and manpower; what he says is timeless. The problem is that while lots of people seem happy to cite him, very few people seem to have read the book (which is a shame).

There is a book coming out this year that provides lots of evidence that the ancient wisdom is wrong or at best harmless, but it does not contain more accurate models to replace what currently exists :-(

February 22, 2020

Jeremy Morgan (JeremyMorgan)

Stop Fearing the Whiteboard. Conquer It. February 22, 2020 12:00 AM

It’s time we stopped fearing, complaining, and arguing about whiteboard coding interviews. With a solid plan, a little skilling up, and some practice, you can master the whiteboard interview. We can argue for days about the validity of this type of interview, but the fact is many organizations require it. So are you going to let this obstacle stop you from getting the job you want? Of course not. Let’s tackle the whiteboard interview, and defeat it.

February 21, 2020

Jan van den Berg (j11g)

Infinite Jest – David Foster Wallace February 21, 2020 03:38 PM

Today is February 21st, David Foster Wallace‘s birthday. So it’s rather fitting that today I finished reading his magnum opus: Infinite Jest. The notoriously long and difficult book from 1996 with visionary insights on modern life. Infinite Jest is one of the biggest books ever written, and it certainly is the biggest book I have ever read.

It took me somewhere between 50 and 60 hours over the course of four months and I — had to — read several other books in between to cool off. So, it was quite the experience. I will try to write some of those personal experiences and observations down here. This post will not discuss or try to dissect the work and themes itself. Many, many, many books and articles have already tried to do that.

No footnotes, I promise.

Why did I read it?

I knew about Wallace and his challenging writing style, and he always looked intimidating. But last year I picked up ‘A Supposedly Fun Thing I’ll Never Do Again’ and it was one of the greatest and funniest things I have ever read (specifically the title essay). And while I was reading that book I tracked down a copy of Infinite Jest. I clearly remember my hesitation. It looked terrifyingly dense. Over 1100 pages in the smallest possible font and hardly any line breaks. And this was just the typesetting, let alone the content! But I bought it anyway, with no real plan to actually read it, yet… But in October I watched ‘The End of the Tour’ — which I greatly enjoyed — so I just had to know what this was all about.

There it was. Fresh.

Did I enjoy it?

Infinite Jest is not your typical enjoyable reader-friendly book. It is also definitely not a funny book. And it certainly wasn’t intended to be. There are multiple horrific scenes of rape, incest, mutilation and drug abuse (and their consequences i.e. faceless babies). Overall it has a very serious tone and message and I don’t think I expected that.

So whether I enjoyed it in the traditional sense is the wrong question, but I am still glad I read it.

Why am I glad?

Infinite Jest is unlike anything else I have ever read.

Not only has Wallace an utterly uniquely distinctive style which is impossible to copy, but above all — this book makes you work! Some books you can read in one sitting, with Infinite Jest I could do 20 pages per hour at most, before I was spent. I could not read this book for hours on end. Because it demands your complete and undivided attention. And this can be exhausting. And at times it felt my life was on hold reading it (would I be able to finish it, where is this going?).

Nothing in the story is left unexplored and Wallace’s vocabulary is unmatched (there are over 20.000 unique words in Infinite Jest, this is only rivaled by the dictionary I would think…). And on top of that, he makes up his own words to fit a specific mood or sentence. This makes the reading experience draining.

However, no word seems wasted. Every word seems vital. It is verbose but still perfectly tight prose.

All these things make it a different, but rewarding, reading experience.

A book book

I was also left with the idea that this experience can only exist on paper (or at most maybe an audiobook). There is no other way to convey the meaning — which just happen to be expressed by words — of this book. You cannot retell the story, you cannot film the scenes, you would lose everything. I know there are also translations, but I worry that a translation wouldn’t provide the same experience.

Wallace demonstrates and justifies with Infinite Jest that writing is an art form of its own and that reading is a different experience from anything else.

Hey, me too Bill! Just read it.

All this seems difficult

On top of all this, it is also mostly a non-lineair story, jumping from the ‘Tennis Academy’ to the ‘Recovery House’ and all over the place. The story seems like loosely connected collection of shards. And you could almost crack it open anywhere and have the same experience as when you start on page one. I think this is deliberate: it makes your brain work. You have to make the connections yourself. Just like there are no explainers or introductions anywhere.

But I suspect this also aided Wallace’s writing? I sense these fragments, sometimes only separated by a double space, are different writing days (?) because they sometimes differ heavily in tone, ideas and intensity.

So what’s it all about then?!

There are many, many different themes and many different ideas in this book. Just as there are a couple of general commonalities. One commonality that stood out specifically for me is that every character in this book is flawed. And trust me, there are many characters. Drug addicts, wheelchair assassins, rapists, dealers, handicapped, deformed, talented athletes who threw their life away, you name it: but every character is flawed.

Suffering

This brings me to what I personally derived as one of the main recurring subjects of the book: suffering. Everyone and everything is suffering from something. The human condition, and in particular its flaws, are at the center of what drives the story.

Wallace himself states that one of the main messages of Infinite Jest is that “pleasure is dangerous“. Specifically: too much pleasure. We are flawed human beings with little to no self-control and a need for pleasure. And this causes problems.

Maybe that’s why he makes the reader work so hard in what could have been a self-referential book title. Pleasure should always be balanced out by hard work.

Breaking its back!

This Wallace guy is quite the character!

Yes, you are nothing short of a genius when you can write like this. And from his collection of essays I had already concluded, he was a chameleon, able to effortlessly jump between different styles. But I have often wondered how deep he had to go as a writer to write Infinite Jest. Because whoever wrote this must have balanced on the brink of madness and peered over (I had the same experience with Virginia Woolf).

But this is also what kept me from enjoying the book to its fullest I think. I don’t mind working hard for a book. But when I read Infinite Jest I was 50% of the time immersed in the story and 50% of the time thinking: how did he do this?! How does his brain work? How can someone write this? It really threw me off.

But either way, in the end, just that may be one of the most rewarding experiences from this book. It’s like you get to spend some time in the mind of a genius, and you may not fully understand the machinery and mechanism but you are certainly in on the outcomes.

Now what?

I heard a great quote: “you might just finish the book and see hints of Infinite Jest in everything that is around you“. I think that is true, this book cannot but leave a mark, and make you see the world through a different lens. And I am glad I am in on it now. And yes, Wallace in 1996 was able to foresee and extrapolate developments with regards to humans and their relation to pleasure (whether this is technology or drugs) that raise questions that are alarmingly relevant today.

At times I thought: when I am finished I am never reading this again! But at other times I thought: this is exceptional I should reread this every so often. (Somewhere in the first few hundreds pages is a story of an addict waiting for a hit. It is quite something). I also made hundreds of yellow marks I want to revisit for various reasons and I want to read more by Wallace. Also there are many Infinite Jest resources I want to read to see what I might have missed (I suspect quite a bit). However, there are also many more books I also want to read of course.

Yeah? Well, you know, that’s just like uh, your opinion, man.

Should *I* read it?

Yes. However, I can’t promise you will enjoy it. And it is an expensive investment of your time (then again, finishing a great video game costs the same amount of time). But the experience of Infinite Jest will be rewarding because it will be unlike anything you have ever read.

Lock the door, turn off your phone and enter the mind of David Foster Wallace.

The post Infinite Jest – David Foster Wallace appeared first on Jan van den Berg.

February 20, 2020

Awn Umar (awn)

plausibly deniable encryption February 20, 2020 12:00 AM

It is safe to assume that in any useful cryptosystem Ck C_k Ck there exists at least one person with access to the key k k k. An adversary with sufficient leverage can bypass the computational cost of a conventional attack by exerting their influence on this person.

xkcd_protocol

xkcd: security

The technique is sometimes referred to as rubber-hose cryptanalysis and it gives the adversary some serious creative freedom. The security properties of the cryptosystem now rely not on the assumed difficulty of mathematical trapdoor functions but on some person’s tolerance to physical or psychological violence. A thief knows that pointing a gun will unlock a safe much faster than using a drill. An adversarial government will similarly seek information using torture and imprisonment rather than computational power.

Many countries have key-disclosure legislation. In the United Kingdom, RIPA was first used against animal-rights activists to unlock data found on machines seized during a raid on their homes. The penalty for refusing to hand over key material is up to two years in prison.

Say Alice has a cryptosystem Ck C_k Ck whose security properties rely on the secrecy of the key k k k. To defend against attacks of this form Alice needs some way to keep k k k a secret. She could,

  1. Claim that k k k is not known. This includes if it has been lost or forgotten.
  2. Claim the ciphertext c c c is random noise and so is not decryptable.
  3. Provide an alternate key j j j under which decryption produces a fake plaintext.

Suppose Mallory is the adversary who wants k k k and suppose Alice makes a claim X X X in order to avoid revealing k k k. Defining success can be tricky as Mallory can ultimately decide not to believe any claim that Alice makes. However we will simply say Mallory wins if she can show ¬X \neg X ¬X and therefore assert that Alice has access to k k k and is able to provide it. So for Alice to win, X X X must be unfalsifiable and hence a plausible defence.

As a side note, if Alice knows and can demonstrate ¬X \neg X ¬X whereas Mallory cannot, then clearly she is missing some necessary information. Kerckhoffs’s principle says that the security of a cryptosystem Ck C_k Ck should rely solely on the secrecy of the key k k k, so in general we want proving ¬X \neg X ¬X to require knowing k k k.

We will ignore weaknesses related to operational security or implementation. For example if Mallory hears Alice admit to Bob that she is lying or if she finds a fragment of plaintext in memory then Alice has lost. However these situations are difficult to cryptographically protect against and so we assume security in this regard.

Pleading ignorance (1) of k k k is an easy strategy for Alice as it leaves little room for dispute and it can be deployed as a tactic almost anywhere. Mallory must show that k k k is known and this is difficult to do without actually producing it. Perhaps the key was on a USB device that has been lost, or was written down on a piece of paper that burned down along with Alice’s house. Mere forgetfulness however implies that the data does exist and the only barrier to retrieving it is in accessing Alice’s memories. This may not be satisfactory.

Asserting the non-existence (2) of the ciphertext is equivalent to claiming that k k k does not exist and so cannot be disclosed. Plausibility comes from the fact that ciphertext is indistinguishable from random noise. This means that given some potential ciphertext c c c an adversary cannot say if c c c is uniformly sampled or if c=Ek(m) c = E_k(m) c=Ek(m) is a valid message m m m encrypted under some key k k k. To prove that c c c is not random noise Mallory must produce k k k and compute m m m, which is assumed to be infeasible.

TrueCrypt and VeraCrypt allow the creation of hidden volumes and hidden operating systems. The idea is that an ordinary encrypted volume will have unused regions of the disk filled with random data, and so a hidden volume can be placed there without revealing its existence.

hidden_volume_layout

On-disk layout of an encrypted VeraCrypt volume.

Suppose we have a boot drive with a standard volume protected by the key k1 k_1 k1 and a hidden volume protected by the key k2 k_2 k2. The existence of the unencrypted boot-loader reveals the fact that the standard volume exists and so Mallory can confidently demand its key. Alice may safely provide Mallory with k1 k_1 k1 thereby revealing the innocuous contents of the standard volume. However when Alice enters k2 k_2 k2, the boot-loader fails to unlock the standard region so instead it tries to decrypt at the offset where the hidden volume’s header would reside. If the hidden volume exists and if the provided key is correct, this operation is successful and the boot-loader proceeds to boot the hidden operating system.

This is an example of providing a decoy decryption (3) but you may notice that Alice also had to claim that the remaining “unused” space on the drive is random noise (2) and not valid ciphertext. The necessity of a secondary claim is not a special case but a general property of systems that try to provide deniability in this way.

xkcd: random number

xkcd_random_number

It's possible to distinguish ciphertext from data from this randomness source.

Providing a plausible reason for the existence of leftover data can be tricky. VeraCrypt relies on the fact that drives are often wiped with random data before being used as encrypted volumes. In other situations we may have to be sneakier.

This strategy does have some practical limitations. If the volume hosts an operating system, the innocuous OS has to be used as frequently as the hidden one to make it seem legitimate. For example if Alice provides the key and Mallory sees that the last login was two years ago, but she knows that Alice logged in the day before, then Mallory can be pretty sure something is off. Also consider what happens if Mallory sees a snapshot of the drive before and after some data is modified in the hidden volume. She then knows that there is data there and that it is not simply the remnants of an earlier wipe.

The Dissident Protocol

Imagine a huge library where every book is full of gibberish. There is a librarian who will help you store and retrieve your data within the library. You give her a bunch of data and a master key. She uses the master key to derive an encryption key and a random location oracle. The data is then split into book-sized pieces, each of which is encrypted with the derived key. Finally each encrypted book is stored at a location provided by the oracle.

More formally, assume “library” means key-value store. Consider a key-derivation function Φ:K→K×K \Phi : K \to K \times K Φ:KK×K and a keyed cryptographic hash function H:K×N→K H : K \times \mathbb{N} \to K H:K×NK, where K K K is the key space. We also define an encryption function E:K×M→C E : K \times M \to C E:K×MC and the corresponding decryption function D:K×C→M D : K \times C \to M D:K×CM, where M M M and C C C are the message space and ciphertext space, respectively.

Alice provides a key k k k which Faythe uses to derive the sub-keys a,b=Φ(k) a, b = \Phi(k) a,b=Φ(k). Alice then provides some data p p p which is split into chunks p1,p2,…,pn p_1, p_2, \ldots, p_n p1,p2,,pn, where every pi p_i pi is padded to the same length. Finally, Faythe stores the entries {Ha(i):Eb(pi)} \{ H_a(i) : E_b(p_i) \} {Ha(i):Eb(pi)} in the key-value store.

For decryption, again Alice provides the key k k k and Faythe computes the sub-keys a,b=Φ(k) a, b = \Phi(k) a,b=Φ(k). She then iterates over i∈N i \in \mathbb{N} iN, retrieving the values ci c_i ci corresponding to the keys Ha(i) H_a(i) Ha(i) and computing Db(ci)=Db(Eb(pi))=pi D_b(c_i) = D_b(E_b(p_i)) = p_i Db(ci)=Db(Eb(pi))=pi, stopping at i=n+1 i = n + 1 i=n+1 where the key-value pair does not exist. The plaintext is then p=p1∥p2∥…∥pn p = p_1 \mathbin\Vert p_2 \mathbin\Vert \ldots \mathbin\Vert p_n p=p1p2pn, after unpadding each pi p_i pi.

Some extra consideration has to go into integrity and authentication to prevent attacks where the data Alice stores is not the data she gets back out. We leave this out for simplicity’s sake.

Suppose the library contains n n n books in total. Mallory cannot say anything about Alice’s data apart from that its total size is less than or equal to the amount of data that can be stored within n n n books. If, under duress, Alice is forced to reveal a decoy key that pieces together data from m m m books, she needs some way to explain the remaining n−m n - m nm books that were not used. She could claim that,

  1. The key for those books has been lost or forgotten.
  2. They are composed of random noise and so cannot be decrypted.
  3. They belong to other people and so the key is not known to her.

This will look mostly familiar. Alice is trying to avoid revealing her actual data by providing a decoy key that unlocks some innocuous data. She then has to make a secondary claim in order to explain the remaining data that was not decrypted under the provided key.

Claiming ignorance (A) has the same trivial plausibility argument and practical limitation as before (1).

Asserting that the leftover books are composed of random bytes (B) requires an explanation for how they came to be there. She could say simply that she added them but this is a can of worms that we want to keep closed. If some software implementation decides how many decoy books to add, it would necessarily leak information to Mallory about the expected frequency of decoys. This value can be compared with Alice’s claim of n−m n - m nm decoys to come up with an indicator of whether Alice is lying.

We have the same problem if the frequency is decided randomly as the value would have to lie within some range. We can get around this by asking Alice herself to decide the frequency, but this is messy and humans are bad at being unpredictable. In any case, this strategy boils down to Alice claiming “I added decoy entries explicitly in order to explain leftover data”, and this would rightly make an adversary extremely suspicious.

A better way to utilise B is for Faythe to replace books that are to be deleted with random data instead of removing them outright. Then Alice can claim that the remaining books have been deleted and therefore the data no longer exists and cannot be decrypted. This way potentially any number of leftover books can be easily explained, but it does mean that the size of our library will only increase over time.

Claim C is new and has some appealing properties but it can’t be used on a personal storage medium—like Alice’s laptop hard drive—as there is unlikely to be a plausible reason for other people’s data to be there. Imagine instead that the “library” is hosted on a service shared by multiple people. Then it is easy for Alice to claim that the remaining entries are not hers. Mallory would need leverage over every other person using the service in order to disprove Alice’s claim. Such a service has to be carefully designed however. For example if it stored how much space Alice is using then this value can be compared with Alice’s claim and Mallory wins.

There are some drawbacks of this scheme. There is an overhead in storing data in discrete, padded chunks. Modifying data in a non-trivial way may be expensive. Overwriting entries instead of removing them uses up storage space that is “wasted” in the sense that it does not hold any useful data. In designing this protocol what I have found is that we have to be extremely careful to avoid losing our deniability. Any implementation has to be verified to ensure that it does not fall short in this regard.

However we now have something that lets you have an arbitrary number of independent “folders” stored amongst numerous indistinguishable packets, with an adversary being unable to infer any information other than the maximum size of the stored data. This is a powerful property but it should be considered as part of the whole picture including your threat model and usability requirements.

There is an experimental client implementing the spirit of this protocol here. As of the time of writing, it is not ready for serious use. However there are some exciting ideas I have for making this into a production ready and usable client in the (hopefully) near future.

February 19, 2020

Marc Brooker (mjb)

Firecracker: Lightweight Virtualization for Serverless Applications February 19, 2020 12:00 AM

Firecracker: Lightweight Virtualization for Serverless Applications

Our second paper for NSDI'20.

In 2018, we announced Firecracker, an open source VMM optimized for multi-tenant serverless and container workloads. We heard some interest from the research community, and in response wrote up our reasoning behind building Firecracker, and how its used inside AWS Lambda.

That paper was accepted to NSDI'20, and is available here. Here's the abstract:

Serverless containers and functions are widely used for deploying and managing software in the cloud. Their popularity is due to reduced cost of operations, improved utilization of hardware, and faster scaling than traditional deployment methods. The economics and scale of serverless applications demand that workloads from multiple customers run on the same hardware with minimal overhead, while preserving strong security and performance isolation. The traditional view is that there is a choice between virtualization with strong security and high overhead, and container technologies with weaker security and minimal overhead. This tradeoff is unacceptable to public infrastructure providers, who need both strong security and minimal overhead. To meet this need, we developed Fire-cracker, a new open source Virtual Machine Monitor (VMM)specialized for serverless workloads, but generally useful for containers, functions and other compute workloads within a reasonable set of constraints. We have deployed Firecracker in two publically available serverless compute services at Amazon Web Services (Lambda and Fargate), where it supports millions of production workloads, and trillions of requests per month. We describe how specializing for serverless in-formed the design of Firecracker, and what we learned from seamlessly migrating Lambda customers to Firecracker.

Like any project the size of Firecracker, it was developed by a team of people from vision to execution. I played only a small role in that, but it's been great to work with the team (and the community) on getting Firecracker out, adding features, and using it in production at pretty huge scale.

Firecracker is a little bit unusual among software projects of having an explicit goal of being simple and well-suited for a relatively small number of tasks. That doesn't mean it's simplistic. Choosing what to do, and what not to do, was some of the most interesting decisions to be made in it's development. I'm particularly proud of how well the team made those decisions, and continues to make them.

February 18, 2020

Dan Luu (dl)

Suspicious discontinuities February 18, 2020 12:00 AM

If you read any personal finance forums late last year, there's a decent chance you ran across a question from someone who was desperately trying to lose money before the end of the year. There are a number of ways someone could do this; one commonly suggested scheme was to buy put options that were expected to expire worthless, allowing the buyer to (probably) take a loss.

One reason people were looking for ways to lose money was that, in the U.S., there's a hard income cutoff for a health insurance subsidy at $48,560 for individuals (higher for larger households; $100,400 for a family of four). There are a number of factors that can cause the details to vary (age, location, household size, type of plan), but across all circumstances, it wouldn't have been uncommon for an individual going from one side of the cut-off to the other to have their health insurance cost increase by roughly $7200/yr. That means if an individual buying ACA insurance was going to earn $55k, they'd be better off reducing their income by $6440 and getting under the $48,560 subsidy ceiling than they are earning $55k.

Although that's an unusually severe example, U.S. tax policy is full of discontinuities that disincentivize increasing earnings and, in some cases, actually incentivize decreasing earnings. Some other discontinuities are the TANF income limit, the Medicaid income limit, the CHIP income limit for free coverage, and the CHIP income limit for reduced-cost coverage. These vary by location and circumstance; the TANF and Medicaid income limits fall into ranges generally considered to be "low income" and the CHIP limits fall into ranges generally considered to be "middle class". These subsidy discontinuities have the same impact as the ACA subsidy discontinuity -- at certain income levels, people are incentivized to lose money.

Anyone may arrange his affairs so that his taxes shall be as low as possible; he is not bound to choose that pattern which best pays the treasury. There is not even a patriotic duty to increase one's taxes. Over and over again the Courts have said that there is nothing sinister in so arranging affairs as to keep taxes as low as possible. Everyone does it, rich and poor alike and all do right, for nobody owes any public duty to pay more than the law demands.

If you agree with the famous Learned Hand quote then losing money in order to reduce effective tax rate, increasing disposable income, is completely legitimate behavior at the individual level. However, a tax system that encourages people to lose money, perhaps by funneling it to (on average) much wealthier options traders by buying put options, seems sub-optimal.

A simple fix for the problems mentioned above would be to have slow phase-outs instead of sharp thresholds. Slow phase-outs are actually done for some subsidies and, while that can also have problems, they are typically less problematic than introducing a sharp discontinuity in tax/subsidy policy.

In this post, we'll look at a variety of discontinuities.

Hardware or software queues

A naive queue has discontinuous behavior. If the queue is full, new entries are dropped. If the queue isn't full, new entries are not dropped. Depending on your goals, this can often have impacts that are non-ideal. For example, in networking, a naive queue might be considered "unfair" to bursty workloads that have low overall bandwidth utilization because workloads that have low bandwidth utilization "shouldn't" suffer more drops than workloads that are less bursty but use more bandwidth (this is also arguably not unfair, depending on what your goals are).

A class of solutions to this problem are random early drop and its variants, which gives incoming items a probability of being dropped which can be determined by queue fullness (and possibly other factors), smoothing out the discontinuity and mitigating issues caused by having a discontinuous probability of queue drops.

This post on voting in link aggregators is fundamentally the same idea although, in some sense, the polarity is reversed. There's a very sharp discontinuity in how much traffic something gets based on whether or not it's on the front page. You could view this as a link getting dropped from a queue if it only receives N-1 votes and not getting dropped if it receives N votes.

College admissions and Pell Grant recipients

Pell Grants started getting used as a proxy for how serious schools are about helping/admitting low-income students. The first order impact is that students above the Pell Grant threshold had a significantly reduced probability of being admitted while students below the Pell Grant threshold had a significantly higher chance of being admitted. Phrased that way, it sounds like things are working as intended.

However, when we look at what happens within each group, we see outcomes that are the opposite of what we'd want if the goal is to benefit students from low income families. Among people who don't qualify for a Pell Grant, it's those with the lowest income who are the most severely impacted and have the most severely reduced probability of admission. Among people who do qualify, it's those with the highest income who are mostly likely to benefit, again the opposite of what you'd probably want if your goal is to benefit students from low income families.

We can see these in the graphs below, which are histograms of parental income among students at two universities in 2008 (first graph) and 2016 (second graph), where the red line indicates the Pell Grant threshold.

Histogram of income distribution of students at two universities in 2008; high incomes are highly overrepresented relative to the general population, but the distribution is smooth

Histogram of income distribution of students at two universities in 2016; high incomes are still highly overrepresented, there's also a sharp discontinuity at the Pell grant threshold; plot looks roughly two upwards sloping piecewise linear functions, with a drop back to nearly 0 at the discontinuity at the Pell grant threshold

A second order effect of universities optimizing for Pell Grant recipients is that savvy parents can do the same thing that some people do to cut their taxable income at the last minute. Someone might put money into a traditional IRA instead of a Roth IRA and, if they're at their IRA contribution limit, they can try to lose money on options, effectively transferring money to options traders who are likely to be wealthier than them, in order to bring their income below the Pell Grant threshold, increasing the probability that their children will be admitted to a selective school.

Election statistics

The following histograms of Russian elections across polling stations shows curious spikes in turnout and results at nice, round, numbers (e.g., 95%) starting around 2004. This appears to indicate that there's election fraud via fabricated results and that at least some of the people fabricating results don't bother with fabricating results that have a smooth distribution.

For finding fraudulent numbers, also see, Benford's law.

p-values

Authors of psychology papers are incentivized to produce papers with p values below some threshold, usually 0.05, but sometimes 0.1 or 0.01. Masicampo et al. plotted p values from papers published in three psychology journals and found a curiously high number of papers with p values just below 0.05.

Histogram of published p-values; spike at p=0.05

The spike at p = 0.05 consistent with a number of hypothesis that aren't great, such as:

  • Authors are fudging results to get p = 0.05
  • Journals are much more likely to accept a paper with p = 0.05 than if p = 0.055
  • Authors are much less likely to submit results if p = 0.055 than if p = 0.05

Head et al. (2015) surveys the evidence across a number of fields.

Andrew Gelman and others have been campaigning to get rid of the idea of statistical significance and p-value thresholds for years, see this paper for a short summary of why. Not only would this reduce the incentive for authors to cheat on p values, there are other reasons to not want a bright-line rule to determine if something is "significant" or not.

Drug charges

The top two graphs in this set of four show histograms of the amount of cocaine people were charged with possessing before and after the passing of the Fair Sentencing Act in 2010, which raised the amount of cocaine necessary to trigger the 10-year mandatory minimum prison sentence for possession from 50g to 280g. There's a relatively smooth distribution before 2010 and a sharp discontinuity after 2010.

The bottom-left graph shows the sharp spike in prosecutions at 280 grams followed by what might be a drop in 2013 after evidentiary standards were changed1.

Birth month and sports

These are scatterplots of football (soccer) players in the UEFA Youth League. The x-axis on both of these plots is how old players are modulo the year, i.e., their birth month normalized from 0 to 1.

The graph on the left is a histogram, which shows that there is a very strong relationship between where a person's birth falls within the year and their odds of making a club at the UEFA Youth League (U19) level. The graph on the right purports to show that birth time is only weakly correlated with actual value provided on the field. The authors use playing time as a proxy for value, presumably because it's easy to measure. That's not a great measure, but the result they find (younger-within-the-year players have higher value, conditional on making the U19 league) is consistent with other studies on sports and discrimination, which ind (for example) that black baseball players were significantly better than white baseball players for decades after desegregation in baseball, French-Canadian defensemen are also better than average (French-Canadians are stereotypically afraid to fight, don't work hard enough, and are too focused on offense).

The discontinuity isn't directly shown in the graphs above because the graphs only show birth date for one year. If we were to plot birth date by cohort across multiple years, we'd expect to see a sawtooth pattern in the probability that a player makes it into the UEFA youth league with a 10x difference between someone born one day before vs. after the threshold.

This phenomenon, that birth day or month is a good predictor of participation in higher-level youth sports as well as pro sports, has been studied across a variety of sports.

It's generally believed that this is caused by a discontinuity in youth sports:

  1. Kids are bucketed into groups by age in years and compete against people in the same year
  2. Within a given year, older kids are stronger, faster, etc., and perform better
  3. This causes older-within-year kids to outcompete younger kids, which later results in older-within-year kids having higher levels of participation for a variety of reasons

This is arguably a "bug" in how youth sports works. But as we've seen in baseball as well as a survey of multiple sports, obviously bad decision making that costs individual teams tens or even hundreds of millions of dollars can persist for decades in the face of people pubicly discussing how bad the decisions are. In this case, the youth sports teams aren't feeder teams to pro teams, so they don't have a financial incentive to select players who are skilled for their age (as opposed to just taller and faster because they're slightly older) so this system-wide non-optimal even more difficult to fix than pro sports teams making locally non-optimal decisions that are completely under their control.

High school exit exam scores

This is a histogram of high school exit exam scores from the Polish language exam. We can see that a curiously high number of students score 30 or just above thirty while curiously low number of students score from 23-29. This is from 2013; other years I've looked at (2010-2012) show a similar discontinuity.

Math exit exam scores don't exhibit any unusual discontinuities in the years I've examined (2010-2013).

An anonymous reddit commenter explains this:

When a teacher is grading matura (final HS exam), he/she doesn't know whose test it is. The only things that are known are: the number (code) of the student and the district which matura comes from (it is usually from completely different part of Poland). The system is made to prevent any kind of manipulation, for example from time to time teachers supervisor will come to check if test are graded correctly. I don't wanna talk much about system flaws (and advantages), it is well known in every education system in the world where final tests are made, but you have to keep in mind that there is a key, which teachers follow very strictly when grading.

So, when a score of the test is below 30%, exam is failed. However, before making final statement in protocol, a commision of 3 (I don't remember exact number) is checking test again. This is the moment, where difference between humanities and math is shown: teachers often try to find a one (or a few) missing points, so the test won't be failed, because it's a tragedy to this person, his school and somewhat fuss for the grading team. Finding a "missing" point is not that hard when you are grading writing or open questions, which is a case in polish language, but nearly impossible in math. So that's the reason why distribution of scores is so different.

As with p values, having a bright-line threshold, causes curious behavior. In this case, scoring below 30 on any subject (a 30 or above is required in every subject) and failing the exam has arbitrary negative effects for people, so teachers usually try to prevent people from failing if there's an easy way to do it, but a deeper root of the problem is the idea that it's necessary to produce a certification that's the discretization of a continuous score.

Procurement auctions

Kawai et al. looked at Japanese government procurement, in order to find suspicious pattern of bids like the ones described in Porter et al. (1993), which looked at collusion in procurement auctions on Long Island (in New York in the United States). One example that's given is:

In February 1983, the New York State Department of Transportation (DoT) held a pro- curement auction for resurfacing 0.8 miles of road. The lowest bid in the auction was $4 million, and the DoT decided not to award the contract because the bid was deemed too high relative to its own cost estimates. The project was put up for a reauction in May 1983 in which all the bidders from the initial auction participated. The lowest bid in the reauction was 20% higher than in the initial auction, submitted by the previous low bidder. Again, the contract was not awarded. The DoT held a third auction in February 1984, with the same set of bidders as in the initial auction. The lowest bid in the third auction was 10% higher than the second time, again submitted by the same bidder. The DoT apparently thought this was suspicious: “It is notable that the same firm submitted the low bid in each of the auctions. Because of the unusual bidding patterns, the contract was not awarded through 1987.”

It could be argued that this is expected because different firms have different cost structures, so the lowest bidder in an auction for one particular project should be expected to be the lowest bidder in subsequent auctions for the same project. In order to distinguish between collusion and real structural cost differences between firms, Kawai et al. (2015) looked at auctions where the difference in bid between the first and second place firms was very small, making the winner effectively random.

In the auction structure studied, bidders submit a secret bid. If the secret bid is above a secret minimum, then the lowest bidder wins the auction and gets the contract. If not, the lowest bid is revealed to all bidders and another round of bidding is done. Kawai et al. found that, in about 97% of auctions, the bidder who submitted the lowest bid in the first round also submitted the lowest bid in the second round (the probability that the second lowest bidder remains second lowest was 26%).

Below, is a histogram of the difference in first and second round bids between the first-lowest and second-lowest bidders (left column) and the second-lowest and third-lowest bidders (right column). Each row has a different filtering criteria for how close the auction has to be in order to be included. In the top row, all auctions that reached the third round were included; in second, and third rows, the normalized delta between the first and second biders was less than 0.05 and 0.01, respectively; in the last row, the normalized delta between the first and the third bidder was less than 0.03. All numbers are normalized because the absolute size of auctions can vary.

We can see that the distributions of deltas between the first and second round are roughly symmetrical when comparing second and third lowest bidders. But when comparing first and second lowest bidders, there's a sharp discontinuity at zero, indicating that second-lowest bidder almost never lowers their bid by more than the first-lower bidder did. If you read the paper, you can see that the same structure persists into auctions that go into a third round.

I don't mean to pick on Japanese procurement auctions in particular. There's an extensive literature on procurement auctions that's found collusion in many cases, often much more blatant than the case presented above (e.g., there are a few firms and they round-robin who wins across auctions, or there are a handful of firms and every firm except for the winner puts in the same losing bid).

Restaurant inspection scores

The histograms below show a sharp discontinuity between 13 and 14, which is the difference between an A grade and a B grade. It appears that some regions also have a discontinuity between 27 and 28, which is the difference between a B and a C and this older analysis from 2014 found what appears to be a similar discontinuity between B and C grades.

Inspectors have discretion in what violations are tallied and it appears that there are cases where restaurant are nudged up to the next higher grade.

Marathon finishing times

A histogram of marathon finishing times (finish times on the x-axis, count on the y-axis) across 9,789,093 finishes shows noticeable discontinuities at every half hour, as well as at "round" times like :10, :15, and :20.

An analysis of times within each race (see section 4.4, figures 7-9) indicates that this is at least partially because people speed up (or slow down less than usual) towards the end of races if they're close to a "round" time2.

Notes

This post doesn't really have a goal or a point, it's just a collection of discontinuities that I find fun.

One thing that's maybe worth noting is that I've gotten a lot of mileage out in my career both out of being suspicious of discontinuities and figuring out where they come from and also out of applying standard techniques to smooth out discontinuities.

For finding discontinuities, basic tools like "drawing a scatterplot", "drawing a histogram", "drawing the CDF" often come in handy. Other kinds of visualizations that add temporality, like flamescope, can also come in handy.

We noted above that queues create a kind of discontinuity that, in some circumstances, should be smoothed out. We also noted that we see similar behavior for other kinds of thresholds and that randomization can be a useful tool to smooth out discontinuities in thresholds as well. Randomization can also be used to allow for reducing quantization error when reducing precision with ML and in other applications.

Thanks to Leah Hanson, Omar Rizwan, Dmitry Belenko, Kamal Marhubi, Danny Vilea, Nick Roberts, Lifan Zeng, Wesley Aptekar-Cassels, Thomas Hauk, @BaudDev, and Michael Sullivan for comments/corrections/discussion.

Also, please feel free to send me other interesting discontinuities!


  1. Most online commentary I've seen about this paper is incorrect. I've seen this paper used as evidence of police malfeasance because the amount of cocaine seized jumped to 280g. This is the opposite of what's described in the paper, where the author notes that, based on drug seizure records, amounts seized do not appear to be the cause of this change. After noting that drug seizures are not the cause, the author notes that prosecutors can charge people for amounts that are not the same as the amount seized and then notes:

    I do find bunching at 280g after 2010 in case management data from the Executive Office of the US Attorney (EOUSA). I also find that approximately 30% of prosecutors are responsible for the rise in cases with 280g after 2010, and that there is variation in prosecutor-level bunching both within and between districts. Prosecutors who bunch cases at 280g also have a high share of cases right above 28g after 2010 (the 5-year threshold post-2010) and a high share of cases above 50g prior to 2010 (the 10-year threshold pre-2010). Also, bunching above a mandatory minimum threshold persists across districts for prosecutors who switch districts. Moreover, when a “bunching” prosecutor switches into a new district, all other attorneys in that district increase their own bunching at mandatory minimums. These results suggest that the observed bunching at sentencing is specifically due to prosecutorial discretion

    This is mentioned in the abstract and then expounded on in the introduction (the quoted passage is from the introduction), so I think that most people commenting on this paper can't have read it. I've done a few surveys of comments on papers on blog posts and I generally find that, in cases where it's possible to identify this (e.g., when the post is mistitled), the vast majority of commenters can't have read the paper or post they're commenting on, but that's a topic for another post.

    There is some evidence that something fishy may be going on in seizures (e.g., see Fig. A8.(c)), but if the analysis in the paper is correct, that impact of that is much smaller than the impact of prosecutorial discretion.

    [return]
  2. One of the most common comments I've seen online about this graph and/or this paper is that this is due to pace runners provided by the marathon. Section 4.4 of the paper gives multiple explanations for why this cannot be the case, once again indicating that people tend to comment without reading the paper. [return]

February 17, 2020

Marc Brooker (mjb)

Physalia: Millions of Tiny Databases February 17, 2020 12:00 AM

Physalia: Millions of Tiny Databases

Avoiding Hard CAP Tradeoffs

A few years ago, when I was still working on EBS, we started building a system called Physalia. Physalia is a custom transactional key-value store, designed to play the role of configuration master in the EBS architecture. Last year, we wrote a paper about Physalia, and were thrilled that it was accepted to NSDI'20.

Millions of Tiny Databases describes our problem and solution in detail. Here's the abstract:

Starting in 2013, we set out to build a new database to act as the configuration store for a high-performance cloud block storage system Amazon EBS. This database needs to be not only highly available, durable, and scalable but also strongly consistent. We quickly realized that the constraints on availability imposed by the CAP theorem, and the realities of operating distributed systems, meant that we didn't want one database. We wanted millions. Physalia is a transactional key-value store, optimized for use in large-scale cloud control planes, which takes advantage of knowledge of transaction patterns and infrastructure design to offer both high availability and strong consistency to millions of clients. Physalia uses its knowledge of datacenter topology to place data where it is most likely to be available. Instead of being highly available for all keys to all clients, Physalia focuses on being extremely available for only the keys it knows each client needs, from the perspective of that client. This paper describes Physalia in context of \amazon \ebs, and some other uses within \awsFull. We believe that the same patterns, and approach to design, are widely applicable to distributed systems problems like control planes, configuration management, and service discovery.

I also wrote a post about Physalia for the Amazon Science blog.

One aspect of Physalia that I'm particular proud of is the work that we put in to correctness. We used TLA+ extensively throughout the design, and as documentation during implementation. As we've published about before, TLA+ is really well suited to these kinds of systems. We also automatically generated unit tests, an approach I haven't seen used elsewhere:

In addition to unit testing, we adopted a number of other testing approaches. One of those approaches was a suite of automatically-generated tests which run the Paxos implementation through every combination of packet loss and re-ordering that a node can experience. This testing approach was inspired by the TLC model checker, and helped usbuild confidence that our implementation matched the formal specification.

Check out our paper if you'd like to learn more.

February 16, 2020

Derek Jones (derek-jones)

Patterns of regular expression usage: duplicate regexs February 16, 2020 06:41 PM

Regular expressions are widely used, but until recently they were rarely studied empirically (i.e., just theory research).

This week I discovered two groups studying regular expression usage in source code. The VTLeeLab has various papers analysing 500K distinct regular expressions, from programs written in eight languages and StackOverflow; Carl Chapman and Peipei Wang have been looking at testing of regular expressions, and also ran an interesting experiment (I will write about this when I have decoded the data).

Regular expressions are interesting, in that their use is likely to be purely driven by an application requirement; the use of an integer literals may be driven by internal housekeeping requirements. The number of times the same regular expression appears in source code provides an insight (I claim) into the number of times different programs are having to solve the same application problem.

The data made available by the VTLeeLab group provides lots of information about each distinct regular expression, but not a count of occurrences in source. My email request for count data received a reply from James Davis within the hour :-)

The plot below (code+data; crates.io has not been included because the number of regexs extracted is much smaller than the other repos) shows the number of unique patterns (y-axis) against the number of identical occurrences of each unique pattern (x-axis), e.g., far left shows number of distinct patterns that occurred once, then the number of distinct patterns that each occur twice, etc; colors show the repositories (language) from which the source was obtained (to extract the regexs), and lines are fitted regression models of the form: NumPatterns = a*MultOccur^b, where: a is driven by the total amount of source processed and the frequency of occurrence of regexs in source, and b is the rate at which duplicates occur.

Number of distinct patterns occurring a given number of times in the source stored in various repositories

So most patterns occur once, and a few patterns occur lots of times (there is a long tail off to the unplotted right).

The following table shows values of b for the various repositories (languages):

StackOverflow   cpan    godoc    maven    npm  packagist   pypi   rubygems
    -1.8        -2.5     -2.5    -2.4    -1.9     -2.6     -2.7     -2.4

The lower (i.e., closer to zero) the value of b, the more often the same regex will appear.

The values are in the region of -2.5, with two exceptions; why might StackOverflow and npm be different? I can imagine lots of duplicates on StackOverflow, but npm (I’m not really familiar with this package ecosystem).

I am pleased to see such good regression fits, and close power law exponents (I would have been happy with an exponential fit, or any other equation; I am interested in a consistent pattern across languages, not the pattern itself).

Some of the code is likely to be cloned, i.e., cut-and-pasted from a function in another package/program. Copy rates as high as 70% have been found. In this case, I don’t think cloned code matters. If a particular regex is needed, what difference does it make whether the code was cloned or written from scratch?

If the same regex appears in source because of the same application requirement, the number of reuses should be correlated across languages (unless different languages are being used to solve different kinds of problems). The plot below shows the correlation between number of occurrences of distinct regexs, for each pair of languages (or rather repos for particular languages; top left is StackOverflow).

Correlation of number of identical pattern occurrences, between pairs of repositories.

Why is there a mix of strong and weakly correlated pairs? Is it because similar application problems tend to be solved using different languages? Or perhaps there are different habits for cut-and-pasted source for developers using different repositories (which will cause some patterns to occur more often, but not others, and have an impact on correlation but not the regression fit).

There are lot of other interesting things that can be done with this data, when connected to the results of the analysis of distinct regexs, but these look like hard work, and I have a book to finish.

February 15, 2020

Jan van den Berg (j11g)

Gung Ho! Turn On the People in Any Organization – Ken Blanchard & Sheldon Bowles February 15, 2020 10:46 PM

Gung Ho! is a management book written by well-know author Ken Blanchard. It was somehow never on my radar, so because of the strange title and my unfamiliarity I wasn’t expecting too much, and I only picked it up because I know Blanchard’s other famous theory.

Gung Ho! Turn On the People in Any Organization – Ken Blanchard & Sheldon Bowles (1997) – 143 pages

But it turned out to be a delightful, short read. This book can help any starting, aspiring or even seasoned manager to get their priorities straight about working with people.

The theory is built around three core ideas (squirrel, beaver, goose). Decide on meaningful work, decide on how you want to accomplish your task and encourage one another.

What makes the story enticing is that it is based in reality (or is it??). It is built around the story of a factory on the brink of a shutdown, and an old wise Indian who sets out to transform the factory with his new boss. It is sort of misty whether this actually happened (the protagonist herself has some pages in the acknowledgements but seems worried about her privacy). But as the book states in the end notes: it does not matter whether this actually happened: this is a universally applicable theory for every successful company. And I agree.

The post Gung Ho! Turn On the People in Any Organization – Ken Blanchard & Sheldon Bowles appeared first on Jan van den Berg.

Gustaf Erikson (gerikson)

Goodbye, Darkness by William Manchester February 15, 2020 10:47 AM

Yet another US Marine Pacific War memoir. While Manchester has a great command of language, the combination with a travelogue doesn’t really work. I still think Sledge’s work is the best I’ve read in this genre so far.

England’s Last War Against France: Fighting Vichy 1940-42 by Colin Smith February 15, 2020 10:47 AM

An informative and entertaining account of British conflict with Vichy France. Has a good overview of the history of that shameful part of French history.

D-Day by Antony Beevor February 15, 2020 10:39 AM

Way less dense than I remember his history of the battle of Crete. Maybe he’s become more fluent, or simply slid into the comfortable narrative style of retelling the “Greatest Generation’s” big battles. This is an ok story. The suffering of French civilians in Normandy is highlighted, which usually doesn’t happen.

Ebook maps suck.

February 12, 2020

Bogdan Popa (bogdan)

The Missing Guide to Racket's Web Server February 12, 2020 10:00 AM

Racket’s built-in web-server package is great, but parts of it are low-level enough that it can be confusing to people who are new to the language. In this post, I’m going to try to clear up some of that confusion by providing some definitions and examples for things beginners might wonder about. Servlets A servlet is a function from a request to a response. It has the contract: 1 (-> request?

February 09, 2020

Ponylang (SeanTAllen)

Last Week in Pony - February 9, 2020 February 09, 2020 04:29 PM

Pony 0.33.2 has been released! The core team is no longer maintaining the ponyc Homebrew formula, since ponyup is now the officially supported installation method.

February 08, 2020

Carlos Fenollosa (carlesfe)

February 07, 2020

Derek Jones (derek-jones)

Source code has a brief and lonely existence February 07, 2020 01:16 PM

The majority of source code has a short lifespan (i.e., a few years), and is only ever modified by one person (i.e., 60%).

Literate programming is all well and good for code written to appear in a book that the author hopes will be read for many years, but this is a tiny sliver of the source code ecosystem. The majority of code is never modified, once written, and does not hang around for very long; an investment is source code futures will make a loss unless the returns are spectacular.

What evidence do I have for these claims?

There is lots of evidence for the code having a short lifespan, and not so much for the number of people modifying it (and none for the number of people reading it).

The lifespan evidence is derived from data in my evidence-based software engineering book, and blog posts on software system lifespans, and survival times of Linux distributions. Lifespan in short because Packages are updated, and third-parties change/delete APIs (things may settle down in the future).

People who think source code has a long lifespan are suffering from survivorship bias, i.e., there are a small percentage of programs that are actively used for many years.

Around 60% of functions are only ever modified by one author; based on a study of the change history of functions in Evolution (114,485 changes to functions over 10 years), and Apache (14,072 changes over 12 years); a study investigating the number of people modifying files in Eclipse. Pointers to other studies that have welcome.

One consequence of the short life expectancy of source code is that, any investment made with the expectation of saving on future maintenance costs needs to return many multiples of the original investment. When many programs don’t live long enough to be maintained, those with a long lifespan have to pay the original investments made in all the source that quickly disappeared.

One benefit of short life expectancy is that most coding mistakes don’t live long enough to trigger a fault experience; the code containing the mistake is deleted or replaced before anybody notices the mistake.

Update: a few days later

I was remiss in not posting some plots for people to look at (code+data).

The plot below shows number of function, in Evolution, modified a given number of times (left), and functions modified by a given number of authors (right). The lines are a bi-exponential fit.

Number of function, in Evolution, modified a given number of times (left), and functions modified by a given number of authors (right).

What is the interval (in hours) between between modifications of a function? The plot below has a logarithmic x-axis, and is sort-of flat over most of the range (you need to squint a bit). This is roughly saying that the probability of modification in hours 1-3 is the same as in hours 3-7, and hours, 7-15, etc (i.e., keep doubling the hour interval). At round 10,000 hours function modification probability drops like a stone.

Number of function, in Evolution, modified a given number of times (left), and functions modified by a given number of authors (right).

Jeremy Morgan (JeremyMorgan)

Become a React Developer in a Weekend February 07, 2020 12:00 AM

If you want to be a React Developer, there’s a lot to learn. You can get started in a weekend. Yes, in a single weekend, you can learn React and start developing some cool applications! Here’s how I would approach it. Friday Night Ok, instead of binge-watching some TV show Friday night (no judgment, I’ve been there), you could spend the time learning the basics of React. If you’ve never touched React or done anything with it, The Big Picture is the place to start.

Dan Luu (dl)

95%-ile isn't that good February 07, 2020 12:00 AM

Reaching 95%-ile isn't very impressive because it's not that hard to do. I think this is one of my most ridiculable ideas. It doesn't help that, when stated nakedly, that sounds elitist. But I think it's just the opposite: most people can become (relatively) good at most things.

Note that when I say 95%-ile, I mean 95%-ile among people who participate, not all people (for many activities, just doing it at all makes you 99%-ile or above across all people). I'm also not referring to 95%-ile among people who practice regularly. The "one weird trick" is that, for a lot of activities, being something like 10%-ile among people who practice can make you something like 90%-ile or 99%-ile among people who participate.

This post is going to refer to specifics since the discussions I've seen about this are all in the abstract, which turns them into Rorschach tests. For example, Scott Adams has a widely cited post claiming that it's better to be a generalist than a specialist because, to become "extraordinary", you have to either be "the best" at one thing or 75%-ile at two things. If that were strictly true, it would surely be better to be a generalist, but that's of course exaggeration and it's possible to get a lot of value out of a specialized skill without being "the best"; since the precise claim, as written, is obviously silly and the rest of the post is vague handwaving, discussions will inevitably devolve into people stating their prior beliefs and basically ignoring the content of the post.

Personally, in every activity I've participated in where it's possible to get a rough percentile ranking, people who are 95%-ile constantly make mistakes that seem like they should be easy to observe and correct. "Real world" activities typically can't be reduced to a percentile rating, but achieving what appears to be a similar level of proficiency seems similarly easy.

We'll start by looking at Overwatch (a video game) in detail because it's an activity I'm familiar with where it's easy to get ranking information and observe what's happening, and then we'll look at some "real world" examples where we can observe the same phenomena, although we won't be able to get ranking information for real world examples1.

Overwatch

At 90%-ile and 95%-ile ranks in Overwatch, the vast majority of players will pretty much constantly make basic game losing mistakes. These are simple mistakes like standing next to the objective instead of on top of the objective while the match timer runs out, turning a probable victory into a certain defeat. See the attached footnote if you want enough detail about specific mistakes that you can decide for yourself if a mistake is "basic" or not2.

Some reasons we might expect this to happen are:

  1. People don't want to win or don't care about winning
  2. People understand their mistakes but haven't put in enough time to fix them
  3. People are untalented
  4. People don't understand how to spot their mistakes and fix them

In Overwatch, you may see a lot of (1), people who don’t seem to care about winning, at lower ranks, but by the time you get to 30%-ile, it's common to see people indicate their desire to win in various ways, such as yelling at players who are perceived as uncaring about victory or unskilled, complaining about people who they perceive to make mistakes that prevented their team from winning, etc.3. Other than the occasional troll, it's not unreasonable to think that people are generally trying to win when they're severely angered by losing.

(2), not having put in time enough to fix their mistakes will, at some point, apply to all players who are improving, but if you look at the median time played at 50%-ile, people who are stably ranked there have put in hundreds of hours (and the median time played at higher ranks is higher). Given how simple the mistakes we're discussing are, not having put in enough time cannot be the case for most players.

A common complaint among low-ranked Overwatch players in Overwatch forums is that they're just not talented and can never get better. Most people probably don't have the talent to play in a professional league regardless of their practice regimen, but when you can get to 95%-ile by fixing mistakes like "not realizing that you should stand on the objective", you don't really need a lot of talent to get to 95%-ile.

While (4), people not understanding how to spot and fix their mistakes, isn't the only other possible explanation4, I believe it's the most likely explanation for most players. Most players who express frustration that they're stuck at a rank up to maybe 95%-ile or 99%-ile don't seem to realize that they could drastically improve by observing their own gameplay or having someone else look at their gameplay.

One thing that's curious about this is that Overwatch makes it easy to spot basic mistakes (compared to most other activities). After you're killed, the game shows you how you died from the viewpoint of the player who killed you, allowing you to see what led to your death. Overwatch also records the entire game and lets you watch a replay of the game, allowing you to figure out what happened and why the game was won or lost. In many other games, you'd have to set up recording software to be able to view a replay.

If you read Overwatch forums, you'll see a regular stream of posts that are basically "I'm SOOOOOO FRUSTRATED! I've played this game for 1200 hours and I'm still ranked 10%-ile, [some Overwatch specific stuff that will vary from player to player]". Another user will inevitably respond with something like "we can't tell what's wrong from your text, please post a video of your gameplay". In the cases where the original poster responds with a recording of their play, people will post helpful feedback that will immediately make the player much better if they take it seriously. If you follow these people who ask for help, you'll often see them ask for feedback at a much higher rank (e.g., moving from 10%-ile to 40%-ile) shortly afterwards. It's nice to see that the advice works, but it's unfortunate that so many players don't realize that watching their own recordings or posting recordings for feedback could have saved 1198 hours of frustration.

It appears to be common for Overwatch players (well into 95%-ile and above) to:

  • Want to improve
  • Not get feedback
  • Improve slowly when getting feedback would make improving quickly easy

Overwatch provides the tools to make it relatively easy to get feedback, but people who very strongly express a desire to improve don't avail themselves of these tools.

Real life

My experience is that other games are similar and I think that "real life" activities aren't so different, although there are some complications.

One complication is that real life activities tend not to have a single, one-dimensional, objective to optimize for. Another is that what makes someone good at a real life activity tends to be poorly understood (by comparison to games and sports) even in relation to a specific, well defined, goal.

Games with rating systems are easy to optimize for: your meta-goal can be to get a high rating, which can typically be achieved by increasing your win rate by fixing the kinds of mistakes described above, like not realizing that you should step onto the objective. For any particular mistake, you can even make a reasonable guess at the impact on your win rate and therefore the impact on your rating.

In real life, if you want to be (for example) "a good speaker", that might mean that you want to give informative talks that help people learn or that you want to give entertaining talks that people enjoy or that you want to give keynotes at prestigious conferences or that you want to be asked to give talks for $50k an appearance. Those are all different objectives, with different strategies for achieving them and for some particular mistake (e.g., spending 8 minutes on introducing yourself during a 20 minute talk), it's unclear what that means with respect to your goal.

Another thing that makes games, at least mainstream ones, easy to optimize for is that they tend to have a lot of aficionados who have obsessively tried to figure out what's effective. This means that if you want to improve, unless you're trying to be among the top in the world, you can simply figure out what resources have worked for other people, pick one up, read/watch it, and then practice. For example, if you want to be 99%-ile in a trick-taking card game like bridge or spades (among all players, not subgroups like "ACBL players with masterpoints" or "people who regularly attend North American Bridge Championships"), you can do this by:

If you want to become a good speaker and you have a specific definition of “a good speaker” in mind, there still isn't an obvious path forward. Great speakers will give directly contradictory advice (e.g., avoid focusing on presentation skills vs. practice presentation skills). Relatively few people obsessively try to improve and figure out what works, which results in a lack of rigorous curricula for improving. However, this also means that it's easy to improve in percentile terms since relatively few people are trying to improve at all.

Despite all of the caveats above, my belief is that it's easier to become relatively good at real life activities relative to games or sports because there's so little delibrate practice put into most real life activities. Just for example, if you're a local table tennis hotshot who can beat every rando at a local bar, when you challenge someone to a game and they say "sure, what's your rating?" you know you're in shellacking by someone who can probably beat you while playing with a shoe brush. You're probably 99%-ile, but someone with no talent who's put in the time to practice the basics is going to have a serve that you can't return as well as be able to kill any shot a local bar expert is able to consitently hit. In most real life activities, there's almost no one who puts in a level of delibrate practice equivalent to someone who goes down to their local table tennis club and practices two hours a week, let alone someone like a top pro, who might seriously train for four hours a day.

To give a couple of concrete examples, I helped Leah prepare for talks from 2013 to 2017. The first couple practice talks she gave were about the same as you'd expect if you walked into a random talk at a large tech conference. For the first couple years she was speaking, she did something like 30 or so practice runs for each public talk, of which I watched and gave feedback on half. Her first public talk was (IMO) well above average for a talk at a large, well regarded, tech conference and her talks got better from there until she stopped speaking in 2017.

As we discussed above, this is more subjective than game ratings and there's no way to really determine a percentile, but if you look at how most people prepare for talks, it's not too surprising that Leah was above average. At one of the first conferences she spoke at, the night before the conference, we talked to another speaker who mentioned that they hadn't finished their talk yet and only had fifteen minutes of material (for a forty minute talk). They were trying to figure out how to fill the rest of the time. That kind of preparation isn't unusual and the vast majority of talks prepared like that aren't great.

Most people consider doing 30 practice runs for a talk to be absurd, a totally obsessive amount of practice, but I think Gary Bernhardt has it right when he says that, if you're giving a 30-minute talk to a 300 person audience, that's 150 person-hours watching your talk, so it's not obviously unreasonable to spend 15 hours practicing (and 30 practice runs will probably be less than 15 hours since you can cut a number of the runs short and/or repeatedly practice problem sections). One thing to note that this level of practice, considered obessive when giving a talk, still pales in comparison to the amount of time a middling table tennis club player will spend practicing.

If you've studied pedagogy, you might say that the help I gave to Leah was incredibly lame. It's known that having laypeople try to figure out how to improve among themselves is among the worst possible ways to learn something, good instruction is more effective and having a skilled coach or teacher give one-on-one instruction is more effective still5. That's 100% true, my help was incredibly lame. However, most people aren't going to practice a talk more than a couple times and many won't even practice a single time (I don't have great data proving this, this is from informally polling speakers at conferences I've attended). This makes Leah's 30 practice runs an extraordinary amount of practice compared to most speakers, which resulted in a relatively good outcome even though we were using one of the worst possible techniques for improvement.

My writing is another example. I'm not going to compare myself to anyone else, but my writing improved dramatically the first couple of years I wrote this blog just because I spent a little bit of effort on getting and taking feedback.

Leah read one or two drafts of almost every post and gave me feedback. On the first posts, since neither one of us knew anything about writing, we had a hard time identifying what was wrong. If I had some awkward prose or confusing narrative structure, we'd be able to point at it and say "that looks wrong" without being able to describe what was wrong or suggest a fix. It was like, in the era before spellcheck, when you misspelled a word and could tell that something was wrong, but every permutation you came up with was just as wrong.

My fix for that was to hire a professional editor whose writing I respected with the instructions "I don't care about spelling and grammar fixes, there are fundamental problems with my writing that I don't understand, please explain to me what they are"6. I think this was more effective than my helping Leah with talks because we got someone who's basically a professional coach involved. An example of something my editor helped us with was giving us a vocabulary we could use to discuss structural problems, the way design patterns gave people a vocabulary to talk about OO design.

Back to this blog's regularly scheduled topic: programming

Programming is similar to the real life examples above in that it's impossible to assign a rating or calculate percentiles or anything like that, but it is still possible to make significant improvements relative to your former self without too much effort by getting feedback on what you're doing.

For example, here's one thing Michael Malis did:

One incredibly useful exercise I’ve found is to watch myself program. Throughout the week, I have a program running in the background that records my screen. At the end of the week, I’ll watch a few segments from the previous week. Usually I will watch the times that felt like it took a lot longer to complete some task than it should have. While watching them, I’ll pay attention to specifically where the time went and figure out what I could have done better. When I first did this, I was really surprised at where all of my time was going.

For example, previously when writing code, I would write all my code for a new feature up front and then test all of the code collectively. When testing code this way, I would have to isolate which function the bug was in and then debug that individual function. After watching a recording of myself writing code, I realized I was spending about a quarter of the total time implementing the feature tracking down which functions the bugs were in! This was completely non-obvious to me and I wouldn’t have found it out without recording myself. Now that I’m aware that I spent so much time isolating which function a bugs are in, I now test each function as I write it to make sure they work. This allows me to write code a lot faster as it dramatically reduces the amount of time it takes to debug my code.

In the past, I've spent time figuring out where time is going when I code and basically saw the same thing as in Overwatch, except instead of constantly making game-losing mistakes, I was constantly doing pointlessly time-losing things. Just getting rid of some of those bad habits has probably been at least a 2x productivity increase for me, pretty easy to measure since fixing these is basically just clawing back wasted time. For example, I noticed how I'd get distracted for N minutes if I read something on the internet when I needed to wait for two minutes, so I made sure to keep a queue of useful work to fill dead time (and if I was working on something very latency sensitive where I didn't want to task switch, I'd do nothing until I was done waiting).

One thing to note here is that it's important to actually track what you're doing and not just guess at what you're doing. When I've recorded what people do and compare it to what they think they're doing, these are often quite different. It would generally be considered absurd to operate a complex software system without metrics or tracing, but it's normal to operate yourself without metrics or tracing, even though you're much more complex and harder to understand than the software you work on.

Jonathan Tang has noted that choosing the right problem dominates execution speed. I don't disagree with that, but doubling execution speed is still decent win that's independent of selecting the right problem to work on and I don't think that discussing how to choose the right problem can be effectively described in the abstract and the context necessary to give examples would be much longer than the already too long Overwatch examples in this post, maybe I'll write another post that's just about that.

Anyway, this is sort of an odd post for me to write since I think that culturally, we care a bit too much about productivity in the U.S., especially in places I've lived recently (NYC & SF). But at a personal level, higher productivity doing work or chores doesn't have to be converted into more work or chores, it can also be converted into more vacation time or more time doing whatever you value.

And for games like Overwatch, I don't think improving is a moral imperative; there's nothing wrong with having fun at 50%-ile or 10%-ile or any rank. But in every game I've played with a rating and/or league/tournament system, a lot of people get really upset and unhappy when they lose even when they haven't put much effort into improving. If that's the case, why not put a little bit of effort into improving and spend a little bit less time being upset?

Some meta-techniques for improving

  • Get feedback and practice
    • Ideally from an expert coach but, if not, this can be from a layperson or even yourself (if you have some way of recording/tracing what you're doing)
  • Guided exercises or exercises with solutions
    • This is very easy to find in books for "old" games, like chess or Bridge.
    • For particular areas, you can often find series of books that have these, e.g., in math, books in the Springer Undergraduate Mathematics Series (SUMS) tend to have problems with solutions

Appendix: other most ridiculable ideas

Here are the ideas I've posted about that were the most widely ridiculed at the time of the post:

My posts on compensation have the dubious distinction of being the posts most frequently called out both for being so obvious that they're pointless as well as for being laughably wrong. I suspect they're also the posts that have had the largest aggregate impact on people -- I've had a double digit number of people tell me one of the compensation posts changed their life and they now make $x00,000/yr more than they used to because they know it's possible to get a much higher paying job and I doubt that I even hear from 10% of the people who make a big change as a result of learning that it's possible to make a lot more money.

When I wrote my first post on compensation, in 2015, I got ridiculed more for writing something obviously wrong than for writing something obvious, but the last few years things have flipped around. I still get the occasional bit of ridicule for being wrong when some corner of Twitter or a web forum that's well outside the HN/reddit bubble runs across my post, but the ratio of “obviously wrong” to “obvious” has probably gone from 20:1 to 1:5.

Opinions on monorepos have also seen a similar change since 2015. Outside of some folks at big companies, monorepos used to be considered obviously stupid among people who keep up with trends, but this has really changed. Not as much as opinions on compensation, but enough that I'm now a little surprised when I meet a hardline anti-monorepo-er.

Although it's taken longer for opinions to come around on CPU bugs, that's probably the post that now gets the least ridicule from the list above.

That markets don't eliminate all discrimination is the one where opinions have come around the least. Hardline "all markets are efficient" folks aren't really convinced by academic work like Becker's The Economics of Discrimination or summaries like the evidence laid out in the post.

The posts on computers having higher latency and the lack of empirical evidence of the benefit of types are the posts I've seen pointed to the most often to defend a ridicuable opinion. I didn't know when I started doing the work for either post and they both happen to have turned up evidence that's the opposite of the most common loud claims (there's very good evidence that advanced type systems improve safety in practice and of course computers are faster in every way, people who think they're slower are just indulging in nostalgia). I don't know if this has changed many opinion. However, I haven't gotten much direct ridicule for either post even though both posts directly state a position I see commonly ridiculed online. I suspect that's partially because both posts are empirical, so there's not much to dispute (though the post on discrimnation is also empirical, but it still gets its share of ridicule).

The last idea in the list is more meta: no one directly tells me that I should use more obscure terminology. Instead, I get comments that I must not know much about X because I'm not using terms of art. Using terms of art is a common way to establish credibility or authority, but that's something I don't really believe in. Arguing from authority doesn't tell you anything; adding needless terminology just makes things more difficult for readers who aren't in the field and are reading because they're interested in the topic but don't want to actually get into the field.

This is a pretty fundamental disagreement that I have with a lot of people. Just for example, I recently got into a discussion with an authority who insisted that it wasn't possible for me to reasonably disagree with them (I suggested we agree to disagree) because they're an authority on the topic and I'm not. It happens that I worked on the formal verification of a system very similar to the system we were discussing, but I didn't mention that because I don't believe that my status as an authority on the topic matters. If someone has such a weak argument that they have to fall back on an infallible authority, that's usually a sign that they don't have a well-reasoned defense of their position. This goes double when they point to themselves as the infallible authority.

I have about 20 other posts on stupid sounding ideas queued up in my head, but I mostly try to avoid writing things that are controversial, so I don't know that I'll write many of those up. If I were to write one post a month (much more frequently than my recent rate) and limit myself to 10% posts on ridiculable ideas, it would take 16 years to write up all of the ridiculable ideas I currently have.

Thanks to Leah Hanson, Hillel Wayne, Robert Schuessler, Michael Malis, Kevin Burke, Jeremie Jost, Pierre-Yves Baccou, Veit Heller, Jeff Fowler, Malte Skarupe, David Turner, Akiva Leffert, Lifan Zeng, John Hergenroder, Wesley Aptekar-Cassels, Chris Lample, Julia Evans, Anja Boskovic, Vaibhav Sagar, Sean Talts, Valentin Hartmann, Sean Barrett, Kevin Shannon, and an anonymous commenter for comments/corrections/discussion.>


  1. The choice of Overwatch is arbitrary among activities I'm familiar with where:

    • I know enough about the activity to comment on it
    • I've observed enough people trying to learn it that I can say if it's "easy" or not to fix some mistake or class of mistake
    • There's a large enough set of rated players is high enough to support the argument
    • Many readers will also be familiar with the activity

    99% of my gaming background comes from 90s video games, but I'm not going to use those as examples because relatively few readers will be familiar with those games. I could also use "modern" board games like Puerto Rico, Dominion, Terra Mystica, ASL etc., but the set of people who played in rated games is very low, which makes the argument less convincing (perhaps people who play in rated games are much worse than people who don't -- unlikely, but difficult to justify without comparing gameplay between rated and unrated games, which is pretty deep into weeds for this post).

    There are numerous activities that would be better to use than Overwatch, but I'm not familiar enough with them to use them as examples. For example, on reading a draft of this post, Kevin Burke noted that he's observed the same thing while coaching youth basketball and multiple readers noted that they've observed the same thing in chess, but I'm not familiar enough with youth basketball or chess to confidently say much about either activity even they'd be better examples because it's likely that more readers are familiar with basketball or chess than with Overwatch.

    [return]
  2. When I first started playing Overwatch (which is when I did that experiment), I ended up getting rated slightly above 50%-ile (for Overwatch players, that was in Plat -- this post is going to use percentiles and not ranks to avoid making non-Overwatch players have to learn what the ranks mean). It's generally believed and probably true that people who play the main ranked game mode in Overwatch are, on average, better than people who only play unranked modes, so it's likely that my actual percentile was somewhat higher than 50%-ile and that all "true" percentiles listed in this post are higher than the nominal percentiles.

    Some things you'll regularly see at slightly above 50%-ile are:

    • Supports (healers) will heal someone who's at full health (which does nothing) while a teammate who's next to them is dying and then dies
    • Players will not notice someone who walks directly behind the team and kills people one at a time until the entire team is killed
    • Players will shoot an enemy until only one more shot is required to kill the enemy and then switch to a different target, letting the 1-health enemy heal back to full health before shooting at that enemy again
    • After dying, players will not wait for their team to respawn and will, instead, run directly into the enemy team to fight them 1v6. This will repeat for the entire game (the game is designed to be 6v6, but in ranks below 95%-ile, it's rare to see a 6v6 engagement after one person on one team dies)
    • Players will clearly have no idea what character abilities do, including for the character they're playing
    • Players go for very high risk but low reward plays (for Overwatch players, a classic example of this is Rein going for a meme pin when the game opens on 2CP defense, very common at 50%-ile, rare at 95%-ile since players who think this move is a good idea tend to have generally poor decision making).
    • People will have terrible aim and will miss four or five shots in a row when all they need to do is hit someone once to kill them
    • If a single flanking enemy threatens a healer who can't escape plus a non-healer with an escape ability, the non-healer will probably use their ability to run away, leaving the healer to die, even though they could easily kill the flanker and save their healer if they just attacked while being healed.

    Having just one aspect of your gameplay be merely bad instead of atrocious is enough to get to 50%-ile. For me, that was my teamwork, for others, it's other parts of their gameplay. The reason I'd say that my teamwork was bad and not good or even mediocre was that I basically didn't know how to play the game, didn't know what any of the characters’ strengths, weaknesses, and abilities are, so I couldn't possibly coordinate effectively with my team. I also didn't know how the game modes actually worked (e.g., under what circumstances the game will end in a tie vs. going into another round), so I was basically wandering around randomly with a preference towards staying near the largest group of teammates I could find. That's above average.

    You could say that someone is pretty good at the game since they're above average. But in a non-relative sense, being slightly above average is quite bad -- it's hard to argue that someone who doesn't notice their entire team being killed from behind while two teammates are yelling "[enemy] behind us!" over voice comms isn't bad.

    After playing a bit more, I ended up with what looks like a "true" rank of about 90%-ile when I'm using a character I know how to use. Due to volatility in ranking as well as matchmaking, I played in games as high as 98%-ile. My aim and dodging were still atrocious. Relative to my rank, my aim was actually worse than when I was playing in 50%-ile games since my opponents were much better and I was only a little bit better. In 90%-ile, two copies of myself would probably lose fighting against most people 2v1 in the open. I would also usually lose a fight if the opponent was in the open and I was behind cover such that only 10% of my character was exposed, so my aim was arguably more than 10x worse than median at my rank.

    My "trick" for getting to 90%-ile despite being a 1/10th aimer was learning how the game worked and playing in a way that maximized the probability of winning (to the best of my ability), as opposed to playing the game like it's an FFA game where your goal is to get kills as quickly as possible. It takes a bit more context to describe what this means in 90%-ile, so I'll only provide a couple examples, but these are representative of mistakes the vast majority of 90%-ile players are making all of the time (with the exception of a few players who have grossly defective aim, like myself, who make up for their poor aim by playing better than average for the rank in other ways).

    Within the game, the goal of the game is to win. There are different game modes, but for the mainline ranked game, they all will involve some kind of objective that you have to be on or near. It's very common to get into a situation where the round timer is ticking down to zero and your team is guaranteed to lose if no one on your team touches the objective but your team may win if someone can touch the objective and not die instantly (which will cause the game to go into overtime until shortly after both teams stop touching the objective). A concrete example of this that happens somewhat regularly is, the enemy team has four players on the objective while your team has two players near the objective, one tank and one support/healer. The other four players on your team died and are coming back from spawn. They're close enough that if you can touch the objective and not instantly die, they'll arrive and probably take the objective for the win, but they won't get there in time if you die immediately after taking the objective, in which case you'll lose.

    If you're playing the support/healer at 90%-ile to 95%-ile, this game will almost always end as follows: the tank will move towards the objective, get shot, decide they don't want to take damage, and then back off from the objective. As a support, you have a small health pool and will die instantly if you touch the objective because the other team will shoot you. Since your team is guaranteed to lose if you don't move up to the objective, you're forced to do so to have any chance of winning. After you're killed, the tank will either move onto the objective and die or walk towards the objective but not get there before time runs out. Either way, you'll probably lose.

    If the tank did their job and moved onto the objective before you died, you could heal the tank for long enough that the rest of your team will arrive and you'll probably win. The enemy team, if they were coordinated, could walk around or through the tank to kill you, but they won't do that -- anyone who knows that will cause them to win the game and can aim well enough to successfully follow through can't help but end up in a higher rank). And the hypothetical tank on your team who knows that it's their job to absorb damage for their support in that situation and not vice versa won't stay at 95%-ile very long because they'll win too many games and move up to a higher rank.

    Another basic situation that the vast majority of 90%-ile to 95%-ile players will get wrong is when you're on offense, waiting for your team to respawn so you can attack as a group. Even at 90%-ile, maybe 1/4 to 1/3 of players won't do this and will just run directly at the enemy team, but enough players will realize that 1v6 isn't a good idea that you'll often 5v6 or 6v6 fights instead of the constant 1v6 and 2v6 fights you see at 50%-ile. Anyway, while waiting for the team to respawn in order to get a 5v6, it's very likely one player who realizes that they shouldn't just run into the middle of the enemy team 1v6 will decide they should try to hit the enemy team with long-ranged attacks 1v6. People will do this instead of hiding in safety behind a wall even when the enemy has multiple snipers with instant-kill long range attacks. People will even do this against multiple snipers when they're playing a character that isn't a sniper and needs to hit the enemy 2-3 times to get a kill, making it overwhelmingly likely that they won't get a kill while taking a significant risk of dying themselves. For Overwatch players, people will also do this when they have full ult charge and the other team doesn't, turning a situation that should be to your advantage (your team has ults ready and the other team has used ults), into a neutral situation (both teams have ults) at best, and instantly losing the fight at worst.

    If you ever read an Overwatch forum, whether that's one of the reddit forums or the official Blizzard forums, a common complaint is "why are my teammates so bad? I'm at [90%-ile to 95%-ile rank], but all my teammates are doing obviously stupid game-losing things all the time, like [an example above]". The answer is, of course, that the person asking the questiiton is also doing obviously stupid game-losing things all the time because anyone who doesn't wins too much to stay at 95%-ile. This also applies to me.

    People will argue that players at this rank should be good because they're better than 95% of other players, which makes them relatively good. But non-relatively, it's hard to argue that someone who doesn't realize that you should step on the objective to probably win the game instead of not touching the objective for a sure loss is good. One of the most basic things about Overwatch is that it's an objective-based game, but the majority of players at 90%-ile to 95%-ile don't play that way.

    For anyone who isn't well into the 99%-ile, reviewing recorded games will reveal game-losing mistakes all the time. For myself, usually ranked 90%-ile or so, watching a recorded game will reveal tens of game losing mistakes in a close game (which is maybe 30% of losses, the other 70% are blowouts where there isn't a single simple mistake that decides the game).

    It's generally not too hard to fix these since the mistakes are like the example above: simple enough that once you see that you're making the mistake, the fix is straightforward because the mistake is straightforward.

    [return]
  3. There are probably some people who just want to be angry at their teammates. Due to how infrequently you get matched with the same players, it's hard to see this in the main rated game mode, but I think you can sometimes see this when Overwatch sometimes runs mini-rated modes.

    Mini-rated modes have a much smaller playerbase than the main rated mode, which has two notable side effects: players with a much wider variety of skill levels will be thrown into the same game and you'll see the same players over and over again if you play multiple games.

    Since you ended up matched with the same players repeatedly, you'll see players make the same mistakes and cause themselves to lose in the same way and then have the same tantrum and blame their teammates in the same way game after game.

    You'll also see tantrums and teammate blaming in the normal rated game mode, but when you see it, you generally can't tell if the person who's having a tantrum is just having a bad day or if it's some other one-off occurrence since, unless you're ranked very high or very low (where there's a smaller pool of closely rated players), you don't run into the same players all that frequently. But when you see a set of players in 15-20 games over the course of a few weeks and you see them lose the game for the same reason a double digit number of times followed by the exact same tantrum, you might start to suspect that some fraction of those people really want to be angry and that the main thing they're getting out of playing the game is a source of anger. You might also wonder about this from how some people use social media, but that's a topic for another post.

    [return]
  4. For example, there will also be players who have some kind of disability that prevents them from improving, but at the levels we're talking about, 99%-ile or below, that will be relatively rare (certainly well under 50%, and I think it's not unreasonable to guess that it's well under 10% of people who choose to play the game). IIRC, there's at least one player who's in the top 500 who's deaf (this is severely disadvantageous since sound cues give a lot of fine-grained positional information that cannot be obtained in any other way), at least one legally blind player who's 99%-ile, and multiple players with physical impairments that prevent them from having fine-grained control of a mouse, i.e., who are basically incapable of aiming, who are 99%-ile.

    There are also other kinds of reasons people might not improve. For example, Kevin Burke has noted that when he coaches youth basketball, some children don't want to do drills that they think make them look foolish (e.g., avoiding learning to dribble with their off hand even during drills where everyone is dribbling poorly because they're using their off hand). When I spent a lot of time in a climbing gym with a world class coach who would regularly send a bunch of kids to nationals and some to worlds, I'd observe the same thing in his classes -- kids, even ones who are nationally or internationally competitive, would sometimes avoid doing things because they were afraid it would make them look foolish to their peers. The coach's solution in those cases was to deliberately make the kid look extremely foolish and tell them that it's better to look stupid now than at nationals. Similarly, but I view these as basically independent of the issue discussed in the post.

    [return]
  5. note that, here, a skilled coach is someone who is skilled at coaching, not necessarily someone who is skilled at the activity. People who are skilled at the activity but who haven't explicitly been taught how to teach or spent a lot of time working on teaching are generally poor coaches. [return]
  6. If you read the acknowledgements section of any of my posts, you can see that I get feedback from more than just two people on most posts (and I really appreciate the feedback), but I think that, by volume, well over 90% of the feedback I've gotten has come from Leah and a professional editor. [return]

February 04, 2020

Patrick Louis (venam)

Command Line Trash February 04, 2020 10:00 PM

lock drawing

NB: This is a repost on this blog of a post made on nixers.net

No this isn’t a post trashing shell scripting.

Handling files on the command line is most of the time a non-reversable process, a dangerous one in some cases (Unix Horror Stories). There are tricks to avoid the unnecessary loss and help in recovering files if need be.

Users do not expect that anything they delete is permanently gone. Instead, they are used to a “Trash can” metaphor. A deleted document ends up in a “Trash can”, and stays there at least for some time — until the can is manually or automatically cleaned.

In this thread we’ll list what ideas we have on this topic, novel or not so novel.


There’s the usual aliasing of rm to mv into a specific directory, a trash can for the command line.
This can be combined with a cron job or timer that cleans files in this directory that are older than a certain time.

You can check the actual XDG trash documentation that goes into great details about what needs to be taken into consideration:
https://specifications.freedesktop.org/trash-spec/trashspec-1.0.html

In $HOME/.local/share/Trash ($XDG_DATA_HOME/Trash) and usually at least split into two directories:

  • files for the actual exact copy of the files and directories (including their permission, and they should not override if two have the same names)
  • info, that contains the information about where and what name the deleted file had, in case it needs to be restored. And also the date it was deleted.

Another way to avoid losing files is to keep backups of the file system. This can be done via a logical volume management be it included in the file system itself (ZFS, btrfs, etc..) or not.

So, what’s your command line trash, how do you deal with avoidable losses.

February 03, 2020

Ponylang (SeanTAllen)

0.33.2 released February 03, 2020 10:37 PM

Pony version 0.33.2 is now available. The release features no breaking changes for users’ Pony code. We recommend updating at your leisure.

February 02, 2020

Gustaf Erikson (gerikson)

Deception Well by Linda Nagata February 02, 2020 09:13 PM

An uneasy melange of Solaris and Herbert’s Destination: Void universe. Not my favorite Nagata novel.

Ponylang (SeanTAllen)

Last Week in Pony - February 2, 2020 February 02, 2020 05:13 PM

Our community Zulip has over 500 members! Ryan A. Hagenson introduces pony-bio, a bioinformatics library for the Pony ecosystem.

Carlos Fenollosa (carlesfe)

February 01, 2020

Gustaf Erikson (gerikson)

The Secrets of Drearcliff Grange School by Kim Newman February 01, 2020 06:13 PM

The first book in the sequence, and the better one, in my opinion. The action is more focussed, the characters introduced, and the Big Bad threat better developed.

January 31, 2020

Gustaf Erikson (gerikson)

The Haunting of Drearcliff Grange School by Kim Newman January 31, 2020 01:50 PM

Apparently the second in a series, which explains the rather abrupt mis-en-scene. I guess is this a YA novel, but I wonder how many in the target audience are up to snuff with interwar British public school slang.

It’s well creepy, though. Newman has come quite a way since I read Anno Dracula back in the day, and I haven’t really read much of his stuff after that. The setting is between the wars, and some people are Unusuals - they can do magic, basically. In a all-girls boarding school, something returns after a disastrous trip to London…

Bogdan Popa (bogdan)

Announcing Try Racket January 31, 2020 10:00 AM

I’d been meaning to play with Racket’s built-in sandboxing capabilities for a while so yesterday I sat down and made Try Racket. It’s a web app that lets you type in Racket code and run it. The code you run is tied to your session and each session is allocated up to 60 seconds of run time per evaluation, with up to 128MB of memory used. Filesystem and network access is not permitted and neither is access to the FFI.

January 30, 2020

Derek Jones (derek-jones)

How are C functions different from Java methods? January 30, 2020 02:58 PM

According to the right plot below, most of the code in a C program resides in functions containing between 5-25 lines, while most of the code in Java programs resides in methods containing one line (code+data; data kindly supplied by Davy Landman):

Number of C/Java functions of a given length and percentage of code in these functions.

The left plot shows the number of functions/methods containing a given number of lines, the right plot shows the total number of lines (as a percentage of all lines measured) contained in functions/methods of a given length (6.3 million functions and 17.6 million methods).

Perhaps all those 1-line Java methods are really complicated. In C, most lines contain a few tokens, as seen below (code+data):

Number of lines containing a given number of C tokens.

I don’t have any characters/tokens per line data for Java.

Is Java code mostly getters and setters?

I wonder what pattern C++ will follow, i.e., C-like, Java-like, or something else? If you have data for other languages, please send me a copy.

January 29, 2020

Gustaf Erikson (gerikson)

The Bohr Maker by Linda Nagata January 29, 2020 06:13 PM

I believe this is Nagata’s debut novel, from 1995, and it holds up well. It’s deep into mid-90s SF tropes and the characterizations may be a bit stereotypical, but the scenes are well described and the action is snappy.

Twice on a Harsh Moon January 29, 2020 11:58 AM

(wow, that title…)

Two works of SF set on our satellite:

Luna - a series by IanMcDonald

New Moon 🌒 Wolf Moon 🌒 Moon Rising

The series comes to a satisfying, if slightly rushed, conclusion. The fact that maybe 80% of the female character’s names start with A doesn’t really help, nor does the sudden introduction of a fourth major faction (the University on Farside) feel very organic.

The Moon is a Harsh Mistress by Robert Heinlein

I think the influence from this book on McDonald’s work is pretty obvious. Both feature a Luna that’s used for extracting resources for Earth[1], both feature a future libertarian[2] society, and both feature the move towards independence.

Of course, Luna is written 40 years after The Moon… , and instead the Moon’s inhabitants being convicts from an authoritarian Earth, the ones in Luna are the ultimate gig workers. They take out loans to finance the trip, and pay for the “4 basics” - air, water, carbon and data. Don’t have enough to cover those? You will die and your mass reclaimed.

I find Heinlein’s view of women retrograde and borderline misogynistic. He gets points for imagining a future melange of languages, the libertarian quasi-utopian is as (im)plausible as McDonald’s, and the depiction of “Adam Selene”, the friendly AI that helps with the independence is well written.

One big difference is that in RHA’s work, libertarianism is an utopia, in Luna it’s a nightmare.


[1] Although Charles Stross has tweeted that the helium economy in Luna makes no scientific sense, the idea that it would be economical to grow wheat on the Moon and send the produce to Earth, as in TMiaHM is even dumber.

[2] in the “original” there are no laws, only contracts sense.

Pete Corey (petecorey)

MongoDB Object Array Lookup Aggregation January 29, 2020 12:00 AM

As part of an ongoing quest to speed up an application I’m working on, I found myself tasked with writing a fairly complicated MongoDB aggregation pipeline. I found no existing documentation on how to accomplish the task at hand, so I figured I should pay it forward and document my solution for future generations.

Widgets and Icons

Imagine we have two MongoDB collections. Our first collection holds information about widgets in our system:


db.getCollection('widgets').insert({
    _id: 1,
    name: 'Name',
    info: [
        {
            iconId: 2,
            text: 'Text'
        }
    ]
});

Every widget has a name and a list of one or more info objects. Each info object has a text field and an associated icon referenced by an iconId.

Our icons collection holds some basic information about each icon:


db.getCollection('icons').insert({
    _id: 2,
    name: 'Icon',
    uri: 'https://...'
});

The goal is to write an aggregation that returns our widgets with the associated icon documents attached to each corresponding info object:


{
    _id: 1,
    name: 'Name',
    info: [
        {
            iconId: 2,
            text: 'Text',
            icon: {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        }
    ]
}

Working Through the Pipeline

The aggregation that accomplishes this goal operates in six stages. Let’s work through each stage one by one. We’ll start by $unwinding our info array:


db.getCollection('widgets').aggregate([
    { $unwind: '$info' }
]);

This creates a new document for every widget/info pair:


{
    _id: 1,
    name: 'Name',
    info: {
        iconId: 2,
        text: 'Text',
    }
}

Next, we’ll $lookup the icon associated with the given iconId:


db.getCollection('widgets').aggregate([
    ...
    {
        $lookup: {
            from: 'icons',
            localField: 'info.iconId',
            foreignField: '_id',
            as: 'info.icon'
        }
    }
]);

Our resulting document will now have a list of icons in the info.icon field:


{
    _id: 1,
    name: 'Name',
    info: {
        iconId: 2,
        text: 'Text',
        icon: [
            {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        ]
    }
}

This is a step in the right direction, but we know that the info to icons relationship will always be a one to one relationship. We’ll always receive exactly one icon as a result of our $lookup.

Armed with this knowledge, we know we can $unwind on info.icon and safely turn our info.icon array into an object:


db.getCollection('widgets').aggregate([
    ...
    { $unwind: '$info.icon' }
]);

{
    _id: 1,
    name: 'Name',
    info: {
        iconId: 2,
        text: 'Text',
        icon: {
            _id: 2,
            name: 'Icon',
            uri: 'https://...'
        }
    }
}

But now we need to roll our info back up into an array. We can accomplish this by $grouping our widgets together based on their _id. However, we need to be careful to preserve the original document to avoid clobbering the entire widget:


db.getCollection('widgets').aggregate([
    ...
    {
        $group: {
            _id: '$_id',
            root: { $mergeObjects: '$$ROOT' },
            info: { $push: '$info' }
        }
    }
]);

Our resulting document contains our info array and the original, pre-$group widget document in the root field:


{
    root: {
        _id: 1,
        name: 'Name',
        info: {
            iconId: 2,
            text: 'Text',
            icon: {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        }
    },
    info: [
        {
            iconId: 2,
            text: 'Text',
            icon: {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        }
    ]
}

The next step in our pipeline is to replace our root document with the root object merged with the actual root document. This will override the info object in root with our newly grouped together info array:


db.getCollection('widgets').aggregate([
    ...
    {
        $replaceRoot: {
            newRoot: {
                $mergeObjects: ['$root', '$$ROOT']
            }
        }
    }
]);

We’re getting close to our goal:


{
    _id: 1,
    name: 'Name',
    info: [
        {
            iconId: 2,
            text: 'Text',
            icon: {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        }
    ],
    root: {
        _id: 1,
        name: 'Name',
        info: {
            iconId: 2,
            text: 'Text',
            icon: {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        }
    }
}

An unfortunate side effect of this merger is that our resulting document still has a root object filled with superfluous data. As a final piece of housecleaning, let’s remove that field:


db.getCollection('widgets').aggregate([
    ...
    {
        $project: {
            root: 0
        }
    }
]);

And with that we’re left with our original goal:


{
    _id: 1,
    name: 'Name',
    info: [
        {
            iconId: 2,
            text: 'Text',
            icon: {
                _id: 2,
                name: 'Icon',
                uri: 'https://...'
            }
        }
    ]
}

Success!

All Together

For posterity, here’s the entire aggregation pipeline in its entirety:


db.getCollection('widgets').aggregate([
    { $unwind: '$info' },
    {
        $lookup: {
            from: 'icons',
            localField: 'info.iconId',
            foreignField: '_id',
            as: 'info.icon'
        }
    },
    { $unwind: '$info.icon' },
    {
        $group: {
            _id: '$_id',
            root: { $mergeObjects: '$$ROOT' },
            info: { $push: '$info' }
        }
    },
    {
        $replaceRoot: {
            newRoot: {
                $mergeObjects: ['$root', '$$ROOT']
            }
        }
    },
    {
        $project: {
            root: 0
        }
    }
]);

I’ll be the first to say that I’m not a MongoDB expert, and I’m even less knowledgeable about building aggregation pipelines. There may be other, better ways of accomplishing this same task. If you know of a better, more efficient pipeline that gives the same results, please let me know!

January 27, 2020

Andrew Gallant (burntsushi)

Posts January 27, 2020 10:55 PM

Wesley Moore (wezm)

New Design 2020 January 27, 2020 04:42 AM

It's been more than 10 years since I started working on the previous design for this website 😅. This feels like a good point to come up with a new one!

The previous design served me well. The uncluttered design focussed on text was fast and responsive. It saw the introduction of new devices like iPad and a gradual increase in mobile display size without needing updating. The new design aims to retain these features while giving it a lighter, more modern feel.

Inspired by Chris Krycho and Shaun Inman I've taken to versioning the website instead of attempting to port all the existing content over to the new technology. This makes the redesign more of a clean slate and leaves old posts appearing how the did when originally posted. The new site is hosted under the /v2/ prefix. This allows all existing pages to stay where they are and retains the www.wezm.net domain. Compared to using a sub-domain it doesn't mess with DNS or search ranking. I have put redirects in place to direct the RSS feeds from the previous version to the new feed.

The new design uses the Manrope variable font for all text. Variable fonts are a fairly recent addition to the web platform but they have good support from fairly recent versions of all modern browsers and operating systems. On older browsers/operating systems the layout will fall back to a sans-serif font. Webfonts generally come with a non-trivial download cost. However, Manrope is 108kB and being a variable font that includes all weights between 200 and 800, as well as italic!

Technology wise, the previous site was built with Nanoc, a Ruby static site compiler. I've been very happy with Nanoc over the years but as my programming interests have shifted away from Ruby to Rust I've wanted to try a Rust static site compiler. I'm now using Zola. Zola is perhaps not quite as flexible as Nanoc but I've been able to achieve everything I wanted to with it. It's super fast and has nice conveniences like live-reload when editing content. Being a single file native binary also makes installation a breeze — no need to juggle Ruby versions or install gems.

Finally, I've now made the repository that the site is generated from public. This is to allow others to see how the site is built and permit corrections/fixes via issue or pull request.

Derek Jones (derek-jones)

How useful are automatically generated compiler tests? January 27, 2020 12:17 AM

Over the last decade, testing compilers using automatically generated source code has been a popular research topic (for those working in the compiler field; Csmith kicked off this interest). Compilers are large complicated programs, and they will always contain mistakes that lead to faults being experienced. Previous posts of mine have raised two issues on the use of automatically generated tests: a financial issue (i.e., fixing reported faults costs money {most of the work on gcc and llvm is done by people working for large companies}, and is intended to benefit users not researchers seeking bragging rights for their latest paper), and applicability issue (i.e., human written code has particular characteristics and unless automatically generated code has very similar characteristics the mistakes it finds are unlikely to commonly occur in practice).

My claim that mistakes in compilers found by automatically generated code are unlikely to be the kind of mistakes that often lead to a fault in the compilation of human written code is based on the observations (I don’t have any experimental evidence): the characteristics of automatically generated source is very different from human written code (I know this from measurements of lots of code), and this difference results in parts of the compiler that are infrequently executed by human written code being more frequently executed (increasing the likelihood of a mistake being uncovered; an observation based on my years working on compilers).

An interesting new paper, Compiler Fuzzing: How Much Does It Matter?, investigated the extent to which fault experiences produced by automatically generated source are representative of fault experiences produced by human written code. The first author of the paper, Michaël Marcozzi, gave a talk about this work at the Papers We Love workshop last Sunday (videos available).

The question was attacked head on. The researchers instrumented the code in the LLVM compiler that was modified to fix 45 reported faults (27 from four fuzzing tools, 10 from human written code, and 8 from a formal verifier); the following is an example of instrumented code:

warn ("Fixing patch reached");
if (Not.isPowerOf2()) {
   if (!(C-> getValue().isPowerOf2()  // Check needed to fix fault
         && Not != C->getValue())) {
      warn("Fault possibly triggered");
   } else { /* CODE TRANSFORMATION */ } } // Original, unfixed code

The instrumented compiler was used to build 309 Debian packages (around 10 million lines of C/C++). The output from the builds were (possibly miscompiled) built versions of the packages, and log files (from which information could be extracted on the number of times the fixing patches were reached, and the number of cases where the check needed to fix the fault was triggered).

Each built package was then checked using its respective test suite; a package built from miscompiled code may successfully pass its test suite.

A bitwise compare was run on the program executables generated by the unfixed and fixed compilers.

The following (taken from Marcozzi’s slides) shows the percentage of packages where the fixing patch was reached during the build, the percentages of packages where code added to fix a fault was triggered, the percentage where a different binary was generated, and the percentages of packages where a failure was detected when running each package’s tests (0.01% is one failure):

Percentage of packages where patched code was reached during builds, and packages with failing tests.

The takeaway from the above figure is that many packages are affected by the coding mistakes that have been fixed, but that most package test suites are not affected by the miscompilations.

To find out whether there is a difference, in terms of impact on Debian packages, between faults reported in human and automatically generated code, we need to compare number of occurrences of “Fault possibly triggered”. The table below shows the break-down by the detector of the coding mistake (i.e., Human and each of the automated tools used), and the number of fixed faults they contributed to the analysis.

Human, Csmith and EMI each contributed 10-faults to the analysis. The fixes for the 10-fault reports in human generated code were triggered 593 times when building the 309 Debian packages, while each of the 10 Csmith and EMI fixes were triggered 1,043 and 948 times respectively; a lot more than the Human triggers :-O. There are also a lot more bitwise compare differences for the non-Human fault-fixes.

Detector  Faults   Reached    Triggered   Bitwise-diff   Tests failed
Human       10      1,990         593         56              1
Csmith      10      2,482       1,043        318              0
EMI         10      2,424         948        151              1
Orange       5        293          35          8              0
yarpgen      2        608         257          0              0
Alive        8      1,059         327        172              0

Is the difference due to a few packages being very different from the rest?

The table below breaks things down by each of the 10-reported faults from the three Detectors.

Ok, two Human fault-fix locations are never reached when compiling the Debian packages (which is a bit odd), but when the locations are reached they are just not triggering the fault conditions as often as the automatic cases.

Detector   Reached    Triggered
Human
              300       278
              301         0
              305         0
                0         0
                0         0
              133        44
              286       231
              229         0
              259        40
               77         0
Csmith
              306         2
              301       118
              297       291
              284         1
              143         6
              291       286
              125       125
              245         3
              285        16
              205       205
EMI      
              130         0
              307       221
              302       195
              281        32
              175         5
              122         0
              300       295
              297       215
              306       191
              287        10

It looks like I am not only wrong, but that fault experiences from automatically generated source are more (not less) likely to occur in human written code (than fault experiences produced by human written code).

This is odd. At best I would expect fault experiences from human and automatically generated code to have the same characteristics.

Ideas and suggestions welcome.

Update: the morning after

I have untangled my thoughts on how to statistically compare the three sets of data.

The bootstrap is based on the idea of exchangeability; which items being measured might we consider to be exchangeable, i.e., being able to treat the measurement of one as being the equivalent to measuring the other.

In this experiment the coding mistakes are not exchangeable, i.e., different mistakes can have different outcomes.

But we might claim that the detection of mistakes is exchangeable; that is, a coding mistake is just as likely to be detected by source code produced by an automatic tool as source written by a Human.

The bootstrap needs to be applied without replacement, i.e., each coding mistake is treated as being unique. The results show that for the sum of the Triggered counts (code+data):

  • treating Human and Csmith as being equally likely to detect the same coding mistake, there is a 18% change of the Human results being lower than 593.
  • treating Human and EMI as being equally likely to detect the same coding mistake, there is a 12% change of the Human results being lower than 593.

So the likelihood of the lower value, 593, of Human Triggered events is expected to occur quite often (i.e., 12% and 18%). Automatically generated code is not more likely to detect coding mistakes than human written code (at least based on this small sample set).

January 26, 2020

Patrick Louis (venam)

Loading of xinitrc,xserverrc,xresources,xdefaults,xprofile,xsession,xmodmap January 26, 2020 10:00 PM

X11 Logo

NB: This is a repost on this blog of a post made on nixers.net

We often hear discussions about X configuration files and their roles. Namely: xinitrc,xserverrc,xresources,xdefaults,xprofile,xsession,xmodmap. So let’s try to clear up this mumbo jumbo of words.

There’s roughly two ways to start your X environment, one is via xinit and the other is via a display manager (fancy login screen). Depending on which one you use, different configuration files will be loaded.

If starting via xinit, or startx which is a wrapper over xinit, then the ~/.xinitrc will be loaded, and if not present will load the global /etc/X11/xinit/xinitrc. This will run all the lines found in it, interpreted by /bin/sh and will stop at the last one. The X session will stop when that last program terminates.

If using the globally available xinitrc it will include in an alphabetical order sub-xinitrc found in the /etc/X11/xinit/xinitrc.d/.

That globally available xinitrc loads two more configurations:

  • Xresources, found in ~/.Xresources or /etc/X11/xinit/.Xresources which consists of the key/value pair accessible for all X clients, the resources. xinit executes xrdb -merge ~/.Xresources or xrdb with the global one.
  • Xmodmap, locally in ~/.Xmodmap and globally in /etc/X11/xinit/.Xmodmap. This will run xmodmap $thefile. So in theory instead of having all those xmodmap lines that we find so commonly in the .xinitrc file (I’m guilty of this too) we can separate them into a .Xmodmap file instead.

xinit/startx will finally start the X server, it does it by executing a script found in ~/.xserverrc or globally /etc/X11/xinit/xserverrc. This consists of simply:

exec /usr/bin/X -nolisten tcp "$@"

However, replacing this xserverrc allows us to start X in different ways.

What about initiating a graphical session from the display manager.

Instead of xinitrc the file loaded at login will be the Xsession file. So similar to xinitrc we have globally a default located at /etc/X11/Xsession.options along with a directory of sub-xsessions to be loaded in /etc/X11/Xsession.d. Also similar to xinit the default Xsession will load the Xresources. As for local configs there are many of them depending on what the type of session, I quote:

If the user has a ~/.xsessionrc file, read it. (used by all sessions types)
If a specific session was selected in the DM (GDM, KDM, WDM, LightDM, …) , run it.
Otherwise, if the user has a ~/.xsession or ~/.Xsession file, run it.
Otherwise, if the /usr/bin/x-session-manager command exists, run it.
Otherwise, if the /usr/bin/x-window-manager command exists, run it.
Otherwise, if the /usr/bin/x-terminal-emulator command exists, run it.

Some specific display manager include in their default Xsession an extra configuration called Xprofile.

For example:

  • GDM - /etc/gdm/Xsession
  • LightDM - /etc/lightdm/Xsession
  • LXDM - /etc/lxdm/Xsession
  • SDDM - /usr/share/sddm/scripts/Xsession

Otherwise, if you want the Xprofile, you have to source the file manually from startx/xinit or XDM or any other display manager.

Now for something unrelated, Xdefaults is the old version of Xresources.

The way it was done in the old days is that Xdefaults was read every single time a client program (Xlib) was started, unlike Xresources which have properties stored in the root window/resource manager (think xrdb). So that means the old method, Xdefaults, couldn’t be used over the network because you needed direct access to the file.

Now that gets a bit complicated because there could be multiple Xdefaults files found in different ways other than ~/.Xdefaults.

I quote:

There also is the $XENVIRONMENT variable, which defaults to ~/.Xdefaults-hostname ($XENVIRONMENT/.Xdefaults) if not set. This is used in the same way as .Xdefaults, but is always read regardless of whether RESOURCE_MANAGER is present. You can use .Xdefaults-hostname files to keep some settings machine-specific while using xrdb for the global ones

The fourth location is the directory pointed to by the $XAPPLRESDIR environment variable. (Oddly, if the variable is not set, $HOME is used as the default.) When a program is started, it looks if any of the following files exist (the file name being the same as the program’s class name):
$XAPPLRESDIR/$LC_CTYPE/XTerm
$XAPPLRESDIR/language/XTerm
$XAPPLRESDIR/XTerm

The fifth location is the system-wide “app-defaults” directories. Again, the app-defaults directories are checked on program startup if they have a file named after the program. For example, XTerm (on Arch Linux) uses:

/etc/X11/$LC_CTYPE/app-defaults/XTerm
/etc/X11/language/app-defaults/XTerm
/etc/X11/app-defaults/XTerm
/usr/share/X11/$LC_CTYPE/app-defaults/XTerm
/usr/share/X11/language/app-defaults/XTerm
/usr/share/X11/app-defaults/XTerm

The app-defaults files are usually installed into /usr/share along with the program itself; administrator overrides would go to /etc.

I hope that helps clear things up.











References:

Key And Trust Store on Unix-like OS January 26, 2020 10:00 PM

lock drawing

NB: This is a repost on this blog of a post made on nixers.net

Let’s have a discussion about all the kinds of trust stores found on Unix-like operating systems.
For those not in the know, trust stores are places where the operating sytems generally, or the specific software, stores private and public keys (asymmetric), trusted CAs, and symmetric keys (decryption keys).

There’s a lot to cover on this topic, I thought of writing an article about this because I couldn’t find anything online that covered it in a generic manner. That’s what gets me writing anyway.


Let’s tackle some of the stuffs regarding TLS PKI (Public Key Infrastructure).

Mozilla maintains a list of trusted CAs in a certificate store that a lot of Unix-like operating system fetch through the package manager and deploy at /etc/ssl/certs.

This location is accessed system wide by a lot of utilities to check the trusted certificates. It sort of has become standard, though as you’ll see in a bit it’s not really.

You also may find a symbolic link there pointing to /etc/ca-certificates/extracted/. This all points to the same thing, there’s even /usr/share/ca-certificates or /usr/lib/mozilla/certificates, /usr/lib64/mozilla/certificates, ~/.mozilla/certificates.

Openssl also stores/read certificate from that location /etc/ssl, that’s where you’ll find openssl.cnf for example. In this directory you can choose to store your private keys associated with certificates you’ve generated yourself in /etc/ssl/private. For obvious reasons, this directory should only be owned by root.

But there’s a catch here, openssl can be compiled with the nss library, which will have its own list of trusted CAs built-in though usually through /usr/lib/libnssckbi.so which, again, has the list of maintained trusted CAs by Mozilla (Mozilla’s Network Security Services).

The Chrome browser also uses nss so you might ask where the trust exclusions are stored when added. They are in $HOME/.pki/nssdb or /etc/pki/nssdb globally in an sqlite3 db.

Firefox also uses an sqlite3 database to store its exclusions. However, it’s not in the .pki directory but right within its profile directory: $HOME/.mozilla/firefox/<profilename>.default/cert_override.txt. Add to this that it has two (or maybe more) sqlite3 dbs in there which are basically copy of the nss trusted certs that are found globally on the system.

Now what about programming languages that want to access the internet in a secure manner through TLS PKI.

Most of them rely on the trusted stores mentioned previously, namely nss or /etc/ssl. However, some don’t.

I’m aware of one well known example with the Java language. It stores its trust store in the $JAVA_HOME/jre/lib/security/cacerts which is a java keystore. The password to this keystore is “changeit”. Java has a concept of security provider, and they are listed in order of preference in the java.security file. Hopefully you can find one of the provider relying on the nss.cfg, and so we have less redundancy within our system.

Let’s also put a hint here about certificate revocation. Sometimes, in specific cases, you can’t always rely on your OS packages to update your trusted CAs and you’ll need a daemon to check CRLs and OCSPs for all the trusted certs you got.

One example is: dirmngr(1)

Now there are two other common places that I’ll tackle too.

Gnupg trust store and ssh trust store. Those are in $HOME/.gnupg and $HOME/.ssh respectively.

Those directories both contains trusted certificates and your private/public pairs.

Let’s mention that almost all things in the crypto world uses a format called ASN.1 with DER encoding or not. GPG, X509, SSH all have it this way with some different formatting in some places.

You can have a look at those here:

And here’s a useful link:

https://wiki.gentoo.org/wiki/Certificates


Outro

So nixers, what do you have to say about trust stores on Unix-like OS. Anything to add to what I’ve mentioned. There’s a lot I’ve deliberately left out. Maybe talk about interfacing with keystores on a hardware security module through pkcs#11, like a yubikey, that could be used for OTP. Maybe we can talk about all the utilities that can be used to manipulate, create, and display in a human readable format the certificates, public/private pairs, and more (openssl, keytool, certutil, etc..). We can also talk about building your own PKI. We can talk about what specific language do to handle cryptographic keystores, what you like about it. Or maybe simply share a link you’ve found useful. Or maybe we can talk about package management and how maintainers should sign their packages. Or just express your opinion about anything.

We could go into secret management, PAM, crypto protocols and different libraries, and MAC (Mandatory access control, think SELinux and others) as a whole but that would be too big for a single thread. If you want to do that we can open a new one. Let’s attack trust and key stores in this thread.

What’s your take on trust and key stores?











Attributions:

  • Unknown. The author of the picture is not given, but it is possible that it is Josef Pokorný, the author of the text of the encyclopedia entry. [Public domain]

Gonçalo Valério (dethos)

Setting up a Content-Security-Policy January 26, 2020 09:08 PM

A couple of weeks ago, I gave a small talk on the Madeira Tech Meetup about a set of HTTP headers that could help website owners protect their assets and their users. The slides are available here, just in case you want to take a look.

The content of the talk is basically a small review about what exists, what each header tries to achieve and how could you use it.

After the talk I remembered that I didn’t review the heades of this blog for quite sometime. So a quick visit to Mozilla Observatory, a tool that lets you have a quick look of some of the security configurations of your website, gave me an idea of what I needed to improve. This was the result:

The Content-Security-Header was missing

So what is a Content Security Policy? On the MDN documentation we can find the following description:

The HTTP Content-Security-Policy response header allows web site administrators to control resources the user agent is allowed to load for a given page.

Mozilla Developer Network

Summing up, in this header we describe with a certain level of detail the sources from where each type of content can be fetched in order to be allowed and included on a given page/app. The main goal of this type of policy is to mitigate Cross-Site Scripting attacks.

In order to start building a CSP for this blog a good approach, in my humble opinion, is to start with the more basic and restrictive policy and then proceed evaluating the need for exceptions and only add them when strictly necessary. So here is my first attempt:

default-src 'self'; object-src 'none'; report-uri https://ovalerio.report-uri.com/r/d/csp/reportOnly

Lets interpret what it says:

  • default-src: This is the default value for all non-mentioned directives. self means “only things that come from this domain”.
  • object-src: No <object>, <embed> or <applet> here.
  • report-uri: All policy violations should be reported by the browser to this URL.

The idea was that all styles, scripts and images should be served by this domain, anything external should be blocked. This will also block inline scripts, styles and data images, which are considered unsafe. If for some reason I need to allow this on the blog I could use unsafe-inline, eval and data: on the directive’s definition but in my opinion they should be avoided.

Now a good way to find out how this policy will affect the website and to understand how it needs to be tuned (or the website changed) we can activate it using the “report only mode:

Content-Security-Policy-Report-Only: <policy>

This mode will generate some reports when you (and other users) navigate through the website, they will be printed on the browser’s console and sent to the defined report-uri, but the resources will be loaded anyway.

Here are some results:

CSP violations logs on the browser consoleExample of the CSP violations on the browser console

As an example below is a raw report from one of those violations:

{
    "csp-report": {
        "blocked-uri": "inline",
        "document-uri": "https://blog.ovalerio.net/",
        "original-policy": "default-src 'self'; object-src 'none'; report-uri https://ovalerio.report-uri.com/r/d/csp/reportOnly",
        "violated-directive": "default-src"
    }
}

After a while I found that:

  • The theme used on this blog used some data: fonts
  • Several inline scripts were being loaded
  • Many inline styles were also being used
  • I have some demos that load content from asciinema.org
  • I often share some videos from Youtube, so I need to allow iframes from that domain
  • Some older posts also embeded from other websites (such as soundcloud)

So for the blog to work fine with the CSP being enforced, I either had to include some exceptions or fix errors. After evaluating the attack surface and the work required to make the changes I ended up with the following policy:

Content-Security-Policy-Report-Only: default-src 'self'; script-src 'self' https://asciinema.org 'sha256-A+5+D7+YGeNGrYcTyNB4LNGYdWr35XshEdH/tqROujM=' 'sha256-2N2eS+4Cy0nFISF8T0QGez36fUJfaY+o6QBWxTUYiHc=' 'sha256-AJyUt7CSSRW+BeuiusXDXezlE1Wv2tkQgT5pCnpoL+w=' 'sha256-n3qH1zzzTNXXbWAKXOMmrBzjKgIQZ7G7UFh/pIixNEQ='; style-src 'self' 'sha256-MyyabzyHEWp8TS5S1nthEJ4uLnqD1s3X+OTsB8jcaas=' 'sha256-OyKg6OHgnmapAcgq002yGA58wB21FOR7EcTwPWSs54E='; font-src 'self' data:; img-src 'self' https://secure.gravatar.com; frame-src 'self' https://www.youtube.com https://asciinema.org; object-src 'none'; report-uri https://ovalerio.report-uri.com/r/d/csp/reportOnly

A lot more complex than I initially expected it to be, but it’s one of the drawbacks of using a “pre-built” theme on a platform that I didn’t develop. I was able (in the available time) to fix some stuff but fixing everything would take a lot more work.

All those sha-256 hashes were added to only allow certain inline scripts and styles without allowing everything using unsafe-inline.

Perhaps in the future I will be able to change to a saner theme/platform, but for the time being this Content-Security-Policy will do the job.

I started enforcing it (by changing Content-Security-Policy-Report-Only to Content-Security-Policy) just before publishing this blog post, so if anything is broken please let me know.

I hope this post has been helpful to you and if you didn’t yet implement this header you should give it a try, it might take some effort (depending on the use case) but in the long run I believe it is totally worth it.

January 25, 2020

Carlos Fenollosa (carlesfe)

January 20, 2020

Gustaf Erikson (gerikson)

With the Old Breed: At Peleliu and Okinawa by Eugene B. Sledge January 20, 2020 01:55 PM

Continuing my deep dive into the rot and shit of the Pacific theatre. Sledge has another background than Leckie (who was a sportswriter as a civilian) and has less facility with words. I believe Leckie spent a lot of time drinking beers with other vets, polishing his anecdotes, while Sledge pushed his memories back - he alludes to frequent nightmares after his experiences.

January 19, 2020

Ponylang (SeanTAllen)

Last Week in Pony - January 19, 2020 January 19, 2020 03:50 PM

Sean T. Allen’s recent PWL on Deny Capabilities for Safe, Fast Actors Talk is available. Microsoft’s Project Verona is now open source.

Eric Faehnrich (faehnrich)

When a Byte is Not 8 Bits January 19, 2020 05:00 AM

I’ve been getting into my backlog of C books and resources when I came across my copy of Portable C by Henry Rabinowitz1 that I obtained after reading the post C Portability Lessons from Weird Machines.

The blog post lists old weird machines with addressable units that might not be your typical 8-bit bytes. One might think that’s well and good, but not a concern since typical modern architectures have 8-bit bytes. That’s not entirely the case.

I work on products that have a Texas Instruments C2000 microcontroller. This is a modern microcontroller in use now. However, it has 16-bit bytes instead of 8.

I understand that the C2000 isn’t a part you see every day, but the fact remains that I have to support code for this.

If you want to play with this part, you can get their LaunchPad kit that has it.

So the addressable units and char are 16 bits. The standard says sizeof(char)==1, so any size is a multiple of 16 bits.

The standard also says int is at least 16 bits, but could for instance be 32. The C2000 just happens to have int be the minimum 16 bits. Interestingly, this means sizeof(int)==1 when we’re used to int being larger than char.

A multi-byte word like a 32-bit unsigned long is then made up of two 16-bit bytes. The C2000 is little-endian, so if we had an unsigned long with the value 0x01020304 at address 0x00000426, it would look like this in memory:

0x00000426 0x0304
0x00000427 0x0102

An example of portability becoming a concern is when we have to take something off of the network. We can’t reuse code to convert network order to the C2000 when it expects 8-bit bytes in the host. We had to write our own just for this.

Endianness is a worry when you have multi-byte words. But also what about 8-bit byte arrays coming in from the network? Do you store each 8-bit network byte in its own 16-bit byte? Or do you pack two 8-bit network bytes into one 16-bit host byte?

Similarly, when we’re sending something out, does the host put just 8-bits in the 16-bit register that holds values going out onto the wire? And is that upper or lower 8 bits? Or pack two 8-bit bytes again?

It’s certainly awkward talking about 8-bit bytes inside our 16-bit bytes, so we just call them octets.

I look forward to learning anything from those portability books that I can apply to the C2000.

  1. That blog post must have driven demand up for that book. When I first ordered it on Amazon for a reasonable price, the seller then canceled my order but I didn’t think much of it. When I searched for another copy, I saw that same seller had the book again, but this time for over $300! It’s a niche area of programming but demand shouldn’t be that crazy.

January 16, 2020

Derek Jones (derek-jones)

for-loop usage at different nesting levels January 16, 2020 04:59 PM

When reading code, starting at the first line of a function/method, the probability of the next statement read being a for-loop is around 1.5% (at least in C, I don’t have decent data on other languages). Let’s say you have been reading the code a line at a time, and you are now reading lines nested within various if/while/for statements, you are at nesting depth d. What is the probability of the statement on the next line being a for-loop?

Does the probability of encountering a for-loop remain unchanged with nesting depth (i.e., developer habits are not affected by nesting depth), or does it decrease (aren’t developers supposed to using functions/methods rather than nesting; I have never heard anybody suggest that it increases)?

If you think the for-loop use probability is not affected by nesting depth, you are going to argue for the plot on the left (below, showing number of loops whose compound-statement contains appearing in C source at various nesting depths), with the regression model fitting really well after 3-levels of nesting. If you think the probability decreases with nesting depth, you are likely to argue for the plot on the right, with the model fitting really well down to around 10-levels of nesting (code+data).

Number of C for-loops whose enclosed compound-statement contains basic blocks nested to a given depth.

Both plots use the same data, but different scales are used for the x-axis.

If probability of use is independent of nesting depth, an exponential equation should fit the data (i.e., the left plot), decreasing probability is supported by a power-law (i.e, the right plot; plus other forms of equation, but let’s keep things simple).

The two cases are very wrong over different ranges of the data. What is your explanation for reality failing to follow your beliefs in for-loop occurrence probability?

Is the mismatch between belief and reality caused by the small size of the data set (a few million lines were measured, which was once considered to be a lot), or perhaps your beliefs are based on other languages which will behave as claimed (appropriate measurements on other languages most welcome).

The nesting depth dependent use probability plot shows a sudden change in the rate of decrease in for-loop probability; perhaps this is caused by the maximum number of characters that can appear on a typical editor line (within a window). The left plot (below) shows the number of lines (of C source) containing a given number of characters; the right plot counts tokens per line and the length effect is much less pronounced (perhaps developers use shorter identifiers in nested code). Note: different scales used for the x-axis (code+data).

Number of lines containing a given number of C tokens.

I don’t have any believable ideas for why the exponential fit only works if the first few nesting depths are ignored. What could be so special about early nesting depths?

What about fitting the data with other equations?

A bi-exponential springs to mind, with one exponential driven by application requirements and the other by algorithm selection; but reality is not on-board with this idea.

Ideas, suggestions, and data for other languages, most welcome.

January 15, 2020

Gustaf Erikson (gerikson)

Dunkirk: Fight to the Last Man by Simon Sebag-Montefiore January 15, 2020 03:20 PM

An accessible read on the fall of France and the evacuation from Dunkirk. This is the first book by Sebag-Montefiore I’ve read and I’m not that impressed.

I did like the attempt to give other viewpoints than the British, though.

Dunkirk-in-memory is weird. I’m sure the recent movie (the reason I wanted to read this book) got a lot of lift from Brexit, and that the Leavers imagine they’re doing something similar. Of course Dunkirk was a crushing defeat, but in that curious British (English?) way, it’s almost more famous than some victories (cf. Scott vs. Amundsen). Perhaps it’s an memory of Thermopylae, as echoed by Hjalmar Gullberg’s poem about the death of Karin Boye:

Ej har Nike med segerkransen
krönt vid flöjtspel och harposlag
perserkonungen, jordens gissel.
Glömd förvittrar hans sarkofag.
Hyllningkören skall evigt handla
om Leonidas’ nederlag.

By far the most chilling parts of the book are the discussions in the War Cabinet on whether Great Britain should seek an armistice with Nazi Germany. Churchill, whatever his faults and motivations, deserves credit for not giving in. Leavers see themselves as heirs to Churchills, but they’re actually followers of Lord Halifax.

January 14, 2020

Mark Fischer (flyingfisch)

MVC website redirecting to logout page immediately after logging in January 14, 2020 06:16 PM

Over the past couple days I have been converting the authentication and authorization method on an existing MVC website to use Auth0, an OpenID provider. During the process of converting the website’s login and logout routines I ran into an issue where no matter what the website would redirect to the logout page immediately after hitting the login route. After much trial and error I finally pinpointed the problem.

In my project’s web.config I had the following code:

<authentication mode="Forms">
	<forms loginUrl="~/Auth/Logout" timeout="2880" />
</authentication>

I changed this to mode="None" and the login page now works flawlessly.

<authentication mode="None"></authentication>

January 13, 2020

Andrew Owen (yumaikas)

A Small Command-line Productivity Tip January 13, 2020 09:00 PM

I really like using the command line for automating tasks. There are some things that a GUI is handier for, but having scripts handy is a very nice place to be.

One trick I’ve started using is writing small wrapper scripts (using batch files at work) for tools like RipGrep or NUnit3-console. This makes it far easier to edit those commands, and save back examples of doing different things, especially for commands with more involved parameter lists. It makes it possible to add small things, like output delimiters between search results, or to do log actions to a file or the like. It also makes it easy to reduce the total amount of typing I have to do in a given command prompt.

An example, using RipGrep, in a file called search.bat on my PATH.

REM banner is a program that spits out "-", as wide as the terminal is
REM it takes a color as its only argument.


banner blue
REM %dp~1 here is a batch-ism for getting the _first_ paramter to a batch script
rg -iw -pcre --glob "!*.xml" %dp~1


REM Other useful argument combos
REM -tcs -tsql -Txml

The nice thing about this approach is that lets me pull up an editor for when I want edit a given command and it makes using multiple arguments to --glob much easier. Were I to start using find on Unix a lot, it’d probably get a similar treatment. It’s definitely a nice thing to have on hand for more involved command-line tools.

Published Jan 12, 2020

Warming up to Unit Testing January 13, 2020 09:00 PM

One of the things that has been a consistent part of my software career is that I often have to approach a given idea or practice at least 3 times before I start to get comfortable with it. It’s happened with managing websites, making video games (my first Ludum Dare was well beyond the 3rd time I’d tried to make a video game), programming languages (both Nim and Factor were languages I approached once, and then again with more experience under my belt), and software develoment techniques. I got a lot more comfortable with Git after I’d had a chance to use Fossil for a while.

All this to say, that I rarely pick up something completely the first time I go at it, so for any younger colleagues that might have been impressed with my mastery of regex or shell scripting, that mastery was not achieved overnight.

Recently, I’ve started to have another go-round at Unit Testing, and this is feeling like the time-around that will have it stick into my habits as well as version control does. And, in the middle of all of this, the reasons why it seems to be sticking around seem to be a convergence of factors, not just a single thing.

For one, some sort of approach to automated testing is being required at work. Were this the only factor, I’d probably pick up a little bit of unit testing practice for work, but it would certainly not be a major factor outside of work. The other thing is that I picked up several books at the end of the year that all ended up talking about unit testing and picking the right abstraction.

The first, and possibly most influential was Mastering Software Technique, and the associated articles, by Noah Gibbs. It doesn’t have anything to do with unit testing specifically, but Gibbs consistently recommended 99 Bottles of OOP (Sandi Metz and Katrina Owen), which does have a decent amount in it about unit testing. I also picked up the book Working Effectively with Legacy Code, by Michael Feathers, mostly because I was looking for books that addressed relatively timeless topics, rather than mostly books that address version X of a given framework.

So, I ended up with a lot of programming books about unit testing. I also found in going through 99 Bottles of OOP, that the unit testing harness available for Ruby is relatively responsive, especially compared to the NUnit Visual Studio Test Explorer. Eventually, some bugs, either in my code, or the Test Explorer, led me to trying the nunit command-line runner. The difference was impressive. It makes me think that the nunit command-line tools get more attention than the test runner, because getting the command-line tools working consistently was a lot easier than keeping the test-runner working.

The last thing that seems to be cementing automated testing in my mind as a worthwhile investment was an article I ran across about Moon Pig. Often, when people tell you to adopt a practice, they don’t do a great job communicating the concrete reasons they adopted it. For me, the story of what and how various things were mocked for testability reasons in Moon Pig, mocking out the storage and time aspects of things, felt like a great starting point for how to mock things for tests, and it included a good, relatively concrete reasons for mocking things out the way they did.

So, now that I have a test-runner that works at a reasonable speed and reliability, and I’m taking in the wisdom of Michael Feathers about Legacy Code, and that I have a good story to refer too when I’m reasoning about how to mock things out, I think unit testing as a practice will be more sticky this time than it was before.

PostScript:

I should also give a shout-out to Kartik Agaram in this. Mu, and how low level he’s taking a test-driven approach to software has definitely informed how I write system-type software, as opposed to business software. PISC’s unit tests weren’t a direct result of hearing about Mu, but Mu’s artificial file system and such definitely got me thinking in that direction.

Published Jan 12, 2020

Ponylang (SeanTAllen)

Last Week in Pony - January 12, 2020 January 13, 2020 01:12 AM

We’ve had a boatload of exciting new RFC proposals in the past few weeks! Check them out at https://github.com/ponylang/rfcs/pulls.

January 12, 2020

Derek Jones (derek-jones)

The dark-age of software engineering research: some evidence January 12, 2020 05:28 PM

Looking back, the 1970s appear to be a golden age of software engineering research, with the following decades being the dark ages (i.e., vanity research promoted by ego and bluster), from which we are slowly emerging (a rough timeline).

Lots of evidence-based software engineering research was done in the 1970s, relative to the number of papers published, and I have previously written about the quantity of research done at Rome and the rise of ego and bluster after its fall (Air Force officers studying for a Master’s degree publish as much software engineering data as software engineering academics combined during the 1970s and the next two decades).

What is the evidence for a software engineering research dark ages, starting in the 1980s?

One indicator is the extent to which ancient books are still venerated, and the wisdom of the ancients is still regularly cited.

I claim that my evidence-based software engineering book contains all the useful publicly available software engineering data. The plot below shows the number of papers cited (green) and data available (red), per year; with fitted exponential regression models, and a piecewise regression fit to the data (blue) (code+data).

Count of papers cited and data available, per year.

The citations+date include works that are not written by people involved in software engineering research, e.g., psychology, economics and ecology. For the time being I’m assuming that these non-software engineering researchers contribute a fixed percentage per year (the BibTeX file is available if anybody wants to do the break-down)

The two straight line fits are roughly parallel, and show an exponential growth over the years.

The piecewise regression (blue, loess was used) shows that the rate of growth in research data leveled-off in the late 1970s and only started to pick up again in the 1990s.

The dip in counts during the last few years is likely to be the result of me not having yet located all the recent empirical research.

January 11, 2020

Gustaf Erikson (gerikson)

Helmet for My Pillow: From Parris Island to the Pacific by Robert Leckie January 11, 2020 02:36 PM

I enjoyed the TV miniseries The Pacific, and this is one of the inspirations for it. Leckie is a good if journeymanlike writer, and the story flows chronologically with no significant pauses. Flashes of class differences, frank discussion of petty criminality and sexual promiscuity, and actual sympathy for the hated enemy enliven the text.

Gergely Nagy (algernon)

2020 January 11, 2020 02:00 PM

I set out to write a year-end retrospective, even ended up with almost a thousand words written, but I got distracted, and never finished. Having sat down to do just that now, I re-read it all, and decided it's best to throw it in the bin. In many ways, 2019 was a terrible year. In some other ways, it was incredible. When I first sat down, I started listing everything that happened, good and bad - but in truth, such a detailed log is not something the great public should read. Yes, yes, that "great" public is likely a handful of people who stumble upon this blog. Doesn't matter. Those who need to know, do. Those who don't, should not.

So instead of looking back, I'll be looking forward.

I will be looking forward to 2020, because we're in our own apartment now, and while not everything's a rosy dream come true (oh, the stories I could tell!), it's a colossal leap forward. It enabled - and continues to enable - us to do so many things we weren't able to do before. We've made friends, plans, and had a lot of fun. I also have a work room, with a lock, making it a million times easier to work from home, and shut out any distractions when immersing myself in keyboard-related work.

As it happens, I've been working for Keyboardio full-time for a while, and it looks like that this might be sustainable for longer than originally planned. Not complaining, mind you, this is a situation I very much enjoy. Doubly so, because I'm also doing similar work for Dygma on the side, and getting paid to do what I love doing is amazing, and I'm incredibly privileged to be in this situation. Every day I wonder when I'll wake up to find out this was just a dream.

Of course, sometimes I have to work on Chrysalis or Bazecor, and then I start to question my life choices, because JavaScript and front-end development in general is something I very much do not enjoy. But alas, that's also work that needs to be done. I guess it's the price to pay for the privilege of being able to work from home, on free software, which I use every single time I interact with the computer, in an area I find great pleasure diving into.

On another note, in 2020, I plan to continue adapting my work environment to be more ergonomic, more comfortable, and more practical. I already have a terrific keyboard, and amazing trackball, an adjustable standing desk. The next thing is adjusting my monitors: they're currently on the desk - I'd love to mount them on the wall instead, on adjustable arms. That'd free up a lot of space on the desk, and would make it easier to have them at the right height, distance, and angle. It would also allow me to reposition more easily: whether I face the window, or away, for example. I'd also love to upgrade at least one of my monitors to something that's easier on my eyes. At a previous job, I loved the 30" 4k monitors, text were crisp, and the high resolution was so much easier on my eyes. I've yet to find one which I like, and can afford. There's a good chance I will need to upgrade my GPU to be able to drive a 4k monitor, too. Plenty of things to figure out here, but this is a goal I'm pursuing this year - because the rest of my work environment is already top notch, and every single upgrade so far made me more productive. It pays off in the long run.

Another thing I want to do in 2020 is write more. Last year, I wasn't exactly prolific, even though there were a lot of things I could've written about - but when it would've made sense to write, I didn't have the resources to do so, and doing it later wasn't a viable strategy. That's just guilt piling up, to the point where I give up and just don't write. So the goal this year is to build writing into my routine, so it does not get postponed. I do enjoy writing! But there was always something more important to do. Writing felt like an administrative burden - it shouldn't. I'm not sure how to tackle this yet.

With working on keyboard-related things full-time (and part-time too), with plans to improve my working conditions, and plans to write more, you can expect more fun stories on these pages. The next one, as a matter of fact, will be the story of two very confused bytes.

Pete Corey (petecorey)

Timing Streams in Node.js January 11, 2020 12:00 AM

On a current client project, I was tasked with optimizing a very large, very slow, very CPU-bound stream-based pipeline. Before I even started to think about optimizing this pipeline, I needed an objective way to measure the execution time of each step of the pipeline.

Imagine the pipeline in question looks something like this:


pipeline(
    httpStream,
    decodeStream,
    parseStream,
    batchStream,
    processStream
);

We’re reading in a stream of JSON-encoded events (httpStream), making sure they’re appropriately decoded (decodeStream), JSON parsing each incoming event (parseStream), batching events together (batchStream), and finally processing each batch of events (processStream).

Ideally I’d like to measure any or all of these individual steps.

However, many of these stream implementations are out of our hands. We can’t easily reach in and add timing code. Thankfully, we can easily write a function that decorates a provided stream with a simple runtime calculation.

Let’s call our decorator function time:


const time = (stream, name) => {
    return stream;
};

Our time function accepts and returns the stream we’ll be decorating, along with a name that describes the provided stream. It should be noted that it’s assumed that stream implements the Readable interface.

What we’re trying to accomplish here is relatively simple. We want to measure the amount of time that elapses between data emission events on our stream. We can use console.time/console.timeEnd and an event listener to make short work of this task:


const time = (stream, name) => {
    let timing = false;
    stream.on('data', () => {
        if (timing) {
            console.timeEnd(name);
        }
        console.time(name);
        timing = true;
    });
    return stream;
};

Every time we receive a 'data' event on our stream, we log the duration since the last received 'data' event, and start a new timer. We’re using a timing flag to ensure that console.timeEnd isn’t called the first time we receive a 'data' event.

Notice that we’re also using the provided name as the label in our console.time/console.timeEnd calls. This keeps us from getting confused when we start measuring multiple stages of our pipeline.

This solution mostly works. Unfortunately, a data event isn’t fired when the stream starts processing its first chunk of data. This means that we’re missing a measurement for this first chunk of execution time. Thankfully, we can capture that missing metric by also listening for a 'resume' event, which is called when the stream starts processing its first chunk of data:


const time = (stream, name) => {
    stream.on('resume', () => {
        console.time(name);
    });
    stream.on('data', () => {
        console.timeEnd(name);
        console.time(name);
    });
    return stream;
};

Notice that we’re no longer concerned about wrapping our console.timeEnd call in a guard in our 'data' event listener. We know that the 'resume' event handler will always call console.time before we reach our 'data' event handler, so we have no need for the timing guard anymore.

We can use our time function by decorating any or all of the stages of our pipeline:


await pipeline(
    httpStream,
    decodeStream,
    parseStream,
    time(batchStream, 'batch'),
    time(processStream, 'process')
);

Now that our runtime durations are finding their way to the logs, we can either use them as-is, or take things a step further and aggregate them for more in-depth data analysis:

...
batch: 258.213ms
process: 512.493ms
batch: 239.112ms
process: 475.293ms
...

As a warning to the reader, I’ll be the first to admit that I’m no stream expert. That said, this utility function proved invaluable to me, so I thought I’d record what I learned and pass it along for posterity.

Stream on.

January 10, 2020

José Padilla (jpadilla)

Podcast: DjangoChat Ep. 45 January 10, 2020 05:14 PM

Back in November I was invited to DjangoChat to talk about my contributions to the Django ecosystem. We talked about how I got started contributing to Django REST Framework and other open source projects, my work on Blimp and FilePreviews, authentication, and my current role at Auth0.

Recording this was a lot of fun. Thanks to William Vincent and Carlton Gibson.

Jeremy Morgan (JeremyMorgan)

The Developer Tool You Can't Live Without January 10, 2020 12:00 AM

I’m introducing a text / code generation tool that you will fall in love with. If you’re a developer or someone who works with text or tabulated data you need this tool. It’s called Nimble Text and it’s awesome. Here’s how the developer of Nimble Text describes it: You can be more awesome at your job by keeping this tool always within reach. NimbleText is a text manipulation and code generation tool available online or as a free download.

January 09, 2020

Patrick Louis (venam)

Will Lebanon Collapse or Will it Survive January 09, 2020 10:00 PM

Cedrus libani

“Collapse”, the word that is on everyone’s lips in Lebanon. What does it mean, will Lebanon fall or survive, and what does the future have in store? “We can predict everything, except the future”, I hear someone say, but can we at least get some possibilities.

Primo, we have to define what is a societal collapse and how it’s related to an economic collapse.
The definitions are broad, a societal collapse can be about a simple change in leadership or governance, a whole change in cultural dynamics like merging with another society and forming an inter-regional structure, the disappearance of traditions and ways of living (aka anomie, you learned a new word today), a population scattering and leaving a geographical area, or the annihilation of a population. Even though we imagine a collapse as being sudden, it can still happen very slowly.

Some scholars, enamored with population studies and social Darwinism, posit that societal collapses are a normal response to population crisis. Their case is that in any society resources will eventually get depleted, be it due to overpopulation or other reasons. When it reaches such state, according to their studies on mammals, the response from the population will be to switch from a cooperation model and parental behavior to a model of competition, dominance, and violence. All together, leading to a societal collapse, which will balance the population and stabilise it again. The cycle oscillating back and forth. Game theorists would also endorse such ideas.
No wonder there’s so much aversion to social Darwinism and cold-hearted economists.

Therefore, an economic collapse is often correlated with a societal collapse, it could happen before or after.

Secondo, what can be the reasons for collapse.
According to Jared Diamond in his popular book “Collapse: How Societies Choose to Fail or Succeed”, there are five key reasons:

  • Environmental degradation (depletion of resources, including overpopulation)
  • Changes in the climate
  • Hostile neighbors
  • Weakened trading partners
  • Absence of cultural resource and attitude to tackle these issues

I can already sense the smirk on your face but before diving into those, let’s mention another position by Joseph Tainter.
Tainter puts four axioms that he says are a must to understand collapses.

  • Human societies are problem-solving organizations;
  • Sociopolitical systems require energy for their maintenance;
  • Increased complexity carries with it increased costs per capita; and
  • Investment in sociopolitical complexity as a problem-solving response reaches a point of declining marginal returns.

The smirk didn’t disappear, am I right?

Let’s see, do we have environmental degradation in Lebanon. Maybe we do:

Do we have a change in climate, maybe we do:

I don’t think I have to even quote articles about our hostile neighbors.

And what about our trading partners, did we loose any?

Most importantly, what about our cultural attitude, what Jared Diamond calls axiological flexibility: the ability of a society to change its cultural values.
Indeed, our sectarian and political divisions implant in us an unprecedented cultural stiffness. The list of things we are banning keeps getting longer.

So that was that, and I feel like I’ve skipped a lot of things..
Regarding Tainter’s view, what can we say, has Lebanese society become too complex for its own good?

How much energy in our system are we using just to maintain our sociopolitical complexity, I’d say quite a lot. And isn’t this a reiteration of our initial definition of collapse: resources getting scarce. Things are getting circular.
While it’s easy to scapegoat a single factor like a certain political group, it’s more about the whole culture that is to blame.

Still, despite all this pessimism, Lebanese strive for better lives. We can’t stop and we can’t let our hopes down because of some environmental determinism explanation. That would be inconsiderate of the humanity in people. A simple explanation can’t possibly explain events in the middle-east, right? What about Black Swan events, those unprovable major game changer events that are inappropriately rationalized later on, as the Lebanese author Nassim Taleb puts it.

Can we somehow grab some hope from somewhere, anything? What can we do to boost the economy and the society.

In my opinion, Lebanon, as a small country, has to take advantage of the intangible economy to thrive. So what is needed to create a boom in the intangible economy?

In the book “Capitalism Without Capital: The Rise of the Intangible Economy”, three societal criteria are a must for a small country to take advantage of the intangible economy.

  • Societal and external trust
  • Small power-distance
  • Societal openness to experience

Apart from this the country can create hubs by having:

  • Low renting price
  • High synergy places, places where it’s pleasant to meet others
  • Unambiguous laws that attract international companies

Oh well… You got that smirk back on your face I presume!

Although it may sound gloomy, I’ll end this article on a note from the sociologist Charles Fritz in his 1961 paper. Surprisingly, he asks “Why do large-scale disasters produce such mentally healthy conditions?” Bizarrely, a catastrophe such as a societal and economic collapse doesn’t necessarily result in social breakdown and trauma but may lead to greater cooperation between groups.

Maybe this is what we’re the spectator of with this Lebanese revolution.











Attributions:

  • ALBA-BALAMAND [CC BY-SA (https://creativecommons.org/licenses/by-sa/4.0)]

Nikola Plejić (nikola)

Book Review: "How to Read a Book" January 09, 2020 01:11 PM

I don't really review the books I read, but I decided to write a short blurb about the oft-recommended "How to Read a Book: The Classic Guide to Intelligent Reading" by Mortimer J. Adler and Charles Van Doren. I've originally posted it on Goodreads, but I'm publishing it here, too, for good measure:

There's a lot to not like about this book: the slightly hermetic style, the occasional sexist slur, the subtly condescending tone, its exclusive--and, grantedly, somewhat apologetic--orientation to the "Western" literary canon, and the fact that the "recommended reading list" includes a single non-male author.

Keeping in mind that it was written in the 1940s, and despite these non-negligible shortcomings, I still find the book thoroughly insightful and valuable for what it is: a manual for analytical and comparative reading of "difficult" books, for whatever definition of "difficult" the reader might choose. It's a deeply practical book, sometimes to a fault, and many of its takeaways might seem obvious. Yet, when outlined in a systematic and formal way, with plenty of examples and illustrations, I believe they give a good framework for approaching demanding literature.

Most importantly, the book forces you to think critically about the act of reading, and this might be its greatest contribution of all: it has certainly made me think about the way I approach books, and it has given me a few new tools to do so.

Gustaf Erikson (gerikson)

The Stars My Destination by Alfred Bester January 09, 2020 10:08 AM

For some reason I’ve not read this classic from 1956 before. I’m glad I did.

Although this follows the basic Count of Monte Cristo formula, it has enough SF concepts for many novels. The central conceit of personal teleportation implies disease spread, new forms of criminality, new forms of urban development, threat of inter-system war - all summarized in breezy paragraphs.

Bester has also thought about the implications for those who because of brain damage or injury cannot “jaunt” - rehabilitation, or degrading slave wage labor at workplaces where jaunting is impractical.

The view of women is from its time, but could be plausibly explained by a neo-Victorian reaction in the novel. The female characters are thinly drawn, but not devoid of agency.

Unrelenting Technology (myfreeweb)

Firefox content process sandboxing with Capsicum: it’s alive! January 09, 2020 01:37 AM

Firefox content process sandboxing with Capsicum: it’s alive! Work in progress, but I have WebGL and audio working :)