Planet Crustaceans

This is a Planet instance for lobste.rs community feeds. To add/update an entry or otherwise improve things, fork this repo.

July 21, 2019

Ponylang (SeanTAllen)

Last Week in Pony - July 21, 2019 July 21, 2019 02:48 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

July 20, 2019

Leo Tindall (LeoLambda)

Simple Elixir Functions July 20, 2019 07:03 PM

I’ve been playing around with Elixir, which is a pretty cool language - specifically, I’m reading Programming Elixir 1.6 which is free for anyone with a .edu email address from The Pragmatic Bookshelf. reverse/1 I enjoy functional programming, so of course the first two functions I made were list operations. First up is reverse(list) (written as reverse/1, meaning it takes one argument), which reverses a list. # reverse/1 # Reverse the given list.

Sevan Janiyan (sevan)

A week of pkgsrc #13 July 20, 2019 04:56 PM

With the lead up to the release of pkgsrc-2019Q2 I picked up the ball with the testing on OS X Tiger again. It takes about a month for a G4 Mac Mini to attempt a bulk build of the entire pkgsrc tree with compilers usually taking up most days without success. To reduce the turnaround …

Awn Umar (awn)

memory retention attacks July 20, 2019 12:00 AM

In my post on implementing an in-memory encryption scheme to protect sensitive information, I referenced a mitigation strategy called a Boojum. It is described by Bruce Schneier, Niels Ferguson and Tadayoshi Kohno in their book, Cryptography Engineering: Design Principles and Practical Applications.

A number of people asked me about the mechanisms of the attack and the scheme, so I am including the relevant parts here. It is an excellent resource and I recommend that you go and buy it.

page one page two page three

[32] Giovanni Di Crescenzo, Niels Ferguson, Russel Impagliazzo, and Markus Jakobsson. How to Forget a Secret. In Christoph Meinel and Sophie Tison, editors, STACS 99, volume 1563 of Lecture Notes in Computer Science, pages 500-509. Springer-Verlag, 1999.

[57] Peter Gutmann. Secure Deletion of Data from Magnetic and Solid-State Memory. In USENIX Security Symposium Proceedings, 1996. Available from http://www.cs.auckland.ac.nz/~pgut001/pubs/secure_del.html.

[59] J. Alex Halderman, Seth D. Schoen, Nadia Heninger, William Clarkson, William Paul, Joseph A. Calandrino, Ariel J. Feldman, Jacob Appelbaum, and Edward W. Felten. Lest We Remember: Cold Boot Attacks on Encryption Keys. In USENIX Security Symposium Proceedings, pages 45-60, 2008.

July 19, 2019

Jeremy Morgan (JeremyMorgan)

You Can Get the Source Code for Apollo 11 and Take a Course on It July 19, 2019 04:47 AM

In software development you'll hear the term "moon shot". If something is a "moon shot" it's something that's extraordinarily difficult, like landing on the moon. We say this about some app doing something cool, but what about the software that... landed us on the moon? What was the original "moon shot" all about? 

The Software That Put Us on the Moon

Margaret Hamilton

Meet Margaret Hamilton). She was the director of Software engineering at MIT Instrumentation Laboratory, which was contracted to build the onboard software for the Apollo Space Program. This is her standing with the stack of source code used to launch us to the moon. Today we complain when Visual Studio runs slow. 

If you're a developer of any kind, you owe a lot of thanks to her, since "Software Engineer" wasn't even a term before she came along. 

So imagine having to write software that had to work the first time it ran, and run perfectly, or people would die. Sounds like a lot of pressure right? Margaret and her team did it. They built it, and it worked. It succeeded. 

They made a piece of machinery land on the moon with a computer that had about .08 percent of the processing power of an iPhone. 

You can get a lot more background and details here.

The Source Code

A few years ago, Ron Burkey released the source code for Apollo 11. He put a ton of work into this. This is the source code for the Apollo Guidance Computer, or AGC. 

Not only can you download the source code, but he created a simulator for the AGC, at the Virtual AGC Page. You can dig deep into the systems and how they work, trust me it's a rabbit hole for geeks. Tons of awesome stuff here. There's even a kinder, gentler introduction to the AGC you can check out to get familiar.

The AGC source code is written in assembler, which is foreign to most of us. I've played around enough with x86 assembler to know it's not my calling, but perusing through a lot of this source code, you can piece together how some of this stuff works.  

Comanche and Luminary

If you dig into the code, you'll see it's divided into two parts, Comanche and Luminary. 

Comanche is the software for the Command Module and Luminary is the Lunar Module

The Command Module was the cone that contained the crew and vital equipment, and was the vessel returned back to earth. 

Apollo 11 Source Code

The Lunar Module well, it was the module that landed on the moon. 

Apollo 11 Source Code

It's very interesting to see how these systems interact. When you look through the source code you can see a lot of cool hints how everything works. 

The DSKY 

The DSKY was the user interface for the computer. You could enter commands through a calculator-like interface and it was connected directly to the AGC. The commands contained a verb and a noun for each command. 

Apollo 11 Source Code

If you dig deep enough into the source code you'll see a lot of references to the DSKY and commands related to it. It was a marvel of engineering for its time. Of course, there is a DSKY simulator if you want to play around. 

How Do I Know so Much About This? 

I may sound like a seasoned expert here, but I just took this free course on the code of the Apollo 11, then started digging in the code and researching stuff. 

Apollo 11 Source Code

Click Here to Take This Course for FREE

For the 50th anniversary of the Apollo 11 Moon Landing, Simon Allardice created this awesome course on Pluralsight exploring the AGC. It's a quick course, and it won't teach you how to write AGC specific assembler code, but it will give you a good idea of how things worked back then and how the whole thing fits together. After watching this you'll get a better understanding of the source code, and start looking for easter eggs. 

Conclusion

Ok if you're a nerd like me, you love stuff like this. Obviously this isn't some kind of actively developed code where you can do pull requests and adding features. Nobody will hire you because you put AGC knowledge on your resume. But it sure is fun to go through this code and figure out how things worked, and look at the comical comments in the code. It's fascinating and well worth the time to explore if you're genuinely curious about this kind of stuff.

You can always yell at me on Twitter if you'd like to discuss it further. 

Enjoy! 

Joe Nelson (begriffs)

History and effective use of Vim July 19, 2019 12:00 AM

This article is based on historical research and on simply reading the Vim user manual cover to cover. Hopefully these notes will help you (re?)discover core functionality of the editor, so you can abandon pre-packaged vimrc files and use plugins more thoughtfully.

physical books

physical books

To go beyond the topics in this blog post, I’d recommend getting a paper copy of the manual and a good pocket reference. I couldn’t find any hard copy of the official Vim manual, and ended up printing this PDF using printme1.com. The PDF is a printer-friendly version of the files $VIMRUNTIME/doc/usr_??.txt distributed with the editor. For a convenient list of commands, I’d recommend the vi and Vim Editors Pocket Reference.

Table of Contents

History

Birth of vi

Vi commands and features go back more than fifty years, starting with the QED editor. Here is the lineage:

  • 1966 : QED (“Quick EDitor”) in Berkeley Timesharing System
  • 1969 Jul: moon landing (just for reference)
  • 1969 Aug: QED -> ed at AT&T
  • 1976 Feb: ed -> em (“Editor for Mortals”) at Queen Mary College
  • 1976 : em -> ex (“EXtended”) at UC Berkeley
  • 1977 Oct: ex gets visual mode, vi

hard copy terminal

You can discover the similarities all the way between QED and ex by reading the QED manual and ex manual. Both editors use a similar grammar to specify and operate on line ranges.

Editors like QED, ed, and em were designed for hard-copy terminals, which are basically electric typewriters with a modem attached. Hard-copy terminals print system output on paper. Output could not be changed once printed, obviously, so the editing process consisted of user commands to update and manually print ranges of text.

video terminal

By 1976 video terminals such as the ADM-3A started to be available. The Ex editor added an “open mode” which allowed intraline editing on video terminals, and a visual mode for screen oriented editing on cursor-addressible terminals. The visual mode (activated with the command “vi”) kept an up-to-date view of part of the file on screen, while preserving an ex command line at the bottom of the screen. (Fun fact: the h,j,k,l keys on the ADM-3A had arrows drawn on them, so that choice of motion keys in vi was simply to match the keyboard.)

Learn more about the journey from ed to ex/vi in this interview with Bill Joy. He talks about how he made ex/vi, and some things that disappointed him about it.

Classic vi is truly just an alter-ego of ex – they are the same binary, which decides to start in ex mode or vi mode based on the name of the executable invoked. The legacy of all this history is that ex/vi is refined by use, requires scant system resources, and can operate under limited bandwidth communication. It is also available on most systems and fully specified in POSIX.

From vi to vim

Being a derivative of ed, the ex/vi editor was intellectual property of AT&T. To use vi on platforms other than Unix, people had to write clones that did not share in the original codebase.

Some of the clones:

  • nvi - 1980 for 4BSD
  • calvin - 1987 for DOS
  • vile - 1990 for DOS
  • stevie - 1987 for Atari ST
  • elvis - 1990 for Minix and 386BSD
  • vim - 1991 for Amiga
  • viper - 1995 for Emacs
  • elwin - 1995 for Windows
  • lemmy - 2002 for Windows

We’ll be focusing on that little one in the middle: vim. Bram Moolenaar wanted to use vi on the Amiga. He began porting Stevie from the Atari and evolving it. He called his port “Vi IMitation.” For a full first-hand account, see Bram’s interview with Free Software Magazine.

By version 1.22 Vim was rechristened “Vi IMproved,” matching and surpassing features of the original. Here is the timeline of the next major versions, with some of their big features:

1991 Nov 2 Vim 1.14: First release (on Fred Fish disk #591).
1992 Vim 1.22: Port to Unix. Vim now competes with Vi.
1994 Aug 12 Vim 3.0: Support for multiple buffers and windows.
1996 May 29 Vim 4.0: Graphical User Interface (largely by Robert Webb).
1998 Feb 19 Vim 5.0: Syntax coloring/highlighting.
2001 Sep 26 Vim 6.0: Folding, plugins, vertical split.
2006 May 8 Vim 7.0: Spell check, omni completion, undo branches, tabs.
2016 Sep 12 Vim 8.0: Jobs, async I/O, native packages.

For more info about each version, see e.g. :help vim8. To see plans for the future, including known bugs, see :help todo.txt.

Version 8 included some async job support due to peer pressure from NeoVim, whose developers wanted to run debuggers and REPLs for their web scripting languages inside the editor.

Vim is super portable. By adapting over time to work on a wide variety of platforms, the editor was forced to keep portable coding habits. It runs on OS/390, Amiga, BeOS and BeBox, Macintosh classic, Atari MiNT, MS-DOS, OS/2, QNX, RISC-OS, BSD, Linux, OS X, VMS, and MS-Windows. You can rely on Vim being there no matter what computer you’re using.

In a final twist in the vi saga, the original ex/vi source code was finally released in 2002 under a BSD free software license. It is available at ex-vi.sourceforge.net.

Let’s get down to business. Before getting to odds, ends, and intermediate tricks, it helps to understand how Vim organizes and reads its configuration files.

Configuration hierarchy

I used to think, incorrectly, that Vim reads all its settings and scripts from the ~/.vimrc file alone. Browsing random “dotfiles” repositories can reinforce this notion. Quite often people publish monstrous single .vimrc files that try to control every aspect of the editor. These big configs are sometimes called “vim distros.”

In reality Vim has a tidy structure, where .vimrc is just one of several inputs. In fact you can ask Vim exactly which scripts it has loaded. Try this: edit a source file from a random programming project on your computer. Once loaded, run

:scriptnames

Take time to read the list. Try to guess what the scripts might do, and note the directories where they live.

Was the list longer than you expected? If you have installed loads of plugins the editor has a lot to do. Check what slows down the editor most at startup by running the following and look at the start.log it creates:

vim --startuptime start.log name-of-your-file

Just for comparison, see how quickly Vim starts without your existing configuration:

vim --clean --startuptime clean.log name-of-your-file

To determine which scripts to run at startup or buffer load time, Vim traverses a “runtime path.” The path is a comma-separated list of directories that each contain a common structure. Vim inspects the structure of each directory to find scripts to run. Directories are processed in the order they appear in the list.

Check the runtimepath on your system by running:

:set runtimepath

My system contains the following directories in the default value for runtimepath. Not all of them even exist in the filesystem, but they would be consulted if they did.

~/.vim
The home directory, for personal preferences.
/usr/local/share/vim/vimfiles
A system-wide Vim directory, for preferences from the system administrator.
/usr/local/share/vim/vim81
Aka $VIMRUNTIME, for files distributed with Vim.
/usr/local/share/vim/vimfiles/after
The “after” directory in the system-wide Vim directory. This is for the system administrator to overrule or add to the distributed defaults.
~/.vim/after
The “after” directory in the home directory. This is for personal preferences to overrule or add to the distributed defaults or system-wide settings.

Because directories are processed by their order in line, the only thing that is special about the “after” directories is that they are at the end of the list. There is nothing magical about the word “after.”

When processing each directory, Vim looks for subfolders with specific names. To learn more about them, see :help runtimepath. Here is a selection of those we will be covering, with brief descriptions.

plugin/
Vim script files that are loaded automatically when editing any kind of file. Called “global plugins.”
autoload/
(Not to be confused with “plugin.”) Scripts in autoload contain functions that are loaded only when requested by other scripts.
ftdetect/
Scripts to detect filetypes. They can base their decision on filename extension, location, or internal file contents.
ftplugin/
Scripts that are executed when editing files with known type.
compiler/
Definitions of how to run various compilers or linters, and of how to parse their output. Can be shared between multiple ftplugins. Also not applied automatically, must be called with :compiler
pack/
Container for Vim 8 native packages, the successor to “Pathogen” style package management. The native packaging system does not require any third-party code.

Finally, ~/.vimrc is the catchall for general editor settings. Use it for setting defaults that can be overridden for particular file types. For a comprehensive overview of settings you can choose in .vimrc, run :options.

Third-party plugins

Plugins are simply Vim scripts that must be put into the correct places in the runtimepath in order to execute. Installing them is conceptually easy: download the file(s) into place. The challenge is that it’s hard to remove or update some plugins because they litter subdirectories in the runtimepath with their scripts, and it can be hard to tell which plugin is responsible for which files.

“Plugin managers” evolved to address this need. Vim.org has had a plugin registry going back at least as far as 2003 (as identified by the Internet Archive). However it wasn’t until about 2008 that the notion of a plugin manager really came into vogue.

These tools add plugins’ separate directories to Vim’s runtimepath, and compile help tags for plugin documentation. Most plugin managers also install and update plugin code from the internet, sometimes in parallel or with colorful progress bars.

In chronological order, here is the parade of plugin managers. I based the date ranges on earliest and latest releases of each, or when no official releases are identified, on the earliest and latest commit dates.

  • Mar 2006 - Jul 2014 : Vimball (A distribution format and associated Vim commands)
  • Oct 2008 - Dec 2015 : Pathogen (Deprecated in favor of native vim packages)
  • Aug 2009 - Dec 2009 : Vimana
  • Dec 2009 - Dec 2014 : VAM
  • Aug 2010 - Nov 2010 : Jolt
  • Oct 2010 - Nov 2012 : tplugin
  • Oct 2010 - Feb 2014 : Vundle (Discontinued after NeoBundle ripped off code)
  • Mar 2012 - Mar 2018 : vim-flavor
  • Apr 2012 - Mar 2016 : NeoBundle (Deprecated in favor of dein)
  • Jan 2013 - Aug 2017 : infect
  • Feb 2013 - Aug 2016 : vimogen
  • Oct 2013 - Jan 2015 : vim-unbundle
  • Dec 2013 - Jul 2015 : Vizardry
  • Feb 2014 - Oct 2018 : vim-plug
  • Jan 2015 - Oct 2015 : enabler
  • Aug 2015 - Apr 2016 : Vizardry 2
  • Jan 2016 - Jun 2018 : dein.vim
  • Sep 2016 - Present : native in Vim 8
  • Feb 2017 - Sep 2018 : minpac
  • Mar 2018 - Mar 2018 : autopac
  • Feb 2017 - Jun 2018 : pack
  • Mar 2017 - Sep 2017 : vim-pck
  • Sep 2017 - Sep 2017 : vim8-pack
  • Sep 2017 - May 2019 : volt
  • Sep 2018 - Feb 2019 : vim-packager
  • Feb 2019 - Feb 2019 : plugpac.vim

The first thing to note is the overwhelming variety of these tools, and the second is that each is typically active for about four years before presumably going out of fashion.

The most stable way to manage plugins is to simply use Vim 8’s built-in functionality, which requires no third-party code. Let’s walk through how to do it.

First create two directories, opt and start, within a pack directory in your runtimepath.

mkdir -p ~/.vim/pack/foobar/{opt,start}

Note the placeholder “foobar.” This name is entirely up to you. It classifies the packages that will go inside. Most people throw all their plugins into a single nondescript category, which is fine. Pick whatever name you like; I’ll continue to use foobar here. You could theoretically create multiple categories too, like ~/.vim/pack/navigation and ~/.vim/pack/linting. Note that Vim does not detect duplication between categories and will double-load duplicates if they exist.

Packages in “start” get loaded automatically, whereas those in “opt” won’t load until specifically requested in Vim with the :packadd command. Opt is good for lesser-used packages, and keeps Vim fast by not running scripts unnecessarily. Note that there isn’t a counterpart to :packadd to unload a package.

For this example we’ll add the “ctrlp” fuzzy find plugin to opt. Download and extract the latest release into place:

curl -L https://github.com/kien/ctrlp.vim/archive/1.79.tar.gz \
	| tar zx -C ~/.vim/pack/foobar/opt

That command creates a ~/.vim/pack/foobar/opt/ctrlp.vim-1.79 folder, and the package is ready to use. Back in vim, create a helptags index for the new package:

:helptags ~/.vim/pack/foobar/opt/ctrlp.vim-1.79/doc

That creates a file called “tags” in the package’s doc folder, which makes the topics available for browsing in Vim’s internal help system. (Alternately you can run :helptags ALL once the package has been loaded, which takes care of all docs in the runtimepath.)

When you want to use the package, load it (and know that tab completion works for plugin names, so you don’t have to type the whole name):

:packadd ctrlp.vim-1.79

Packadd includes the package’s base directory in the runtimepath, and sources its plugin and ftdetect scripts. After loading ctrlp, you can press CTRL-P to pop up a fuzzy find file matcher.

Some people keep their ~/.vim directory under version control and use git submodules for each package. For my part, I simply extract packages from tarballs and track them in my own repository. If you use mature packages you don’t need to upgrade them often, plus the scripts are generally small and don’t clutter git history much.

Backups and undo

Depending on user settings, Vim can protect against four types of loss:

  1. A crash during editing (between saves). Vim can protect against this one by periodically saving unwritten changes to a swap file.
  2. Editing the same file with two instances of Vim, overwriting changes from one or both instances. Swap files protect against this too.
  3. A crash during the save process itself, after the destination file is truncated but before the new contents have been fully written. Vim can protect against this with a “writebackup.” To do this, it writes to a new file and swaps it with the original on success, in a way that depends on the “backupcopy” setting.
  4. Saving new file contents but wanting the original back. Vim can protect against this by persisting the backup copy of the file after writing changes.

Before examining sensible settings, how about some comic relief? Here are just a sampling of comments from vimrc files on GitHub:

  • “Do not create swap file. Manage this in version control”
  • “Backups are for pussies. Use version control”
  • “use version control FFS!”
  • “We live in a world with version control, so get rid of swaps and backups”
  • “don’t write backup files, version control is enough backup”
  • “I’ve never actually used the VIM backup files… Use version control”
  • “Since most stuff is on version control anyway”
  • “Disable backup files, you are using a version control system anyway :)”
  • “version control has arrived, git will save us”
  • “disable swap and backup files (Always use version control! ALWAYS!)”
  • “Turn backup off, since I version control everything”

The comments reflect awareness of only the fourth case above (and the third by accident), whereas the authors generally go on to disable the swap file too, leaving one and two unprotected.

Here is the configuration I recommend to keep your edits safe:

" Protect changes between writes. Default values of
" updatecount (200 keystrokes) and updatetime
" (4 seconds) are fine
set swapfile
set directory^=~/.vim/swap//

" protect against crash-during-write
set writebackup
" but do not persist backup after successful write
set nobackup
" use rename-and-write-new method whenever safe
set backupcopy=auto
" patch required to honor double slash at end
if has("patch-8.1.0251")
	" consolidate the writebackups -- not a big
	" deal either way, since they usually get deleted
	set backupdir^=~/.vim/backup//
end

" persist the undo tree for each file
set undofile
set undodir^=~/.vim/undo//

These settings enable backups for writes-in-progress, but do not persist them after successful write because version control etc etc. Note that you’ll need to mkdir ~/.vim/{swap,undodir,backup} or else Vim will fall back to the next available folder in the preference list. You should also probably chmod the folders to keep the contents private, because the swap files and undo history might contain sensitive information.

One thing to note about the paths in our config is that they end in a double slash. That ending enables a feature to disambiguate swaps and backups for files with the same name that live in different directories. For instance the swap file for /foo/bar will be saved in ~/.vim/swap/%foo%bar.swp (slashes escaped as percent signs). Vim had a bug until a fairly recent patch where the double slash was not honored for backupdir, and we guard against that above.

We also have Vim persist the history of undos for each file, so that you can apply them even after quitting and editing the file again. While it may sound redundant with the swap file, the undo history is complementary because it is written only when the file is written. (If it were written more frequently it might not match the state of the file on disk after a crash, so Vim doesn’t do that.)

Speaking of undo, Vim maintains a full tree of edit history. This means you can make a change, undo it, then redo it differently and all three states are recoverable. You can see the times and magnitude of changes with the :undolist command, but it’s hard to visualize the tree structure from it. You can navigate to specific changes in that list, or move in time with :earlier and :later which take a time argument like 5m, or the count of file saves, like 3f. However navigating the undo tree is an instance when I think a plugin – like undotreeis warranted.

Enabling these disaster recovery settings can bring you peace of mind. I used to save compulsively after most edits or when stepping away from the computer, but now I’ve made an effort to leave documents unsaved for hours at a time. I know how the swap file works now.

Some final notes: keep an eye on all these disaster recovery files, they can pile up in your .vim folder and use space over time. Also setting nowritebackup might be necessary when saving a huge file with low disk space, because Vim must otherwise make an entire copy of the file temporarily. By default the “backupskip” setting disables backups for anything in the system temp directory.

Vim’s “patchmode” is related to backups. You can use it in directories that aren’t under version control. For instance if you want to download a source tarball, make an edit and send a patch over a mailing list without bringing git into the picture. Run :set patchmod=.orig and any file ‘foo’ Vim is about to write will be backed up to ‘foo.orig’. You can then create a patch on the command line between the .orig files and the new ones.

Include and path

Most programming languages allow you to include one module or file from another. Vim knows how to track program identifiers in included files using the configuration settings path, include, suffixesadd, and includeexpr. The identifier search (see :help include-search) is an alternative to maintaining a tags file with ctags for system headers.

The settings for C programs work out of the box. Other languages are supported too, but require tweaking. That’s outside the scope of this article, see :help include.

If everything is configured right, you can press [i on an identifier to display its definition, or [d for a macro constant. Also when you press gf with the cursor on a filename, Vim searches the path to find it and jump there. Because the path also affects the :find command, some people have the tendency to add ‘**/*’ or commonly accessed directories to the path in order to use :find like a poor man’s fuzzy finder. Doing this slows down the identifier search with directories which aren’t relevant to that task.

A way to get the same level of crappy find capability, without polluting the path, is to just make another mapping. You can then press <Leader><space> (which is typically backslash space) then start typing a filename and use tab or CTRL-D completion to find the file.

" fuzzy-find lite
nmap <Leader><space> :e ./**/

Just to reiterate: the path parameter was designed for header files. If you want more proof, there is even a :checkpath command to see whether the path is functioning. Load a C file and run :checkpath. It will display filenames it was unable to find that are included transitively by the current file. Also :checkpath! with a bang dumps the whole hierarchy of files included from the current file.

By default path has the value “.,/usr/include,,” meaning the working directory, /usr/include, and files that are siblings of the active buffer. The directory specifiers and globs are pretty powerful, see :help file-searching for the details.

In my C ftplugin (more on that later), I also have the path search for include files within the current project, like ./src/include or ./include .

setlocal path=.,,*/include/**3,./*/include/**3
setlocal path+=/usr/include

The ** with a number like **3 bounds the depth of the search in subdirectories. It’s wise to add depth bounds where you can to avoid identifier searches that lock up.

Here are other patterns you might consider adding to your path if :checkpath identifies that files can’t be found in your project. It depends on your system of course.

  • More system includes: /usr/include/**4,/usr/local/include/**3
  • Homebrew library headers: /usr/local/Cellar/**2/include/**2
  • Macports library headers: /opt/local/include/**
  • OpenBSD library headers: /usr/local/lib/\*/include,/usr/X11R6/include/\*\*3

See also: :he [, :he gf, :he :find.

Edit ⇄ compile cycle

The :make command runs a program of the user’s choice to build a project, and collects the output in the quickfix buffer. Each item in the quickfix records the filename, line, column, type (warning/error) and message of each output item. A fairly idomatic mapping uses bracket commands to move through quickfix items:

" quickfix shortcuts
nmap ]q :cnext<cr>
nmap ]Q :clast<cr>
nmap [q :cprev<cr>
nmap [Q :cfirst<cr>

If, after updating the program and rebuilding, you are curious what the error messages said last time, use :colder (and :cnewer to return). To see more information about the currently selected error use :cc, and use :copen to see the full quickfix buffer. You can populate the quickfix yourself without running :make with :cfile, :caddfile, or :cexpr.

Vim parses output from the build process according to the errorformat string, which contains scanf-like escape sequences. It’s typical to set this in a “compiler file.” For instance, Vim ships with one for gcc in $VIMRUNTIME/compiler/gcc.vim, but has no compiler file for clang. I created the following definition for ~/.vim/compiler/clang.vim:

" formatting variations documented at
" https://clang.llvm.org/docs/UsersManual.html#formatting-of-diagnostics
"
" It should be possible to make this work for the combination of
" -fno-show-column and -fcaret-diagnostics as well with multiline
" and %p, but I was too lazy to figure it out.
"
" The %D and %X patterns are not clang per se. They capture the
" directory change messages from (GNU) 'make -w'. I needed this
" for building a project which used recursive Makefiles.

CompilerSet errorformat=
	\%f:%l%c:{%*[^}]}{%*[^}]}:\ %trror:\ %m,
	\%f:%l%c:{%*[^}]}{%*[^}]}:\ %tarning:\ %m,
	\%f:%l:%c:\ %trror:\ %m,
	\%f:%l:%c:\ %tarning:\ %m,
	\%f(%l,%c)\ :\ %trror:\ %m,
	\%f(%l,%c)\ :\ %tarning:\ %m,
	\%f\ +%l%c:\ %trror:\ %m,
	\%f\ +%l%c:\ %tarning:\ %m,
	\%f:%l:\ %trror:\ %m,
	\%f:%l:\ %tarning:\ %m,
	\%D%*\\a[%*\\d]:\ Entering\ directory\ %*[`']%f',
	\%D%*\\a:\ Entering\ directory\ %*[`']%f',
	\%X%*\\a[%*\\d]:\ Leaving\ directory\ %*[`']%f',
	\%X%*\\a:\ Leaving\ directory\ %*[`']%f',
	\%DMaking\ %*\\a\ in\ %f

CompilerSet makeprg=make

To activate this compiler profile, run :compiler clang. This is typically done in an ftplugin file.

Another example is running GNU Diction on a text document to identify wordy and commonly misused phrases in sentences. Create a “compiler” called diction.vim:

CompilerSet errorformat=%f:%l:\ %m
CompilerSet makeprg=diction\ -s\ %

After you run :compiler diction you can use the normal :make command to run it and populate the quickfix. The final mild convenience in my .vimrc is a mapping to run make:

" real make
map <silent> <F5> :make<cr><cr><cr>
" GNUism, for building recursively
map <silent> <s-F5> :make -w<cr><cr><cr>

Diffs and patches

Vim’s internal diffing is powerful, but it can be daunting, especially the three-way merge view. In reality it’s not so bad once you take time to study it. The main idea is that every window is either in or out of “diff mode.” All windows put in diffmode (with :difft[his]) get compared with all other windows already in diff mode.

For example, let’s start simple. Create two files:

echo "hello, world" > h1
echo "goodbye, world" > h2

vim h1 h2

In vim, split the arguments into their own windows with :all. In the top window, for h1, run :difft. You’ll see a gutter appear, but no difference detected. Move to the other window with CTWL-W CTRL-W and run :difft again. Now hello and goobye are identified as different in the current chunk. Continuing in the bottom window, you can run :diffg[et] to get “hello” from the top window, or :diffp[ut] to send “goodbye” into the top window. Pressing ]c or [c would move between chunks if there were more than one.

A shortcut would be running vim -d h1 h2 instead (or its alias, vimdiff h1 h2) which applies :difft to all windows. Alternatively, load just h1 with vim h1 and then :diffsplit h2. Remember that fundamentally these commands just load files into windows and set the diff mode.

With these basics in mind, let’s learn to use Vim as a three-way mergetool for git. First configure git:

git config merge.tool vimdiff
git config merge.conflictstyle diff3
git config mergetool.prompt false

Now, when you hit a merge conflict, run git mergetool. It will bring Vim up with four windows. This part looks scary, and is where I used to flail around and often quit in frustration.

+-----------+------------+------------+
|           |            |            |
|           |            |            |
|   LOCAL   |    BASE    |   REMOTE   |
+-----------+------------+------------+
|                                     |
|                                     |
|             (edit me)               |
+-------------------------------------+

Here’s the trick: do all the editing in the bottom window. The top three windows simply provide context about how the file differs on either side of the merge (local / remote), and how it looked prior to either side doing any work (base).

Move within the bottom window with ]c, and for each chunk choose whether to replace it with text from local, base, or remote – or whether to write in your own change which might combine parts from several.

To make it easier to pull changes from the top windows, I set some mappings in my vimrc:

" shortcuts for 3-way merge
map <Leader>1 :diffget LOCAL<CR>
map <Leader>2 :diffget BASE<CR>
map <Leader>3 :diffget REMOTE<CR>

We’ve already seen :diffget, and here our bindings pass an argument of the buffer name that identifies which window to pull from.

Once done with the merge, run :wqa to save all the windows and quit. If you want to abandon the merge instead, run :cq to abort all changes and return an error code to the shell. This will signal to git that it should ignore your changes.

Diffget can also accept a range. If you want to pull in all changes from one of the top windows rather than working chunk by chunk, just run :1,$+1diffget {LOCAL,BASE,REMOTE}. The “+1” is required because there can be deleted lines “below” the last line of a buffer.

The three-way marge is fairly easy after all. There’s no need for plugins like Fugitive, at least for presenting a simplified view for resolving merge conflicts.

Finally, as of patch 8.1.0360, Vim is bundled with the xdiff library and can create diffs internally. This can be more efficient than shelling out to an external program, and allows for a choice of diff algorithms. The “patience” algorithm often produces more human-readable output than the default, “myers.” Set it in your .vimrc like so:

if has("patch-8.1.0360")
	set diffopt+=internal,algorithm:patience
endif

Buffer I/O

See if this sounds familiar: you’re editing a buffer and want to save it as a new file, so you :w newname. After editing some more, you :w, but it writes over the original file. What you want for this scenario is :saveas newname, which does the write but also changes the filename of the buffer for future writes. Alternately, the :file newname command will change the filename without doing a write.

It also pays off to learn more about the read and write commands. Becuase r and w are Ex commands, they work with ranges. Here are some variations you might not know about:

:w >>foo append the whole buffer to a file
:.w >>foo append current line to a file
:$r foo read foo into the end of the buffer
:0r foo read foo into the start, moving existing lines down
:.,$w foo write current line and below to a file
:r !ls read ls output into cursor position
:w !wc send buffer to wc and display output
:.!tr ‘A-Za-z’ ‘N-ZA-Mn-za-m’ apply ROT-13 to current line
:w|so % chain commands: write and then source buffer
:e! throw away unsaved changes, reload buffer
:hide edit foo edit foo, hide current buffer if dirty

Useless fun fact: we piped a line to tr in an example above to apply a ROT-13 cypher, but Vim has that functionality built in with the the g? command. Apply it to a motion, like g?$.

Filetypes

Filetypes are a way to change settings based on the type of file detected in a buffer. They don’t need to be automatically detected though, we can manually enable them to interesting effect. An example is doing hex editing. Any file can be viewed as raw hexadecimal values. GitHub user the9ball created a clever ftplugin script that filters a buffer back and forth through the xxd utility for hex editing.

The xxd utility was bundled as part of Vim 5 for convenience. The Vim todo.txt file mentions they want to make it more seamless to edit binary files, but xxd can take us pretty far.

Here is code you can put in ~/.vim/ftplugin/xxd.vim. Its presence in ftplugin means Vim will execute the script when filetype (aka “ft”) becomes xxd. I added some basic comments to the script.

" without the xxd command this is all pointless
if !executable('xxd')
	finish
endif

" don't insert a newline in the final line if it
" doesn't already exist, and don't insert linebreaks
setlocal binary noendofline
silent %!xxd -g 1
%s/\r$//e

" put the autocmds into a group for easy removal later
augroup ftplugin-xxd
	" erase any existing autocmds on buffer
	autocmd! * <buffer>

	" before writing, translate back to binary
	autocmd BufWritePre <buffer> let b:xxd_cursor = getpos('.')
	autocmd BufWritePre <buffer> silent %!xxd -r

	" after writing, restore hex view and mark unmodified
	autocmd BufWritePost <buffer> silent %!xxd -g 1
	autocmd BufWritePost <buffer> %s/\r$//e
	autocmd BufWritePost <buffer> setlocal nomodified
	autocmd BufWritePost <buffer> call setpos('.', b:xxd_cursor) | unlet b:xxd_cursor

	" update text column after changing hex values
	autocmd TextChanged,InsertLeave <buffer> let b:xxd_cursor = getpos('.')
	autocmd TextChanged,InsertLeave <buffer> silent %!xxd -r
	autocmd TextChanged,InsertLeave <buffer> silent %!xxd -g 1
	autocmd TextChanged,InsertLeave <buffer> call setpos('.', b:xxd_cursor) | unlet b:xxd_cursor
augroup END

" when filetype is set to no longer be "xxd," put the binary
" and endofline settings back to what they were before, remove
" the autocmds, and replace buffer with its binary value
let b:undo_ftplugin = 'setl bin< eol< | execute "au! ftplugin-xxd * <buffer>" | execute "silent %!xxd -r"'

Try opening a file, then running :set ft. Note what type it is. Then:set ft=xxd. Vim will turn into a hex editor. To restore your view, :set ft=foo where foo was the original type. Note that in hex view you even get syntax highlighting because $VIMRUNTIME/syntax/xxd.vim ships with Vim by default.

Notice the nice use of “b:undo_ftplugin” which is an opportunity for filetypes to clean up after themselves when the user or ftdetect mechanism switches away from them to another filetype. (The example above could use a little work because if you :set ft=xxd then set it back, the buffer is marked as modified even if you never changed anything.)

Ftplugins also allow you to refine an existing filetype. For instance, Vim already has some good defaults for C programming in $VIMRUNTIME/ftplugin/c.vim. I put these extra options in ~/.vim/after/ftplugin/c.vim to add my own settings on top:

" the smartest indent engine for C
setlocal cindent
" my preferred "Allman" style indentation
setlocal cino="Ls,:0,l1,t0,(s,U1,W4"

" for quickfix errorformat
compiler clang
" shows long build messages better
setlocal ch=2

" auto-create folds per grammar
setlocal foldmethod=syntax
setlocal foldlevel=10

" local project headers
setlocal path=.,,*/include/**3,./*/include/**3
" basic system headers
setlocal path+=/usr/include

setlocal tags=./tags,tags;~
"                      ^ in working dir, or parents
"                ^ sibling of open file

" the default is menu,preview but the preview window is annoying
setlocal completeopt=menu

iabbrev #i #include
iabbrev #d #define
iabbrev main() int main(int argc, char **argv)

" add #include guard
iabbrev #g _<c-r>=expand("%:t:r")<cr><esc>VgUV:s/[^A-Z]/_/g<cr>A_H<esc>yypki#ifndef <esc>j0i#define <esc>o<cr><cr>#endif<esc>2ki

Notice how the script uses “setlocal” rather than “set.” This applies the changes to just the current buffer rather than the whole Vim instance.

This script also enables some light abbreviations. Like I can type #g and press enter and it adds an include guard with the current filename:

#ifndef _FILENAME_H
#define _FILENAME_H

/* <-- cursor here */

#endif

You can also mix filetypes by using a dot (“.”). Here is one application. Different projects have different coding conventions, so you can combine your default C settings with those for a particular project. The OpenBSD source code follows the style(9) format, so let’s make a special openbsd filetype. Combine the two filetypes with :set ft=c.openbsd on relevant files.

To detect the openbsd filetype we can look at the contents of buffers rather than just their extensions or locations on disk. The telltale sign is that C files in the OpenBSD source contain /* $OpenBSD: in the first line.

To detect them, create ~/.vim/after/ftdetect/openbsd.vim:

augroup filetypedetect
        au BufRead,BufNewFile *.[ch]
                \  if getline(1) =~ 'OpenBSD;'
                \|   setl ft=c.openbsd
                \| endif
augroup END

The Vim port for OpenBSD already includes a special syntax file for this filetype: /usr/local/share/vim/vimfiles/syntax/openbsd.vim. If you recall, the /usr/local/share/vim/vimfiles directory is in the runtimepath and is set aside for files from the system administrator. The provided openbsd.vim script includes a function:

function! OpenBSD_Style()
	setlocal cindent
	setlocal cinoptions=(4200,u4200,+0.5s,*500,:0,t0,U4200
	setlocal indentexpr=IgnoreParenIndent()
	setlocal indentkeys=0{,0},0),:,0#,!^F,o,O,e
	setlocal noexpandtab
	setlocal shiftwidth=8
	setlocal tabstop=8
	setlocal textwidth=80
endfun

We simply need to call the function at the appropriate time. Create ~/.vim/after/ftplugin/openbsd.vim:

call OpenBSD_Style()

Now opening any C or header file with the characteristic comment at the top will be recognized as type c.openbsd and will use indenting options that conform with the style(9) man page.

Don’t forget the mouse

This is a friendly reminder that despite our command-line machismo, the mouse is in fact supported in Vim, and can do some things more easily than the keyboard. Mouse events work even over SSH thanks to xterm turning mouse events into stdin escape codes.

To enable mouse support, set mouse=n. Many people use mouse=a to make it work in all modes, but I prefer to enable it only in normal mode. This avoids creating visual selections when I click links with a keyboard modifier to open them in my browser.

Here are things the mouse can do:

  • Open or close folds (when foldcolumn > 0).
  • Select tabs (beats gt gt gt…)
  • Click to complete a motion, like d<click!>. Similar to the easymotion plugin but without any plugin.
  • Jump to help topics with double click.
  • Drag the status line at the bottom to change cmdheight.
  • Drag edge of window to resize.
  • Scroll wheel.

Misc editing

This section could be enormous, but I’ll stick to a few tricks I learned. The first one that blew me away was :set virtualedit=all. It allows you to move the cursor anywhere in the window. If you enter characters or insert a visual block, Vim will add whatever spaces are required to the left of the inserted characters to keep them in place. Virtual edit mode makes it simple to edit tabular data. Turn it off with :set virtualedit=.

Next are some movement commands. I used to rely a lot on } to jump by paragraphs, and just muscle my way down the page. However the ] character makes more precise motions: by function ]], scope ]}, paren ‘])’, comment ]/, diff block ]c. This series is why the quickfix mapping ]q mentioned earlier fits the pattern so well.

For big jumps I used to try things like 1000j, but in normal mode you can actually just type a percentage and Vim will go there, like 50%. Speaking of scroll percentage, you can see it at any time with CTRL-G. Thus I now do :set noruler and ask to see the info as needed. It’s less cluttered. Kind of the opposite of the trend of colorful patched font powerlines.

After jumping around between tags, files, or within a file, there are some commands to get your bearings. Try :ls, :tags, :jumps, and :marks. Jumping through tags actually creates a stack, and you can press CTRL-T to pop one back. I used to always press CTRL-O to back out of jumps, but it is not as direct as popping the tag stack.

In a project directory that has been indexed with ctags, you can open the editor directly to a tag with -t, like vim -t main. To find tags files more flexibly, set the tags configuration variable. Note the semicolon in the example below that allows Vim to search the current directory upward to the home directory. This way you could have a more general system tags file outside the project folder.

set tags=./tags,**5/tags,tags;~
"                          ^ in working dir, or parents
"                   ^ in any subfolder of working dir
"           ^ sibling of open file

There are some buffer tricks too. Switching to a buffer with :bu can take a fragment of the buffer name, not just a number. Sometimes it’s harder to memorize those numbers than remember the name of a source file. You can navigate buffers with marks too. If you use a capital letter as the name of a mark, you can jump to it across buffers. You could set a mark H in a header, C in a source file, and M in a Makefile to go from one buffer to another.

Do you ever get mad after yanking a word, deleting a word somewhere else, trying paste the first word in, and then discovering your original yank is overwritten? The Vim registers are underappreciated for this. Inspect their contents with :reg. As you yank text, previous yanks are rotated into the registers "0 - "9. So "0p pastes the next-to-last yank/deletion. The special registers "+ and "* can copy/paste from/to the system clipboard. They usually mean the same thing, except in some X11 setups that distinguish primary and secondary selection.

Another handy hidden feature is the command line window. It it’s a buffer that contains your previous commands and searches. Bring it up with q: or q/. Once inside you can move to any line and press enter to run it. However you can also edit any of the lines before pressing enter. Your changes won’t affect the line (the new command will merely be added to the bottom of the list).

This article could go on and on, so I’m going to call it here. For more great topics, see these help sections: views-sessions, viminfo, TOhtml, ins-completion, cmdline-completion, multi-repeat, scroll-cursor, text-objects, grep, netrw-contents.

vim logo

vim logo

July 18, 2019

Awn Umar (awn)

encrypting secrets in memory July 18, 2019 12:00 AM

In a previous post I talked about designing and implementing an in-memory data structure for storing sensitive information in Go. The latest version of memguard adds something new: encryption.

But why? Well, there are limitations to the old solution of using guarded heap allocations for everything.

  1. A minimum of three memory pages have to be allocated for each value: two guard pages sandwiching n≥1 n \geq 1 n1 data pages.
  2. Some systems impose an upper limit on the amount of memory that an individual process is able to prevent from being swapped out to disk.

Memory layout of guarded heap allocation.

Typical layout of a 32 byte guarded heap allocation.

So it is worth looking into the use of encryption to protect information. After all, ciphertext does not have to be treated with much care and authentication guarantees immutability for free. The problem of recovering a secret is shifted to recovering the key that protects it.

But there is the obvious problem. Where and how do you store the encryption key?

We use a scheme described by Bruce Schneier in Cryptography Engineering. The procedure is sometimes referred to as a Boojum. I will formally define it below for convenience.

Define B={i:0≤i≤255} B = \{i : 0 \leq i \leq 255\} B={i:0i255} to be the set of values that a byte may take. Define h:Bn→B32 h : B^n \to B^{32} h:BnB32 for n∈N0 n \in \mathbb{N}_0 nN0 to be a cryptographically-secure hash function. Define the binary XOR operator ⊕:Bn×Bn→Bn \oplus : B^n \times B^n \to B^n :Bn×BnBn.

Suppose k∈B32 k \in B^{32} kB32 is the key we want to protect and Rn∈B32 R_n \in B^{32} RnB32 is some random bytes sourced from a suitable CSPRNG. We initialise the two partitions:

x1=R1y1=h(x1)⊕k \begin{aligned} x_1 &= R_1\\ y_1 &= h(x_1) \oplus k\\ \end{aligned} x1y1=R1=h(x1)k

storing each inside its own guarded heap allocation. Then every m m m milliseconds we overwrite each value with:

xn+1=xn⊕Rn+1=R1⊕R2⊕⋯⊕Rn+1yn+1=yn⊕h(xn)⊕h(xn+1)=h(xn)⊕h(xn)⊕h(xn+1)⊕k=h(xn+1)⊕k \begin{aligned} x_{n+1} &= x_n \oplus R_{n+1}\\ &= R_1 \oplus R_2 \oplus \cdots \oplus R_{n+1}\\ y_{n+1} &= y_n \oplus h(x_n) \oplus h(x_{n+1})\\ &= h(x_n) \oplus h(x_n) \oplus h(x_{n+1}) \oplus k\\ &= h(x_{n+1}) \oplus k\\ \end{aligned} xn+1yn+1=xnRn+1=R1R2Rn+1=ynh(xn)h(xn+1)=h(xn)h(xn)h(xn+1)k=h(xn+1)k

and so on. Then by the properties of XOR,

k=h(xn)⊕yn=h(xn)⊕h(xn)⊕k=0⊕k=k \begin{aligned} k &= h(x_n) \oplus y_n\\ &= h(x_n) \oplus h(x_n) \oplus k\\ &= 0 \oplus k\\ &= k\\ \end{aligned} k=h(xn)yn=h(xn)h(xn)k=0k=k

It is clear from this that our iterative overwriting steps do not affect the value of k k k, and the proof also gives us a way of retrieving k k k. My own implementation of the protocol in fairly idiomatic Go code is available here.

An issue with the Boojum scheme is that it has a relatively high overhead from two guarded allocations using six memory pages in total, and we have to compute and write 64 bytes every m m m milliseconds. However we only store a single global key, and the overhead can be tweaked by scaling m m m as needed. Its value at the time of writing is 8 milliseconds.

The authors of the Boojum claim that it defends against cold boot attacks, and I would speculate that there is also some defence against side-channel attacks due to the fact that k k k is split across two different locations in memory and each partition is constantly changing. Those attacks usually have an error rate and are relatively slow.

OpenBSD added a somewhat related mitigation to their SSH implementation that stores a 16 KiB (static) “pre-key” that is hashed to derive the final key when it is needed. I investigated incorporating it somehow but decided against it. Both schemes have a weak point when the key is in “unlocked” form so mimimising this window of opportunity is ideal.

In memguard the key is initialised when the program starts and then hangs around in the background—constantly flickering—until it is needed. When some data needs to be encrypted or decrypted, the key is unlocked and used for the operation and then it is destroyed.

Diagram showing the high-level structure of the scheme.

High-level overview of the encryption scheme.

The documentation provides a relatively intuitive guide to the package’s functionality. The Enclave stores ciphertext, the LockedBuffer stores plaintext, and core.Coffer implements the Boojum. Examples are available in the examples sub-package.

The most pressing issue at the moment is that the package relies on cryptographic primitives implemented by the Go standard library which does not secure its own memory and may leak values that are passed to it. There has been some discussion about this but for now it seems as though rewriting crucial security APIs to use specially allocated memory is the only feasible solution.

If you have any ideas and wish to contribute, please do get in touch or open a pull request.

Pepijn de Vos (pepijndevos)

VHDL to PCB July 18, 2019 12:00 AM

When learning to program FPGAs using VHDL or Verilog, you also learn that these hardware description languages can be used to design ASICs (application specific integrated circuit). But this is only something big corporations with millions of dollars can afford, right? Even though I later learned it only costs thousands, not millions, to make an ASIC on an older process, it is still far away from hobby budgets.

I had been keeping an eye on Yosys, the open source HDL synthesis tool, which can apparently do ASIC by giving it a liberty file that specifies the logic cells your foundry supports. Meanwhile I also toyed with the idea of making a 7400 series computer, and I wondered if you could write a liberty file for 7400 chips. I had kind of dismissed the idea, but then ZirconiumX came along and did it.

It suffices to say this revived my interest in the idea and a lively discussion and many pull requests followed. First some small changes, then simulations to verify the synthesized result is still correct, and finally a KiCad netlist generator.

You see, generating a Yosys netlist is nice, but eventually these 7400 chips have to end up on a PCB somehow. Normally you draw your schematic in Eeschema, generate a netlist, and import that to Pcbnew. But instead I used skidl to generate the netlist directly. Then all there is to do is add the inputs and outputs and run the autorouter (or do it manually of course).

I decided to do a proof-of-concept “application specific interconnected circuit”, with the goal of making something fun in under 10 chips. (a Risc-V CPU currently sits at about 850) I settled on a fading PWM circuit to drive an LED. I manually added a 555-based clock, and ordered a PCB for a few bucks. A few weeks later, this was the result. It worked on the first try! This feeling is even more amazing than with software, and shows that as long as there are no compiler/library bugs or DRC errors, logic simulations are a good way to prove your PCB design.

To follow along at home you need to install Yosys. A recent release might work, but it’s getting better every day, so building from source is recommended. Then you can just git clone 74xx-liberty and go. There are a number of Verilog programs in benchmarks in case you’d rather make PicoRV32 in 7400 chips.

cd stat
make pwmled.stat # synthesize and run stat
../ic_count.py pwmled.stat # count number of chips used
cd ../sim
make pwmled.vcd # synth to low-level verilog and simulate
gtkwave pwmled.vcd # show test bench results
cd ../kicad
make pwmled.net # generate kicad netlist

But this was all done in Verilog, so where is the VHDL, you might wonder. Well, Yosys does not really support VHDL yet, but Tristan Gingold is hard at work making GHDL synthesize VHDL as a Yosys plugin. I think this is very important work, so I’ve been contributing there as well. After some pull requests I was able to port the breathing LED to VHDL.

Getting VHDL to work in Yosys is a bit of effort. First you need to compile GHDL, which requires installing a recent version of GNAT. Then you need to install ghdlsynth-beta as a plugin, allowing you to run yosys -m ghdl. My fork of 74xx-liberty contains additional make rules for doing the above synthesization for VHDL files, which does something like this before calling the 7400 synthesis script.

cd stat
ghdl -a ../benchmarks/pwmled.vhd # analyse VHDL file
yosys -m ghdl -p "ghdl pwmled; show" # load pwmled entity, show graph

yosys dot graph

A huge thank you to all the people working tirelessly to make open source hardware design a reality. You’re awesome!

July 16, 2019

Jeremy Morgan (JeremyMorgan)

Forget What You've Heard, Now Is the Best Time to Become a Coder July 16, 2019 04:15 PM

Do you want to be a coder? Are you on the fence about trying it? Nervous to get started?

The time is now. Time to pull the trigger. 

There has never been a better time to become a coder.

And I'm going to tell you how to get started.  

My History as a Coder

I started writing code professionally in 2002. Before that, I was building (terrible) websites for myself and friends. I even ran a business for a few years crapping out HTML/Perl sites in the late 90s for good money. I built shoddy software in Visual Basic for businesses.

To learn how to do this, I bought expensive books on HTML, Perl, Visual Basic and Unix. I read them cover to cover and marked them up. Google didn’t exist yet. I consulted Usenet news groups where they insulted you for asking "stupid questions" and figured out most of the stuff the hard way. I learned Unix so I could host my customer's sites and destroyed a lot of stuff before figuring it out. 

I was living in a rural area and had very few friends that were even in to this kind of thing so I had few people to bounce ideas and questions off of. I hadn't yet gone to college and met other technical folks. 

I'm not telling you this so you'll think I'm awesome or some kind of grizzled veteran who is better than you. It was hard, and I wasn't very good for many years, but I still got the job done. I spent countless nights chugging Surge and hacking away trying to figure out things. I got good. I figured things out. But it sucked.

 

Your History

Your story doesn't have to go that way. The world is at your fingertips. You have all the information to get where I am right in front of you. Some grizzled veterans like myself say things like "new developers don't even have to work for it" and pretend like we're so much better because of our struggles. Not true. In reality because you have this information readily available, you'll get better faster. The developer I was at the 3 year mark will be a rookie compared to you.


Today's beginning developers are positioned to be the best generation of developers yet.

That's part of the reason that NOW is the best time to start. You have Google, Stack Overflow, Dev.to, Social Media, Pluralsight, YouTube, you name it. NOW is exponentially better than 1995 as far a starting point goes. 

The Market: We need developers! 

Another great reason now is the time: we are in a talent shortage. Everyone I talk to is looking for developers. Junior, Senior, mid-level, whatever. Can you code? You'll get a job. It doesn't even have to be in the language they're using. 

In the mid 2000s being a developer I had to know everything under the sun to get in the door to interview. They had crowds lined up fighting for every job. In 2019 if you learn the basics, throw some projects on GitHub and start sending out resumes you'll get that call. 

According to code.org, there are 504,199 open computing jobs nationwide. There were 63,744 computer science graduates entering the workforce last year. 

There are more jobs available than coders to fill them.

The numbers are in your favor. 

What do you need? 

  • Do you need a Computer Science Degree? : No
  • Do you need expensive training? No
  • Do you need a MacBook Pro?: No 

You don't need any of these things. If you want to get started, you can do it with a ChromeBook. 

For instance: 

Want to learn JavaScript?

  1. Go to JavaScript.Info and start from the beginning. 
  2. Create an account at JSFiddle and start hacking away. 

Don't worry about details like getting your own website, server, etc. Just do it.   Want to learn some Python? You can install it on your old Dell laptop and start going. CSS? PHP? C#? These are available to download free and will run on nearly any machine built in the last 10 years. 

At some point you will want a faster computer and more in depth training. But to get started this is all you'll need. 

Next Steps

Disclaimer: So, remember how I talked about how hard it was to learn tech? It's what drives me now. This stage of my career is dedicated towards teaching people, and improving their tech skills. In the interest of transparency I work for a company called Pluralsight which is the leading tech skills platform. This is not the only reason I recommend them, I was a customer long before becoming and employee and it's boosted my career tremendously.

So for next steps after taking some tutorials and getting your feet wet, do the following to break into this industry. 

Phase 1

  • Determine what you want to develop (Web, Mobile, Desktop)
  • Find as many tutorials as you can on the subject.  (You can contact me if you need help with this)
  • Create a GitHub account
  • Start uploading code samples you build with tutorials

Phase 2

  • Start a project -No matter how stupid it seems. Make an app to store your favorite jokes, or todo lists.
  • Create a HackerRank account - Start tackling some puzzles. 
  • Pick problems from Project Euler and write code to solve them. 
  • Get to know Stack Overflow - Search it when you have a problem and answer questions if you know it!

Once you get to this point, you'll start feeling comfortable. Keep working and improving your craft. Dig deeper on the tech you've chosen. Build things, even small things. Then think about getting your first job. 

Phase 3

And many more. If you're targeting a specific role, this is a great way to get your skills up. 

  • Build something useful - Build something that solves a problem for you, a friend or your employer. Nothing teaches like you like building something real.
  • Start teaching - Nothing helps you learn a subject like teaching it. When you get comfortable with your knowledge, share it.

Conclusion

Stop making excuses and don't listen to people who tell you you can't or shouldn't learn to code. If you want it bad enough you can do it and get paid for it. Teaching others is a deep passion of mine so if you get stuck on something or need some advice, hit me up I'd be glad to help. 

Now start learning!  



What is your DevOps IQ?

what's your devops score


My Devops Skill IQ is 232. Can you beat it? Take the test now to find out your Devops IQ score!!

Bogdan Popa (bogdan)

Announcing deta July 16, 2019 07:00 AM

I just released the first version of deta, a Racket library for mapping between database tables and structs. A bit like Elixir’s Ecto.

You can install it from the package server with:

raco pkg install deta

Check it out!

July 15, 2019

Derek Jones (derek-jones)

2019 in the programming language standards’ world July 15, 2019 08:26 PM

Last Tuesday I was at the British Standards Institute for a meeting of IST/5, the committee responsible for programming language standards in the UK.

There has been progress on a few issues discussed last year, and one interesting point came up.

It is starting to look as if there might be another iteration of the Cobol Standard. A handful of people, in various countries, have started to nibble around the edges of various new (in the Cobol sense) features. No, the INCITS Cobol committee (the people who used to do all the heavy lifting) has not been reformed; the work now appears to be driven by people who cannot let go of their involvement in Cobol standards.

ISO/IEC 23360-1:2006, the ISO version of the Linux Base Standard, has been updated and we were asked for a UK position on the document being published. Abstain seemed to be the only sensible option.

Our WG20 representative reported that the ongoing debate over pile of poo emoji has crossed the chasm (he did not exactly phrase it like that). Vendors want to have the freedom to specify code-points for use with their own emoji, e.g., pineapple emoji. The heady days, of a few short years ago, when an encoding for all the world’s character symbols seemed possible, have become a distant memory (the number of unhandled logographs on ancient pots and clay tablets was declining rapidly). Who could have predicted that the dream of a complete encoding of the symbols used by all the world’s languages would be dashed by pile of poo emoji?

The interesting news is from WG9. The document intended to become the Ada20 standard was due to enter the voting process in June, i.e., the committee considered it done. At the end of April the main Ada compiler vendor asked for the schedule to be slipped by a year or two, to enable them to get some implementation experience with the new features; oops. I have been predicting that in the future language ‘standards’ will be decided by the main compiler vendors, and the future is finally starting to arrive. What is the incentive for the GNAT compiler people to pay any attention to proposals written by a bunch of non-customers (ok, some of them might work for customers)? One answer is that Ada users tend to be large bureaucratic organizations (e.g., the DOD), who like to follow standards, and might fund GNAT to implement the new document (perhaps this delay by GNAT is all about funding, or lack thereof).

Right on cue, C++ users have started to notice that C++20’s added support for a system header with the name version, which conflicts with much existing practice of using a file called version to contain versioning information; a problem if the header search path used the compiler includes a project’s top-level directory (which is where the versioning file version often sits). So the WG21 committee decides on what it thinks is a good idea, implementors implement it, and users complain; implementors now have a good reason to not follow a requirement in the standard, to keep users happy. Will WG21 be apologetic, or get all high and mighty; we will have to wait and see.

July 14, 2019

Ponylang (SeanTAllen)

Last Week in Pony - July 14, 2019 July 14, 2019 11:06 AM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

July 13, 2019

eta (eta)

Building the Physics Penitentiary July 13, 2019 11:00 PM

I “recently”1 took part in the Weizmann Institute’s international “safe-cracking” tournament, as part of a team of five people2 who placed highly enough in the UK version of the tournament to get to fly over to Israel and have a shot it it over there. Since I did most/all of the electronics and control system for the damn thing, I thought I’d write a blog post about it, because it’s pretty cool!

Essentially, the idea behind the competition is to build ‘safes’ - essentially boxes with physics-related puzzles and challenges that must be completed in order to ‘crack’ the safe, with the point being that only an understanding of the physics involved would let you unlock it. Our safe, the Physics Penitentiary3, looked a bit like this:

Picture of the safe

Safe background

So, I’m not a physicist by any means (which led to some fun discussions about what a random CS student was doing taking part in a physics competition, but we’ll gloss over that…), which means I have only a loose understanding of the physics. That aside, what was in the safe roughly looked like:

  • one Crookes radiometer / light mill (the weird glass thing in the very centre, surrounded by a cage)
  • a pump, hooked up to a container full of ethanol, arranged to deposit said ethanol on the radiometer
  • a soundproof(-ish) section to the right, with two speakers facing one another, plus a microphone
  • a control panel with lots of fun buttons for the user to press
  • a tube with two servo-controlled flags blocking it, which was used to dispense marbles (‘prisoners’) upon successful completion of the challenges

In addition to this, you were allowed:

  • two torches, one with a tungsten bulb, one LED
  • a canister of compressed air

With this, you had to ‘crack’ the safe by figuring out how the Crookes radiometer worked - essentially, due to some physicsey explanation I don’t quite comprehend4, the radiometer would spin if it was heated (e.g. by shining an inefficient tungsten bulb on it), and would also spin, but in the opposite direction, if you cooled it again (e.g. by inverting the canister of compressed air to get at the incredibly cold gas and spraying that on the surface of the radiometer). We also added the sound-based challenge seen to the right, where you had to get the two sound waves to match frequency and wave type (e.g. sine, square, or triangle) in order to cancel each other out5 - to make it a bit harder, and also to fit with the ‘prison’ theme.

Radiometer spinning

Above: nifty GIF of the radiometer doing its thing. Image courtesy of Nevit Dilmen on Wikimedia.

And that was all well and good; the thing spun one way if you used the light, and spun the other way if you used the inverted gas canister.

However, one important detail is still missing: how did the safe actually detect which way the radiometer was spinning, if at all? In planning, we sort of hand-waved this issue away with vague talk of ‘lasers’, ‘infrared’ and ‘proximity sensor hackery’, but it turned out that detecting the spin direction was one of the hardest challenges we had to solve - and the solution was nearly all in code as well, which makes it interesting to talk about!

You spin me right round, baby, right round

So, we had a problem. The problem was made harder by a number of restrictions:

  • The radiometer was a completely sealed partial vacuum chamber, so we couldn’t get in there to add sensors or machinery of any sort. Anything we’d want to use to detect the spin would have to be outside.
  • It’s curved glass, so using a laser, IR beam or something like that probably wouldn’t work too well (though we didn’t test it) due to the light being bent (i.e. refraction).
  • The method used to cool the radiometer (shooting freezing cold liquid at the glass through the grating at the top) caused it to frost over with an opaque white layer of ice. If any sensors were in the way of this blast, they probably wouldn’t have fared too well.
    • To remedy this issue, the ethanol dispenser would let you spray ethanol on the glass to defrost it, making it clear again - although, of course, this also meant the sensors had random extra liquids to contend with…

In the first revision of the safe (the one that we presented at the UK finals), this was accomplished using a generic chinesium proximity sensor we managed to procure from eBay pointed at the radiometer, and some very very questionable heuristics - to give you an idea, we only managed to get the heuristics to work semi-reliably on the very day of the competition itself, and even then, the output of the sensor detection algorithm was fed into an array of 5 recent readings, and we’d take the majority value out of these five to smooth out the inconsistencies (!)6. Very little science was involved in getting this to work; we7 just threw code at the problem until something semi-worked.

Needless to say, such a solution was not going to fly (quite literally8) at the international competition, so, after our miraculous success at the national finals, we needed to think of something else.

The solution we eventually settled on involved using a Vishay VCNL4010 proximity sensor (and SV-Zanshin’s wonderful Arduino library). This sensor had a number of nice features that made us choose it:

  • The actual sensor block (the small black square in the linked image) was very tiny - in fact, just smaller than one of the blades of the radiometer. This meant we could position it such that the block lined up exactly with the blades, so we’d get actually decent readings.
  • It had a built-in ambient light sensor, and combined this data with the IR proximity data to avoid ambient light messing up the sensor readings, which was an issue with the old sensor. As a bonus, this feature actually worked - shining a light on the radiometer didn’t seem to affect the sensor at all!

Sensor closeup

Above: one of the pictures we took of the sensor alignment during testing. Note the neat use of braided sheathing to make cable management not painful…

It’s graph time!

With the actual hardware sorted out, as best we could (positioning the sensor was painstakingly hard, and required multiple attempts to get it to work properly), the next remaining challenge was actually making sense of the proximity data.

Of course, a proximity sensor only gives you a reading indicating how near or far something is, in arbitrary units; you still need to do some processing to figure out spin direction (spin frequency is relatively easy to determine, but we didn’t really care about that much here). In theory, this should be relatively easy: if the sensor is spinning in one direction, the proximity should be increasing (modulo the spike down every now and then), and vice versa for the opposite direction. Indeed, if you plot the readings on the graph, you get something looking like:

Plot of radiometer proximity data

Above: a plot of the radiometer proximity data, taken from the sensor. The first half of the graph shows it spinning in one direction, then it changes at around the 4000 mark on the x-axis to spin in the other direction.

Looking at this (probably somewhat tiny) plot, a pattern is somewhat distinguishable: the troughs of each of the spikes have a different gradient, depending on which direction the radiometer is spinning in - somewhat as we’d expect, from the description we gave above. This reduces the problem to:

  • How do we filter out just the bit at the bottom of each spike that we’re interested in from all the rest of the stuff?
  • How do we figure out the gradient of said interesting points?

The first bullet point was solved with some hacky application of elementary statistics - we kept a buffer of the last 256 data points received from the sensor, calculated their mean and standard deviation, and excluded points that were above mean - (stddev / 4) (with the divisor determined experimentally; 4 seemed to work well enough). This isolated the points to just the few points at the bottom of the trough (or something approximating that).

Radiometer proximity data with gradient lines drawn

For the second, it was time for…some more elementary stats! As might be obvious, a simple linear regression sufficed, giving us a regression line and its gradient to work with. To prevent data points from being calculated from not enough data, we also prevented regressions from being carried out on data with an insufficiently high range, and rejected results with a low R-value. This finally gave us something vaguely capable of determining the spin direction - and we’d also require 3 or so concordant readings in order to actually come to a judgement, to minimize the chance of random errors. (Remember, a false positive would result in the challenge instantly unlocking, so we wanted to be sure; better to have the cracker wait for 5 seconds or so than to have it randomly unlock out of nowhere!)

Oh, and the issues with the user being able to spray frost (or ethanol) at the sensor were resolved by…judicious use of tape covering the sensor, so nothing could get in at it (!). We also added some tissue paper, so that the ethanol wouldn’t run everywhere and ruin the rest of the safe.

Radiometer closeup

Above: If you look closely, you can just about make out the tape on the other side of the radiometer…

What about the rest of the electronics?

This whole thing was done using STM32F103 microcontrollers, actually - the first revision of the safe used a Blue Pill (see linked blog post for more), while the second revision used the somewhat more reliable Maple Mini. The STM32s were definitely a big plus; they were pretty small, for one, which meant we could stuff the electronics underneath the radiometer (an Arduino Uno would definitely not have fit in!), and their speed (and 32-bit address bus…) made doing calculations like the above regression stuff actually feasible. I used the STM32duino IDE and tooling, primarily because writing drivers (for things like the VCNL4010) didn’t seem like my idea of a fun time (!).

Electronics stuffed into the radiometer well

Above: most of the electronics and cabling, stuffed into a well below the radiometer during construction. Connecting all the wires was pretty difficult!

To put things into context, the original revision didn’t use a PCB at all, and instead had tons of flying wires connecting everything on a piece of protoboard; we decided that making a proper PCB for the second revision might be a good idea, so we did:

Artsy shot of the PCB, with theta.eu.org logo in focus

Above: artsy shot of the PCB made for the project, with gratutitous theta.eu.org logo in focus.

In total, the electronics ended up being made up of:

  • 1 Maple Mini STM32F103 microcontroller
  • 1 VCNL4010 proximity sensor breakout
  • 2 AD9833 waveform generator modules, for the frequency challenge
    • The first iteration of the safe initially used digital potentiometers (!) to control some hacked up tone generator kits we had lying around; using the AD9833s was definitely a better idea.
    • However, finding breakout boards with this IC on them wasn’t an easy task9.
  • 1 SparkFun microphone breakout
  • 2 amplifiers, to drive the speakers
  • 2 servo motors, to control release of the marbles
  • 1 Tracopower TSR-2450 voltage regulator, for power input
    • I love these modules (for their efficiency); we used one to step down the 9V barrel jack input to a more manageable 5V.
  • Some well-placed debug (UART) and programming (SWDIO) headers, to get at the Maple Mini
  • Assorted push switches and LEDs for the control panel
  • Probably a few passives and other things like that

Conclusion

Right, that’s probably about enough information for one blog post; it’s been upwards of 2000 words, after all! Hopefully some of it was vaguely informative / interesting; it’s also worth keeping in mind that the actual process of building this thing was significantly more chaotic and panicked than it might seem (c.f. the footnotes below). However, it was definitely a fun thing to build (despite not doing well in the international finals, sadly!)

Do let me know using the comment thing below if you have any more questions about the safe, or feedback; it’s always nice to receive comments!


  1. (read: a couple of months ago) 

  2. This means your odds of figuring out who the hell I actually am in real life are now about 1 in 5. 

  3. Not my choice of name. 

  4. The explanations behind the radiometer are apparently still disputed; if you do a search for it, there are a number of conflicting reasons why it behaves like this! 

  5. If you’re wondering how that actually worked in practice given differences in phase, etc., you’d be onto something. We may or may not have bodged this part a bit to get it to work… 

  6. It also stopped giving any readings at all if you shone a light at the sensor, which caused some fun issues when it came to cracking the safe. 

  7. When I say ‘we’, I really mean ‘I’, but it sounds nicer this way. 

  8. Transporting the safe to Israel was done, of course, on a plane; the sensor positioning was so sensitive that we thought it’d definitely get screwed up by being bumped around… 

  9. When one of them broke in the week before the international final, trying to hunt around Amazon for another one wasn’t as easy as we would have hoped! 

Jeff Carpenter (jeffcarp)

How are Words Represented in Machine Learning? Part 1 July 13, 2019 01:31 AM

Machine learning on human languages is a super exciting space right now. Applications are exploding—just think of how many natural language ML models it takes to run a smart assistant, from transforming spoken audio to text, to finding the exact part of a web page that answers your question, to choosing the correct words with the correct grammar to reply to you. At work I recently had the opportunity to build an NLP system.

July 12, 2019

Gustaf Erikson (gerikson)

June July 12, 2019 06:57 PM

Kungsträdgården

Stensund

Jun 2018 | Jun 2017 | Jun 2016 | Jun 2015 | Jun 2014 | Jun 2013 | Jun 2012 | Jun 2011 | Jun 2010 | Jun 2009

Leo Tindall (LeoLambda)

What Is Rust's unsafe? July 12, 2019 05:00 PM

I’ve seen a lot of misconceptions around what the unsafe keyword means for the utility and validity of Rust and its marketing as a “safe systems language”. The truth is a lot more complicated than a single pithy tweet can possibly sum up, unfortunately; here it is as I see it. Basically, the unsafe keyword does not turn off the advanced type system that keeps Rust code honest. It only allows a few select “superpowers”, like dereferencing raw pointers.

Jan van den Berg (j11g)

About WordPress, emojis, MySQL and latin1, utf8 and utf8mb4 character sets July 12, 2019 11:55 AM

PSA: the MySQL utf8 character set is not real Unicode utf8. Instead use utf8mb4.

So you landed here because some parts of your website are garbled. And this happened after a server or website migration. You exported your database and imported this export or dump on the new server. And now your posts look like this:

Strange characters!

When they should look like this:

Emojis!

These are screenshots from this website. This website was an old WordPress installation that still used the latin1 character set. The WordPress installation was up to date, but automatic WordPress updates will never update or change the database character set. So this will always remain what it was on initial installation (which was latin1 in my case).

And latin1 can not store (real) Unicode characters e.g. emojis. You need a Unicode character set. So, just use utf8, right?

Problem 1: exporting and importing the database with the wrong character set

When exporting/dumping a database with mysqldump, this will use the default MySQL servers’ character set (set in my.cnf). In my case this was set to utf8. But by not explicitly telling the mysqldump to use the correct character set for a particular database (which was latin1) my dumped data was messed up.

So when I restored the dump (on the new server) some text was garbled and emojis had completely disappeared from blog posts.

I fixed this with help of these links. Key here is: explicitly set the correct character for a database when exporting this database. Then: change all instances in the dump file from the old character set to the new character set and import this file.

Problem 2: some emojis work, some don’t

After this my text was fine. I exported using the old character set and imported using utf8, what could go wrong! But some emojis were still missing, but others were not?! This was a head-scratcher.

There is a question mark instead of an emoji

How can this be, I had set my database character set to utf8 (with utf8_general_ci collation). This is Unicode, right? Wrong!

MySQL utf8 does not support complete Unicode. MySQL utf8 uses only 3 bytes per character.

Full Unicode support needs 4 bytes per character. So your MySQL installation needs to use the utf8mb4 character set (and utf8mb4_unicode_ci collation) to have real and full Unicode support.

Some strange decisions were made in the 2002. 🤯 Which has given a lot of people headaches.

So, MySQL utf8 only uses three bytes per character, but the explanation for the image with the one missing emoji is, that *some* emojis (like the❤) will be expressed correctly with only three bytes.

Problem 3: but I used emojis before, in my latin1 WordPress installation?!

Yes, things worked fine, right? And here is the thing: if you dumped your database with the correct (old) character set and imported correctly with this old character set, things would still work!

But you said latin1 does not support (4 byte character) emojis!?

Correct!

However WordPress is smart enough to figure this out: when using emojis in post titles or posts it will check the database character set and change the emoji to (hexadecimal) HTML code — which can be expressed and stored just fine in latin1.

But how do you explain this image? This was after incorrectly dumping and importing a database.

Wait, what?! The URL has the correct emoji but the post title does not!?!

The post URL has two emojis, but one emoji is missing from the title?! I already explained why the emoji is missing from the post title, so how can that emoji be present in the post URL? This is because WordPress stores the post titles differently from the URLs.

How the title and the post name (url) are stored

So the post title field stores real Unicode symbols and the post_name (URL) field stores them encoded. So there you have it 🤓!

The post About WordPress, emojis, MySQL and latin1, utf8 and utf8mb4 character sets appeared first on Jan van den Berg.

Dan Luu (dl)

Files are fraught with peril July 12, 2019 12:00 AM

This is a psuedo-transcript for a talk given at Deconstruct 2019. To make this accessible for people on slow connections as well as people using screen readers, the slides have been replaced by in-line text (the talk has ~120 slides; at an average of 20 kB per slide, that's 2.4 MB; in Nigeria, 1GB of data costs ~1.2x median daily income).

Let's talk about files! Most developers seem to think that files are easy. Just for example, let's take a look at the top reddit r/programming comments from when Dropbox announced that they were only going to support ext4 on Linux (the most widely used Linux filesystem). For people not familiar with reddit r/programming, I suspect r/programming is the most widely read English langauge programming forum in the world.

The top comment reads:

I'm a bit confused, why do these applications have to support these file systems directly? Doesn't the kernel itself abstract away from having to know the lower level details of how the files themselves are stored?

The only differences I could possibly see between different file systems are file size limitations and permissions, but aren't most modern file systems about on par with each other?

The #2 comment (and the top replies going two levels down) are:

#2: Why does an application care what the filesystem is?

#2: Shouldn't that be abstracted as far as "normal apps" are concerned by the OS?

Reply: It's a leaky abstraction. I'm willing to bet each different FS has its own bugs and its own FS specific fixes in the dropbox codebase. More FS's means more testing to make sure everything works right . . .

2nd level reply: What are you talking about? This is a dropbox, what the hell does it need from the FS? There are dozenz of fssync tools, data transfer tools, distributed storage software, and everything works fine with inotify. What the hell does not work for dropbox exactly?

another 2nd level reply: Sure, but any bugs resulting from should be fixed in the respective abstraction layer, not by re-implementing the whole stack yourself. You shouldn't re-implement unless you don't get the data you need from the abstraction. . . . DropBox implementing FS-specific workarounds and quirks is way overkill. That's like vim providing keyboard-specific workarounds to avoid faulty keypresses. All abstractions are leaky - but if no one those abstractions, nothing will ever get done (and we'd have billions of "operating systems").

In this talk, we're going to look at how file systems differ from each other and other issues we might encounter when writing to files. We're going to look at the file "stack" starting at the top with the file API, which we'll see is nearly impossible to use correctly; move down to the filesystem, which we'll see has serious bugs that cause data loss and data corruption; and then we'll look at disks and see that disks can easily corrupt data at a rate five million times greater than claimed in vendor datasheets.

File API

Writing one file

Let's say we want to write a file safely, so that we don't want to get data corruption. For the purposes of this talk, this means we'd like our write to be "atomic" -- our write should either fully complete, or we should be able to undo the write and end up back where we started. Let's look at an example from Pillai et al., OSDI’14.

We have a file that contains the text a foo and we want to overwrite foo with bar so we end up with a bar. We're going to make a number of simplifications. For example, you should probably think of each character we're writing as a sector on disk (or, if you prefer, you can imagine we're using a hypothetical advanced NVM drive). Don't worry if you don't know what that means, I'm just pointing this out to note that this talk is going to contain many simplifications, which I'm not going to call out because we only have twenty-five minutes and the unsimplified version of this talk would probably take about three hours.

To write, we might use the pwrite syscall. This is a function provided by the operating system to let us interact with the filesystem. Our invocation of this syscall looks like:

pwrite(
  [file], 
  “bar”, // data to write
  3,     // write 3 bytes
  2)     // at offset 2

pwrite takes the file we're going to write, the data we want to write, bar, the number of bytes we want to write, 3, and the offset where we're going to start writing, 2. If you're used to using a high-level language, like Python, you might be used to an interface that looks different, but underneath the hood, when you write to a file, it's eventually going to result in a syscall like this one, which is what will actually write the data into a file.

If we just call pwrite like this, we might succeed and get a bar in the output, or we might end up doing nothing and getting a foo, or we might end up with something in between, like a boo, a bor, etc.

What's happening here is that we might crash or lose power when we write. Since pwrite isn't gauranteed to be atomic, if we crash, we can end up with some fraction of the write completing, causing data corruption. One way to avoid this problem is to store an "undo log" that will let us restore corrupted data. Before we're modify the file, we'll make a copy of the data that's going to be modified (into the undo log), then we'll modify the file as normal, and if nothing goes wrong, we'll delete the undo log.

If we crash while we're writing the undo log, that's fine -- we'll see that the undo log isn't complete and we know that we won't have to restore because we won't have started modifying the file yet. If we crash while we're modifying the file, that's also ok. When we try to restore from the crash, we'll see that the undo log is complete and we can use it to recover from data corruption:

creat(/d/log) // Create undo log
write(/d/log, "2,3,foo", 7) // To undo, at offset 2, write 3 bytes, "foo"
pwrite(/d/orig, “bar", 3, 2) // Modify original file as before
unlink(/d/log) // Delete log file

If we're using ext3 or ext4, widely used Linux filesystems, and we're using the mode data=journal (we'll talk about what these modes mean later), here are some possible outcomes we could get:

d/log: "2,3,f"
d/orig: "a foo"

d/log: ""
d/orig: "a foo"

It's possible we'll crash while the log file write is in progress and we'll have an incomplete log file. In the first case above, we know that the log file isn't complete because the file says we should start at offset 2 and write 3 bytes, but only one byte, f, is specified, so the log file must be incomplete. In the second case above, we can tell the log file is incomplete because the undo log format should start with an offset and a length, but we have neither. Either way, since we know that the log file isn't complete, we know that we don't need to restore.

Another possible outcome is something like:

d/log: "2,3,foo"
d/orig: "a boo"

d/log: "2,3,foo"
d/orig: "a bar"

In the first case, the log file is complete we crashed while writing the file. This is fine, since the log file tells us how to restore to a known good state. In the second case, the write completed, but since the log file hasn't been deleted yet, we'll restore from the log file.

If we're using ext3 or ext4 with data=ordered, we might see something like:

d/log: "2,3,fo"
d/orig: "a boo"

d/log: ""
d/orig: "a bor"

With data=ordered, there's no guarantee that the write to the log file and the pwrite that modifies the original file will execute in program order. Instesad, we could get

creat(/d/log) // Create undo log
pwrite(/d/orig, “bar", 3, 2) // Modify file before writing undo log!
write(/d/log, "2,3,foo", 7) // Write undo log
unlink(/d/log) // Delete log file

To prevent this re-ordering, we can use another syscall, fsync. fsync is a barrier (prevents re-ordering) and it flushes caches (which we'll talk about later).

creat(/d/log)
write(/d/log, “2,3,foo”, 7)
fsync(/d/log) // Add fsync to prevent re-ordering
pwrite(/d/orig, “bar”, 3, 2)
fsync(/d/orig) // Add fsync to prevent re-ordering
unlink(/d/log)

This works with ext3 or ext4, data=ordered, but if we use data=writeback, we might see something like:

d/log: "2,3,WAT"
d/orig: "a boo"

Unfortunately, with data=writeback, the write to the log file isn't guaranteed to be atomic and the filesystem metadata that tracks the file length can get updated before we've finished writing the log file, which will make it look like the log file contains whatever bits happened to be on disk where the log file was created. Since the log file exists, when we try to restore after a crash, we may end up "restoring" random garbage into the original file. To prevent this, we can add a checksum (a way of making sure the file is actually valid) to the log file.

creat(/d/log)
write(/d/log,“…[✓∑],foo”,7) // Add checksum to log file to detect incomplete log file
fsync(/d/log)
pwrite(/d/orig, “bar”, 3, 2)
fsync(/d/orig)
unlink(/d/log)

This should work with data=writeback, but we could still see the following:

d/orig: "a boo"

There's no log file! Although we created a file, wrote to it, and then fsync'd it. Unfortunately, there's no guarantee that the directory will actually store the location of the file if we crash. In order to make sure we can easily find the file when we restore from a crash, we need to fsync the parent of the newly created log.

creat(/d/log)
write(/d/log,“…[✓∑],foo”,7)
fsync(/d/log)
fsync(/d) /// fsync parent directory
pwrite(/d/orig, “bar”, 3, 2)
fsync(/d/orig)
unlink(/d/log)

There are a couple more things we should do. We shoud also fsync after we're done (not shown), and we also need to check for errors. These syscalls can return errors and those errors need to be handled appropriately. There's at least one filesystem issue that makes this very difficult, but since that's not an API usage thing per se, we'll look at this again in the Filesystems section.

We've now seen what we have to do to write a file safely. It might be more complicated than we like, but it seems doable -- if someone asks you to write a file in a self-contained way, like an interview question, and you know the appropriate rules, you can probably do it correctly. But what happens if we have to do this as a day-to-day part of our job, where we'd like to write to files safely every time to write to files in a large codebase.

API in practice

Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope write to files safely, like datbases and version control systems: Leveldb, LMDB, GDBM, HSQLDB, Sqlite, PostgreSQL, Git, Mercurial, HDFS, Zookeeper. They then wrote a static analysis tool that can find incorrect usage of the file API, things like incorrectly assuming that operations that aren't atomic are actually atomic, incorrectly assuming that operations that can be re-ordered will execute in program order, etc.

When they did this, they found that every single piece of software they tested except for SQLite in one particular mode had at least one bug. This isn't a knock on the developers of this software or the software -- the programmers who work on things like Leveldb, LBDM, etc., know more about filesystems than the vast majority programmers and the software has more rigorous tests than most software. But they still can't use files safely every time! A natural follow-up to this is the question: why the file API so hard to use that even experts make mistakes?

Concurrent programming is hard

There are a number of reasons for this. If you ask people "what are hard problems in programming?", you'll get answers like distributed systems, concurrent programming, security, aligning things with CSS, dates, etc.

And if we look at what mistakes cause bugs when people do concurrent programming, we see bugs come from things like "incorrectly assuming operations are atomic" and "incorrectly assuming operations will execute in program order". These things that make concurrent programming hard also make writing files safely hard -- we saw examples of both of these kinds of bugs in our first example. More generally, many of the same things that make concurrent programming hard are the same things that make writing to files safely hard, so of course we should expect that writing to files is hard!

Another property writing to files safely shares with concurrent programming is that it's easy to write code that has infrequent, non-deterministc failures. With respect to files, people will sometimes say this makes things easier ("I've never noticed data corruption", "your data is still mostly there most of the time", etc.), but if you want to write files safely because you're working on software that shouldn't corrupt data, this makes things more difficult by making it more difficult to tell if your code is really correct.

API inconsistent

As we saw in our first example, even when using one filesystem, different modes may have significantly different behavior. Large parts of the file API look like this, where behavior varies across filesystems or across different modes of the same filesystem. For example, if we look at mainstream filesystems, appends are atomic, except when using ext3 or ext4 with data=writeback, or ext2 in any mode and directory operations can't be re-ordered w.r.t. any other operations, except on btrfs. In theory, we should all read the POSIX spec carefully and make sure all our code is valid according to POSIX, but if they check filesystem behavior at all, people tend to code to what their filesystem does and not some abtract spec.

Docs unclear

In our first example, we saw that we can get different behavior from using different data= modes. If we look at the manpage (manual) on what these modes mean in ext3 or ext4, we get:

journal: All data is committed into the journal prior to being written into the main filesystem.

ordered: This is the default mode. All data is forced directly out to the main file system prior to its metadata being committed to the journal.

writeback: Data ordering is not preserved – data may be written into the main filesystem after its metadata has been committed to the journal. This is rumoured to be the highest-throughput option. It guarantees internal filesystem integrity, however it can allow old data to appear in files after a crash and journal recovery.

If you want to know how to use your filesystem safely, and you don't already know what a journaling filesystem is, this definitely isn't going to help you. If you know what a journaling filesystem is, this will give you some hints but it's still not sufficient. It's theoretically possible to figure everything out from reading the source code, but this is pretty impractical for most people who don't already know how the filesystem works.

For English-language documentation, there's lwn.net and the Linux kernel mailing list (LKML). LWN is great, but they can't keep up with everything, so you LKML is the place to go if you want something comprehensive. Here's an example of an exchange on LKML about filesystems:

Dev 1: Personally, I care about metadata consistency, and ext3 documentation suggests that journal protects its integrity. Except that it does not on broken storage devices, and you still need to run fsck there.
Dev 2: as the ext3 authors have stated many times over the years, you still need to run fsck periodically anyway.
Dev 1: Where is that documented?
Dev 2: linux-kernel mailing list archives.
FS dev: Probably from some 6-8 years ago, in e-mail postings that I made.

While the filesystem developers tend to be helpful and they write up informative responses, most people probably don't keep up with the past 6-8 years of LKML.

Performance / correctness conflict

Another issue is that the file API has an inherent conflict between performance and correctness. We noted before that fsync is a barrier (which we can use to enforce ordering) and that it flushes caches. If you've ever worked on the design of a high-performance cache, like a microprocessor cache, you'll probably find the bundling of these two things into a single primitive to be unusual. A reason this is unusual is that flushing caches has a significant performance cost and there are many cases where we want to enforce ordering without paying this performance cost. Bundling these two things into a single primitive forces us to pay the cache flush cost when we only care about ordering.

Chidambaram et al., SOSP’13 looked at the performance cost of this by modifying ext4 to add a barrier mechanism that doesn't flush caches and they found that, if they modified software appropriately and used their barrier operation where a full fsync wasn't necessary, they were able to achieve performance roughly equivalent to ext4 with cache flushing entirely disabled (which is unsafe and can lead to data corruption) without sacrificing safety. However, making your own filesystem and getting it adopted is impractical for most people writing user-level software. Some databases will bypass the filesystem entirely or almost entirely, but this is also impractical for most software.

That's the file API. Now that we've seen that it's extraordinarily difficult to use, let's look at filesystems.

Filesystem

If we want to make sure that filessytems work, one of the most basic tests we could do is to inject errors are the layer below the filesystem to see if the filesystem handles them properly. For example, on a write, we could have the disk fail to write the data and return the appropriate error. If the filesystem drops this error or doesn't handle ths properly, that means we have data loss or data corrupiton. This is analogous to the kinds of distributed systems faults Kyle Kingsbury talked about in his distributed systems testing talk yesterday (although these kinds of errors are much more straightforward to test).

Prabhakaran et al., SOSP’05 did this and found that, for most filesystems tested, almost all write errors were dropped. The major exception to this was on ReiserFS, which did a pretty good job with all types of errors tested, but ReiserFS isn't really used today for reasons beyond the scope of this talk.

We (Wesley Aptekar-Cassels and I) looked at this again in 2017 and found that things had improved significantly. Most filesystems (other than JFS) could pass these very basic tests on error handling.

Another way to look for errors is to look at filesystems code to see if it handles internal errors correctly. Gunawai et al., FAST’08 did this and found that internal errors were dropped a significant percentage of the time. The technique they used made it difficult to tell if functions that could return many different errors were correctly handling each error, so they also looked at calls to functions that can only return a single error. In those cases, depending on the function, errors were dropped rouhgly 2/3 to 3/4 of the time, depending on the function.

Wesley and I also looked at this again in 2017 and found significant improvement -- errors for the same functions Gunawi et al. looked at were "only" ignored 1/3 to 2/3 of the time, depending on the function.

Gunawai et al. also looked at comments near these dropped errors and found comments like "Just ignore errors at this point. There is nothing we can do except to try to keep going." (XFS) and "Error, skip block and hope for the best." (ext3).

Now we've seen that while filesystems used to drop even the most basic errors, they now handle then correctly, but there are some code paths where errors can get dropped. For a concrete example of a case where this happens, let's look back at our first example. If we get an error on fsync, unless we have a pretty recent Linux kernel (Q2 2018-ish), there's a pretty good chance that the error will be dropped and it may even get reported to the wrong process!

On recent Linux kernels, there's a good chance the error will be reported (to the correct process, even). Wilcox, PGCon’18 notes that an error on fsync is basically unrecoverable. The details for depending on filesystem -- on XFS and btrfs, modified data that's in the filesystem will get thrown away and there's no way to recover. On Linux zfs, the system hangs. On ext4, the data isn't thrown away, but it's marked as unmodified, so the filesystem won't try to write it back to disk later, and if there's memory pressure, the data can be thrown out at any time. If you're feeling adventerous, you can try to recover the data before it gets thrown out with various tricks (e.g., by forcing the filesystem to mark it as modified again, or by writing it out to another device, which will force the filesystem to write the data out even though it's marked as unmodified), but there's no guarantee you'll be able to recover the data before it's thrown out.

In general, there isn't a good way to recover from this on Linux. Postgres, MySQL, and MongoDB (widely used databases) will crash themselves and the user is expected to restore from the last checkpoint. Most software will probably just silently lose or corrupt data. And fsync is a relatively good case -- for example, syncfs simply doesn't return errors on Linux at all, leading to silent data loss and data corruption.

BTW, when Craig Ringer first proposed that Postgres should crash on fsync error, the first response on the Postgres dev mailing list was:

Surely you jest . . . If [current behavior of fsync] is actually the case, we need to push back on this kernel brain damage

But after talking through the details, everyone agreed that crashing was the only good option. One of the many unfortunate things is that most disk errors are transient. Since the filesystem discards critical information that's necessary to proceed without data corruption on any error, transient errors that could be retried instead force software to take drastic measures.

And while we've talked about Linux, this isn't unique to Linux. Fsync error handling (and error handling in general) is broken on many different operating systems. At the time Postgres "discovered" the behavior of fsync on Linux, FreeBSD had arguably correct behavior, but OpenBSD and NetBSD behaved the same as Linux (true error status dropped, retrying causes success response, data lost).

Now that we've seen that, for many years, filesystems failed to handle errors in some of the most straightforward and simple cases and that there are cases that still aren't handled correctly today, let's look at disks.

Disk

Flushing

We've seen that it's easy to not realize we have to call fsync when we have to call fsync, and that even if we call fsync appropriately, bugs may prevent fsync from actually working. Rajimwale et al., DSN’11 into whether or not disks actually flush when you ask them to flush, assuming everything above the disk works correctly (their paper is actually mostly about something else, they just discuss this briefly at the beginning). Someone from Microsoft anonymously told them "[Some disks] do not allow the file system to force writes to disk properly" and someone from Seagate, a disk manufacturer, told them "Some disks do not allow the file system to force writes to disk properly". Bairavasundaram et al., FAST’07 also found the same thing when they looked into disk reliability.

Error rates

We've seen that filessytems sometimes don't handle disk errors correctly. If we want to know how serious this issue is, we should look at the rate at which disks emit errors. Disk datasheets will usually an uncorrectable bit error rate of 1e-14 for consuemr HDDs (often called spinning metal or spinning rust disks), 1e-15 for enterprise HDDs, 1e-15 for consumer SSDs, and 1e-16 for enterprise SSDs. This means that, on average, we expect to see one unrecoverable data error every 1e14 bits we read on an HDD.

To get an intuition for what this means in practice, 1TB is now a pretty normal disk size. If we read a full drive once, that's 1e12 bytes, or almost 1e13 bits (technically 8e12 bits), which means we should expect to see one unrecoverable error every if we buy a 1TB HDD and read the entire disk ten-ish times. Nowadays, we can buy 10TB HDDs, in which case we'd expect to see an error (technically, 8/10th errors) on every read of an entire consumer HDD.

In practice, observed data rates are are significantly higher. Narayanan et al., SYSTOR’16 (Microsoft) observed SSD error rates from 1e-11 to 6e-14, depending on the drive model. Meza et al., SIGMETRICS’15 (FB) observed even worse SSD error rates, 2e-9 to 6e-11 depending on the model of drive. Depending on the type of drive, 2e-9 is 2 gigabits, or 250 MB, 500 thousand to 5 million times worse than stated on datasheets depending on the class of drive.

Bit error rate is arguably a bad metric for disk drives, but this is the metric disk vendors claim, so that's what we have to compare against if we want an apples-to-apples comparison. See Bairavasundaram et al., SIGMETRICS'07, Schroeder et al., FAST'16, and others for other kinds of error rates.

One thing to note is that it's often claimed that SSDs don't have problems with corruption because they use error correcting codes (ECC), which can fix data corruption issues. "Flash banishes the specter of the unrecoverable data error", etc. The thing this misses is that modern high-density flash devices are very unreliable and need ECC to be usable at all. Grupp et al., FAST’12 looked at error rates of the kind of flash the underlies SSDs and found errors rates from 1e-1 to 1e-8. 1e-1 is one error every ten bits, 1e-8 is one error every 100 megabits.

Power loss

Another claim you'll hear is that SSDs are safe against power loss and some types of crashes because they now have "power loss protectection" -- there's some mechanism in the SSDs that can hold power for long enough during an outage that the internal SSD cache can be written out safely.

Luke Leighton tested this by buying 6 SSDs that claim to have power loss protection and found that four out of the six models of drive he tested failed (every drive that wasn't an Intel drive). If we look at the details of the tests, when drives fail, it appears to be because they were used in a way that the implementor of power loss protection didn't expect (writing "too fast", although well under the rate at which the drive is capable of writing, or writing "too many" files in parallel). When a drive advertises that it has power loss protection, this appears to mean that someone spent some amount of effort implementing something that will, under some circumstances, prevent data loss or data corruption under power loss. But, as we saw in Kyle's talk yesterday on distributed systems, if you want to make sure that the mechanism actually works, you can't rely on the vendor to do rigorous or perhaps even any semi-serious testing and you have to test it yourself.

Retention

If we look at SSD datasheets, a young-ish drive (one with 90% of its write cycles remaining) will usually be specced to hold data for about ten years after a write. If we look at a worn out drive, one very close to end-of-life, it's specced to retain data for one year to three months, depending on the class of drive. I think people are often surprised to find that it's within spec for a drive to lose data three months after the data is written.

These numbers all come from datasheets and specs, as we've seen, datasheets can be a bit optimistic. On many early SSDs, using up most or all of a drives write cycles would cause the drive to brick itself, so you wouldn't even get the spec'd three month data retention.

Corollaries

Now that we've seen that there are significant problems at every level of the file stack, let's look at a couple things that follow from this.

What to do?

What we should do about this is a big topic, in the time we have left, one thing we can do instead of writing to files is to use databases. If you want something lightweight and simple that you can use in most places you'd use a file, SQLite is pretty good. I'm not saying you should never use files. There is a tradeoff here. But if you have an application where you'd like to reduce the rate of data corruption, considering using a database to store data instead of using files.

FS support

At the start of this talk, we looked at this Dropbox example, where most people thought that there was no reason to remove support for most Linux filesystems because filesystems are all the same. I believe their hand was forced by the way they want to store/use data, which they can only do with ext, but even if that wasn't the case, perhaps you can see why software that's attempting to sync data to disk reliably and with decent performance might not want to support every single filesystem in the universe for an OS that, for their product, is relatively niche.

FAQ

While putting this talk to together, I read a bunch of different online discussions about how to write to files safely. For discussions outside of specialized communities (e.g., LKML, the Postgres mailing list, etc.), many people will drop by to say something like "why is everyone making this so complicated? You can do this very easily and completely safely with this one weird trick". Let's look at the most common "one weird trick"s from two thousand internet comments on how to write to disk safely.

Rename

The most frequently mentioned trick is to rename instead of overwriting. If you remember our single-file write example, we made a copy of the data that we wanted to overwrite before modifying the file. The trick here is to do the opposite:

  1. Make a copy of the entire file
  2. Modify the copy
  3. Rename the copy on top of the original file

This trick doesn't work. People seem to think that this is safe becaus the POSIX spec says that rename is atomic, but that only means rename is atomic with respect to normal operation, that doesn't mean it's atomic on crash. This isn't just a theoretical problem; if we look at mainstream Linux filesystems, most have at least one mode where rename isn't atomic on crash. Rename also isn't gauranteed to execute in program order, as people sometimes expect.

The most mainstream exception where rename is atomic on crash is probably btrfs, but even there, it's a bit subtle -- as noted in Bornholt et al., ASPLOS’16, rename is only atomic on crash when renaming to replace an existing file, not when renaming to create a new file. Also, Mohan et al., OSDI’18 found numerous rename atomicity bugs on btrfs, some quite old and some introduced the same year as the paper, so you want not want to rely on this without extensive teesting, even if you're writing btrfs specific code.

Append

The second most frequently mentioned trick is to only ever append (instead of sometimes overwriting). This also doesn't work. As noted in Pillai et al., OSDI’14 and Bornholt et al., ASPLOS’16, appends don't gaurantee ordering or atomicity and believing that appends are safe is the cause of some bugs.

Conclusion

In conclusion, computers don't work (but you probably already know this if you're here at Gary-conf). This talk happened to be about files, but there are many areas we could've looked into where we would've seen similar things.

One thing I'd like to note before we finish is that, IMO, the underlying problem isn't technical. If you look at what huge tech companies do (companies like FB, Amazon, MS, Google, etc.), they often handle writes to disk pretty safely. They'll make sure that they have disks where power loss protection actually work, they'll have patches into the OS and/or other instrumentation to make sure that errors get reported correctly, there will be large distributed storage groups to make sure data is replicated safely, etc. We know how to make this stuff pretty reliable. It's hard, and it takes a lot of time and effort, i.e., a lot of money, but it can be done.

If you ask someone who works on that kind of thing why they spend mind boggling sums of money to enure (or really, increase the probability of) correctness, you'll often get an answer like "we have a zillion machines and if you do the math on the rate of data corruption, if we didn't do all of this, we'd have data corruption every minute of every day. It would be totally untenable". A huge tech company might have, what, order of ten million machines? The funny thing is, if you do the math for how many consumer machines there are out there and much consumer software runs on unreliable disks, the math is similar. There are many more consumer machines; they're typically operated at much lighter load, but there are enough of them that, if you own a widely used piece of desktop/laptop/workstation software, the math on data corruption is pretty similar. Without "extreme" protections, we should expect to see data corrupiton all the time.

But if we look at how consumer software works, it's usually quite unsafe with respect to handling data. IMO, the key difference here is that when a huge tech company loses data, whether that's data on who's likely to click on which ads or user emails, the company pays the cost, directly or indirectly and the cost is large enough that it's obviously correct to spend a lot of effort to avoid data loss. But when consumers have data corruption on their own machines, they're mostly not sophisticated enough to know who's at fault, so the company can avoid taking the brunt of the blame. If we have a global optimization function, the math is the same -- of course we should put more effort into protecting data on consumer machines. But if we're a company that's locally optimizing for our own benefit, the math works out differently and maybe it's not worth it to spend a lot of effort on avoiding data corruption.

Yesterday, Ramsey Nasser gave a talk where he made a very compelling case that something was a serious problem, which was followed up by a comment that his proposed solution will have a hard time getting adoption. I agree with both parts -- he discussed an important problem, and it's not clear how solving that problem will make anyone a lot of money, so the problem is likely to go unsolved.

With GDPR, we've seen that regulation can force tech companies to protect people's privacy in a way they're not naturally inclined to do, but regulation is a very big hammer and the unintended consequences can often negate or more than negative the benefits of regulation. When we look at the history of regulations that are designed to force companies to do the right thing, we can see that it's often many years, sometimes decades, before the full impact of the regulation is understood. Designing good regulations is hard, much harder than any of the technical problems we've discussed today.

Acknowledgements

Thanks to Leah Hanson, Gary Bernhardt, Kamal Marhubi, Rebecca Isaacs, Jesse Luehrs, Tom Crayford, Wesley Aptekar-Cassels, Rose Ames, and Benjamin Gilbert for their help with this talk!

Sorry we went so fast. If there's anything you missed you can catch it in the pseudo-transcript at danluu.com/deconstruct-files.

This "transcript" is pretty rough since I wrote it up very quickly this morning before the talk. I'll try to clean it within a few weeks, which will include adding material that was missed, inserting links, adding references that were missed, etc.

Thanks to Anatole Shaw for comments/corrections/discussion on this transcript.

July 11, 2019

Frederic Cambus (fcambus)

Fuzzing DNS zone parsers July 11, 2019 01:00 PM

In my never-ending quest to improve the quality of my C codebases, I've been using AFL to fuzz statzone, the zone parser I use to generate monthly statistics on StatDNS. It helped me to find and fix a NULL pointer dereference.

I initially used the .arpa zone file as input, but then remembered that OpenDNSSEC bundles a special zone for testing purposes, containing a lot of seldom used resource records types, and decided to use this one too.

Out of curiosity, I decided to try fuzzing other DNS zone parsers. I started with validns 0.8, and within seconds the fuzzer found multiple NULL pointer dereferences.

validns

The first occurrence happens in the name2findable_name() function, and can be triggered with the following input:

arpa                    86400   IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2019021500 1800 900 604800 86400
arpa.                   86400   IN      RRSIG   SOA 8 1 86400 20190228000000 20190214230000 49906 arpa. Qot7qHAA2QhNmAz3oJUIGmxGJrKnWsIzEvZ92R+LV03K7YTFozio2U7Z534RZBhc0UJvlF1YenrbM6ugmF0z55CJD9JY7cFicalFPOkIuWslSl62vuIWHLwN5sA7VZ0ooVN2ptQpPHDa3W/9OPJRF0YqjBBBwD7IiL7V560rbXM=

With the above input, the following call to strlen(3) in rr.c results in a NULL pointer dereference because 's' ends up being NULL:

static unsigned char *name2findable_name(char *s)
{
    int l = strlen(s);

The second occurrence happens in the nsec_validate_pass2() function, and can be triggered with the following input:

arpa.                   86400   IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2019021500 1800 900 604800 86400
arpa.                   86400   IN      NSEC    a

With the above input, the following call to strcasecmp(3) in rr.c results in a NULL pointer dereference because 'rr->next_domain' ends up being NULL:

if (strcasecmp(rr->next_domain, zone_apex) == 0) {

Given those encouraging results, I went on to fuzz BIND, NSD and Knot zone parsers, using their zone validation tools named-checkzone, nsd-checkzone, and kzonecheck respectively.

While the fuzzers didn't produce any crash for BIND and Knot after running for 3 days and 11 hours, they did produce some valid ones for NSD, and I decided to continue on nsd-checkzone and stop the other fuzzers.

nsd-checkzone

I let AFL complete one cycle, and as I didn't need the box for anything else at this time, I decided to let it run for a few more days. I ended the process after 16 days and 19 hours, completing 2 cycles with 167 unique crashes.

nsd-checkzone

After sorting and analyzing the crashes, I had two valid issues to report.

The first one is an out-of-bounds read caused by improper validation of array index, in the rdata_maximum_wireformat_size() function, in rdata.c.

The second one is a stack-based buffer overflow in the dname_concatenate() function in dname.c, which got assigned CVE-2019-13207.

=================================================================
==7395==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffcd6a9763f at pc 0x0000004dadbc bp 0x7ffcd6a97510 sp 0x7ffcd6a96cc0
WRITE of size 8 at 0x7ffcd6a9763f thread T0
    #0 0x4dadbb in __asan_memcpy (/home/fcambus/nsd/nsd-checkzone+0x4dadbb)
    #1 0x534251 in dname_concatenate /home/fcambus/nsd/dname.c:464:2
    #2 0x69e61f in yyparse /home/fcambus/nsd/./zparser.y:1024:12
    #3 0x689fd1 in zonec_read /home/fcambus/nsd/zonec.c:1623:2
    #4 0x6aedd1 in check_zone /home/fcambus/nsd/nsd-checkzone.c:61:11
    #5 0x6aea07 in main /home/fcambus/nsd/nsd-checkzone.c:127:2
    #6 0x7fa60ece6b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
    #7 0x41c1d9 in _start (/home/fcambus/nsd/nsd-checkzone+0x41c1d9)

Address 0x7ffcd6a9763f is located in stack of thread T0 at offset 287 in frame
    #0 0x533f8f in dname_concatenate /home/fcambus/nsd/dname.c:458

  This frame has 1 object(s):
    [32, 287) 'temp' (line 459) <== Memory access at offset 287 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow (/home/fcambus/nsd/nsd-checkzone+0x4dadbb) in __asan_memcpy
Shadow bytes around the buggy address:
  0x10001ad4ae70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001ad4ae80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001ad4ae90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001ad4aea0: 00 00 00 00 f1 f1 f1 f1 00 00 00 00 00 00 00 00
  0x10001ad4aeb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x10001ad4aec0: 00 00 00 00 00 00 00[07]f3 f3 f3 f3 f3 f3 f3 f3
  0x10001ad4aed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001ad4aee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001ad4aef0: 00 00 00 00 f1 f1 f1 f1 00 00 00 00 00 00 00 00
  0x10001ad4af00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001ad4af10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==7395==ABORTING

Both issues have been fixed and will be part of the NSD 4.2.2 release.

I also fuzzed ldns using ldns-read-zone for 12 days and 7 hours but the only crashes it produced were in fact only triggering assertions.

It's been an interesting journey so far, and while finding issues is still relatively easy, time required to sort crashes and distinguish between valid, duplicates, and false positives takes a lot of time. Nonetheless, reading 3rd party source code and analyzing what is going on and why the program crashes is both very instructing and rewarding.

For the time being, I plan to continue fuzzing stuff and will write more about my findings.

July 09, 2019

Mark J. Nelson (mjn)

C.A.R. Hoare's distant connection to the C. Hoare & Co private bank July 09, 2019 12:00 PM

A recent article on UK private bank C. Hoare & Co. made the rounds, noting that it has been run for more than 300 years by the Hoare family. Somehow this found its way into tech discussion boards, which led some people to wonder whether pioneering British computer scientist C.A.R. "Tony" Hoare is from the same Hoare family.

I investigated briefly, and as it turns out the answer appears to be: yes, but with only a distant connection to the branch of the family running the bank, now five generations removed.

The hint for how to track this down came from a mention in the book Reflections on the Work of C.A.R. Hoare (Springer, 2010) that C.A.R. Hoare was of a "somewhat upper-class" background, with a footnote pointing to thepeerage.com, a site primarily focusing on genealogy of the British upper classes.

From this, I was able to trace his family history back along the paternal line as follows:

At this point the genealogy seems to connect to someone who has additional biographical details in Wikipedia, assuming that the Charles James Hoare is the same one described in this article (besides the same name, the approximate timeframe and occupations match). Wikipedia describes Charles James Hoare as "third son of Henry ('Harry') Hoare, banker, of Fleet Street, London, a Partner in C. Hoare & Co", so there's the connection to the banking family, at a great-great-great grandfather (five generations back) distance, around the late 1700s or early 1800s.

The generations after that appear unrelated to banking though: clergyman, clergyman, military officer, colonial civil servant, and finally professor. (The colonial-civil-servant occupation explains the somewhat well known fact that C.A.R. Hoare was born in Sri Lanka.)

Not that this distant connection is necessarily very important biographical information, but there you have it, assuming I haven't made an error in reading the sources, and assuming that they also haven't made any errors.

Tobias Pfeiffer (PragTob)

Looking for a job! July 09, 2019 08:26 AM

It’s that time: I’m looking for a job! You can find my CV and all relevant links over at my website: CV Web, CV PDF Quick links that might interest you: website, github, twitter and blog Who am I and why would you want to hire me? My name is Tobi, but online I’m better […]

Pete Corey (petecorey)

The Many Ways to Define Verbs in J July 09, 2019 12:00 AM

I recently watched an interesting 12tone video on the “Euler Gradus Suavitatis” which is a formula devised by Leonhard Euler to analytically compute the consonance, or translated literally, the “degree of pleasure” of a given musical interval.

In this formula, n can be any positive integer who’s prime factorization is notated as:

In reality, n can be any ratio of two or more positive integers. Simple intervalic ratios, like an octave, are represented as 1:2, where n would be 2. For more complicated intervals, like a perfect fifth (3:2), we need to find the least common multiple of both components (6) before passing it into our Gradus Suavitatis function. We can even do the same for entire chords, like a major triad (4:5:6), who’s least common multiple would be 60.

The Gradus Suavitatis in J

Intrigued, I decided to implement a version of the Gradus Suavitatis function using the J programming language.

Let’s start by representing our interval as a list of integers. Here we’ll start with a major triad:

   4 5 6
4 5 6

Next, we’ll find the least common multiple of our input:

   *./ 4 5 6
60

From there, we want to find the prime factors of our least common multiple:

   q: *./ 4 5 6
2 2 3 5

Since our primes are duplicated in our factorization, we can assume that their exponent is always 1. So we’ll subtract one from each prime, and multiply by one (which is a no-op):

   _1&+ q: *./ 4 5 6
1 1 2 4

Or even better:

   <: q: *./ 4 5 6
1 1 2 4

Finally, we sum the result and add one:

   1 + +/ <: q: *./ 4 5 6
9

And we get a Gradus Suavitatis, or “degree of pleasure” of 9 for our major triad. Interestingly, a minor triad (10:12:15) gives us the same value:

   1 + +/ <: q: *./ 10 12 15
9

Gradus Suavitatis as a Verb

To make playing with our new Gradus Suavitatis function easier, we should wrap it up into a verb.

One way of building a verb in J is to use the 3 : 0 or verb define construct, which lets us write a multi-line verb that accepts y and optionally x as arguments. We could write our Gradus Suavitatis using verb define:

   egs =. verb define
   1 + +/ <: q: *./ y
   )

While this is all well and good, I wasn’t happy that we needed three lines to represent our simple function as a verb.

Another option is to use caps ([:) to construct a verb train that effectively does the same thing as our simple chained computation:

   egs =. [: 1&+ [: +/ [: <: [: q: *./
   egs 4 5 6
9

However the use of so many caps makes this solution feel like we’re using the wrong tool for the job.

Raul on Twitter showed me the light and made me aware of the single-line verb definition syntax, which I somehow managed to skip over while reading the J Primer and J for C Programmers. Using the verb def construct, we can write our egs verb on a single line using our simple right to left calculation structure:

   egs =. verb def '1 + +/ <: q: *./ y'
   egs 4 5 6
9

It turns out that there’s over two dozen forms of “entity” construction in the J language. I wish I’d known about these sooner.

Regardless of how we define our verb, the Gradus Suavitatis is an interesting function to play with. Using the plot utility, we can plot the “degree of pleasure” of each of the interval ratios from 1:1 to 1:100 and beyond:

July 08, 2019

Derek Jones (derek-jones)

Complexity is a source of income in open source ecosystems July 08, 2019 10:19 PM

I am someone who regularly uses R, and my interest in programming languages means that on a semi-regular basis spend time reading blog posts about the language. Over the last year, or so, I had noticed several patterns of behavior, and after reading a recent blog post things started to make sense (the blog post gets a lot of things wrong, but more of that later).

What are the patterns that have caught my attention?

Some background: Hadley Wickham is the guy behind some very useful R packages. Hadley was an academic, and is now the chief scientist at RStudio, the company behind the R language specific IDE of the same name. As Hadley’s thinking about how to manipulate data has evolved, he has created new packages, and has been very prolific. The term Hadley-verse was coined to describe an approach to data manipulation and program structuring, based around use of packages written by the man.

For the last nine-months I have noticed that the term Tidyverse is being used more regularly to describe what had been the Hadley-verse. And???

Another thing that has become very noticeable, over the last six-months, is the extent to which a wide range of packages now have dependencies on packages in the HadleyTidyverse. And???

A recent post by Norman Matloff complains about the Tidyverse’s complexity (and about the consistency between its packages; which I had always thought was a good design principle), and how RStudio’s promotion of the Tidyverse could result in it becoming the dominant R world view. Matloff has an academic world view and misses what is going on.

RStudio, the company, need to sell their services (their IDE is clunky and will be wiped out if a top of the range product, such as Jetbrains, adds support for R). If R were simple to use, companies would have less need to hire external experts. A widely used complicated library of packages is a god-send for a company looking to sell R services.

I don’t think Hadley Wickam intentionally made things complicated, any more than the creators of the Microsoft server protocols added interdependencies to make life difficult for competitors.

A complex package ecosystem was probably not part of RStudio’s product vision, at least for many years. But sooner or later, RStudio management will have realised that simplicity and ease of use is not in their interest.

Once a collection of complicated packages exist, it is in RStudio’s interest to get as many other packages using them, as quickly as possible. Infect the host quickly, before anybody notices; all the while telling people how much the company is investing in the community that it cares about (making lots of money from).

Having this package ecosystem known as the Hadley-verse gives too much influence to one person, and makes it difficult to fire him later. Rebranding as the Tidyverse solves these problems.

Matloff accuses RStudio of monopoly behavior, I would have said they are fighting for survival (i.e., creating an environment capable of generating the kind of income a VC funded company is expected to make). Having worked in language environments where multiple, and incompatible, package ecosystems existed, I can see advantages in there being a monopoly. Matloff is also upset about a commercial company swooping in to steal their precious, a common academic complaint (academics swooping in to steal ideas from commercially developed software is, of course, perfectly respectable). Matloff also makes claims about teachability of programming that are not derived from any experimental evidence, but then everybody makes claims about programming languages without there being any experimental evidence.

RStudio management rode in on the data science wave, raising money from VCs. The wave is subsiding and they now need to appear to have a viable business (so they can be sold to a bigger fish), which means there has to be a visible market they can sell into. One way to sell in an open source environment is for things to be so complicated, that large companies will pay somebody to handle the complexity.

July 07, 2019

Andrew Owen (yumaikas)

The Kinship of Midnight Travel July 07, 2019 07:25 PM

There is a strange sort of kinship that I feel for other people I see when I’m traveling from one place to another after midnight. Part of it is that travel after everyone is supposed to be in bed means that I see things without all the people around, denuded of the hustle and bustle that often gives a place it’s charm or stress.

But then, spotting that lone trucker at 1 AM, or the lady working the airline check in counter starting at 3:30 AM, or the security guard at 4:30 AM, I know that I’m with people that also know what it means to see a place without it’s crowds. They also have seen the empty streets, the fog, the sprinklers running in the night. They know what it is to wake before the sun, or to chase the hours into the night.

Sometimes this happens in airports, today, it happened as I was driving from Carthage, MO to Joplin, MO. I saw a trucker pulling in off the highway onto one of the commercial districts in Joplin, and was reminded of all the times I’ve traveled in the dark, either at the end of long day in airports, or at the start of a long day on the road. And, I knew, that in some small way, that trucker also knew what it was like to travel at night.

For them, it likely has lost any of the romance or novelty that I still give it. But there’s still something to be said for that most remote of connections, if for no other reason that it’s just me and that other driver on the road, neither of us lost in the crowd, both wanting to reach a destination, yet still travelling in the night.

Thing is, I won’t ever meet that person. But we still shared a space, both members of the midnight traveling clan, a space that most people actively opt-out of. Many interesting bits of life are often found at the edges, where only a few people are paying attention, rather than in the rush of the crowds, where everyone already is, and has already seen them.

Ponylang (SeanTAllen)

Last Week in Pony - July 7, 2019 July 07, 2019 09:29 AM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

July 06, 2019

Jeff Carpenter (jeffcarp)

3 Tips for New Technical Interviewers July 06, 2019 10:00 PM

One year ago I conducted my first software engineering interview at Google. In that first interview I gave, I guarantee you I was more nervous than the candidate I was interviewing. Those first few interviews were particularly nerve-wracking. A lot was on the line—I didn’t want to screw up this person’s career by being a bad interviewer! Since then I’ve conducted a great deal more interviews and learned a lot about how to interview candidates successfully.

Ponylang (SeanTAllen)

0.29.0 Released July 06, 2019 02:32 PM

We are proud to announce the release of Pony version 0.29.0. This release includes several improvements to the standard library’s TCP support. This version contains breaking API changes. We recommend upgrading as soon as possible.

Jan van den Berg (j11g)

Thomas Dekker: The Descent (Mijn Gevecht) – Thijs Zonneveld July 06, 2019 08:35 AM

I finished this book in one sitting. Partly because Zonneveld has a pleasant writing style. But also because the rather recent story of a hugely talented and (very) young cyclist who early on in his career got involved with dope and raced towards destruction is fascinating.

Thomas Dekker: The Descent (Mijn Gevecht) – Thijs Zonneveld (2016) – 220 pages

It’s the (auto)biography of Thomas Dekker but it is just as much the biography of the cycling world in the early 2000s. And this world, as we now know, was rotten to the core. This book helped uncover parts of it when it came out in 2016. And many more books about this subject have come out since and around that time.

The book is telling and doesn’t hold back, for anything of anyone. Even Dekker himself doesn’t come across as a particular likeable character. Arrogant, cocky, egotistical and self-destructive to a fault. A very bright star who burned out VERY quickly.

He only did one Tour de France and his actual relevant career was only a few short years. The book came out three years ago, and even this year’s Tour de France will have riders older than Dekker is at the moment. So there is a sense of what could have been.

One important takeaway is the notion that using dope is a gradual (non-conscious) thing. Driven by ego and desire to win. But even more important is the notion that there is no such thing as a casual doper. You either dope or you don’t.

As a cycling enthusiast it’s not necessarily what you want to read. But it is what it is.

The post Thomas Dekker: The Descent (Mijn Gevecht) – Thijs Zonneveld appeared first on Jan van den Berg.

Slaughterhouse Five – Kurt Vonnegut July 06, 2019 08:33 AM

Slaughterhouse Five is a well-known classic. And I had been wanting to read it for quite some time now, and now that I finally did, I must say it was absolutely not what I expected.

In a good way.

Slaughterhouse Five – Kurt Vonnegut (1969) – 220 pages

The book is a sort of autobiographical non-chronological story about the bombing of Dresden, but it is also about time travel, space travel and aliens and different thoughts on philosophy. So yes, there is quite a lot to unpack in this wondrously written meta-fiction novel.

As a reader you have to work hard to keep up with all the time and place switches. But the fantastical and funny storytelling make that easy. (Vonnegut has a certain dry comedic style that I suspect people like Douglas Adams must have been inspired by.)

But even the — sometimes — nihilistic black humour can’t hide that Vonnegut is actually trying to tell or show the reader something. What that is, is largely up to the reader — and also what makes this a postmodern book. One of the things the book itself claims to be, is an anti-war book. And that is certainly also true. So it goes.

The post Slaughterhouse Five – Kurt Vonnegut appeared first on Jan van den Berg.

July 05, 2019

Simon Zelazny (pzel)

Plain text & un*x tools are still the best July 05, 2019 10:00 PM

I did some freelance work for a friendly company that had a problem with its production dataset. They had a Postgres database, running on a 1TB drive, holding historic data in a ~500GB table.

The historic data wasn't needed for anything aside from archival reasons, so the developers had tried — in between feature work — to remove the old rows while maintaining DB uptime. Unfortunately, all their previous attempts at clearing out obsolete data had failed.

Here's what they tried (on clones of the production VM):

  • DELETE FROM samples WHERE timestamp < 1234567890
  • VACUUM
  • VACUUM FULL
  • INSERT INTO new_samples SELECT * FROM samples WHERE (..), followed by TRUNCATE TABLE samples and rename

All of these experiments failed to inspire confidence, as they either filled up the available hard drive space with temporary data, or took so long that the potential downtime was deemed unacceptable (looking at you, VACUUM FULL). In particular, any operation that tried to rebuild the samples table afresh would fail. This was due to the fact that the disk was too small to hold two copies of the table, as dictated by the atomicity requirement of SQL operations.

After some experimentation, we determined that the fastest way to achieve our goals of deleting old data, slimming down index bloat, and reclaiming space for the OS, was to:

  1. export the entire DB to a text file of SQL statements (a.k.a a dump)
  2. remove the existing database completely
  3. clean up the dump in text form
  4. import the trimmed-down dump into a fresh database

This operation proved to be the fastest and most efficient in terms of disk-space required, although we were forced to incur downtime.

The joy of AWK

A concise AWK script was perfect for the job:

1.   BEGIN { in_samples=0; min_ts=1514764800; }
2.   /^\\.$/ { if (in_samples==1) { in_samples=0; } }
3.   // {
4.     if (in_samples) {
5.       if ($1 >= min_ts) { print } else { } ;
6.     } else {
7.       print
8.     }
9.   }
10.  /COPY public.samples/ { in_samples=1; }

In short, the script can be in 2 states:

in_samples := 0 | 1

It starts out in in_samples=0 (1.), and copies over every line of text (7.). If it finds the first line of the beginning of the samples table (10.), it switches to in_samples=1. In this state, it will only copy over samples that are NEWER than January 1, 2018 (5.). If it finds the end-of-table-data marker and is in_samples, it will exit this state (2.). Unless there are two tables in the dump named public.samples (there aren't), the script will never enter in_samples=1 again, and will simply copy over all other rows verbatim (line 7., again).

It's important to note that awk evaluates all clauses in order for every line of input (except for the special BEGIN and END clauses), so some lines of input might 'hit' several awk statements. This is beneficial, but also means that the order of the clauses is important. (Consider what would happen if the clause on line 10. had been placed before the clause on line 3.?)

Summing up

The entire operation of pg_dumpall, awk, and psql import took around 3 hours, which was acceptable downtime for this database as part of scheduled nighttime maintenance. The space taken up by the Postgres data directory went down from ~760GB to ~200GB.

That day I learned that plain text is still the lingua franca of UN*X, and that 40-year-old tools are still excellent at what they were built to do.

Leo Tindall (LeoLambda)

Speedy Desktop Apps With GTK and Rust July 05, 2019 09:00 PM

The web platform is the delivery mechanism of choice for a ton of software these days, either through the web browser itself or through Electron, but that doesn’t mean there isn’t a place for a good old fashioned straight-up desktop application in the picture. Fortunately, it’s easier than ever to write a usable, pretty, and performant desktop app, using my language of choice (Rust) and the wildly successful cross-platform GUI framework GTK.

July 04, 2019

Derek Jones (derek-jones)

How much is a 1-hour investment today worth a year from now? July 04, 2019 02:49 PM

Today, I am thinking of investing 1-hour of effort adding more comments to my code; how much time must this investment save me X-months from now, for today’s 1-hour investment to be worthwhile?

Obviously, I must save at least 1-hour. But, the purpose of making an investment is to receive a greater amount at a later time; ‘paying’ 1-hour to get back 1-hour is a poor investment (unless I have nothing else to do today, and I’m likely to be busy in the coming months).

The usual economic’s based answer is based on compound interest, the technique your bank uses to calculate how much you owe them (or perhaps they owe you), i.e., the expected future value grows exponentially at some interest rate.

Psychologists were surprised to find that people don’t estimate future value the way economists do. Hyperbolic discounting provides a good match to the data from experiments that asked subjects to value future payoffs. The form of the equation used by economists is: e^{-kD}, while hyperbolic discounting has the form 1/{1+kD}, where: k is a constant, and D the period of time.

The simple economic approach does not explicitly include the risk that one of the parties involved may cease to exist. Including risk is non-trivial, banks handle the risk that you might disappear by asking for collateral, or adding something to the interest rate charged.

The fact that humans, and some other animals, have been found to use hyperbolic discounting suggests that evolution has found this approach, to discounting time, increases the likelihood of genes being passed on to the next generation. A bird in the hand is worth two in the bush.

How do software developers discount investment in software engineering projects?

The paper Temporal Discounting in Technical Debt: How do Software Practitioners Discount the Future? describes a study that specifies a decision that has to be made and two options, as follows:

“You are managing an N-years project. You are ahead of schedule in the current iteration. You have to decide between two options on how to spend our upcoming week. Fill in the blank to indicate the least amount of time that would make you prefer Option 2 over Option 1.

  • Option 1: Implement a feature that is in the project backlog, scheduled for the next iteration. (five person days of effort).
  • Option 2: Integrate a new library (five person days effort) that adds no new functionality but has a 60% chance of saving you person days of effort over the duration of the project (with a 40% chance that the library will not result in those savings).

Subjects are then asked six questions, each having the following form (for various time frames):

“For a project time frame of 1 year, what is the smallest number of days that would make you prefer Option 2? ___”

The experiment is run twice, using professional developers from two companies, C1 and C2 (23 and 10 subjects, respectively), and the data is available for download :-)

The following plot shows normalised values given by some of the subjects from company C1, for the various time periods used (y-axis shows PresentValue/FutureValue). On a log scale, values estimated using the economists exponential approach would form a straight line (e.g., close to the first five points of subject M, bottom right), and values estimated using the hyperbolic approach would have the concave form seen for subject C (top middle) (code+data).

Normalised returned required for various elapsed years.

Subject B is asking for less, not more, over a longer time period (several other subjects have the same pattern of response). Why did Subject E (and most of subject G’s responses) not vary with time? Perhaps they were tired and were not willing to think hard about the problem, or perhaps they did not think the answer made much difference. The subjects from company C2 showed a greater amount of variety. Company C1 had some involvement with financial applications, while company C2 was involved in simulations. Did this domain knowledge spill over into company C1’s developers being more likely to give roughly consistent answers?

The experiment was run online, rather than an experimenter being in the room with subjects. It is possible that subjects would have invested more effort if a more formal setting, with an experimenter who had made the effort to be present. Also, if an experimenter had been present, it would have been possible to ask question to clarify any issues.

Both exponential and hyperbolic equations can be fitted to the data, but given the diversity of answers, it is difficult to put any weight in either regression model. Some subjects clearly gave responses fitting a hyperbolic equation, while others gave responses fitted approximately well by either approach, and other subjects used. It was possible to fit the combined data from all of company C1 subjects to a single hyperbolic equation model (the most significant between subject variation was the value of the intercept); no such luck with the data from company C2.

I’m very please to see there has been a replication of this study, but the current version of the paper is a jumble of ideas, and is thin on experimental procedure. I’m sure it will improve.

What do we learn from this study? Perhaps that developers need to learn something about calculating expected future payoffs.

Jeff Carpenter (jeffcarp)

Side Projects July 04, 2019 04:14 AM

Here are some side projects I’ve worked on. Running Pace Calculator 2019 Is an offline-enabled running pace calculator. I wrote about how I made it here. → Visit pacing.run JLPT Daily N3 Twitter bot 2013 to present I run a Twitter bot that tweets vocabulary for the JLPT N3 test. → Visit @jlptdaily3 on Twitter rcrd 2011 to present rcrd is my barebones app for doing quantified-self. → Visit rcrd.

July 02, 2019

Bogdan Popa (bogdan)

Announcing racket-sentry July 02, 2019 07:00 AM

I just released the first version of sentry, a Racket library that lets you capture exceptions using the Sentry API.

Jeff Carpenter (jeffcarp)

Brief: Machine Learning July 02, 2019 04:14 AM

Briefs are where I try condensing my notes on a subject into a concise list of explanations (Feyman Technique-style), each fewer than 250 words. 🚨 This brief is a work in progress. 🚨 Overview Types of Prediction Classification and regression. Types of Models Supervised is where you know all the labels beforehand, unsupervised is where you don’t. Semi-supervised is where you know some. Evaluating Models precision, recall NLP Text preprocessing: GloVe, BERT Binary classification A classification task where the model has 2 choices.

Andreas Zwinkau (qznc)

Definitions of Software Architecture July 02, 2019 12:00 AM

Software architecture documents the shared understanding of a software system.

Read full article!

July 01, 2019

Stig Brautaset (stig)

Speeding up tests on CircleCI for a Python Django project July 01, 2019 05:49 PM

I outline how we reduced the time to run a Django application's CI test suite from about 13 minutes to under 4 minutes.

Kevin Burke (kb)

Using a Bernzomatic TS8000 Kitchen Torch to Sear Meat July 01, 2019 04:43 AM

For a company trying to sell and explain a product, a lot of this information was amazingly difficult to find so I wrote this.

For the rest of this we'll assume you're cooking a steak but the same advice applies to most other meats/cuts.

Why Do You Want A Torch to Sear Meat

To get a good texture on a steak, you need to get the outside of the steak really hot. If you keep the steak on the heat source for too long, though, it will start to cook the inside too much, and you'll end up with a well done steak.

A torch helps with this problem by making it possible to sear the outside of the steak very quickly which means the outside can get a nice sear while the inside stays at medium rare.

What to Get

You want a torch that can char the surface of a steak in under a minute. Most of the torches can't actually get that hot!

I used a butane crème brûlée torch for years and would get frustrated my steaks didn't look like the ones on Serious Eats, unless I torched them for 5-6 minutes. The torch wasn't hot enough. Using a cast iron pan by itself, even on the highest setting is also not likely to get hot enough. If your steak does not look as black as the one in Serious Eats the outside is not getting hot enough.

I read the Wirecutter guide and got the Bernzomatic TS8000. This torch is definitely hot enough; with propane it's about 3x as hot as my crème brûlée torch. I used it tonight to sear a tri tip. It is so much better.

Is that it? (No, you need fuel)

No. The Bernzomatic torch is just a torch; you still need to buy fuel. The TS8000 works with either butane or propane. You don't want butane though, you want propane, because it gets 50% hotter than butane does.

Every photo on the Internet of both the TS8000 and propane fuel tanks either shows them connected or with the caps on. You need a propane tank that can connect to this, on the bottom of the TS8000.

The top of your propane canister should look like this, with the cap off:

Bernzomatic sells a compatible 14 oz propane tank. You can also use the green Coleman camping gas tanks. The advantage of the Coleman tank is it is cheap and your grocery/hardware store probably has them. It weighs 16 ounces full, which sounds heavy but, it's fine, it's what I used.

I tried to find a small tank - 5-10 ounces - but didn't find anything promising. You'll probably get something in the 12-16 ounce range. Just put it on the end of the torch and sear with the whole tank on the end of the torch.

Summary

  • Get the Bernzomatic TS8000, it is one of the few torches that gets hot enough

  • Use propane, not butane, with it

  • Coleman camping gas tanks are fine

That's it. Here is the manual for the TS8000. Happy searing!

Pete Corey (petecorey)

Building My Own Spacemacs July 01, 2019 12:00 AM

I switched from Vim to Spacemacs several years ago and have never looked back. At first, Spacemacs offered an amazingly polished and complete editing environment that I immediately fell in love with. But over the years my Spacemacs environment began to fall apart. My config grew stale, I found myself having to write clumsy custom layers to incorporate unsupported Emacs packages, and the background radiation of bugs and glitches started to become unignorable.

At the suggestion of Andrew on Twitter, I decided to ditch Spacemacs completely and roll my own Emacs configuration.

I couldn’t be happier with the result!

Before we dive into some of the details, here’s my entire Emacs init.el file at the time of posting this article. My current configuration is modeled after the tiny subset of Spacemacs I found myself using on a day-to-day basis. You can get an idea of my typical workflow by looking at my General key mappings.

I work on several projects throughout the day, so I use Projectile to manage operations within each of those projects. I use Perspective to isolate buffers between those projects. The projectile-persp-switch-project package acts as a nice bridge between the two projects:

"p" 'projectile-command-map
"pp" 'projectile-persp-switch-project
"pf" 'counsel-projectile

Spacemacs has the home-grown concept of a “layout” that lets you switch between numbered workspaces. I was worried I would miss this feature, as a common workflow for me was to hit SPC p l <workspace number> to switch between various workspaces. Thankfully, I’ve found that projectile-persp-switch-project works just as well if not better that Spacemac’s “layout” system. I hit SPC p p and then type one or two characters to narrow down onto the project I want. I no longer have to remember workspace numbers, and I don’t need to worry about exceeding ten workspaces.

When I’m not using counsel-projectile to find files within a project, I use deer to navigate directories:

"a" '(:ignore t :which-key "Applications")
"ar" 'ranger
"ad" 'deer

The only other external application I use is a terminal, but I use it so frequently I’ve mapped it to SPC ', just like Spacemacs:

"'" 'multi-term

I’ve also included a few cosmetic customizations to build out a distraction-free code editing environment.

As you can see, I have a fairly simple workflow, and my custom Emacs configuration reflects that simplicity. Not only is my configuration simpler, but I’ve noticed that startup times have reduced significantly, and the number of in-editor bugs and glitches I’ve encountered as dropped to zero. I’ve also developed a much deeper understanding of Emacs itself and the Emacs ecosystem.

I couldn’t be happier!

Andreas Zwinkau (qznc)

One Letter Programming Languages July 01, 2019 12:00 AM

If you are looking for a free name, there is none.

Read full article!

Robin Schroer (sulami)

My Thoughts on spec July 01, 2019 12:00 AM

In the beginning of this year I started a new job, and I am now fortunate enough to be writing Clojure full-time. I believe that Clojure is a very well crafted language and enables developers like few others, but I have also some grievances to report. I want to prefix this by saying that I love Clojure despite its faults, and this is more of a constructive criticism piece than anything else.

What Is spec?

What I want to discuss today is clojure.spec, the gradual typing solution shipped with Clojure since 1.9.I am of course aware that spec is officially in alpha, and that there is a second alpha version which might address some of my points here. But as I have not tried alpha 2 yet, and a lot of people are using alpha 1, we will be mostly looking at alpha 1 in this post. Still, feel free to contact me and tell me how alpha 2 will improve things.

In case you are not familiar with spec, here is a quick run-down of how it works:

Clojure is a dynamically typed language, meaning no (or very few) type annotations in the source code, unlike say Haskell or most curl-brace languages. While this removes a lot of visual clutter, and the language is designed in such a way that most functions can operate on most types for maximum flexibility, this also means that sometimes things break at run time due to type errors. To make dealing with types easier, spec allows you to place type definitions in strategic places, say at entry- and exit points of an app or a module to ensure that your idea of structure and type of your data still lines up with reality.

The benefit of this approach over static typing is that you can encapsulate parts of your software with type safety without having to deal with the encumbrance in the internals. You define a contract to the outside, but stay flexible in your implementation.

In practice it looks like this:

(ns spec-demo
  (:require [clojure.spec.alpha :as s]
            [clojure.spec.gen.alpha :as gen]))
            
(s/def ::name string?)
(s/def ::age pos-int?)

(s/def ::person
  (s/keys :req-un [::name
                   ::age]))
                
(s/valid? ::person
          {:name "John"
           :age 35})
;; => true

(s/explain ::person
           {:name 35})
;; => 35 - failed: string? in: [name] at: [:name] spec: :spec-demo/name
;;    {:name 35} - failed: (contains? % :age) spec: :spec-demo/person

(gen/generate
  (s/gen ::person))
;; => {:name "3o311eSXA4Dm9cLkENIt3J5Gb", :age 3318}

This snippet demonstrates creating a spec person which has sub-specs name and age, validates some data against the spec, explains why some data does not validate, and also generates some random data compliant with the spec. The last bit is incredibly useful for testing, as you can imagine. There are some more advanced features than what I have shown here, like specs for in- and output types of functions, but this is the gist of it.

So What Is Wrong with spec?

specs as Macros

spec is implemented using macros, which are compile-time expansions, think template meta-programming in C++. This brings some limitations, which are especially noticeable in Clojure where, like in most other Lisps, code is treated as data and many parts of your software are introspectable at run time. specs are, once defined, not easy to extract data from.

Contrast this with schema, a Clojure library which does essentially the same thing, and got started before spec. schema “schemas” are just hash maps, which are easily constructed and introspected at run time. schema has other problems, but the community seems to generally flock to spec as the officially blessed version.

The situation I have now repeatedly seen is one where I have a spec defining a specific representation of an entity, where for example one spec extends another one. A practical example here is when augmenting an entity with additional data, so that the output representation is a modified version of the input type.

You can define a spec as a superset of another spec, and only in that direction, using s/or, but at that point you are already stuck in the spec an cannot get the list of fields easily out of your specs. It would be nice to be able to define one spec in terms of flat data, which would also be available at run time. This would be incredibly helpful for use with something like select-keys.

There are ways of working around this, for example by using eval, or wrapping your spec macros in another layer of macros, but both of these are more hacks than solutions. spec alpha 2 promises to solve this particular problem by separating symbolic specs from spec objects, the former being just flat data which can also be generated programmatically.

If spec alpha 2 works out the way I hope, this will be solved soon, and the documentation looks promising so far.

Namespacing

spec is very intent on you using namespaces for your keywords, which in general is a good idea. Many developers, including myself, have been using the namespace of a key to give context about its use, for example :person/name is a different spec from :company/name. The problem with this is the overloading of namespaced keywords, which are an existing Clojure feature. These namespaces person and company do not actually exist, and I do not want to clutter all my keys with a long prefix like :myapp.entities.person/name to make it match a real namespace.

Now namespaced keywords can have any namespace, and the namespace does not actually have to exist for everything related to keywords to work well, except when it does. If you want to use a spec from another namespace, but alias that namespace to shorten it, you need to require that namespace, for which it needs to exist. As a workaround to this I have created “fake” namespaces in the past using a helper.

This actually leads me to another problem, which is the question of where to place specs in the first place. spec comes with a global, centralised registry for specs, which in alpha 1 you cannot opt out of. In theory this allows you to define/register your spec once in any place you like, and then access it from anywhere without having to know where it comes from or requiring it. This, while having the potential for being too opaque and magical, is actually a very good feature in my opinion. It is trivial to override specs when reloading them, and I have not experienced any issues with evaluation order yet. Due to specs referring to other specs by their name, you can define dependencies of a spec after the fact, and the system will pick them up accordingly.

My current solution for this is having a file & namespace for every entity which can be required and aliased normally, the only problem with this being relationships between specs. As soon as one spec needs to include another spec, dependencies get muddled, so I have experimented with having a special namespace for specs which are shared across many other specs, but this is far from ideal. I wish there was a cleaner way to do this, especially leveraging the global registry.

Function spec Definitions

I mentioned above that spec also allows instrumenting functions, but the semantics for this are a bit wonky in my opinion. See for yourself:

(defn double [x]
  (* 2 x))
  
(s/fdef double
  :args (s/cat :x int?)
  :ret int?
  :fn #(= (:ret %)
          (-> % :args :x (* 2))))

This naive spec restricts in- and output types to integers, which is okay in this case. The :fn key describes the relationship between in- and output, and is in this case actually absurdly strict, but this is just an example. There are two issues I have with this:

First the :fn definition tends to be very elaborate and hard to understand at a glance. Even in this simple case, there is a lot going on in there, and the use of anonymous functions does not help it. In practice, this key is optional and I omit it almost always, because I cannot think of any formal assertions I want to make about the output which are also reasonably simple to encode. And if you want to make several separate assertions about the output, you almost cannot avoid breaking up the spec into pieces, at which point you have predicates which exist purely for spec.

The other issue I have is that this is decoupled from the actual function definition. In theory you can place the spec in a different namespace and refer to the function using its fully qualified name, and this is tempting, especially when looking at my previous point about these specs having the potential to be far more longer than the actual function definitions. But then your function exists conceptually in two places, and these two places have to be kept in sync. If you move, rename, or modify the function in almost any way, you have to modify the spec, too, but first you have to find the spec.

The problem I can see with this is that :fn can be used to make assertions about the functions, which can in almost all cases be made in unit tests as well. In fact, unit tests are meant for exactly this, asserting single assumptions about units at a time. The condition above could just as well be a test called the result equals twice the input value. Functions are usually only instrumented locally and/or during testing, as they incur non-negligible overhead, and I would argue that they do not provide significant assurances over unit tests.

I would much rather drop the :fn key, and include type declarations in the actual function definition, which is incidentally how schema works:

(s/defn double :- s/Int
  [x :- s/Int]
  (* 2 x))

In the end, I am looking forward to the next iteration of spec, and hope it addresses as many issues as possible, and I am already blown away by how much better it is than alternatives available in other languages.

June 30, 2019

Derek Jones (derek-jones)

Medieval guilds: a tax collection bureaucracy June 30, 2019 11:45 PM

The medieval guild is sometimes held up as the template for an institution dedicated to maintaining high standards, and training the next generation of craftsmen.

“The European Guilds: An economic analysis” by Sheilagh Ogilvie takes a chainsaw (i.e., lots of data) to all the positive things that have been said about medieval guilds (apart from them being a money making machine for those on the inside).

Guilds manipulated markets (e.g., drove down the cost of input items they needed, and kept the prices they charged high), had little or no interest in quality, charged apprentices for what little training they received, restricted entry to their profession (based on the number of guild masters the local population could support in a manner expected by masters), and did not hesitate to use force to enforce the rules of the guild (should a member appear to threaten the livelihood of other guild members).

Guild wars is not the fiction of an online game, guilds did go to war with each other.

Given their focus on maximizing income, rather than providing customer benefits, why did guilds survive for so many centuries? Guilds paid out significant sums to influence those in power, i.e., bribes. Guilds paid annual sums for the exclusive rights to ply their trade in geographical areas; it’s all down on Vellum.

Guilds provided the bureaucracy needed to collect money from the populace, i.e., they were effectively tax collectors. Medieval rulers had a high turn-over, and most were not around long enough to establish a civil service. In later centuries, the growth of a country’s population led to the creation of government departments, that were stable enough to perform tax collecting duties more efficiently that guilds; it was the spread of governments capable of doing their own tax collecting that killed off guilds.

June 29, 2019

jfb (jfb)

Movie Review - A Simple Favor (2019) June 29, 2019 12:00 AM

Untitled code{white-space: pre-wrap;} span.smallcaps{font-variant: small-caps;} span.underline{text-decoration: underline;} div.column{display: inline-block; vertical-align: top; width: 50%;}

I didn’t really know anything about this, going in. It was tonally all over the map; the performances were good, it was competently put together. The pure aesthetics of watching Henry Golding and Blake Lively interact was pretty great; but at the end of the day, the plot was stupid, the pacing weird, and the tone veered too radically between farce and thriller to work for me.

Any time you “solve” the big mystery with a twin, you have failed, screenwriter. Sure, that might be the plot failing of the source material, but it’s still dumb of hell.

I give this movie two stars: one for the aforementioned aesthetics, and one for Anna Kendrick’s teeth.

Rating: ★ ★

June 28, 2019

Grzegorz Antoniak (dark_grimoire)

Enabling lldb on Mojave June 28, 2019 06:00 AM

I have an impression that each version of macOS system introduces additional obstacles for developer. From one point of view I understand that they just want to make the system more secure, and that should be the biggest priority above all (well, maybe not above common sense). But it's hard …

June 27, 2019

Gonçalo Valério (dethos)

8 useful dev dependencies for django projects June 27, 2019 11:15 PM

In this post I’m gonna list some very useful tools I often use when developing a Django project. These packages help me improve the development speed, write better code and also find/debug problems faster.

So lets start:

Black

This one is to avoid useless discussions about preferences and taste related to code formatting. Now I just simply install black and let it care of these matters, it doesn’t have any configurations (with one or two exceptions) and if your code does not have any syntax errors it will be automatically formatted according to a “style” that is reasonable.

Note: Many editors can be configured to automatically run black on every file save.

https://github.com/python/black

PyLint

Using a code linter (a kind of static analysis tool) is also very easy, can be integrated with your editor and allows you to catch many issues without even running your code, such as, missing imports, unused variables, missing parenthesis and other programming errors, etc. There are a few other In this case pylint does the job well and I never bothered to switch.

https://www.pylint.org/

Pytest

Python has a unit testing framework included in its standard library (unittest) that works great, however I found out that there is an external package that makes me more productive and my tests much more clear.

That package is pytest and once you learn the concepts it is a joy to work with. A nice extra is that it recognizes your older unittest tests and is able to execute them anyway, so no need to refactor the test suite to start using it.

https://docs.pytest.org/en/latest/

Pytest-django

This package, as the name indicates, adds the required support and some useful utilities to test your Django projects using pytest. With it instead of python manage.py test, you will execute just pytest like any other python project.

https://pytest-django.readthedocs.io

Django-debug-toolbar

Debug toolbar is a web panel added to your pages that lets you inspect your requests content, database queries, template generation, etc. It provides lots of useful information in order for the viewer to understand how the whole page rendering is behaving.

It can also be extended with other plugin that provide more specific information such as flamegraphs, HTML validators and other profilers.

https://django-debug-toolbar.readthedocs.io

Django-silk

If you are developing an API without any HTML pages rendered by Django, django-debug-toobar won’t provide much help, this is where django-silk shines in my humble opinion, it provides many of the same metrics and information on a separate page that can be inspected to debug problems and find performance bottlenecks.

https://github.com/jazzband/django-silk

Django-extensions

This package is kind of a collection of small scripts that provide common functionality that is frequently needed. It contains a set of management commands, such as shell_plus and runserver_plus that are improved versions of the default ones, database visualization tools, debugger tags for the templates, abstract model classes, etc.

https://django-extensions.readthedocs.io

Django-mail-panel

Finally, this one is an email panel for the django-debug-toolbar, that lets you inspect the sent emails while developing your website/webapp, this way you don’t have to configure another service to catch the emails or even read the messages on terminal with django.core.mail.backends.console.EmailBackend, which is not very useful if you are working with HTML templates.

https://github.com/scuml/django-mail-panel

Awn Umar (awn)

mutable strings in go June 27, 2019 12:00 AM

According to the Go language specification:

Strings are immutable: once created, it is impossible to change the contents of a string.

We’ll see about that.

data := []byte("yellow submarine")
str := *(*string)(unsafe.Pointer(&data))

for i := range data {
    data[i] = 'x'
    fmt.Printf("%s\n%s\n\n", string(data), str)
}

Try it out yourself.

June 24, 2019

Simon Zelazny (pzel)

Ruby apps under runit: notes to self June 24, 2019 10:00 PM

I was recently migrating nearly ten-year-old Ruby software from an Ubuntu 10:04 VM to something more recent. It took me way longer than I expected, due to two major snags.

Snag one: Bundler and per-user runsvdir

The apps in their original Lucid Lynx setting were run from a bash script that resembled this:

while (/bin/true); do
  (bundle exec 'thin -C config_foo.yml -R config.ru start' 2>&1)
  echo "Server died at `date`. Respawning.." >&2
  sleep 2
done

The app stayed up for years, so I guess this unsophisticated approach was good enough. For ease of deployment (and improved logging, see point #2 below), I decided to move all ruby web apps on this VPS to a user-local runit supervision tree.

In practice, this means that there's a /etc/service/webuser-sv directory, containing a runit service which launches a nested runsvdir program as the webuser-sv user.

$ cat /etc/service/webuser-sv
#!/bin/sh
exec 2>&1
exec chpst -uwebuser runsvdir /home/webuser/service

Now, I can define all of webuser's ruby apps as entries under /home/webuser/service/*, and have them supervised without bash hackery.

The snag was that the apps would crash with this error, but only when run as part of the runit supervision tree:

2019-06-25_11:27:37.51241 `/` is not writable.
2019-06-25_11:27:37.51244 Bundler will use `/tmp/bundler/home/unknown' as your home directory temporarily.

But if I ran the runit run scripts by hand, the apps started up and worked correctly.

After a lot of false starts and pointless github issue rabbit-holes, I realized that the runsvdir process managing the user-local supervision tree was launched with chpst by the master runit supervisor. Specifying a UID for chpst with -u does not automatically mean that the profile for this user gets loaded. In particular, not even $HOME was configured in the runit supervisor environment.

Bundler needs '$HOME' to be set, otherwise it gets confused.

Hence, my runit run files now look like this:

exec 2>&1
export RUBYOPT=-W0
export HOME=/home/web
export LANG=en_US.UTF-8
cd /home/web/webapp.1.com
exec bundle exec rackup -E production -s thin -o 0.0.0.0 -p 4567 config.ru

Duplication could be further removed by setting up an envdir, and running the user-level runsvdir with this envdir passed to chpst ... but the above solution is good enough for today.

Snag two: Don't ever redirect $stdin and $stderr

Runit has a wonderfully clean approach to logging, predating the 12-factor app, but very similar in spirit. Run your app in the foreground, and pipe its output to your log processor. This way applications themselves never need to concern themselves with logging frameworks and infrastructure.

Now ten years ago, when I launched these apps from bash scripts, they apps themselves definitely needed to know what to do with their logs. Hence, these two monstrous lines:

settings.configure do |s|
    $stdout = File.open("log/log_out.txt","a")
    $stderr = File.open("log/log_err.txt","a")

These two lines had me stumped as to why an app was silently crashing at startup with nothing in the runit logs. After removing the two lines I was able to see the real reason for the crash (some missing assets).

Sure, the exception had been logged all along to log/log_err.txt, but I'd completely forgotten to look there, expecting the logging to be handled by runit's log facility.

Never redirect $stdout and $stdin inside the app, kids. Your future self will thank you.

Chris Allen (bitemyapp)

Abusing instance local functional dependencies for nicer number literals June 24, 2019 12:00 AM

I demonstrated how to make kilobyte/megabyte constants in Haskell and a Lobsters user asked how it worked. This is a bloggification of my reply to them explaining the trick.

June 23, 2019

Ponylang (SeanTAllen)

Last Week in Pony - June 23, 2019 June 23, 2019 06:06 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

Chris Double (doublec)

Getting Started with Mercury June 23, 2019 11:00 AM

Mercury is a logic programming language, similar to Prolog, but with static types. It feels like a combination of SML and Prolog at times. It was designed to help with programming large systems - that is large programs, large teams and better reliability, etc. The commercial product Prince XML is written in Mercury.

I've played around with Mercury in the past but haven't done anything substantial with it. Recently I picked it up again. This post is a short introduction to building Mercury, and some example "Hello World" style programs to test the install.

Build

Mercury is written in the Mercury language itself. This means it needs a Mercury compiler to bootstrap from. The way I got a build going from source was to download the source for a release of the day version, build that, then use that build to build the Mercury source from github. The steps are outlined in the README.bootstrap file, but the following commands are the basic steps:

$ wget http://dl.mercurylang.org/rotd/mercury-srcdist-rotd-2019-06-22.tar.gz
$ tar xvf mercury-srcdist-rotd-2019-06-22.tar.gz
$ cd mercury-srcdist-rotd-2019-06-22
$ ./configure --enable-minimal-install --prefix=/tmp/mercury
$ make
$ make install
$ cd ..
$ export PATH=/tmp/mercury/bin:$PATH

With this minimal compiler the main source can be built. Mercury has a number of backends, called 'grades' in the documentation. Each of these grades makes a number of tradeoffs in terms of generated code. They define the platform (C, assembler, Java, etc), whether GC is used, what type of threading model is available (if any), etc. The Adventures in Mercury blog has an article on some of the different grades. Building all of them can take a long time - multiple hours - so it pays to limit it if you don't need some of the backends.

For my purposes I didn't need the CSharp backend, but wanted to explore the others. I was ok with the time tradeoff of building the system. To build from the master branch of the github repository I did the following steps:

$ git clone https://github.com/Mercury-Language/mercury
$ cd mercury
$ ./prepare.sh
$ ./configure --enable-nogc-grades --disable-csharp-grade \
              --prefix=/home/myuser/mercury
$ make PARALLEL=-j4
$ make install PARALLEL=-j4
$ export PATH=/home/myuser/mercury/bin:$PATH

Change the prefix to where you want Mercury installed. Add the relevant directories to the PATH as specified by the end of the build process.

Hello World

A basic "Hello World" program in Mercury looks like the following:

:- module hello.

:- interface.
:- import_module io.
:- pred main(io, io).
:- mode main(di, uo) is det.

:- implementation.
main(IO0, IO1) :-
    io.write_string("Hello World!\n", IO0, IO1).

With this code in a hello.m file, it can be built and run with:

$ mmc --make hello
Making Mercury/int3s/hello.int3
Making Mercury/ints/hello.int
Making Mercury/cs/hello.c
Making Mercury/os/hello.o
Making hello    
$ ./hello
Hello World!

The first line defines the name of the module:

:- module hello.

Following that is the definitions of the public interface of the module:

:- interface.
:- import_module io.
:- pred main(io, io).
:- mode main(di, uo) is det.

We publically import the io module, as we use io definitions in the main predicate. This is followed by a declaration of the interface of main - like C this is the user function called by the runtime to execute the program. The definition here declares that main is a predicate, it takes two arguments, of type io. This is a special type that represents the "state of the world" and is how I/O is handled in Mercury. The first argument is the "input world state" and the second argument is the "output world state". All I/O functions take these two arguments - the state of the world before the function and the state of the world after.

The mode line declares aspects of a predicate related to the logic programming side of things. In this case we declare that the two arguments passed to main have the "destructive input" mode and the "unique output" mode respectively. These modes operate similar to how linear types work in other languages, and the reference manual has a section describing them. For now the details can be ignored. The is det portion identifies the function as being deterministic. It always succeeds, doesn't backtrack and only has one result.

The remaining code is the implementation. In this case it's just the implementation of the main function:

main(IO0, IO1) :-
    io.write_string("Hello World!\n", IO0, IO1).

The two arguments to main, are the io types representing the before and after representation of the world. We call write_string to display a string, passing it the input world state, IO0 and receiving the new world state in IO1. If we wanted to call an additional output function we'd need to thread these variables, passing the obtained output state as the input to the new function, and receiving a new output state. For example:

main(IO0, IO1) :-
    io.write_string("Hello World!\n", IO0, IO1),
    io.write_string("Hello Again!\n", IO1, IO2).

This state threading can be tedious, especially when refactoring - the need to renumber or rename variables is a pain point. Mercury has syntactic sugar for this called state variables, enabling this function to be written like this:

main(!IO) :-
    io.write_string("Hello World!\n", !IO),
    io.write_string("Hello Again!\n", !IO).

When the compiler sees !Variable_name in an argument list it creates two arguments with automatically generated names as needed.

Another syntactic short cut can be done in the pred and mode lines. They can be combined into one line that looks like:

:- pred main(io::di, io::uo) is det.

Here the modes di and uo are appended to the type prefixed with a ::. The resulting program looks like:

- module hello.

:- interface.
:- import_module io.
:- pred main(io::di, io::uo) is det.

:- implementation.
main(!IO) :-
    io.write_string("Hello World!\n", !IO),
    io.write_string("Hello Again!\n", !IO).

Factorial

The following is an implementation of factorial:

:- module fact.

:- interface.
:- import_module io.
:- pred main(io::di, io::uo) is det.

:- implementation.
:- import_module int.
:- pred fact(int::in, int::out) is det.

fact(N, X) :-
  ( N = 1 -> X = 1 ; fact(N - 1, X0), X = N * X0 ).

main(!IO) :-
  fact(5, X),
  io.print("fact(5, ", !IO),
  io.print(X, !IO),
  io.print(")\n", !IO).

In the implementation section here we import the int module to access functions across machine integers. The fact predicate is declared to take two arguments, both of type int, the first an input argument and the second an output argument.

The definition of fact uses Prolog syntax for an if/then statement. It states that if N is 1 then (the -> token) the output variable, X is 1. Otherwise (the ; token), calculate the factorial recursively using an intermediate variable X0 to hold the temporary result.

There's a few other ways this could be written. Instead of the Prolog style if/then, we can use an if/then syntax that Mercury has:

fact(N, X) :-
  ( if N = 1 then X = 1 else fact(N - 1, X0), X = N * X0 ).

Instead of using predicates we can declare fact to be a function. A function has no output variables, instead it returns a result just like functions in standard functional programming languages. The changes for this are to declare it as a function:

:- func fact(int) = int.
fact(N) = X :-
  ( if N = 1 then X = 1 else X = N * fact(N - 1) ).

main(!IO) :-
  io.print("fact(5, ", !IO),
  io.print(fact(5), !IO),
  io.print(")\n", !IO)

Notice now that the call to fact looks like a standard function call and is inlined into the print call in main. A final syntactic shortening of function implementations enables removing the X return variable name and returning directly:

fact(N) = (if N = 1 then 1 else N * fact(N - 1)).

Because this implementation uses machine integers it won't work for values that can overflow. Mercury comes with an arbitrary precision integer module, integer, that allows larger factorials. Replacing the use of the int module with integer and converting the static integer numbers is all that is needed:

:- module fact.

:- interface.
:- import_module io.
:- pred main(io::di, io::uo) is det.

:- implementation.
:- import_module integer.
:- func fact(integer) = integer.

fact(N) = (if N = one then one else N * fact(N - one)).

main(!IO) :-
  io.print("fact(1000, ", !IO),
  io.print(fact(integer(1000)), !IO),
  io.print(")\n", !IO).

Conclusion

There's a lot more to Mercury. These are just first steps to test the system works. I'll write more about it in later posts. Some further reading:

Bit Cannon (wezm)

Announcing Desktop Institute June 23, 2019 05:42 AM

Since publishing, A Tiling Desktop Environment, I’ve continued to think about the topic, absorb the comments I received, try out some of the suggestions, and poke around the code bases of some existing window managers and Wayland compositors.

This weekend I set up a new website to document the thinking and research I’ve been doing. It’s called Desktop Institute, and has a fun domain: desktop.institute. Check it out for a more info on what I have planned as well as a roadmap for future posts.

June 21, 2019

Indrek Lasn (indreklasn)

How To Fetch Data From An API With React Hooks June 21, 2019 01:18 PM

React hooks let us write pure functional components without ever using the class syntax. Usually, but not always, the less code we have to…

June 20, 2019

Indrek Lasn (indreklasn)

How did you get this title formatting? June 20, 2019 06:59 PM

How did you get this title formatting?

How To Use Redux with React Hooks June 20, 2019 10:44 AM

React Redux released hooks with the 7.1.0 version. This means we get to use the latest best practices with React.

Marc Brooker (mjb)

When Redundancy Actually Helps June 20, 2019 12:00 AM

When Redundancy Actually Helps

Redundancy can harm more than it helps.

Just after I joined the EBS team at AWS in 2011, the service suffered a major disruption lasting more than two days to full recovery. Recently, on Twitter, Andrew Certain said:

We were super dependent on having a highly available network to make the replication work, so having two NICs and a second network fabric seemed to be a way to improve availability. But the lesson of this event is that only some forms of redundancy improve availability.

I've been thinking about the second part of that a lot recently, as my team starts building a new replicated system. When does redundancy actually help availability? I've been breaking that down into four rules:

  1. The complexity added by introducing redundancy mustn't cost more availability than it adds.
  2. The system must be able to run in degraded mode.
  3. The system must reliably detect which of the redundant components are healthy and which are unhealthy.
  4. The system must be able to return to fully redundant mode.

This might seem like obvious, even tautological, but each serves as the trigger of deeper thinking and conversation.

Don't add more risk than you take away

Andrew (or Kerry Lee, I'm not sure which) introduced this to the EBS team as don't be weird.

This isn't a comment on people (who are more than welcome to be weird), but on systems. Weirdness and complexity add risk, both risk that we don't understand the system that we're building, and risk that we don't understand the system that we are operating. When adding redundancy to a system, it's easy to fall into the mistake of adding too much complexity, and underestimating the ways in which that complexity adds risk.

You must be able to run in degraded mode

Once you've failed over to the redundant component, are you sure it's going to be able to take the load? Even in one of the simplest cases, active-passive database failover, this is a complex question. You're going from warm caches and full buffers to cold caches and empty buffers. Performance can differ significantly.

As systems get larger and more complex, the problem gets more difficult. What components do you expect to fail? How many at a time? How much traffic can each component handle? How do we stop our cost reduction and efficiency efforts from taking away the capacity needed to handle failures? How do we continuously test that the failover works? What mechanism do we have to make sure there's enough failover capacity? There's typically at least as much investment in answering these questions as building the redundant system in the first place.

Chaos testing, gamedays, and other similar approaches are very useful here, but typically can't test the biggest failure cases in a continuous way.

You've got to fail over in the right direction

When systems suffer partial failure, it's often hard to tell what's healthy and what's unhealthy. In fact, different systems in different parts of the network often completely disagree on health. If your system sees partial failure and fails over towards the truly unhealthy side, you're in trouble. The complexity here comes from the distributed systems fog of war: telling the difference between bad networks, bad software, bad disks, and bad NICs can be surprisingly hard. Often, systems flap a bit before falling over.

The system must be able to return to fully redundant mode

If your redundancy is a single shot, it's not going to add much availability in the long term. So you need to make sure the system can safely get from one to two, or N to N+1, or N to 2N. This is relatively easy in some kinds of systems, but anything with a non-zero RPO or asynchronous replication or periodic backups can make it extremely difficult. In small systems, human judgement can help. In larger systems, you need an automated plan. Most likely, you're going to make a better automated plan during daylight in the middle of the week during your design phase than at 3AM on a Saturday while trying to fix the outage.

June 19, 2019

Derek Jones (derek-jones)

A zero-knowledge proofs workshop June 19, 2019 11:33 PM

I was at the Zero-Knowledge proofs workshop run by BinaryDistict on Monday and Tuesday. The workshop runs all week, but is mostly hacking for the remaining days (hacking would be interesting if I had a problem to code, more about this at the end).

Zero-knowledge proofs allow person A to convince person B, that A knows the value of x, without revealing the value of x. There are two kinds of zero-knowledge proofs: an interacting proof system involves a sequence of messages being exchanged between the two parties, and in non-interactive systems (the primary focus of the workshop), there is no interaction.

The example usually given, of a zero-knowledge proof, involves Peggy and Victor. Peggy wants to convince Victor that she knows how to unlock the door dividing a looping path through a tunnel in a cave.

The ‘proof’ involves Peggy walking, unseen by Victor, down path A or B (see diagram below; image from Wikipedia). Once Peggy is out of view, Victor randomly shouts out A or B; Peggy then has to walk out of the tunnel using the path Victor shouted; there is a 50% chance that Peggy happened to choose the path selected by Victor. The proof is iterative; at the end of each iteration, Victor’s uncertainty of Peggy’s claim of being able to open the door is reduced by 50%. Victor has to iterate until he is sufficiently satisfied that Peggy knows how to open the door.

Alibaba example cave loop.

As the name suggests, non-interactive proofs do not involve any message passing; in the common reference string model, a string of symbols, generated by person making the claim of knowledge, is encoded in such a way that it can be used by third-parties to verify the claim of knowledge. At the workshop we got an overview of zk-SNARKs (zero-knowledge succinct non-interactive argument of knowledge).

The ‘succinct’ component of zk-SNARK is what has made this approach practical. When non-interactive proofs were first proposed, the arguments of knowledge contained around one-terabyte of data; these days common reference strings are around a kilobyte.

The fact that zero-knowledge ‘proof’s are possible is very interesting, but do they have practical uses?

The hackathon aspect of the workshop was designed to address the practical use issue. The existing zero-knowledge proofs tend to involve the use of prime numbers, or the factors of very large numbers (as might be expected of a proof system that is heavily based on cryptographic techniques). Making use of zero-knowledge proofs requires mapping the problem to a form that has a known solution; this is very hard. Existing applications involve cryptography and block-chains (Zcash is a cryptocurrency that has an option that provides privacy via zero-knowledge proofs), both heavy users of number theory.

The workshop introduced us to two languages, which could be used for writing zero-knowledge applications; ZoKrates and snarky. The weekend before the workshop, I tried to install both languages: ZoKrates installed quickly and painlessly, while I could not get snarky installed (I was told that the first two hours of the snarky workshop were spent getting installs to work); I also noticed that ZoKrates had greater presence than snarky on the web, in the form of pages discussing the language. It seemed to me that ZoKrates was the market leader. The workshop presenters included people involved with both languages; Jacob Eberhardt (one of the people behind ZoKrates) gave a great presentation, and had good slides. Team ZoKrates is clearly the one to watch.

As an experienced hack attendee, I was ready with an interesting problem to solve. After I explained the problem to those opting to use ZoKrates, somebody suggested that oblivious transfer could be used to solve my problem (and indeed, 1-out-of-n oblivious transfer does offer the required functionality).

My problem was: Let’s say I have three software products, the customer has a copy of all three products, and is willing to pay the license fee to use one of these products. However, the customer does not want me to know which of the three products they are using. How can I send them a product specific license key, without knowing which product they are going to use? Oblivious transfer involves a sequence of message exchanges (each exchange involves three messages, one for each product) with the final exchange requiring that I send three messages, each containing a separate product key (one for each product); the customer can only successfully decode the product-specific message they had selected earlier in the process (decoding the other two messages produces random characters, i.e., no product key).

Like most hackathons, problem ideas were somewhat contrived (a few people wanted to delve further into the technical details). I could not find an interesting team to join, and left them to it for the rest of the week.

There were 50-60 people on the first day, and 30-40 on the second. Many of the people I spoke to were recent graduates, and half of the speakers were doing or had just completed PhDs; the field is completely new. If zero-knowledge proofs take off, decisions made over the next year or two by the people at this workshop will impact the path the field follows. Otherwise, nothing happens, and a bunch of people will have interesting memories about stuff they dabbled in, when young.

Indrek Lasn (indreklasn)

I reckon those are for testing purposes only. June 19, 2019 05:43 PM

I reckon those are for testing purposes only. Check out the Move programming section to programmatically handle transactions:

https://developers.libra.org/docs/move-overview#move-transaction-scripts-enable-programmable-transactions

I’m Giving Out My Best Business Ideas Hoping Someone Will Build Them — Part II June 19, 2019 02:36 PM

This is a series where I post ideas and problems I want to see solved. The ideas range from tech to anything. The main idea behind the…

June 18, 2019

Indrek Lasn (indreklasn)

Thanks, Madhu. I love exploring and writing about cool new tech. Keep on rockin’ June 18, 2019 08:55 PM

Thanks, Madhu. I love exploring and writing about cool new tech. Keep on rockin’

Getting started with the Facebook Libra programming language June 18, 2019 03:06 PM

Facebook revealed it&aposs new global cryptocurrency and programming environment called Libra. Libra will let you buy things or send money to…

June 17, 2019

Indrek Lasn (indreklasn)

I’m Giving Out My Best Business Ideas Hoping Someone Will Build Them June 17, 2019 09:07 PM

Yup, I’m giving out some of my best ideas. I’m not a saint, but I definitely belong to the group of doing things that matter and create a…

Bogdan Popa (bogdan)

racket/gui saves the day June 17, 2019 07:00 AM

Yesterday, I bought an icon pack containing over 3,000 (!) SVG files and macOS utterly failed me when I tried to search the unarchived folder.

empty search screen

So I did what any self-respecting Racketeer would do. I used this as an excuse to play around with Racket’s built-in GUI library!

the final product

Marc Brooker (mjb)

Is Anatoly Dyatlov to blame? June 17, 2019 12:00 AM

Is Anatoly Dyatlov to blame?

Without a good safety culture, operators are bound to fail.

(Spoiler warning: containers spoilers for the HBO series Chernobyl, and for history).

Recently, I enjoyed watching HBO's new series Chernobyl. Like everybody else on the internet, I have some thoughts about it. I'm not a nuclear physicist or engineer, but I do think a lot about safety and the role of operators.

The show tells the story of the accident at Chernobyl in April 1986, the tragic human impact, and the cleanup and investigation in the years that followed. One of the villains in the show is Anatoly Dyatlov, the deputy chief engineer of the plant. Dyatlov was present in the control room of reactor 4 when it exploded, and received a huge dose of radiation (the second, or perhaps third, large dose in his storied life of being near reactor accidents). HBO's portrayal of Dyatlov is of an arrogant and aggressive man whose refusal to listen to operators was a major cause of the accident. Some first-hand accounts agree2, 3, 6, and others disagree1. Either way, Dyatlov spent over three years in prison for his role in the accident.

There's little debate that the reactor's design was deeply flawed. The International Nuclear Safety Advisory Group (INSAG) found4 that certain features of the reactor "had a primary influence on the course of the accident and its consequences". During the time before the accident, operators had put the reactor into a mode where it was unstable, with reactivity increases leading to higher temperatures, and further reactivity increases. The IAEA (and Russian scientists) also found that the design of the control rods was flawed, both in that they initially increased (rather than decreasing) reactivity when first inserted, and in that they machinery to insert them moved too slowly. They also found issues with the control systems, cooling systems, and the fact that some critical safety measures could be manually disabled. Authorities had been aware of many of these issues since an accident at the Ignalina plant in 19834, page 13, but no major design or operational practice changes had been made by the time of the explosion in 1986.

In the HBO series' telling of the last few minutes before the event, Dyatlov was shown to dismiss concerns from his team that the reactor shouldn't be run for long periods of time at low power. Initially, Soviet authorities claimed that the dangers of doing this was made clear to operators (and Dyatlov ignored procedures). Later investigations by IAEA found no evidence that running the reactor in this dangerous mode was forbidden 4, page 11. The same is true of other flaws in the plant. Operators weren't clearly told that pushing the emergency shutdown (aka SCRAM, aka AZ-5) button could temporarily increase the reaction rate in some parts of the reactor. The IAEA also found that the reactors were safe in "steady state", and the accident would not have occurred without the actions of operators.

Who is to blame for the 1986 explosion at Chernobyl?

In 1995, Dyatlov wrote an article in which he criticized both the Soviet and IAEA investigations5, and asked a powerful question:

How and why should the operators have compensated for design errors they did not know about?

If operators make mistakes while operating systems which have flaws they don't know about, is that "human error"? Does it matter if their ignorance of those flaws is because of their own inexperience, bureaucratic incompetence, or some vast KGB-lead conspiracy? Did Dyatlov deserve death for his role in the accident, as the series suggests? As Richard Cook says in "How Complex Systems Fail"7:

Catastrophe requires multiple failures – single point failures are not enough. ... Each of these small failures is necessary to cause catastrophe but only the combination is sufficient to permit failure.

And

After accidents, the overt failure often appears to have been inevitable and the practitioner’s actions as blunders or deliberate willful disregard of certain impending failure. ... That practitioner actions are gambles appears clear after accidents; in general, post hoc analysis regards these gambles as poor ones. But the converse: that successful outcomes are also the result of gambles; is not widely appreciated.

If Dyatlov and the other operators of the plant had known about the design issues with the reactor that had been investigated following the accident at Ignalina in 1983, would they have made the same mistake? It's hard to believe they would have. If the reactor design had been improved following the same accident, would the catastrophe had occurred? The consensus seems to be that it wouldn't have, and if it did then it would have taken a different form.

From "How Complex Systems Fail":

Most initial failure trajectories are blocked by designed system safety components. Trajectories that reach the operational level are mostly blocked, usually by practitioners.

The show's focus on the failures of practitioners to block the catastrophe, and maybe on their unintentional triggering of the catastrophe seems unfortunate to me. The operators - despite their personal failings - had not been set up for success, either by arming them with the right knowledge, or by giving them the right incentives.

From my perspective, the show is spot-on in it's treatment of the "cost of lies". Lies, and the incentive to lie, almost make it impossible to build a good safety culture. But not lying is not enough. A successful culture needs to find the truth, and then actively use it to both improve the system and empower operators. Until the culture can do that, we shouldn't be surprised when operators blunder or even bluster their way into disaster.

Footnotes

  1. BBC, Chernobyl survivors assess fact and fiction in TV series, 2019
  2. Svetlana Alexievich, "Voices from Chernobyl".
  3. Serhii Plokhy, "Chernobyl: The History of a Nuclear Catastrophe". This is my favorite book about the disaster (I've probably read over 20 books on it), covering a good breadth of history and people without being too dramatic. There are a couple of minor errors in the book (like confusing GW and GWh in multiple places), but those can be overlooked.
  4. INSAG-7 The Chernobyl Accident: Updating of INSAG-1, IAEA, 1992
  5. Anatoly Dyatlov, Why INSAG has still got it wrong, NEI, 1995
  6. Adam Higginbotham, "Midnight in Chernobyl: The Untold Story of the World's Greatest Nuclear Disaster"
  7. Richard Cook, How Complex Systems Fail

June 16, 2019

Derek Jones (derek-jones)

Lehman ‘laws’ of software evolution June 16, 2019 09:32 PM

The so called Lehman laws of software evolution originated in a 1968 study, and evolved during the 1970s; the book “Program Evolution: processes of software change” by Lehman and Belady was published in 1985.

The original work was based on measurements of OS/360, IBM’s flagship operating system for the computer industries flagship computer. IBM dominated the computer industry from the 1950s, through to the early 1980s; OS/360 was the Microsoft Windows, Android, and iOS of its day (in fact, it had more developer mind share than any of these operating systems).

In its day, the Lehman dataset not only bathed in reflected OS/360 developer mind-share, it was the only public dataset of its kind. But today, this dataset wouldn’t get a second look. Why? Because it contains just 19 measurement points, specifying: release date, number of modules, fraction of modules changed since the last release, number of statements, and number of components (I’m guessing these are high level programs or interfaces). Some of the OS/360 data is plotted in graphs appearing in early papers, and can be extracted; some of the graphs contain 18, rather than 19, points, and some of the values are not consistent between plots (extracted data); in later papers Lehman does point out that no statistical analysis of the data appears in his work (the purpose of the plots appears to be decorative, some papers don’t contain them).

One of Lehman’s early papers says that “… conclusions are based, comes from systems ranging in age from 3 to 10 years and having been made available to users in from ten to over fifty releases.“, but no other details are given. A 1997 paper lists module sizes for 21 releases of a financial transaction system.

Lehman’s ‘laws’ started out as a handful of observations about one very large software development project. Over time ‘laws’ have been added, deleted and modified; the Wikipedia page lists the ‘laws’ from the 1997 paper, Lehman retired from research in 2002.

The Lehman ‘laws’ of software evolution are still widely cited by academic researchers, almost 50-years later. Why is this? The two main reasons are: the ‘laws’ are sufficiently vague that it’s difficult to prove them wrong, and Lehman made a large investment in marketing these ‘laws (e.g., publishing lots of papers discussing these ‘laws’, and supervising PhD students who researched software evolution).

The Lehman ‘laws’ are not useful, in the sense that they cannot be used to make predictions; they apply to large systems that grow steadily (i.e., the kind of systems originally studied), and so don’t apply to some systems, that are completely rewritten. These ‘laws’ are really an indication that software engineering research has been in a state of limbo for many decades.

Indrek Lasn (indreklasn)

Demystifying React Hooks June 16, 2019 11:33 AM

You probably heard about the new concept for React called hooks. Hooks were released in React version 16.8 and they let us write stateful…

June 15, 2019

Stig Brautaset (stig)

Digital Minimalism June 15, 2019 02:07 PM

I introduce Cal Newport's book, and how it's helping me take control of where I spend my limited currency in today's attention economy.

June 14, 2019

Jeff Carpenter (jeffcarp)

On Being Injured (Again) June 14, 2019 09:55 PM

tl;dr: I signed up for the SF Marathon (this would have been my first marathon), then overtrained, got injured, and am currently recovering. I’m probably going to defer my registration to 2020 and become a cheering squad this year. (╯°□°)╯︵ ┻━┻ This is a cycle I’ve been through over and over again. I literally wrote about this in 2015. Being injured massively sucks. I can’t exercise the way I usually do, and I don’t get see my running buddies.

Indrek Lasn (indreklasn)

How to set up a powerful API with GraphQL, Koa, and MongoDB — deploying to production June 14, 2019 07:58 PM

Our GraphQL runs smoothly locally, but what if we want to share it with the world?

June 13, 2019

Nikita Voloboev (nikivi)

Indrek Lasn (indreklasn)

Working on a startup is frustrating and here’s how you can combat it June 13, 2019 08:15 AM

There’s no doubt starting a new startup can feel lonely, depressive, and frustrating. That’s mostly because we have no idea where we will…

June 12, 2019

Indrek Lasn (indreklasn)

4 Daily habits successfully happy people have June 12, 2019 10:23 AM

What makes successful people happy? Some might say wealth, money equals happiness, right?

June 09, 2019

Ponylang (SeanTAllen)

Last Week in Pony - June 9, 2019 June 09, 2019 08:39 AM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

June 08, 2019

Bogdan Popa (bogdan)

Announcing marionette June 08, 2019 12:45 PM

I just released the first version of marionette (named after the protocol it implements), a Racket library that lets you remotely control the Firefox web browser. Think “puppeteer”, but for Firefox.

Derek Jones (derek-jones)

PCTE: a vestige of a bygone era of ISO standards June 08, 2019 02:24 AM

The letters PCTE (Portable Common Tool Environment) might stir vague memories, for some readers. Don’t bother checking Wikipedia, there is no article covering this PCTE (although it is listed on the PCTE acronym page).

The ISO/IEC Standard 13719 Information technology — Portable common tool environment (PCTE) —, along with its three parts, has reached its 5-yearly renewal time.

The PCTE standard, in itself, is not interesting; as far as I know it was dead on arrival. What is interesting is the mindset, from a bygone era, that thought such a standard was a good idea; and, the continuing survival of a dead on arrival standard sheds an interesting light on ISO standards in the 21st century.

PCTE came out of the European Union’s first ESPRIT project, which ran from 1984 to 1989. Dedicated workstations for software developers were all the rage (no, not those toy microprocessor-based thingies, but big beefy machines with 15inch displays, and over a megabyte of memory), and computer-aided software engineering (CASE) tools were going to provide a huge productivity boost.

PCTE is a specification for a tool interface, i.e., an interface whereby competing CASE tools could provide data interoperability. The promise of CASE tools never materialized, and they faded away, removing the need for an interface standard.

CASE tools and PCTE are from an era where lots of managers still thought that factory production methods could be applied to software development.

PCTE was a European funded project coordinated by a (at the time) mainframe manufacturer. Big is beautiful, and specifications with clout are ISO standards (ECMA was used to fast track the document).

At the time Ada was the language that everybody was going to be writing in the future; so, of course, there is an Ada binding (there is also a C one, cannot ignore reality too much).

Why is there still an ISO standard for PCTE? All standards are reviewed every 5-years, countries have to vote to keep them, or not, or abstain. How has this standard managed to ‘live’ so long?

One explanation is that by being dead on arrival, PCTE never got the chance to annoy anybody, and nobody got to know anything about it. Standard’s committees tend to be content to leave things as they are; it would be impolite to vote to remove a document from the list of approved standards, without knowing anything about the subject area covered.

The members of IST/5, the British Standards committee responsible (yes, it falls within programming languages), know they know nothing about PCTE (and that its usage is likely to be rare to non-existent) could vote ABSTAIN. However, some member countries of SC22 might vote YES, because while they know they know nothing about PCTE, they probably know nothing about most of the documents, and a YES vote does not require any explanation (no, I am not suggesting some countries have joined SC22 to create a reason for flunkies to spend government money on international travel).

Prior to the Internet, ISO standards were only available in printed form. National standards bodies were required to hold printed copies of ISO standards, ready for when an order to arrive. When a standard having zero sales in the last 5-years, came up for review a pleasant person might show up at the IST/5 meeting (or have a quiet word with the chairman beforehand); did we really want to vote to keep this document as a standard? Just think of the shelf space (I never heard them mention the children dead trees). Now they have pdfs occupying rotating rust.

June 06, 2019

Indrek Lasn (indreklasn)

Absolutely. I recommend doing the dependency updates monthly. June 06, 2019 09:59 AM

Absolutely. I recommend doing the dependency updates monthly.

June 05, 2019

Jeff Carpenter (jeffcarp)

Setting Up a Recruiter Auto-reply Bot June 05, 2019 08:16 PM

If you’re a software engineer, you’re likely familiar with unsolicited emails from recruiters. Most are probably template emails. Some of them are funny, some are thoughtful, and some of them ask you to move 3000 miles, take a 50% pay cut, and code in a language you don’t know. Impact Recruiter emails have a measurable impact on productivity. If I were to hand-write a response to each one (taking 2 minutes), and I got 1 recruiter email a day, that’s 12 hours of work, or more than one full work day of each year… gone.

June 04, 2019

Jan van den Berg (j11g)

The Fall (De Val) – Matthias M.R. Declercq June 04, 2019 06:07 PM

Matthias M.R. Declercq pulled of two remarkable things. Not only did he manage to find this extraordinary story about friendship, ambition and sacrifice, he was also able to write it down in exceptional fashion.

De Val – Matthias M.R. Declercq (2017) – 296 paginas

The events described in ‘The Fall’ (‘De Val’) are real, but the book is not necessarily a biography. The story revolves around a group of five Belgian riders (flandriens) who are pretty well known in the cycling circuit. Some are even minor celebrities. Their lives and events — and especially the fall — are pretty well known and in some cases were front page news. As a writer you could easily overlook these stories because they were already so heavily documented.

But Declercq shows to have a keen eye for the story behind a story, and he was able to look past known facts and look for a deeper, collective connection between these five riders. And from their humble shared beginnings, Declercq takes the reader on a journey for each individual rider.

He does so with finesse. There is a dignified distance in his writing style (like a reporter) and this strikes the right tone of being an interested witness rather than a thrill seeker (the latter being the fate of many sport books of recent years). By doing so we get to hear the human perspective behind the stories. All these riders have lives, parents, wives, children and they sacrifice a lot. Which might be easy to forget when watching the Giro.

The fact that the often dramatic and heroic sport of cycling is a central subject, of course helps the book, but it is mostly Declercqs’ writing that make this book stand out. I love cycling and I love good books by good writers. This book has both.

The post The Fall (De Val) – Matthias M.R. Declercq appeared first on Jan van den Berg.

Indrek Lasn (indreklasn)

Here are 3 super simple developer tips that will supercharge your project June 04, 2019 01:37 PM

GitHub repository code quality checks and automatic deployment

Saving small amounts of time here and there leads up to saving a big chunk of time. The more time you can save with automation, the more you can focus on other areas of your project and getting more done. Here are some of the most time-saving tips and tools I use at my current company.

Continuous Integration

Code checks with Github and CircleCi

Every time you push code to the repository there should be automatic checks to run all unit tests, check ESLint rules, build the project to ensure the new deployment of application will be successful.

Let’s say you work in a small team of 3 developers. Each time someone commits new code to the repository, you would have to run all code quality checks manually. Who in the right mind does that and has time for it? That’s why we have continuous integration.

Check out this tutorial on how I setup CI for my projects.

How to setup continuous integration (CI) with React, CircleCI, and GitHub

Automated dependency updates

Automated dependency updates at www.getnewly.com

Have you ever had the chance to update dependencies for an application that’s running in production?

It’s a nightmare, some packages have peer dependencies, some have bugs, some are not compatible. Worst case scenario when updating packages is that application won’t work anymore.

Automated dependency updates to the rescue!

Instead of updating once a month and hoping everything will continue to work, wake up to new pull requests of daily version updates. If a package updates, the bot will create a PR with the new version, changelogs, release notes and package commits. Very useful if I may say.

I use dependabot (https://dependabot.com/)

Code formatter + (Prettier)

To semicolon or not to semicolon? Why fight over petty things when you can save energy and focus on your next unicorn idea?

I fell in love with Prettier ever since I started using it. Prettier combined with ESLint and Stylelint will make your developer soul happy.

I found it most convenient setting up Prettier with ESLint with Wes Bos’s approach. Here’s the full tutorial on how to do that.

https://medium.com/media/9fd7e0babcf87b91ce28ca24b7e42487/href

Thanks for reading! ❤

Check out my Twitter if you have any questions or know how to improve the developer environment even further.

Indrek Lasn (@lasnindrek) | Twitter

Don’t forget to follow Newly publication for more awesome stuff!

Here are some of my previous article you might enjoy;


Here are 3 super simple developer tips that will supercharge your project was originally published in Newly on Medium, where people are continuing the conversation by highlighting and responding to this story.

June 03, 2019

Jeremy Morgan (JeremyMorgan)

Thinking About Reusable Code June 03, 2019 06:09 AM

The mythical "reusable code" idea has existed for decades. It showed up shortly after the first lines of code were written. We preach re-usability and sometimes strive for it but it rarely becomes a reality. I've seen various levels of success with this over the years. Everything from "we have a reusable library that 75% of us use" to "we have shared code libraries here, but never use them in your projects".

A recent discussion led me to think about this. Why don't more software development organizations have shared code libraries? It is the pointy-haired bosses preventing it? Team conflicts? Or is it the developers themselves? 

We can't blame the pointy-haired bosses for this one. If you explain the basic concepts of reusable code to management, most would agree it's a great idea. Building something once, so you don't have to build it repeatedly? Easy to find the upside here.

Team conflicts can also contribute to it, which is usually people disagreeing about who gets to determine what code is shared. Developers themselves can also be opposed to it, due not having enough time to build them.

All of these are contributing factors to lack of adoption, but the question you should ask is do we need reusable code libraries at all?


What Are Shared Libraries and Why Do We Care?

If you have tasks your developers are building that contain code you can use for something else, you put that code in its own "library" to use later. This can be a DLL, a folder of snippets, a Node module, whatever. Connecting to a database? There's no reason to write that code for every piece of software that accesses a database. Create a DB_Connect class, and put that in a file you can copy to another project later.

It's easy. You take a function and if it's abstract enough, parameterize it and make it available for other projects to use. When you start your project, you don't have to write code to connect to the database, pull the library and enter your parameters. Here are some upsides:

  • You write code once and use it multiple times (saving cycle times)
  • If tested thoroughly, it cuts regression errors
  • It enforces standards other groups can follow

These are all great reasons to use shared libraries. Countless articles and books have been written about code reuse and most of you are familiar with them. The biggest selling point for this is not having to code "boring stuff" over and over and have wild variations of the same methods in the wild. This frees up time to work on exciting features.


How Are They Used?

Here are some ways shared libraries are used in business:

  • Internal Use Code that shared with internal groups and used for projects
  • External Use Code that designed for programmers outside the organization to use
  • Forked - Users can take your initial code and "fork" it to make it work for their specific needs

This falls in line with the DRY principle of software development. Don't repeat yourself.


Do We Want Shared Code Libraries?

Why isn't everyone doing this? Your organization may avoid shared libraries for a good reason. Not every project or team benefits from this and it's not the magic bullet to solve all development problems. Here are some reasons not to use code libraries.

  • Distributed Monolith Problem - If you put something into a library and everyone uses it, that doesn't mean it isn't bad. An inefficient database connector design means everyone is using an inefficient database connector. Changes to your implementation can still have negative cascading effects. A small change to an API could mean every single system is now broken.

  • Dependency Hell - If every piece of software is relying on your library, there's a tight coupling in place. There will be a myriad of dependencies in place, for instance, relying on a framework like Angular. If you have software built to use a specific version, upgrading it could produce a chain reaction amongst your applications.

  • Inflexible Design - Sometimes you're asked to use a design in a certain way that's different from what you actually need. Because "everyone else is doing it" you're forced to make your code work around it, which can then increase development time. Wait, why did we build shared libraries in the first place?

  • Programming Language Lock-In - Too many times I've heard "well our library is written in ____ so you have to use that" which means you're locked into a single programming language. This can lead to using the wrong tool for the job. So don't assume because your team doesn't use shared libraries means they're unenlightened and don't want better production.


What to Examine if You Want to Use Reusable Code Libraries

If you're considering creating reusable code libraries, you should first see if it makes sense to do so.

  • Take a hard look at the design - Then look at it again. Will you really benefit from it? Do you have repeated code now? Are there any issues that may encourage tight coupling? 
  • Closely examine your dependencies - Are you creating dependencies others will have to work around? How often do they change? Can they bypass the ones you create? 
  • Start abstract as possible - You want your libraries to be highly abstract so that they perform expected functions, but can be overridden or extended to meet specialized use cases. Give them the basics but let them evolve it how they'd like.

How to Get Developers to Implement Them

So you've decided you want a shared library. Here are some ways you can help teams adopt your library:

  • Steal it from an existing project - This is the easiest way to start a new library. Take an existing project and remove everything that's specialized and keep everything that can be re-used and feels solid.

  • Thoroughly test it - The last thing you want to do is introduce problems into the software. After building your abstract library, test it thoroughly so you can rely on it, and help developers find regressions. Don't hand them garbage and expect them to use it.

  • Examine all current use cases - By digging in and understanding how people will use your library you can determine whether it will benefit them, and whether to make changes. Do this on a method by method basis to ensure that you're asking them to use something that will work for them with minimum effort.  Like most anything in business, success will come from great communication. Top down design is not your friend here, you'll want to examine the needs of the group and design a library from those needs.


Conclusion

Most of the organizations I've been involved with do not use a shared code library. They're usually in some form of working towards one, and only two where they were implemented and working well.

At one end of the spectrum you have a pattern where all repeated code is in a library. On the other you'll have no code library, and everyone builds software for their own projects. You'll find success somewhere between the two. How far you lean toward each extreme will depend on your needs.

There are abstract functions that should always be shared. Some that come to mind are authentication/authorization, database connections and logging. It should be either a shared library you developed, or one that's open sourced and built by someone else. Everything else should be examined, considering the overall design.

Don't jump to conclusions and take on a large project to build a giant code library that will make developers lives worse. But don't exclude the idea altogether. Be purposeful and observant to find which strategy will work best for your projects. 



What is your DevOps IQ?

what's your devops score


My Devops Skill IQ is 232. Can you beat it? Take the test now to find out your Devops IQ score!!

Andreas Zwinkau (qznc)

Thinking in Systems by Donella Meadows June 03, 2019 12:00 AM

Book review: A shallow introduction to Systems Thinking.

Read full article!

June 02, 2019

Gustaf Erikson (gerikson)

Gokberk Yaltirakli (gkbrk)

Gopher Server in Rust June 02, 2019 06:06 PM

I find Gopher really cool. I think it’s a really nice way to organize information into trees and hierarchies, and as we all know programmers can’t resist trees. So recently I took an interest in Gopher and started writing my own server.

Gopher, like HTTP, is a network protocol for retrieving information over the internet. One crucial difference is, it hasn’t been commercialized by adtech companies. This is probably because it doesn’t provide many opportunities for tracking, and it doesn’t have a significantly large user base.

But recently it’s been gaining traction; so we should provide a decent landscape for new gophers, full of oxidised servers. Since I started using Gopher more often, it’s beneficial for me if there’s more content out there. So I’m writing this blog post to walk you through how to write your own server. We’ll be doing this in Rust.

Before we jump into the details of the protocol, let’s set up a server that responds with “Hello world”. This will provide a skeleton that we can fill with Gopher-related code later.

Handling connections

fn handle_client(stream: TcpStream) -> io::Result<()> {
    write!(stream, "Hello world!")?;
    Ok(())
}

fn main() -> io::Result<()> {
    let listener = TcpListener::bind(format!("0.0.0.0:70")?;

    for stream in listener.incoming() {
        thread::spawn(move || handle_client(stream?));
    }

    Ok(())
}

In this skeleton, pretty much all of our code is going to be in the handle_client function. If we look at the RFC for Gopher; we can see that after establishing a connection, the client sends the selector for the resource they are interested in. Like /ProgrammingLanguages/Python. Let’s read one line from the socket and look at which selector they want.

Gopher protocol

let mut line = String::new();
BufReader::new(stream.try_clone()?).read_line(&mut line)?;
let line = line.trim();

At this point, a traditional server would check the filesystem for the selector and a fancy web framework would go through the registered routes and possibly check some regular expressions. But for our toy server, a simple match statement will be more than enough.

let mut menu = GopherMenu::with_write(&stream);

match line {
    "/" | "" => {
        menu.info("Amazing home page of amazingness")?;
    }
    x => {
        menu.info("Page not found")?;
    }
}
menu.end()?;

In the code above, GopherMenu comes from a crate called gophermap. It’s a crate that can parse and generate Gopher menus.

Relative links

For relative links, we need to know the server address. Let’s put that in a constant and write a small helper.

const HOST: &str = "gkbrk.com";

let menu_link = |text: &str, selector: &str| {
    menu.write_entry(ItemType::Directory, text, selector, HOST, 70)
};

match line {
    "/" | "" => {
        menu.info("Hi!")?;
        menu.info("Try going to page 1")?;
        menu_link("Page One", "/page1")?;
        menu_link("Go to unknown link", "/test")?;
    }
    "/page1" => {
        menu.info("Yay! You found the secret page")?;
        menu_link("Home page", "/")?;
    }
    x => {
        menu.info(&format!("Unknown link: {}", x))?;
        menu_link("Home page", "/")?;
    }
};
menu.end()?;

Now we can link between our pages and start building proper pages. Hopefully this was a good start. If anyone reading this sets up their own Gopherspace, please let me know by leaving a comment or sending me an email.

Ponylang (SeanTAllen)

Last Week in Pony - June 2, 2019 June 02, 2019 11:35 AM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

June 01, 2019

Bit Cannon (wezm)

A Tiling Desktop Environment June 01, 2019 11:48 PM

I’ve been thinking about graphical shells recently. One of the great things about open source desktops1 is there is a plethora of choice when it comes to graphical shells. However they seem to fall into two camps:

  1. Full featured desktop environments that stick to the conventional stacking window metaphor.
  2. Narrowly featured window manager based environments that include tools like tiling window managers often optimised for efficient keyboard use.

I am currently using the second of these through the Awesome window manager. I’m really enjoying the keyboard centric operation, and almost never needing to manually position newly spawned windows. Each workspace (desktop) gets its own layout2, which describes how windows are laid out. For example, my most commonly used layout has one master window that takes up half the screen then additional windows are stacked on the right half. The split between the two halves of the screen is easily adjusted with Superh and Superl. Layouts can be changed on the fly with Superspace to suit your work.

Another aspect I enjoy about Awesome is its snappiness. This is largely due to the lack of animation. Switching workspaces is instant, without any unnecessary flourishes. It seems that the animations used in many graphical shells these days tend to reduce the perceived performance of the system. Look how fast this iPhone is when animations are disabled.

The drawback to window manager based environments is that you give up the cohesive, full featured nature of a desktop environment. For example, these are features of GNOME that I had to research, install, and configure after moving to Awesome (items in italic are ones I haven’t actually taken the time to implement yet):

  • Compositor
  • Volume and brightness status and control
  • Network status and control
  • SSH agent, GPG agent, polkit agent
  • Screenshot tools
  • Media controls
  • Notifications
  • HiDPI support
    • Cursor sizing
  • Automounting of external drives
  • Automatic multi-monitor support
    • Desktop gracefully adapting to monitors being added and removed
  • Screen locking
  • Power management
    • Battery status
    • Low power warnings
  • Clipboard preservation
    • I.e. clipboard source can exit and you can still paste
  • Color management

Even with many of these implemented the components don’t always work as nicely as in GNOME. For example, my XPS 15 has a built-in 4K display and I connect it to an external 4K display at work. When dunst shows a notification on the built-in display the text is sized wrong, when it shows on the external display it is correct, even though the displays are identical resolution.

On the flip side Awesome has these things in its favour:

  • Lower resource usage (mostly RAM)
  • Alternate window management layouts:
    • Stacking
    • Tiling
    • Floating
    • Maximised
    • Full screen
    • And more
  • Keyboard oriented
  • Keyboard bindings completely customisable
  • Better use of screen space
    • I have no title bars on windows
    • Top bar is very short and can be instantly toggled with Superb

So all this makes me wonder, where is the middle ground? Where is the desktop environment for professionals?

Pro Desktop

Mac OS is a popular choice for developers in some circles and has the cohesive full-featured experience that I mentioned above. I conducted an informal survey on Twitter to try to see what things Mac users are adding to the system to make it work better for them:

Hey Mac power users! I’m doing some research: What tools do you install to make the UI/graphical shell work better for you? Things that come to mind are: Alfred, Divvy, Spectacle, FastScripts, LaunchBar, chunkwm, that kind of thing.

The responses almost all included one or more of these elements:

  • Window management (often via the keyboard)
  • Keyboard remapping
  • Automation
  • Application launching
  • System stats

Some open source desktops have all these features but the ones that have them all seem to lack the polish and consistency of Mac OS, GNOME, or KDE and require a large investment in researching, installing, and configuring each desired feature. The ones that have the polish and consistency lack the customisation and keyboard control.

So where does that leave me? I want a desktop environment like GNOME but with more control over window management and more keyboard control.

Perhaps there is room for something that takes the place of gnome-shell in the typical GNOME desktop but built for this use case. gnome-shell is built on mutter and there are other desktop shells built on this too such as gala, and Budgie, so perhaps it would be possible to use mutter as the base window manager and compositor and build upon it.

I’ve been considering starting such a project but before diving in decided to write this post and do some more research to help clarify my thoughts. Something for me to ponder. 🤔

Comments


Why don’t you just…

Inevitably some folks will be thinking, “why don’t you just…”. Below are a few of these that I’ve thought of already. I may add more as time goes on.

Use KDE

One possible option is using KDE with an alternate window manager. Although this does prevent you from using Wayland. I am fan of Wayland but not yet a user. I believe it is the future of the graphics stack on open source desktops and I think its architecture makes sense give the way computers are used today.

My problem with KDE is the aesthetic. KDE and Qt really don’t seem to align with me. That’s not to say they’re bad or even ugly, it’s just not to my liking. I suppose as an ex-Mac user I feel more at home with GNOME/GTK. On the other hand it seems like someone familiar with Windows would feel more at home with KDE/Qt.

Things like menus attached to windows, icons on buttons, icons on menu items, Application launcher menu (“Start” button), bottom task bar, and apply buttons in configuration dialogs all feel very foreign to my Mac using past. Sure some of these may be configurable but I’m not sure I’d ever feel at home.

KDE neon: Icons on buttons, Apply button for configuration, task bar, Start-esque menu.

KDE neon: Icons in menus.

For comparison here is GNOME showing the the same things. I prefer that it is less busy and to be honest more like Mac OS in some ways.

GNOME: No icons in menus, no task bar.

GNOME: No Apply button in settings.

Use Xfce

It is possible to use an alternate window manager with Xfce. However, while Xfce has made recent progress on HiDPI support it’s still a mishmash of blurry icons, and tiny controls in places.

Xubuntu 19.10 with 2x scaling: Blurry icons, tiny controls.

Use the gTile GNOME extension

gTile is more of a manual window resizer. It allows you to position windows on a grid but it doesn’t appear to have anything approaching Awesome’s layouts.

Stick with Awesome

It’s true that Awesome is working for me but it does feel a bit like I’m back in the dark ages needing to find and configure things that I’ve previously taken for granted. It is nice to build your own environment like this but the little imperfections like the dunst notifications mentioned above, or handling of external displays have me wanting more.


  1. I’m referring to these as open source desktops and not Linux desktops since they work on other systems too, like BSDs, and OpenIndiana. [return]
  2. My first Awesome includes a litte information in the layouts. [return]

Pierre Chapuis (catwell)

Truncating an Alembic migrations history June 01, 2019 07:00 PM

In projects that use SQLAlchemy and Alembic via Flask-Migrate, you may want to truncate the migrations history. By that I mean: rewrite all the migrations up to some point as a single initial migration, to avoid replaying them every single time you create a new database instance. Of course, you only want to do that if you have already migrated all your database instances at least up to that point.

As far as I know, there is no Alembic feature to do this automatically. However, I found a way to avoid having to write the migration by hand. Here is an example of how you can achieve this with a project using Git, PostgreSQL, and environment variables for configuration.

First, checkout a commit of your project where the first migration you want to keep is the current migration, and create a temporary branch. Then, take a note of the ID of that migration (for instance abcd12345678), delete the whole migrations directory and reinitialize Alembic.

git checkout $my_commit
git checkout -b tmp-alembic
rm -rf migrations
flask db init

At this point, using Git, revert changes to files where you should keep your changes, such as script.py.mako and env.ini. Then, create a temporary empty database to work with.

git checkout migrations/script.py.mako
git checkout migrations/env.py
createdb -T template0 my-temp-db

Now create the initial migration that corresponds to your model, with the ID that you noted previously, e.g.:

MY_DATABASE_URI="postgresql://postgres@localhost/my-empty-db" \
    flask db migrate --rev-id abcd12345678

Finally, you can delete the temporary database, commit your changes to your temporary branch, merge it into your main development branch and delete it:

dropdb my-empty-db
git commit
git checkout dev
git merge tmp-alembic
git branch -D tmp-alembic

Ponylang (SeanTAllen)

0.28.1 Released June 01, 2019 01:01 PM

Pony 0.28.1 is here! It includes a couple high-priority bug fixes. Updating as soon as possible is recommended. Please note, the Homebrew PR hasn’t yet been merged so you can’t update using Homebrew until that is done. All other supported platforms are good to go!

May 31, 2019

Jan van den Berg (j11g)

Why We Sleep – Matthew Walker May 31, 2019 07:07 PM

Why We Sleep by Matthew Walker is one of the most profound books I have ever read. It has directly impacted my attitude towards sleep and subsequently altered my behaviour. Books that change your behaviour are rare and this is one of them. You should read it.

Why We Sleep – Matthew Walker (2017) – 368 pages

We all know that sleep is important. But Walker dissects study, after study, after study to describe how important sleep exactly is, and what the devastating effects of too little sleep are. Walker presents what we know about sleep — which is still a large research area — and he comes to quite sobering, startling and stunning conclusions about the importance of sleep.

The shock and awe approach could leave you with a sense of defeat of how we approach sleep related problems. Because, individually and as a society, we handle it very poorly. But Walkers’ optimism towards the end regarding solutions, does provide a little bit of comfort.

Modern man has dug quite a hole for himself, with blue LED lights, caffeine, alcohol, iPads and online distractions, which all disrupt our (sleep) lives more than we can even begin to imagine. But Walkers’ solution is not to shy away from technology, but rather to embrace and expand it explicitly towards better sleep. So fortunately there is at least also some direction in this book. We badly need it.

Please read this book.

The post Why We Sleep – Matthew Walker appeared first on Jan van den Berg.

May 30, 2019

Derek Jones (derek-jones)

Cognitive capitalism chapter reworked May 30, 2019 01:22 AM

The Cognitive capitalism chapter of my evidence-based software engineering book took longer than expected to polish; in fact it got reworked, rather than polished (which still needs to happen, and there might be more text moving from other chapters).

Changing the chapter title, from Economics to Cognitive capitalism, helped clarify lots of decisions about the subject matter it ought to contain (the growth in chapter page count is more down to material moving from other chapters, than lots of new words from me).

I over-spent time down some interesting rabbit holes (e.g., real options), before realising that no public data was available, and unlikely to be available any time soon. Without data, there is not a lot that can be said in a data driven book.

Social learning is a criminally under researched topic in software engineering. Some very interesting work has been done by biologists (e.g., Joseph Henrich, and Kevin Laland), in the last 15 years; the field has taken off. There is a huge amount of social learning going on in software engineering, and virtually nobody is investigating it.

As always, if you know of any interesting software engineering data, please let me know.

Next, the Ecosystems chapter.

May 29, 2019

Jan van den Berg (j11g)

Humor schept evenwicht (Humor creates balance) – Jaap Bakker May 29, 2019 07:50 PM

Jaap Bakker, a local storyteller from a small rural town in the Netherlands (Urk), has written down anecdotes and jokes from the last hundred years or so. Either things he experienced first hand or that were told to him. So expect hundreds of fun little stories. Stories anyone can identify with, about human interaction and small town life, that make you smile, laugh or even burst out.

Humor Schept Evenwicht – Jaap Bakker (2005) – 87 pages

Needless to say, I am biased about this book. Since all stories are rooted in my hometown, and therefore very relatable. But I do fear some stories would probably need added context for outsiders to make sense. So it could have used some outside editing to make it more coherent. But nonetheless, I thought it was a delightful read.

The post Humor schept evenwicht (Humor creates balance) – Jaap Bakker appeared first on Jan van den Berg.

Gokberk Yaltirakli (gkbrk)

Writing a Simple IPFS Crawler May 29, 2019 04:20 PM

IPFS is a peer-to-peer protocol that allows you to access and publish content in a decentralized fashion. It uses hashes to refer to files. Short of someone posting hashes on a website, discoverability of content is pretty low. In this article, we’re going to write a very simple crawler for IPFS.

It’s challenging to have a traditional search engine in IPFS because content rarely links to each other. But there is another way than just blindly following links like a traditional crawler.

Enter DHT

In IPFS, the content for a given hash is found using a Distributed Hash Table. Which means our IPFS daemon receives requests about the location of IPFS objects. When all the peers do this, a key-value store is distributed among them; hence the name Distributed Hash Table. Even though we won’t get all the queries, we will still get a fraction of them. We can use these to discover when people put files on IPFS and announce it on the DHT.

Fortunately, IPFS lets us see those DHT queries from the log API. For our crawler, we will use the Rust programming language and the ipfsapi crate for communicating with IPFS. You can add ipfsapi = "0.2" to your Cargo.toml file to get the dependency.

Using IPFS from Rust

Let’s test if our IPFS daemon and the IPFS crate are working by trying to fetch and print a file.

let api = IpfsApi::new("127.0.0.1", 5001);

let bytes = api.cat("QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u")?;
let data = String::from_utf8(bytes.collect())?;

println!("{}", data);

This code should grab the contents of the hash, and if everything is working print “Hello World”.

Getting the logs

Now that we can download files from IPFS, it’s time to get all the logged events from the daemon. To do this, we can use the log_tail method to get an iterator of all the events. Let’s get everything we get from the logs and print it to the console.

for line in api.log_tail()? {
    println!("{}", line);
}

This gets us all the loops, but we are only interested in DHT events, so let’s filter a little. A DHT announcement looks like this in the JSON logs.

{
  "duration": 235926,
  "event": "handleAddProvider",
  "key": "QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u",
  "peer": "QmeqzaUKvym9p8nGXYipk6JpafqqQAnw1ZQ4xBoXWcCrLb",
  "session": "ffffffff-ffff-ffff-ffff-ffffffffffff",
  "system": "dht",
  "time": "2018-03-12T00:32:51.007121297Z"
}

We are interested in all the log entries with the event handleAddProvider. And the hash of the IPFS object is key. We can filter the iterator like this.

let logs = api.log_tail()
        .unwrap()
        .filter(|x| x["event"].as_str() == Some("handleAddProvider"))
        .filter(|x| x["key"].is_string());

for log in logs {
    let hash = log["key"].as_str().unwrap().to_string();
    println!("{}", hash);
}

Grabbing the valid images

As a final step, we’re going to save all the valid image files that we come across. We can use the image crate. Basically; for each object we find, we’re going to try parsing it as an image file. If that succeeds, we likely have a valid image that we can save.

Let’s write a function that loads an image from IPFS, parses it with the image crate and saves it to the images/ folder.

fn check_image(hash: &str) -> Result<(), Error> {
    let api = IpfsApi::new("127.0.0.1", 5001);

    let data: Vec<u8> = api.cat(hash)?.collect();
    let img = image::load_from_memory(data.as_slice())?;

    println!("[!!!] Found image on hash {}", hash);

    let path = format!("images/{}.jpg", hash);
    let mut file = File::create(path)?;
    img.save(&mut file, image::JPEG)?;

    Ok(())
}

And then connecting to our main loop. We’re checking each image in a seperate thread because IPFS can take a long time to resolve a hash or timeout.

for log in logs {
    let hash = log["key"].as_str().unwrap().to_string();
    println!("{}", hash);

    thread::spawn(move|| check_image(&hash));
}

Possible improvements / future work

  • File size limits: Checking the size of objects before downloading them
  • More file types: Saving more file types. Determining the types using a utility like file.
  • Parsing HTML: When the object is valid HTML, parse it and index the text in order to provide search

Evolving Neural Net classifiers May 29, 2019 04:20 PM

As a research interest, I play with evolutionary algorithms a lot. Recently I’ve been messing around with Neural Nets that are evolved rather than trained with backpropagation.

Because this is a blog post, and to further demonstrate that literally anything can result in evolution, I’m going to be using a hill climbing algorithm. Here’s the gist of it.

  1. Initially, we will start with a Neural Network with random weights.
  2. We’re going to clone the network, pick a weight and change it to a random number.
  3. Evaluate the old network and the new network and get their scores
  4. If the new network has done better or the same as the old one, replace the old network with it
  5. Repeat until the results are satisfactory

The algorithm

The algorithm is shown below. All it does is split the given data into training and test parts, randomly change the neural network weights until the score improves, and then use the test data to determine how good we did.

def train_and_test(X, y, nn_size, iterations=1000, test_size=None, stratify=None):
    random.seed(445)
    np.random.seed(445)
    net = NeuralNetwork(nn_size)

    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=test_size, random_state=42, stratify=stratify
    )

    score = 0
    for i in range(iterations):
        score = net.get_score(X_train, y_train)

        new = net.clone()
        new.mutate()
        new_score = new.get_score(X_train, y_train)

        if new_score >= score:
            net = new
            score = new_score

    print(f"Training set: {len(X_train)} elements. Error: {score}")

    score = net.get_classify_score(X_test, y_test)

    print(f"Test set: {score} / {len(X_test)}. Score: {score / len(X_test) * 100}%")

Iris flower dataset

If you are learning about classifiers, the Iris flower dataset is probably the first thing you’re going to test. It is like the “Hello World” of classification basically.

The dataset includes petal and sepal size measurements from 3 different Iris species. The goal is to get measurements and classify which species they are from.

You can find more information on the dataset here.

data = pandas.read_csv("IRIS.csv").values

name_to_output = {
    "Iris-setosa": [1, 0, 0],
    "Iris-versicolor": [0, 1, 0],
    "Iris-virginica": [0, 0, 1],
}

rows = data.shape[0]
data_input = data[:, 0:4].reshape((rows, 4, 1)).astype(float)
data_output = np.array(list(map(lambda x: name_to_output[x], data[:, 4]))).reshape(
    (rows, 3)
)

train_and_test(data_input, data_output, (4, 4, 3), 10000, 0.2)
Training set: 120 elements. Error: -5.697678436657024
Test set: 29 / 30. Score: 96.66666666666667%

96% accuracy isn’t bad such a simple algorithm. But it has that accuracy when it trains with 120 samples and tests with 30. Let’s see if it’s good at generalization by turning our train/test split into 0.03/0.97.

As you can see below; just by training on 4 samples, our network is able to classify the rest of the data with a 94% accuracy.

train_and_test(data_input, data_output, (4, 4, 3), 10000, 0.97)
Training set: 4 elements. Error: -0.8103166051741318
Test set: 138 / 146. Score: 94.52054794520548%

Cancer diagnosis dataset

This dataset has includes some data/measurements about tumors, and classifies them as either Benign (B) or Malignant (M).

You can find the dataset and more information about it here.

data = pandas.read_csv("breast_cancer.csv").values[1:]

rows = data.shape[0]

name_to_output = {"B": [1, 0], "M": [0, 1]}

data_input = data[:, 2:32].reshape((rows, 30, 1)).astype(float) / 100
data_output = np.array(list(map(lambda x: name_to_output[x], data[:, 1]))).reshape(
    (rows, 2)
)

train_and_test(data_input, data_output, (30, 30, 15, 2), 10000, 0.3)
Training set: 397 elements. Error: -5.626705318006574
Test set: 159 / 171. Score: 92.98245614035088%

To see if the network is able to generalize, let’s train it on 11 samples and test it on 557. You can see below that it has an 86% accuracy after seeing a tiny amount of samples.

train_and_test(data_input, data_output, (30, 30, 15, 2), 10000, 0.98)
Training set: 11 elements. Error: -0.2742514647152907
Test set: 481 / 557. Score: 86.35547576301616%

Glass classification dataset

This dataset has some material measurements, like how much of each element was found in a piece of glass. Using these measurements, the goal is to classify which of the 8 glass types it was from.

This dataset doesn’t separate cleanly, and there aren’t a lot of samples you get. So I cranked up the iteration number and added more hidden layers. Deep learning baby!

You can find more information on the dataset here.

data = pandas.read_csv("glass.csv").values[1:]

rows = data.shape[0]
data_input = data[:, :-1].reshape((rows, 9, 1)).astype(float)
data_output = np.array(list(map(lambda x: np.eye(8)[int(x)], data[:, -1]))).reshape((rows, 8))

train_and_test(data_input, data_output, (9, 9, 9, 9, 8), 20000, 0.3, stratify=data_output)
Training set: 149 elements. Error: -8.261249669954738
Test set: 47 / 64. Score: 73.4375%

After I saw this result, I wasn’t super thrilled about it. But after I went through the other solutions on Kaggle and looked at their results, I found out that this wasn’t bad compared to other classifiers.

But where’s the Neural Network code?

Here it is. While it’s a large chunk of code, I find that this is the least interesting part of the project. This is basically a bunch of matrices getting multiplied and mutated randomly. You can find a bunch of tutorials/examples of this on the internet.

import numpy as np
import random
import pandas
from sklearn.model_selection import train_test_split

class NeuralNetwork:
    def __init__(self, layer_sizes):
        self.layer_sizes = layer_sizes
        weight_shapes = [(a, b) for a, b in zip(layer_sizes[1:], layer_sizes[:-1])]
        self.weights = [
            np.random.standard_normal(s) / s[1] ** 0.5 for s in weight_shapes
        ]
        self.biases = [np.random.rand(s, 1) for s in layer_sizes[1:]]

    def predict(self, a):
        for w, b in zip(self.weights, self.biases):
            a = self.activation(np.matmul(w, a) + b)
        return a

    def get_classify_score(self, images, labels):
        predictions = self.predict(images)
        num_correct = sum(
            [np.argmax(a) == np.argmax(b) for a, b in zip(predictions, labels)]
        )
        return num_correct

    def get_score(self, images, labels):
        predictions = self.predict(images)
        predictions = predictions.reshape(predictions.shape[0:2])
        return -np.sum(np.abs(np.linalg.norm(predictions-labels)))

    def clone(self):
        nn = NeuralNetwork(self.layer_sizes)
        nn.weights = np.copy(self.weights)
        nn.biases = np.copy(self.biases)
        return nn

    def mutate(self):
        for _ in range(self.weighted_random([(20, 1), (3, 2), (2, 3), (1, 4)])):
            l = self.weighted_random([(l.flatten().shape[0], i) for i, l in enumerate(self.weights)])
            shape = self.weights[l].shape
            layer = self.weights[l].flatten()
            layer[np.random.randint(0, layer.shape[0]-1)] = np.random.uniform(-2, 2)
            self.weights[l] = layer.reshape(shape)

            if np.random.uniform() < 0.01:
                b = self.weighted_random([(b.flatten().shape[0], i) for i, b in enumerate(self.biases)])
                shape = self.biases[b].shape
                bias = self.biases[b].flatten()
                bias[np.random.randint(0, bias.shape[0]-1)] = np.random.uniform(-2, 2)
                self.biases[b] = bias.reshape(shape)

    @staticmethod
    def activation(x):
        return 1 / (1 + np.exp(-x))

    @staticmethod
    def weighted_random(pairs):
        total = sum(pair[0] for pair in pairs)
        r = np.random.randint(1, total)
        for (weight, value) in pairs:
            r -= weight
            if r <= 0: return value

Phone Location Logger May 29, 2019 04:20 PM

If you are using Google Play Services on your Android phone, Google receives and keeps track of your location history. This includes your GPS coordinates and timestamps. Because of the privacy implications, I have revoked pretty much all permissions from Google Play Services and disabled my Location History on my Google settings (as if they would respect that).

But while it might be creepy if a random company has this data, it would be useful if I still have it. After all, who doesn’t want to know the location of a park that they stumbled upon randomly on a vacation 3 years ago.

I remember seeing some location trackers while browsing through F-Droid. I found various applications there, and picked one that was recently updated. The app was a Nextcloud companion app, with support for custom servers. Since I didn’t want a heavy Nextcloud install just to keep track of my location, I decided to go with the custom server approach.

In the end, I decided that the easiest path is to make a small CGI script in Python that appends JSON encoded lines to a text file. Because of this accessible data format, I can process this file in pretty much every programming language, import it to whatever database I want and query it in whatever way I see fit.

The app I went with is called PhoneTrack. You can find the APK and source code links on F-Droid. It replaces the parameters in the URL, logging every parameter looks like this: https://example.com/cgi-bin/locationrecorder.py ?acc=%ACC&alt=%ALT&batt=%BATT&dir=%DIR&lat=%LAT&lon=%LON&sat=%SAT&spd=%SPD &timestamp=%TIMESTAMP

Here’s the script in all it’s glory.

import cgi
import json

PATH = '/home/databases/location.txt'

print('Content-Type: text/plain\n')
form = cgi.FieldStorage()

# Check authentication token
if form.getvalue('token') != 'SECRET_VALUE':
    raise Exception('Nope')

obj = {
    'accuracy':   form.getvalue('acc'),
    'altitude':   form.getvalue('alt'),
    'battery':    form.getvalue('batt'),
    'bearing':    form.getvalue('dir'),
    'latitude':   form.getvalue('lat'),
    'longitude':  form.getvalue('lon'),
    'satellites': form.getvalue('sat'),
    'speed':      form.getvalue('spd'),
    'timestamp':  form.getvalue('timestamp'),
}

with open(PATH, 'a+') as log:
    line = json.dumps(obj)
    log.write(f'{line}\n')

Reverse Engineering the Godot File Format May 29, 2019 04:20 PM

I’ve been messing around with the Godot game engine recently. After writing some examples that load the assets and map data from files, I exported it and noticed that Godot bundled all the resources into a single .pck file. It was packing all the game resources and providing them during runtime as some sort of virtual file system.

Of course; after I was finished for the day with learning gamedev, I was curious about that file. I decided to give myself a small challenge and parse that file using only hexdump and Python. I opened the pack file with my hex editor and I was met with a mixture of binary and ASCII data. Here’s the beginning of the file from hexdump -C game.pck.

00000000  47 44 50 43 01 00 00 00  03 00 00 00 01 00 00 00  |GDPC............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000050  00 00 00 00 1e 00 00 00  40 00 00 00 72 65 73 3a  |........@...res:|
00000060  2f 2f 2e 69 6d 70 6f 72  74 2f 61 6d 67 31 5f 66  |//.import/amg1_f|
00000070  72 31 2e 70 6e 67 2d 32  32 66 30 32 33 32 34 65  |r1.png-22f02324e|
00000080  63 62 39 34 32 34 39 37  62 61 61 31 38 30 37 33  |cb942497baa18073|
00000090  37 36 62 37 31 64 30 2e  73 74 65 78 00 3d 00 00  |76b71d0.stex.=..|
000000a0  00 00 00 00 28 04 00 00  00 00 00 00 de a0 23 30  |....(.........#0|
000000b0  cf 7c 59 5c fb 73 5d f6  a7 f8 12 a7 40 00 00 00  |.|Y\.s].....@...|
000000c0  72 65 73 3a 2f 2f 2e 69  6d 70 6f 72 74 2f 61 6d  |res://.import/am|
000000d0  67 31 5f 6c 66 31 2e 70  6e 67 2d 62 33 35 35 35  |g1_lf1.png-b3555|
000000e0  34 66 64 31 39 36 37 64  65 31 65 62 32 63 64 31  |4fd1967de1eb2cd1|
000000f0  32 33 32 65 32 31 38 33  33 30 32 2e 73 74 65 78  |232e2183302.stex|

Immediately we can see that the file starts with some magic bytes, and before our ASCII filename begins there’s a lot of zeros. They might be keeping that space as a padding or for future extensions, or maybe it can even contain data depending on your settings, but for our immediate goal it doesn’t matter.

What looks interesting here is the two integers right before our path. 1e 00 00 00 and 40 00 00 00, probably lengths or counts since they are before real data. Saying these two are little-endian unsigned integers would be a good assumption, because otherwise they would be huge numbers that have no business being the length of anything.

The first number is 30 and the second one is 64. Now, what’s the name of that file? res://.import/amg1_fr1.png-22f02324ecb942497baa1807376b71d0.stex. Exactly 64 bytes. That means we now know that the paths are prefixed by their length.

If we look at the next path, we can see that a similar pattern of being length prefixed still applies. The first integer we found, 30, is most likely the number of files we have. And a rough eyeballing of the file contents reveals that to be the case.

Let’s get a little Python here and try to read the file with our knowledge so far. We’ll read the first integer and loop through all the files, trying to print their names.

pack = open('game.pck', 'rb')
pack.read(0x54) # Skip the empty padding

file_count = struct.unpack('<I', pack.read(4))[0]

name_len = struct.unpack('<I', pack.read(4))[0]
name = pack.read(name_len).decode('utf-8')

print(f'The first file is {name}')

Running this code produces the following output, success!

The first file is res://.import/amg1_fr1.png-22f02324ecb942497baa1807376b71d0.stex

Now let’s try to loop file_count times and see if our results are good. One thing we should notice is the data following the ASCII text, before the new one begins. If we miss that, we will read the rest of the data wrong and end up with garbage. Let’s go back to our hexdump and count how many bytes we need to skip until the next length. Looks like we have 32 extra bytes. Let’s account for those and print everything.

for i in range(file_count):
  name_len = struct.unpack('<I', pack.read(4))[0]
  name = pack.read(name_len)
  pack.read(32)
  print(name)
...
b'res://MapLoader.gd.remap'
b'res://MapLoader.gdc\x00'
b'res://MapLoader.tscn'
b'res://Player.gd.remap\x00\x00\x00'
b'res://Player.gdc'
...

Much better, the only issue is the trailing null bytes on some file names. This shouldn’t be a huge issue though, these are probably random padding and they don’t even matter if we consider the strings null-terminated. Let’s just get rid of trailing null bytes.

name = pack.read(name_len).rstrip(b'\x00').decode('utf-8')

After this change, we can get a list of all the resource files in a Godot pack file.

Getting the file contents

Sure, getting the list of files contained in the file is useful. But it’s not super helpful if our tool can’t get the file contents as well.

The file contents are stored separately from the file names. The thing we parsed so far is only the file index, like a table of contents. It’s useful to have it that way so when the game needs a resource at the end of the file, the game engine won’t have to scan the whole file to get there.

But how can we get find where the contents are without going through the whole thing? With offsets of course. Every entry we read from the index, along with the file name, contains the offset and the size of the file. It’s not the easiest thing to explain how you discover something after the fact, but it’s a combination of being familiar with other file formats and a bunch of guesswork. Now I’d like to direct your attention to the 32 bytes we skipped earlier.

Since we already have the file names, we can make assumptions like text files being smaller than other resources like images and sprites. This can be made even easier by putting files with known lengths there, but just for the sake of a challenge let’s pretend that we can’t create these files.

After each guess, we can easily verify this by checking with hexdump or plugging the new logic in to our Python script.

The 4 byte integer that follows the file name is our offset. If we go to the beginning of the file and count that many bytes, we should end up where our file contents begin.

offset = struct.unpack('<I', pack.read(4))[0]

This is followed by 4 empty bytes and then another integer which is our size. Those 4 bytes might be used for other purposes, but again they are irrelevant for our goal.

pack.read(4)
size = struct.unpack('<I', pack.read(4))[0]

The offset is where the file contents began, and the size is how many bytes it is. So the range between offset and offset + size is the file contents. And because we ended up reading 12 bytes from the file, we should make our padding 20 instead of 32.

Reading a File

To finish up, let’s read the map file I put in my game. It’s a plaintext file with JSON contents, so it should be easy to see if everything looks complete.

pack = open('game.pck', 'rb')
pack.read(0x54) # Skip the empty padding

file_count = struct.unpack('<I', pack.read(4))[0]

for i in range(file_count):
  name_len = struct.unpack('<I', pack.read(4))[0]
  name = pack.read(name_len).rstrip(b'\x00').decode('utf-8')

  offset = struct.unpack('<I', pack.read(4))[0]
  pack.read(4)
  size = struct.unpack('<I', pack.read(4))[0]
  pack.read(20)

  print(name)

  if name == 'res://maps/map01.tres':
    pack.seek(offset)
    content = pack.read(size).decode('utf-8')
    print(content)
    break
...
res://Player.gd.remap
res://Player.gdc
res://Player.tscn
res://block.gd.remap
res://block.gdc
res://block.tscn
res://default_env.tres
res://icon.png
res://icon.png.import
res://maps/map01.tres
{
    "spawn_point": {"x": 5, "y": 3},
    "blocks": [
        {"x": 5, "y": 5, "msg": "Welcome to Mr Jumpy Man", "jumpLimit": 0},
        {"x": 7, "y": 5, "msg": "We're still waiting on the trademark"},
        {"x": 9, "y": 5, "texture": "platformIndustrial_001.png"},
        {"x": 11, "y": 6, "msg": "If you fall down,\nyou will be teleported to the start point"},
...

Everything looks good! Using this code, we should be able to extract the resources of Godot games.

Finishing words

Now; before the angry comments section gets all disappointed, let me explain. I am fully aware that Godot is open source. Yes, I could’ve looked at the code to know exactly how it works. No, that wouldn’t be as fun. Kthxbye.

Mastodon Bot in Common Lisp May 29, 2019 04:20 PM

If you post a programming article to Hacker News, Reddit or Lobsters; you will notice that soon after it gets to the front page, it gets posted to Twitter automatically.

But why settle for Twitter when you can have this on Mastodon? In this article we will write Mastodon bot that regularly checks the Lobste.rs front page and posts new links to Mastodon.

Since this is a Mastodon bot, let’s start by sending a post to our followers.

Sending a Mastodon post

To interact with Mastodon, we are going to use a library called Tooter. To get the API keys you need; just log in to Mastodon, go to Settings > Development > New Application. Once you create an application, the page will show all the API keys you need.

(defun get-mastodon-client ()
  (make-instance 'tooter:client
                 :base "https://botsin.space"
                 :name "lobsterbot"
                 :key "Your client key"
                 :secret "Your client secret"
                 :access-token "Your access token")
  )

This function will create a Mastodon client whenever you call it. Now, let’s send our first message.

(tooter:make-status (get-mastodon-client) "Hello world!")

Now that we can send messages, the next step in our project is to fetch the RSS feed.

Fetching the RSS feed

Fetching resources over HTTP is really straightforward with Common Lisp, the drakma library provides an easy-to-use function called http-request. In order to get the contents of my blog, all you need to do is

(drakma:http-request "https://gkbrk.com")

So let’s write a function that takes a feed URL and returns the RSS items.

There is one case we need to handle with this. When you are fetching text/html, drakma handles the decoding for you; but it doesn’t do this when we fetch application/rss. Instead, it returns a byte array.

(defvar feed-path "https://lobste.rs/rss")

(defun get-rss-feed ()
  "Gets rss feed of Lobste.rs"
  (let* ((xml-text (babel:octets-to-string (drakma:http-request feed-path)))
         (xml-tree (plump:parse xml-text)))
    (plump:get-elements-by-tag-name xml-tree "item")
    ))

This function fetches an RSS feed, parses the XML and returns the <item> tags in it. In our case, these tags contain each post Lobste.rs.

Creating structs for the links

A struct in Common Lisp is similar to a struct in C and other languages. It is one object that stores multiple fields.

(defstruct lobsters-post
  title
  url
  guid
  )

Getting and setting fields of a struct can be done like this.

; Pretend that we have a post called p
(setf (lobsters-post-title p) "An interesting article") ; Set the title
(print (lobsters-post-title p))                         ; Print the title

Let’s map the RSS tags to our struct fields.

(defun find-first-element (tag node)
  "Search the XML node for the given tag name and return the text of the first one"
  (plump:render-text (car (plump:get-elements-by-tag-name node tag)))
  )

(defun parse-rss-item (item)
  "Parse an RSS item into a lobsters-post"
  (let ((post (make-lobsters-post)))
    (setf (lobsters-post-title post) (find-first-element "title" item))
    (setf (lobsters-post-url post) (find-first-element "link" item))
    (setf (lobsters-post-guid post) (find-first-element "guid" item))
    post
    ))

Now, we can make the previous get-rss-feed function return lobsters-post‘s instead of raw XML nodes.

(defun get-rss-feed ()
  "Gets rss feed of Lobste.rs"
  (let* ((xml-text (babel:octets-to-string (drakma:http-request *feed-url*)))
         ; Tell the parser that we want XML tags instead of HTML
         ; This is needed because <link> is a self-closing tag in HTML
         (plump:*tag-dispatchers* plump:*xml-tags*)
         (xml-tree (plump:parse xml-text))
         (items (plump:get-elements-by-tag-name xml-tree "item"))
         )
    (reverse (map 'list #'parse-rss-item items))
    ))

Posting the first link to Mastodon

(defun share-post (item)
  "Takes a lobsters-post and posts it on Mastodon"
  (tooter:make-status (get-mastodon-client) (format nil "~a - ~a ~a"
                                                    (lobsters-post-title item)
                                                    (lobsters-post-guid item)
                                                    (lobsters-post-url item)))
  )

(share-post (car (get-rss-feed)))

Keeping track of shared posts

We don’t want our bot to keep posting the same links. One solution to this is to keep all the links we already posted in a file called links.txt.

Every time we come accross a link, we will record it to our “database”. This basically appends the link follewed by a newline to the file. Not very fancy, but certainly enough for our purposes.

(defun record-link-seen (item)
  "Writes a link to the links file to keep track of it"
  (with-open-file (stream "links.txt"
                          :direction :output
                          :if-exists :append
                          :if-does-not-exist :create)
    (format stream "~a~%" (lobsters-post-guid item)))
  )

In order to filter our links before posting, we will go through each line in that file and check if our link is in there.

(defun is-link-seen (item)
  "Returns if we have processed a link before"
  (with-open-file (stream "links.txt"
                          :if-does-not-exist :create)
    (loop for line = (read-line stream nil)
       while line
       when (string= line (lobsters-post-guid item)) return t))
  )

Now let’s wrap this all up by creating a task that

  • Fetches the RSS feed
  • Gets the top 10 posts
  • Filters out the links that we shared before
  • Posts them to Mastodon
(defun run-mastodon-bot ()
  (let* ((first-ten (subseq (get-rss-feed) 0 10))
         (new-links (remove-if #'is-link-seen first-ten))
         )
    (loop for item in new-links do
         (share-post item)
         (record-link-seen item))
    ))

How you schedule this to run regularly is up to you. Set up a cron job, make a timer or just run it manually all the time.

You can find the full code here.

Fetching ActivityPub Feeds May 29, 2019 04:20 PM

Mastodon is a federated social network that uses the ActivityPub protocol to connect separate communities into one large network. Both Mastodon and the ActivityPub protocol are increasing in usage every day. Compared to formats like RSS, which are pull-based, ActivityPub is push-based. This means rather than your followers downloading your feed regularly to check if you have shared anything, you send each follower (or each server as an optimization) the content you shared.

While this decreases latency in your followers receiving your updates, it does complicate the implementation of readers. But fortunately, it is still possible to pull the feed of ActivityPub users. Just like the good old days.

In this article; we’re going to start from a handle like leo@niu.moe, and end up with a feed of my latest posts.

WebFinger

First of all, let’s look at how the fediverse knows how to find the ActivityPub endpoint for a given handle. The way this is done is quite similar to email.

To find the domain name, let’s split the handle into the username and domain parts.

handle           = 'leo@niu.moe'
username, domain = handle.split('@')

Next, we need to make a request to the domain’s webfinger endpoint in order to find more data about the account. This is done by performing a GET request to /.well-known/webfinger.

wf_url = 'https://{}/.well-known/webfinger'.format(domain)
wf_par = {'resource': 'acct:{}'.format(handle)}
wf_hdr = {'Accept': 'application/jrd+json'}

# Perform the request
wf_resp = requests.get(wf_url, headers=wf_hdr, params=wf_par).json()

Now we have our WebFinger response. We can filter this data in order to find the correct ActivityPub endpoint. We need to do this because webfinger can return a variety of URLs, not just ActivityPub.

Filtering the endpoints

The response we get from WebFinger looks like this.

{
  "subject": "acct:leo@niu.moe",
  "aliases": [
    "https://niu.moe/@leo",
    "https://niu.moe/users/leo"
  ],
  "links": [
    {
      "rel": "http://webfinger.net/rel/profile-page",
      "type": "text/html",
      "href": "https://niu.moe/@leo"
    },
    {
      "rel": "http://schemas.google.com/g/2010#updates-from",
      "type": "application/atom+xml",
      "href": "https://niu.moe/users/leo.atom"
    },
    {
      "rel": "self",
      "type": "application/activity+json",
      "href": "https://niu.moe/users/leo"
    }
  ]
}

Depending on the server, there might be more or less entries in the links key. What we are intereted in is the URL with the type application/activity+json. Let’s go through the array and find the link URL we’re looking for.

matching = (link['href'] for link in wf_resp['links'] if link['type'] == 'application/activity+json')
user_url = next(matching, None)

Fetching the feed link

We can fetch our feed URL using requests like before. One detail to note here is the content type that we need to specify in order to get the data in the format we want.

as_header = {'Accept': 'application/ld+json; profile="https://www.w3.org/ns/activitystreams"'}
user_json = requests.get(user_url, headers=as_header).json()

user_json is a dictionary that contains information about the user. This information includes the username, profile summary, profile picture and other URLs related to the user. One such URL is the “Outbox”, which is basically a feed of whatever that user shares publicly.

This is the final URL we need to follow, and we will have the user feed.

feed_url  = user_json['outbox']

In ActivityPub, the feed is an OrderedCollection. And those can be paginated. The first page can be empty, or have all the content. Or it can be one event for each page. This is completely up to the implementation. In order to handle this transparently, let’s write a generator that will fetch the next pages when they are requested.

def parse_feed(url):
    feed = requests.get(url, headers=as_header).json()

    if 'orderedItems' in feed:
        for item in feed['orderedItems']:
            yield item

    next_url = None
    if 'first' in feed:
        next_url = feed['first']
    elif 'next' in feed:
        next_url = feed['next']

    if next_url:
        for item in parse_feed(next_url):
            yield item

Now; for the purposes of a blog post and for writing simple feed parsers, this code works with most servers. But this is not a fully spec-complient function for grabbing all the pages of content. Technically next and first can be lists of events instead of other links, but I haven’t come across that in the wild. It is probably a good idea to write your code to cover more edge cases when dealing with servers on the internet.

Printing the first 10 posts

The posts in ActivityPub contain HTML and while this is okay for web browsers, we should strip the HTML tags before printing them to the terminal.

Here’s how we can do that with the BeautifulSoup and html modules.

def clean_html(s):
    text = BeautifulSoup(s, 'html.parser').get_text()
    return html.unescape(text)

i = 0
for item in parse_feed(feed_url):
    try:
        # Only new tweets
        assert item['type'] == 'Create'
        content = item['object']['content']
        text = clean_html(content)

        print(text)
        i += 1
    except:
        continue

    if i == 10:
        break

Future Work

Mastodon is not the only implementation of ActivityPub, and each implementation can do things in slightly different ways. While writing code to interact with ActivityPub servers, you should always consult the specification document.

Useful Links

Plaintext budgeting May 29, 2019 04:20 PM

For the past ~6 months, I’ve been using an Android application to keep track of my daily spending. To my annoyance, I found out that the app doesn’t have an export functionality. I didn’t want to invest more time in a platform that I couldn’t get my data out of, so I started looking for another solution.

I’ve looked into budgeting systems before, and I’ve seen both command-line (ledger) and GUI systems (GNUCash). Now; both of these are great software, and I can appreciate how Double-entry bookkeeping is a useful thing for accounting purposes. But while they are powerful, they’re not as simple as they could be.

I decided to go with CSV files. CSV is one of the most universal file formats, it’s simple and obvious. I can process it with pretty much every programming language and import it to pretty much every spreadsheet software. Or… I could use a shell script to run calculations with SQLite.

If I ever want to migrate to another system; it will probably be possible to convert this file with a shell script, or even a sed command.

I create monthly CSV files in order to keep everything nice and tidy, but the script adapts to everything from a single CSV file to one file for each day/hour/minute.

Here’s what an example file looks like:

Date,Amount,Currency,Category,Description
2019-04-02,5.45,EUR,Food,Centra
2019-04-03,2.75,EUR,Transport,Bus to work

And here’s the script:

#!/bin/sh

days=${1:-7}

cat *.csv | sed '/^Date/d' > combined.csv.temp

output=$(sqlite3 <<EOF
create table Transactions(Date, Amount, Currency, Category, Description);
.mode csv
.import combined.csv.temp Transactions
.mode list

select 'Amount spent today:',
coalesce(sum(Amount), 0) from Transactions where Date = '$(date +%Y-%m-%d)';

select '';
select 'Last $days days average:',
sum(Amount)/$days, Currency from Transactions where Date > '$(date --date="-$days days" +%Y-%m-%d)'
group by Currency;

select '';
select 'Last $days days by category';
select '=======================';

select Category, sum(Amount) from Transactions
where Date > '$(date --date="-$days days" +%Y-%m-%d)'
group by Category order by sum(Amount) desc;
EOF
      )

rm combined.csv.temp

echo "$output" | sed 's/|/ /g'

This is the output of the command

[leo@leo-arch budget]$ ./budget.sh
Amount spent today: 8.46

Last 7 days average: 15.35 EUR

Last 7 days by category
=======================
Groceries 41.09
Transport 35.06
Food 31.35
[leo@leo-arch budget]$ ./budget.sh 5
Amount spent today: 8.46

Last 5 days average: 11.54 EUR

Last 5 days by category
=======================
Groceries 29.74
Transport 17.06
Food 10.9
[leo@leo-arch budget]$

Rendering GPS traces May 29, 2019 04:20 PM

If you ask a bunch of people to upload GPS traces when they walk/drive and you combine those traces, you can get a rudimentary map. In fact, this is one of the primary data sources of OpenStreetMap. The data for those is freely available, so we can use it in a small project.

To draw simple outlines, iterating over the GPS track points and putting them on an image should be enough. It will give us the main roads and the general city structure, and will be possible to recognize when compared to an actual map.

To begin, let’s get the coordinates of the place we’ll be mapping. In my case, this will be Sheffield. If you go to OpenStreetMap and hit Export, it will let you select and area with a bounding box and get the coordinates of it. We’ll get the coordinates and write it to to our script.

# Area format is left, bottom, right, top
AREA = [-1.4853, 53.3730, -1.4557, 53.3893]

The other thing we should get out of the way is the output size. We should go with a nice 720p picture.

WIDTH  = 1280
HEIGHT = 720

Getting the GPS data

OpenStreetMap provides an API that we can use in order to fetch GPS track data. It gives us the data for a given region in the XML format.

A small disclaimer about the API. It’s normally meant for editing, which means you should try to keep your usage very light. While making this project and iterating on the code, I kept all my API calls in a local cache. I’ll write about this in the future.

You can find the documentation for the API here. Here’s the gist of it.

bbox = ','.join(AREA)
url = 'https://api.openstreetmap.org/api/0.6/trackpoints'
xml = requests.get(url, params={'bbox': bbox}).text

This should get an XML document with the GPS trackpoints. Let’s parse it and get the latitude/longitude pairs. The coordinates are held in <trkpt> tags.

root = ET.fromstring(xml)
selector = './/{http://www.topografix.com/GPX/1/0}trkpt'

for trkpt in root.findall(selector):
  print(trkpt.attrib['lat'], trkpt.attrib['lon'])

As you can see, this is relatively straightforward. The XML selector might look weird. It just means get all the trkpt elements that belong to the namespace of that URL.

Plotting the data

Let’s start with a small example, creating an empty image and drawing a pixel in it.

img = Image.new('L', (WIDTH, HEIGHT), color='white')
img.putpixel((5, 5), 1)

This code will draw a single black pixel on an empty image. The rest of the way should be looking pretty clear now, go through the lat/lon pairs and plot them as pixels. But before we get to that step, there is one more hurdle to get through. And that is mapping the GPS coordinates to image coordinates.

Mapping coordinates for our map

The problem is, we have a 1280x720 image. And we can’t ask Python to put a pixel on (52.6447, -8.6337). We already know the exact area of the map we’re drawing, and the size of our output. What we need to do is get those two ranges and interpolate where a given coordinate falls on our image. For this, we can use the interp function from numpy.

y, x = point
x = int(interp(x, [AREA[0], AREA[2]], [0, WIDTH]))
y = int(interp(y, [AREA[1], AREA[3]], [HEIGHT, 0]))

try:
    img.putpixel((x, y), 1)
except:
    # In case math goes wrong
    pass

Drawing the map

We are know able to get GPS trackpoints and know how to map them to image coordinates. So let’s loop through everything and put the pixels on our map. Since each page has a limit of 5000 points, we should also iterate through the pages.

for page in range(15):
  for point in get_points(AREA, page):
    y, x = point
    x = int(interp(x, [AREA[0], AREA[2]], [0, WIDTH]))
    y = int(interp(y, [AREA[1], AREA[3]], [HEIGHT, 0]))
    try:
      img.putpixel((x, y), 1)
    except:
      pass

Getting points with pagination

Here’s a generator function to return OpenStreetMap trackpoints with pagination.

def get_points(area, page=0):
  bbox = ','.join(map(str, area))
  xml = sess.get('https://api.openstreetmap.org/api/0.6/trackpoints',
    params={'bbox': bbox, 'page': page}).text
  root = ET.fromstring(xml)

  for trkpt in root.findall('.//{http://www.topografix.com/GPX/1/0}trkpt'):
    yield trkpt.attrib['lat'], trkpt.attrib['lon']

Results

Rendering of Tokyo Rendering of Limerick Rendering of Sheffield

If you come up with any cool-looking renders, or better ways to plot this data, either leave a comment about it or send me an email about.

Free Hotel Wifi with Python and Selenium May 29, 2019 04:20 PM

Recently I took my annual leave and decided to visit my friend during the holidays. I stayed at a hotel for a few days but to my surprise, the hotel charged money to use their wifi. In $DEITY‘s year 2000 + 18, can you imagine?

But they are not so cruel. You see, these generous people let you use the wifi for 20 minutes. 20 whole minutes. That’s almost half a Minecraft video.

If they let each device use the internet for a limited amount of time, they must have a way of identifying each device. And a router tells devices apart is by their MAC addresses. Fortunately for us, we can change our MAC address easily.

Enter macchanger

There is a really useful command-line tool called macchanger. It lets you manually change, randomize and restore the MAC address of your devices. The idea here is randomizing our MAC regularly (every 20 minutes) in order to use the free wifi over and over indefinitely.

There are 3 small commands you need to run. This is needed because macchanger can’t work while your network interface is connected to the router.

# Bring network interface down
ifconfig wlp3s0 down

# Get random MAC address
macchanger -r wlp3s0

# Bring the interface back up
ifconfig wlp3s0 up

In the commands above, wlp3s0 is the name of my network interface. You can find yours by running ip a. If you run those commands, you can fire up your browser and you will be greeted with the page asking you to pay or try it for 20 minutes. After your time is up, you can run the commands again and keep doing it.

But this is manual labor, and doing it 3 times an hour is too repetitive. Hmm. What’s a good tool to automate repetitive stuff?

Enter Selenium

First, lets get those commands out of the way. Using the os module, we can run macchanger from our script.

import os

interface = 'wlp3s0'

os.system(f'sudo ifconfig {interface} down')
os.system(f'sudo macchanger -r {interface}')
os.system(f'sudo ifconfig {interface} up')

After these commands our computer should automatically connect to the network as a completely different device. Let’s fire up a browser and try to use the internet.

d = webdriver.Chrome()
d.get('http://example.com')
d.get('https://www.wifiportal.example/cp/sponsored.php')

The sponsored.php URL is where I ended up after pressing the Free Wifi link, so the script should open the registration form for us. Let’s fill the form.

In my case, all it asked for was an email address and a full name. If there are more fields, you can fill them in a similar fashion.

num   = random.randint(0, 99999)
email = f'test{num}@gmail.com'

d.find_element_by_name('email').send_keys(email)
d.find_element_by_name('name').send_keys('John Doe\n')

This should fill the form and press enter to submit it. Afterwards, the portal asked me if I wanted to subscribe to their emails or something like that. Of course, we click Reject without even reading it and close the browser.

d.find_elements_by_class_name('reject')[0].click()
d.close()

After this, you should have an internet connection. You can either run the script whenever you notice your connection is gone, or put it on a cron job / while loop.

Generating Vanity Infohashes for Torrents May 29, 2019 04:20 PM

In the world of Bittorrent, each torrent is identified by an infohash. It is basically the SHA1 hash of the torrent metadata that tells you about the files. And people, when confronted with something that’s supposed to be random, like to control it to some degree. You can see this behaviour in lots of different places online. People try to generate special Bitcoin wallets, Tor services with their nick or 4chan tripcodes that look cool. These are all done by repeatedly generating the hash until you find a result that you like. We can do the exact same thing with torrents as well.

The structure of torrent files

Before we start tweaking our infohash, let’s talk about torrent files first. A torrent file is a bencoded dictionary. It contains information about the files, their names, how large they are and hashes for each piece. This is stored in the info section of the dictionary. The rest of the dictionary includes a list of trackers, the file comment, the creation date and other optional metadata. The infohash is quite literally the SHA1 hash of the info section of the torrent. Any modification to the file contents changes the infohash, while changing the other metadata doesn’t.

This gives us two ways of affecting the hash without touching the file contents. The first one is adding a separate key called vanity and chaning the value of it. While this would be really flexible and cause the least change that the user can see, it adds a non-standard key to our dictionary. Fortunately, torrent files are supposed to be flexible and handle unknown keys gracefully.

The other thing we can do is to add a prefix to the file name. This should keep everything intact aside from a random value in front of our filename.

Parsing the torrent file

First of all, let’s read our torrent file and parse it. For this purpose, I’m using the bencoder module.

import bencoder

target = 'arch-linux.torrent'
with open(target, 'rb') as torrent_file:
    torrent = bencoder.decode(torrent_file.read())

Calculating the infohash

The infohash is the hash of the info section of the file. Let’s write a function to calculate that. We also encode the binary of the hash with base 32 to bring it to the infohash format.

import hashlib
import base64

def get_infohash(torrent):
    encoded = bencoder.encode(torrent[b'info'])
    sha1 = hashlib.sha1(encoded).hexdigest()
    return sha1

Prefixing the name

Let’s do the method with prefixing the name first. We will start from 0 and keep incrementing the name prefix until the infohash starts with cafe.

original_name = torrent[b'info'][b'name'].decode('utf-8')

vanity = 0
while True:
    torrent[b'info'][b'name'] = '{}-{}'.format(vanity, original_name)
    if get_infohash(torrent).startswith('cafe'):
        print(vanity, get_infohash(torrent))
        break
    vanity += 1

This code will increment our vanity number in a loop and print it and the respective infohash when it finds a suitable one.

Adding a separate key to the info section

While the previous section works well, it still causes a change that is visible to the user. Let’s work around that by modifying the data in a bogus key called vanity.

vanity = 0
while True:
    torrent[b'info'][b'vanity'] = str(vanity)
    if get_infohash(torrent).startswith('cafe'):
        print(vanity, get_infohash(torrent))
        break
    vanity += 1

Saving the modified torrent files

While it is possible to do the modification to the file yourself, why not go all the way and save the modified torrent file as well? Let’s write a function to save a given torrent file.

def save_torrent(torrent, name):
    with open(name, 'wb+') as torrent_file:
        torrent_file.write(bencoder.encode(torrent))

You can use this function after finding an infohash that you like.

Cool ideas for infohash criteria

  • Release groups can prefix their infohashes with their name/something unique to them
  • Finding smaller infohashes - should slowly accumulate 0’s in the beginning
  • Infohashes with the least entropy - should make them easier to remember
  • Infohashes with the more digits
  • Infohashes with no digits