Planet Crustaceans

This is a Planet instance for lobste.rs community feeds. To add/update an entry or otherwise improve things, fork this repo.

October 15, 2019

Gustaf Erikson (gerikson)

The Big Short: Inside the Doomsday Machine by Michael Lewis October 15, 2019 12:13 PM

I caught the movie adaptation of this book on a flight, and wanted to get some more background. Honestly, the movie does a good job of summarizing the contents, but also adds some humanizing touches such as the visit to the ground zero of the housing market: Florida. Both are recommended.

October 14, 2019

Pete Corey (petecorey)

Rendering a React Application Across Multiple Containers October 14, 2019 12:00 AM

A few of my recent articles have been embedding limited builds of Glorious Voice Leader directly into the page. At first, this presented an interesting challenge. How could I render a single React application across multiple container nodes, while maintaining shared state between all of them?

While the solution I came up with probably isn’t best practice, it works!

As a quick example, imagine you have a simple React component that manages a single piece of state. The user can change that state by pressing one of two buttons:


const App = () => {
  let [value, setValue] = useState("foo");
  return (
    <div>
      <button onClick={() => setValue("foo")}>
        Value is "{value}". Click to change to "foo"!
      </button>
      <button onClick={() => setValue("bar")}>
        Value is "{value}". Click to change to "bar"!
      </button>
    </div>
  );
};

Normally, we’d render our App component into a container in the DOM using ReactDOM.render:


ReactDOM.render(<App />, document.getElementById('root'));

But what if we want to render our buttons in two different div elements, spread across the page? Obviously, we could build out two different components, one for each button, and render these components in two different DOM containers:


const Foo = () => {
  let [value, setValue] = useState("foo");
  return (
    <button onClick={() => setValue("foo")}>
      Value is "{value}". Click to change to "foo"!
    </button>
  );
};

const Bar = () => {
  let [value, setValue] = useState("foo");
  return (
    <button onClick={() => setValue("bar")}>
      Value is "{value}". Click to change to "bar"!
    </button>
  );
};

ReactDOM.render(<Foo />, document.getElementById('foo'));
ReactDOM.render(<Bar />, document.getElementById('bar'));

But this solution has a problem. Our Foo and Bar components maintain their own versions of value, so a change in one component won’t affect the other.

Amazingly, it turns out that we can create an App component which maintains our shared state, render that component into our #root container, and within App we can make additional calls to ReactDOM.render to render our Foo and Bar components. When we call ReactDOM.render we can pass down our state value and setters for later use in Foo and Bar:


const App = () => {
  let [value, setValue] = useState("foo");
  return (
    <>
      {ReactDOM.render(
        <Foo value={value} setValue={setValue} />,
        document.getElementById("foo")
      )}
      {ReactDOM.render(
        <Bar value={value} setValue={setValue} />,
        document.getElementById("bar")
      )}
    </>
  );
};

Our Foo and Bar components can now use the value and setValue props provided to them instead of maintaining their own isolated state:


const Foo = ({ value, setValue }) => {
  return (
    <button onClick={() => setValue("foo")}>
      Value is "{value}". Click to change to "foo"!
    </button>
  );
};

const Bar = ({ value, setValue }) => {
  return (
    <button onClick={() => setValue("bar")}>
      Value is "{value}". Click to change to "bar"!
    </button>
  );
};

And everything works! Our App is “rendered” to our #root DOM element, though nothing actually appears there, and our Foo and Bar components are rendered into #foo and #bar respectively.

Honestly, I’m amazed this works at all. I can’t imagine this is an intended use case of React, but the fact that it’s still a possibility made my life much easier.

Happy hacking.

October 13, 2019

Derek Jones (derek-jones)

Comparing expression usage in mathematics and C source October 13, 2019 10:11 PM

Why does a particular expression appear in source code?

One reason is that the expression is the coded form of a formula from the application domain, e.g., E=mc^2.

Another reason is that the expression calculates an algorithm/housekeeping related address, or offset, to where a value of interest is held.

Most people (including me, many years ago) think that the majority of source code expressions relate to the application domain, in one-way or another.

Work on a compiler related optimizer, and you will soon learn the truth; most expressions are simple and calculate addresses/offsets. Optimizing compilers would not have much to do, if they only relied on expressions from the application domain (my numbers tool throws something up every now and again).

What are the characteristics of application domain expression?

I like to think of them as being complicated, but that’s because it used to be in my interest for them to be complicated (I used to work on optimizers, which have the potential to make big savings if things are complicated).

Measurements of expressions in scientific papers is needed, but who is going to be interested in measuring the characteristics of mathematical expressions appearing in papers? I’m interested, but not enough to do the work. Then, a few weeks ago I discovered: An Analysis of Mathematical Expressions Used in Practice, by Clare So; an analysis of 20,000 mathematical papers submitted to arXiv between 2000 and 2004.

The following discussion uses the measurements made for my C book, as the representative source code (I keep suggesting that detailed measurements of other languages is needed, but nobody has jumped in and made them, yet).

The table below shows percentage occurrence of operators in expressions. Minus is much more common than plus in mathematical expressions, the opposite of C source; the ‘popularity’ of the relational operators is also reversed.

Operator  Mathematics   C source
=         0.39          3.08
-         0.35          0.19 
+         0.24          0.38
=        0.06          0.04
>         0.041         0.11
         0.037         0.22

The most common single binary operator expression in mathematics is n-1 (the data counts expressions using different variable names as different expressions; yes, n is the most popular variable name, and adding up other uses does not change relative frequency by much). In C source var+int_constant is around twice as common as var-int_constant

The plot below shows the percentage of expressions containing a given number of operators (I've made a big assumption about exactly what Clare So is counting; code+data). The operator count starts at two because that is where the count starts for the mathematics data. In C source, around 99% of expressions have less than two operators, so the simple case completely dominates.

Percentage of expressions containing a given number of operators.

For expressions containing between two and five operators, frequency of occurrence is sort of about the same in mathematics and C, with C frequency decreasing more rapidly. The data disagrees with me again...

Ponylang (SeanTAllen)

Last Week in Pony - October 13, 2019 October 13, 2019 03:19 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

Carlos Fenollosa (carlesfe)

October 12, 2019

Carlos Fenollosa (carlesfe)

US Software companies comply with international law, to their great regret October 12, 2019 04:58 PM

This week has been very heavy on China-related software scandals:

On Apple's side, as usual, ther has been more media coverage:

US companies and entities are forced to apply international law, sometimes breaking universal human rights.

This is a difficult topic. On one hand, States are sovereign. On the other, we should push for a better world. However, to which degree has a private company the right to ignore state rulings? They can, and suffer the consequences. That would be consistent. Are they ready to boycott a whole country, or risk a banishment from that country?

As an individual, the take home message is that if you delegate some of your tasks to a private company, or you rely on a private company in some degree, you risk being unable to access your data or virtual possessions at any time. Be it due to international law, or to some stupid enforcement or terms-of-service bullshit.

Please follow the HN discussions on the "via" links above, they are very informative.

Tags: internet, law

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

October 10, 2019

Tobias Pfeiffer (PragTob)

Video & Slides: Functioning Among Humans (Heart of Clojure) October 10, 2019 03:41 PM

Back in July I had a great time at Heart of Clojure – the first conference who finally allowed me to share my thoughts on the importance of people skills and important people skills. And they were so nice to even record it, so here it is! Slides can be viewed here, on speaker deck, […]

Chris Allen (bitemyapp)

Why I use and prefer GitLab CI October 10, 2019 12:00 AM

In the past, I talked about how to make your CI builds faster using Drone CI. I don't use DroneCI any more and haven't for a couple years now so I wanted to talk about what I use now.

October 07, 2019

Benaiah Mischenko (benaiah)

Configuring Go Apps with TOML October 07, 2019 08:10 PM

Configuring Go Apps with TOML

So you’ve been writing an application in Go, and you’re getting to the point where you have a lot of different options in your program. You’ll likely want a configuration file, as specifying every option on the command-line can get difficult and clunky, and launching applications from a desktop environment makes specifying options at launch even more difficult.

This post will cover configuring Go apps using a simple, INI-like configuration language called TOML, as well as some related difficulties and pitfalls.

TOML has quite a few implementations, including several libraries for Go. I particularly like BurntSushi’s TOML parser and decoder, as it lets you marshal a TOML file directly into a struct. This means your configuration can be fully typed and you can easily do custom conversions (such as parsing a time.Duration) as you read the config, so you don’t have to do them in the rest of your application.

Configuration location

The first question you should ask when adding config files to any app is "where should they go?". For tools that aren’t designed to be run as a service, as root, or under a custom user (in other words, most of them), you should be putting them in the user’s home directory, so they’re easily changed. A few notes:

  • Even if you currently have only one file, you should use a folder and put the config file within it. That way, if and when you do need other files there, you won’t have to clutter the user’s home directory or deal with loading config files that could be in two different locations (Emacs, for instance, supports both ~/.emacs.d/init.el and ~/.emacs for historical reasons, which ends up causing confusing problems when both exist).

  • You should name your configuration directory after your program.

  • You should typically prefix your config directory with a . (but see the final note for Linux, as configuration directories within XDG_CONFIG_HOME should not be so prefixed).

  • On most OSs, putting your configuration files in the user’s “home” directory is typical. I recommend the library go-homedir, rather than the User.Homedir available in the stdlib from os/user. This is because use of os/user uses cgo, which, while useful in many situations, also causes a number of difficulties that can otherwise be avoided - most notably, cross-compilation is no longer simple, and the ease of deploying a static Go binary gets a number of caveats.

  • On Linux specifically, I strongly encourage that you do not put your configuration directory directly in the user’s home directory. Most commonly-used modern Linux distributions use the XDG Base Directory Specification from freedesktop.org, which specifies standard locations for various directories on an end-user Linux system. (Despite this, many applications don’t respect the standard and put their configurations directly in ~ anyway). By default, this is ~/.config/, but it can also be set with the XDG_CONFIG_HOME environment variable. Directories within this should not use a leading ., as the directory is already hidden by default.

The following function should get you the correct location for your config directory on all platforms (if there’s a platform with a specific convention for config locations which I’ve missed, I’d appreciate you letting me know so I can update the post - my email is at the bottom of the page).

import (
    "path/filepath"
    "os"
    "runtime"

    "github.com/mitchellh/go-homedir"
)

var configDirName = "example"

func GetDefaultConfigDir() (string, error) {
    var configDirLocation string

    homeDir, err := homedir.Dir()
    if err != nil {
        return "", err
    }

    switch runtime.GOOS {
    case "linux":
        // Use the XDG_CONFIG_HOME variable if it is set, otherwise
        // $HOME/.config/example
        xdgConfigHome := os.Getenv("XDG_CONFIG_HOME")
        if xdgConfigHome != "" {
            configDirLocation = xdgConfigHome
        } else {
            configDirLocation = filepath.Join(homeDir, ".config", configDirName)
        }

    default:
        // On other platforms we just use $HOME/.example
        hiddenConfigDirName := "." + configDirName
        configDirLocation = filepath.Join(homeDir, hiddenConfigDirName)
    }

    return configDirLocation, nil
}

Within the config folder, you can use any filename you want for your config - I suggest config.toml.

Loading the config file

To load a config file, you’ll first want to define the what config values you’ll use. burntsushi/toml will ignore options in the TOML file that you don’t use, so you don’t have to worry about that causing errors. For instance, here’s the proposed configuration for a project I’m maintaining, wuzz (the keybindings aren’t currently implemented, but I’ve left them in for the sake of demonstration):

type Config struct {
    General GeneralOptions
    Keys    map[string]map[string]string
}

type GeneralOptions struct {
    FormatJSON             bool
    Insecure               bool
    PreserveScrollPosition bool
    DefaultURLScheme       string
}

It’s pretty simple. Note that we use a named struct for GeneralOptions, rather than making Config.General an anonymous struct. This makes nesting options simpler and aids tooling.

Loading the config is quite easy:

import (
    "errors"
    "os"
    
    "github.com/BurntSush/toml"
)

func LoadConfig(configFile string) (*Config, error) {
    if _, err := os.Stat(configFile); os.IsNotExist(err) {
        return nil, errors.New("Config file does not exist.")
    } else if err != nil {
        return nil, err
    }
    
    var conf Config
    if _, err := toml.DecodeFile(configFile, &conf); err != nil {
        return nil, err
    }

    return &conf, nil
}

toml.DecodeFile will automatically populate conf with the values set in the TOML file. (Note that we pass &conf to toml.DecodeFile, not conf - we need to populate the struct we actually have, not a copy). Given the above Config type and the following TOML file…

[general]
defaultURLScheme = "https"
formatJSON = true
preserveScrollPosition = true
insecure = false

[keys]

  [keys.general]
  "C-j" = "next-view"
  "C-k" = "previous-view"
  
  [keys.response-view]
  "<down>" = "scroll-down"

…we’ll get a Config like the following:

Config{
    General: GeneralOptions{
        DefaultURLScheme:       "https",
        FormatJSON:             true,
        PreserveScrollPosition: true,
        Insecure:               false,
    },
    Keys: map[string]map[string]string{
        "general": map[string]string{
            "C-j": "next-view",
            "C-k": "previous-view",
        },
        "response-view": map[string]string{
            "<down>": "scroll-down",
        },
    },
}

Automatically decoding values

wuzz actually uses another value in its config - a default HTTP timeout. In this case, though, there’s no native TOML value that cleanly maps to the type we want - a time.Duration. Fortunately, the TOML library we’re using supports automatically decoding TOML values into custom Go values. To do so, we’ll need a type that wraps time.Duration:

type Duration struct {
    time.Duration
}

Next we’ll need to add an UnmarshalText method, so we satisfy the toml.TextUnmarshaler interface. This will let toml know that we expect a string value which will be passed into our UnmarshalText method.

func (d *Duration) UnmarshalText(text []byte) error {
    var err error
    d.Duration, err = time.ParseDuration(string(text))
    return err
}

Finally, we’ll need to add it to our Config type. This will go in Config.General, so we’ll add it to GeneralOptions:

type GeneralOptions struct {
    Timeout                Duration
    // ...
}

Now we can add it to our TOML file, and toml.DecodeFile will automatically populate our struct with a Duration value!

Input:

[general]
timeout = "1m"
# ...

Equivalent output:

Config{
    General: GeneralOptions{
        Timeout: Duration{
            Duration: 1 * time.Minute
        },
        // ...
    }
}

Default config values

We now have configuration loading, and we’re even decoding a text field to a custom Go type - we’re nearly finished! Next we’ll want to specify defaults for the configuration. We want values specified in the config to override our defaults. Fortunately, toml makes really easy to do.

Remember how we passed in &conf to toml.DecodeFile? That was an empty Config struct - but we can also pass one with its values pre-populated. toml.DecodeFile will set any values that exist in the TOML file, and ignore the rest. First we’ll create the default values:

import (
    "time"
)
var DefaultConfig = Config{
    General: GeneralOptions{
        DefaultURLScheme:       "https",
        FormatJSON:             true,
        Insecure:               false,
        PreserveScrollPosition: true,
        Timeout: Duration{
            Duration: 1 * time.Minute,
        },
    },
    // You can omit stuff from the default config if you'd like - in
    // this case we don't specify Config.Keys
}

Next, we simply modify the LoadConfig function to use DefaultConfig:

func LoadConfig(configFile string) (*Config, error) {
    if _, err := os.Stat(configFile); os.IsNotExist(err) {
        return nil, errors.New("Config file does not exist.")
    } else if err != nil {
        return nil, err
    }

    conf := DefaultConfig
    if _, err := toml.DecodeFile(configFile, &conf); err != nil {
        return nil, err
    }

    return &conf, nil
}

The important line here is conf := DefaultConfig - now when conf is passed to toml.DecodeFile it will populate that.

Summary

I hope this post helped you! you should now be able to configure Go apps using TOML with ease.

If this post was helpful to you, or you have comments or corrections, please let me know! My email address is at the bottom of the page. I’m also looking for work at the moment, so feel free to get in touch if you’re looking for developers.

Complete code

package config

import (
    "errors"
    "path/filepath"
    "os"
    "runtime"
    "time"

    "github.com/BurntSushi/toml"
    "github.com/mitchellh/go-homedir"
)

var configDirName = "example"

func GetDefaultConfigDir() (string, error) {
    var configDirLocation string

    homeDir, err := homedir.Dir()
    if err != nil {
        return "", err
    }

    switch runtime.GOOS {
    case "linux":
        // Use the XDG_CONFIG_HOME variable if it is set, otherwise
        // $HOME/.config/example
        xdgConfigHome := os.Getenv("XDG_CONFIG_HOME")
        if xdgConfigHome != "" {
            configDirLocation = xdgConfigHome
        } else {
            configDirLocation = filepath.Join(homeDir, ".config", configDirName)
        }

    default:
        // On other platforms we just use $HOME/.example
        hiddenConfigDirName := "." + configDirName
        configDirLocation = filepath.Join(homeDir, hiddenConfigDirName)
    }

    return configDirLocation, nil
}

type Config struct {
    General GeneralOptions
    Keys    map[string]map[string]string
}

type GeneralOptions struct {
    DefaultURLScheme       string
    FormatJSON             bool
    Insecure               bool
    PreserveScrollPosition bool
    Timeout                Duration
}

type Duration struct {
    time.Duration
}

func (d *Duration) UnmarshalText(text []byte) error {
    var err error
    d.Duration, err = time.ParseDuration(string(text))
    return err
}

var DefaultConfig = Config{
    General: GeneralOptions{
        DefaultURLScheme:       "https",
        FormatJSON:             true,
        Insecure:               false,
        PreserveScrollPosition: true,
        Timeout: Duration{
            Duration: 1 * time.Minute,
        },
    },
}

func LoadConfig(configFile string) (*Config, error) {
    if _, err := os.Stat(configFile); os.IsNotExist(err) {
        return nil, errors.New("Config file does not exist.")
    } else if err != nil {
        return nil, err
    }

    conf := DefaultConfig
    if _, err := toml.DecodeFile(configFile, &conf); err != nil {
        return nil, err
    }

    return &conf, nil
}

If you’d like to leave a comment, please email benaiah@mischenko.com

Andrew Owen (yumaikas)

What 8 years of side projects has taught me October 07, 2019 08:10 PM

I’ve been a professional software developer for almost 8 years now. I’ve been paid to write a lot of software in those years. Far more interesting to me has been the recurring themes that have come up in my side-projects and in the software I’ve been personally compelled to write.

Lesson 0: Programming in a void is worthless

When I wanted to learn programming, I always had to come to the keyboard with a purpose. I couldn’t just sit down and start writing code, I had to have built an idea of where I was going.

For that reason, I’ve always used side-projects as a mean of learning programming languages. When I wanted to learn QBasic, there were a number of games, one based in space, another was a fantasy game. When I wanted to learn Envelop Basic, I attempted making a Yahtzee Clone, a Space Invaders Clone, a Hotel running game, and made a MicroGame based on the ones I’d seen videos of in WarioWare DIY.

When I wanted to learn C#, I went through a book, but I also, at the same time, worked on building a Scientific Calculator. (put a pin in that idea). When I wanted to learn Go, I wrote the CMS for the blog you’re reading right now. To pick up Lua, I used Love2D in several game jams. The only reason I have more than a passing familiarity with Erlang is because I used it for idea.junglecoder.com

Most times, when I’ve tried to learn a programming technology without a concrete goal to get something built, it is hard for me to maintain interest. That hasn’t kept me from trying out a lot of things in the past, but it’s the ones that allowed me to build useful or interesting things that have stuck with me the most. Right now, for side-projects, that list includes Go, Lua, Tcl, and Bash.

Lesson 1: Building a programming language is hard, but rewarding

Ever since I first started cutting my teeth on C#, the ideas of parsing have held a certain fascination to me. Like I said before, I started wanting to write a scientific calculator. But, because I was a new programmer, with no idea of what building a scientific calculator should look like, I did a lot of inventing things from first principles. It felt like a divine revelation when I reasoned out a add/multiply algorithm for parsing numbers. It also took me the better part of two weeks to puzzle it out.

I was so proud of that, in fact, that I copied that code into one of my work projects, a fact which really amuses me now that I know about the existance of Regex and Int.Parse().

Eventually, I worked out a very basic notion of doing recursive descent parsing, and some tree evaluation, so that, on a good day, I had a basic, but working, math expression evaluator.

Working on that calculator, however, set me on a course of wanting to understand how programming languages worked. In the process of wanting to understand them, I’ve looked over papers on compilers more than once, but never quite had the patience to actually write one out. In the process of wanting to make a programming language, I ended up writing two before PISC. One that was a “I want to write something in a night that is Turing Complete” langauge that was basically a bastard version of assembly. At the time, I called it SpearVM. I had intended it to be a compilation target for some higher level language, but it mostly just served as stepping stone for the next two projects.

The second one was a semester-long moonshot project where I wanted to try to make a visual programming language, using either Java Swing or JavaFX, inspired by Google’s Blockly environment. Unfortunately, I could not figure out nesting, so I giving the ideas I’d had in SpearVM a visual representation, and using that for my class assignment.

The combination of all of these experiences, and discovering the Factor language, set me thinking about trying to build a programming language that was stack-based, especially since parsing it seemed a far easier task than what I’d been trying to do until then. A couple late nights later, and I’d built out a prototype in Go.

I’ve had a number of co-workers impressed that I’ve written a scripting language. Thing is, it took me like 7 false starts to find a way to do it that made sense to me (and that was a stack-based language with almost 0 lexing). It’s only now, that I’m on the other side of that learning experience, that I’d feel comfortable approach writing a language with C-like syntax. PISC, as I’ve had time to work on it, has actually started to develop more things like parsing, lexing, and even compiling. In fact, I’ve got a small prototype of a langauge called Tinscript that isn’t nearly so post-fix oriented as PISC, though it’s still stack based.

And, PISC, to boot, is still what I’d consider easy-mode when it comes to developing a programming language. Factor, Poprc, or even a run-of-the mill C-like language all strike me as projects that take more tenacity to pull off.

Lesson 2: Organizing my thoughts is important, but tricky to figure out

If the early years of my programming side-projects often focused on how to build programming languages, and how to better understand computers, the more recent years have a had a much stronger focus on how to harness computers to augment my mind. A big focus here was for me to find ways to reduce the working set I needed in my mind at any given time. This has resulted in no less than 7 different systems for trying to help keep track of things.

  • A TCL application for launching programs I used on a regular basis.
  • A C# application for the same, but with a few more bells and whistles
  • Trying out Trello
  • Trying out various online outliners, ultimately being satisfied with none of them.
  • A months-long foray into trying to learn and apply Org-mode and Magit in Emacs, and ultimately giving up due to slowness on Windows, and the fact that my org-files kept getting too messy.
  • ideas.junglecoder.com, a place meant for me to shunt my stray thoughts to get them off my mind during the work day.
  • Another TCL application called PasteKit, which was designed to help me juggle all of the 4-6 digit numbers I was juggling at one of my jobs.
  • A C# version of PasteKit, that also had a customizeable list of launchers
  • .jumplist.sh
  • Bashmarks, but for CMD.exe

These are all approaches I’ve invested non-trivial amounts of time into over the past three years, trying to figure out a way to organize my thoughts as a software developer, but none of them lasted much longer than a month or so.

All of this came to a head during Thanksgiving weekend of 2018. My work at Greenshades often involved diving deep into tickets and opening a lot of SQL scripts in SSMS, and I had found no good way to organize them all. So, in a move that felt rather desperate at the time, I wrote a C# program that was a simple journal, but one that had a persistent search bar, and stored all of its entries in a SQLite database. And I used a simple tagging scheme for the entries, of marking them with things like @ticket65334, and displaying the most recent 5 notes.

It was finally a system that seemed to actually work for how I liked to think about things. The UI was a fairly simple 3-column layout. In the leftmost column, I had a “scratchpad” where I kept daily notes of what I’d worked on, in the middle I had my draft pad, and on the right I had the feed of notes, based on the search I’d done. I also had a separate screen dedicated to searching through all the notes that had previously been recorded.

There were several benefits to how this system worked:

  • It allowed me to forget things by putting notes under a different tags. That meant they wouldn’t show up on my focused feed, but that I could get back to them later.
  • It allowed me to regain context after getting interrupted much easier. - It gave me a virtual rubber duck. Since I often was trying to figure out issues by writing them to my teammates anyway, the journal gave me a very good first port of call when my stream of consciousness got blocked by an obstacle. This helped dramatically with keeping me off distracting websites like Hackernews or Reddit.
  • It allowed old information to fall out of relevance. One of the biggest problems with all the various tracking systems I’d used before, especially Trello and Org-mode, is that the system filled up, it was hard for old items to fall out of relevance without also being just a bit harder to access. Due to the nature of the feed, this system made it much more natural for information to fall off. And if I wanted it to stick around, I could just copy the relevant bits to a new note, which I often do.

All of this added up to me feeling like I’d found a missing piece of my mind. Almost like I’d created a REPL for my thought process.

Unfortunately, I don’t have that C# version any more. I do have a Go/Lua version, which is webapp based, though I still need to put some time into making the feedback-loop for it tighter, since my first versions weren’t quite as tightly focused on that, as much as they were focused on replicating the UI layout of the C# version. I’d argue that the tight feedback loop that the C# version had would be more important now, and I’ve slowly been working on adding it back.

The nice thing about the Go/Lua journal is that it’s far more flexible than the C# version, due to being able to write pages in Lua. Which means I’ll be able to

Lesson 3: Search is a great tool for debugging and flexible organization

Exhaustive string search of both code and notes has proven to be a surprisingly effective tool for understanding and cataloging large systems for me. To this end, Gills (my journaling software), Everything Search (search over the paths and file names on your laptop) and RipGrep have been extremely handy tools to have on hand. The nice thing about search as a tool is that it can be adapted into other things quite nicely. In fact, I would argue that fast search, both via Google, and via the tools I’d mentioned above, is one of the more influential changes we’ve seen in programming in the last 20 years.

Coda: Sticky ideas

8 years is a long time, and there are lot more ideas that I’d like to get into later. However, these are the ideas and things I’ve worked on that have proven to be surprisingly sticky. Perhaps they might help you, or give you some ideas of where to focus.

Published September 9th, 2019

Jan van den Berg (j11g)

Iedere dag vrij – Bob Crébas October 07, 2019 08:00 PM

I remember exactly where I was when, in 2004, I heard that Dutch ad site marktplaats.nl was sold for a staggering 224,5 million euros to eBay. A polder Cinderella story.

This success was, however, no accident. Of course, luck was involved, but this is true of all successful businesses. A few years after this deal Bob Crébas (don’t forget the acute accent), wrote down his experiences that lead to this deal. And this has resulted in a very fun auto-biography.

Iedere dag vrij (Every day off) – Bob Crébas (2006) – 238 pages

Bob

A former, staunch anti-nuclear energy, jobless and musically inclined hippie, from a farmer family, who was not particularly concerned with appearance and image, first grew a thrift-store chain into a multi-million euro business before deciding to jump on the internet bandwagon. And then he defied several odds (there were *many* competitors) before striking gold with this internet thing.

I read this book in one sitting: it is an absolutely fun, well-written and energizing story. And the internet deal (my primary interest) is only a small part of this story. Which is proof that this is a balanced story and that there is more to the writer than just this one deal.

I particularly liked how he weaved the entrepreneurial and pioneering spirit of the new land and the zeitgeist of the 60s and 70s into this story. And of course, the bands he played in are absolutely fantastic fun to read about.

Bob comes off as an interesting character and this auto-biography seems to be the accumulation of someone who is able to define and articulate his life philosophy succinctly: it’s about being not having, and it’s about creating and giving and not taking.

The post Iedere dag vrij – Bob Crébas appeared first on Jan van den Berg.

A Moveable Feast – Ernest Hemingway October 07, 2019 07:49 PM

Hemingway, the writers’ writer, is famously known for having spent his early years in Paris. Freshly married, this struggling and then unknown writer was honing his craft and subsequently defining what it means to be a writer in a vibrant post World War I Paris. Where he wrote his first big novel.

A Moveable Feast – Ernest Hemingway (1964) – 192 pages

In later life Hemingway wrote up his 5 year experience in a couple of loosely related stories. Which involve interactions with other writers (mainly Scott Fitzgerald) and poets. And which offer very specific details (drinks, prices, addresses etc.). Which is almost odd, since the stories were written some forty years later? These stories were posthumously bundled and released as this memoir.

This memoir offers a perfect insight in understanding Hemingway better and his writing. Blunt, scarce, dead-serious (to a point of being humourless even), and without pretense, A Moveable Feast is quintessential Hemingway and a must-read for anyone who wants to better understand him.

The post A Moveable Feast – Ernest Hemingway appeared first on Jan van den Berg.

Pete Corey (petecorey)

Generating Guitar Chords with Cartesian Products October 07, 2019 12:00 AM

Given two or more lists, like [1, 2] and [3, 4], the Cartesian product of those lists contains all ordered combinations of the elements within those lists: [1, 3], [1, 4], [2, 3], and [2, 4]. This may not seem like much, but Cartesian products are an algorithmic superpower. Maybe it’s J’s subtle influence over my programming style, but I find myself reaching more and more for Cartesian products in the algorithms I write, and I’m constantly awed by the simplicity and clarity the bring to my solutions.

As an example of how useful they can be, let’s look at the problem of generating all possible guitar chord voicings, like I do in Glorious Voice Leader. As a quick aside, if you want to know more about Glorious Voice Leader, check out last week’s post!

Imagine we’re trying to generate all possible C major chord voicings across a guitar’s fretboard. That is, we’re trying to find all playable combinations of the notes C, E, and G. How would we do this?

One approach, as you’ve probably guessed, is to use Cartesian products!

Let’s assume that we have a function, findNoteOnFretboard, that gives us all the locations (zero-based string/fret pairs) of a given note across the fretboard. For example, if we pass it a C (0 for our purposes), we’ll receive an array of string/fret pairs pointing to every C note on the fretboard:


[[0,8],[1,3],[1,15],[2,10],[3,5],[3,17],[4,1],[4,13],[5,8]]

Plotted on an actual guitar fretboard, we’d see all of our C notes exactly where we’d expect them to be:

Now imagine we’ve done this for each of our notes, C, E, and G:


let cs = findNoteOnFretboard(frets, strings, tuning)(0);
let es = findNoteOnFretboard(frets, strings, tuning)(4);
let gs = findNoteOnFretboard(frets, strings, tuning)(7);

The set of all possible voicings of our C major chord, or voicings that contain one of each of our C, E, and G notes, is just the cartesian product of our cs, es, and gs lists!


let voicings = _.product(cs, es, gs);

We’re using the lodash.product here, rather than going through the process of writing our own Cartesian product generator.

We can even generalize this to any given array of notes, and wrap it up in a function:


const voicings = (
  notes,
  tuning = [40, 45, 50, 55, 59, 64],
  frets = 18,
  strings = _.size(tuning)
) =>
  _.chain(notes)
    .map(findNoteOnFretboard(frets, strings, tuning))
    .thru(notesOnFretboard => _.product(...notesOnFretboard))
    .value();

Finding Notes on the Fretboard

So that’s great and all, but how do we implement our findNoteOnFretboard function? With Cartesian products, of course! We’ll generate a list of every string and fret position on the fretboard by computing the Cartesian product of each of our possible string and fret values:


const findNoteOnFretboard = (frets, strings, tuning) => note =>
  _.chain(_.product(_.range(strings), _.range(frets)))
    .value();

Next, we’ll need to filter down to just the string/fret pairs that point to the specified note:


const isNote = (note, tuning) => ([string, fret]) =>
  (tuning[string] + fret) % 12 === note;

const findNoteOnFretboard = (frets, strings, tuning) => note =>
  _.chain(_.product(_.range(strings), _.range(frets)))
    .filter(isNote(note, tuning))
    .value();

The isNote helper function returns whether the note at the given string/fret is the note we’re looking for, regardless of octave.

Filtering Out Doubled Strings

Currently, our chord voicing generator looks like this:


const isNote = (note, tuning) => ([string, fret]) =>
  (tuning[string] + fret) % 12 === note;

const findNoteOnFretboard = (frets, strings, tuning) => note =>
  _.chain(_.product(_.range(strings), _.range(frets)))
  .filter(isNote(note, tuning))
  .value();

const voicings = (
  notes,
  tuning = [40, 45, 50, 55, 59, 64],
  frets = 18,
  strings = _.size(tuning)
) =>
  _.chain(notes)
    .map(findNoteOnFretboard(frets, strings, tuning))
    .thru(notesOnFretboard => _.product(...notesOnFretboard))
    .value();

Not bad. We’ve managed to generate all possible voicings for a given chord in less than twenty lines of code! Unfortunately, we have a problem. Our solution generates impossible voicings!

The first problem is that it can generate voicings with two notes on the same string:

On a stringed instrument like the guitar, it’s impossible to sound both the C and E notes simultaneously. We’ll need to reject these voicings by looking for voicings with “doubled strings”. That is, voicings with two or more notes played on the same string:


const voicings = (
  notes,
  tuning = [40, 45, 50, 55, 59, 64],
  frets = 18,
  strings = _.size(tuning)
) =>
  _.chain(notes)
    .map(findNoteOnFretboard(frets, strings, tuning))
    .thru(notesOnFretboard => _.product(...notesOnFretboard))
    .reject(hasDoubledStrings)
    .value();

Our hasDoubledStrings helper simply checks if the size of the original voicing doesn’t match the size of our voicing after removing duplicated strings:


const hasDoubledStrings = chord =>
  _.size(chord) !==
  _.chain(chord)
    .map(_.first)
    .uniq()
    .size()
    .value();

Filtering Out Impossible Stretches

Unfortunately, our solution has one last problem. It can generate chords that are simply too spread out for any human to play. Imagine trying to stretch your hand enough to play this monster of a voicing:

No good. We’ll need to reject these voicings that have an unplayable stretch:


const voicings = (
  notes,
  tuning = [40, 45, 50, 55, 59, 64],
  frets = 18,
  maxStretch = 5,
  strings = _.size(tuning)
) =>
  _.chain(notes)
    .map(findNoteOnFretboard(frets, strings, tuning))
    .thru(notesOnFretboard => _.product(...notesOnFretboard))
    .reject(hasDoubledStrings)
    .reject(hasUnplayableStretch(maxStretch))
    .value();

Let’s keep things simple for now and assume that an “unplayable stretch” is anything over five frets in distance from one note in the voicing to another.


const hasUnplayableStretch = maxStretch => chord => {
  let [, min] = _.minBy(chord, ([string, fret]) => fret);
  let [, max] = _.maxBy(chord, ([string, fret]) => fret);
  return max - min > maxStretch;
};

Expansion and Contraction

Our voicings function now generates all possible voicings for any given set of notes. A nice way of visualizing all of these voicings on the fretboard is with a heat map. Here are all of the C major voicings we’ve generated with our new Cartesian product powered voicings function:

The darker the fret, the more frequently that fret is used in the set of possible voicings. Click any fret to narrow down the set of voicings.

The Cartesian product, at least in the context of algorithms, embodies the idea of expansion and contraction. I’ve found that over-generating possible results, and culling out impossibilities leads to incredibly clear and concise solutions.

Be sure to add the Cartesian product to your programming tool box!

#cs, #double, #stretch, #all { width: 100%; } #all { cursor: pointer; } #cs .fretboard, #cs canvas, #double .fretboard, #double canvas, #stretch .fretboard, #stretch canvas, #all .fretboard, #all canvas { width: 120% !important; margin-left: -10%; }

October 06, 2019

Derek Jones (derek-jones)

Cost ratio for bespoke hardware+software October 06, 2019 09:32 PM

What percentage of the budget for a bespoke hardware/software system is spent on software, compared to hardware?

The plot below has become synonymous with this question (without the red line, which highlights 1973), and is often used to claim that software costs are many times more than hardware costs.

USAF bespoke hardware/Software cost ratio from 1955 to 1980.

The paper containing this plot was published in 1973 (the original source is a Rome period report), and is an extrapolation of data I assume was available in 1973, into what was then the future. The software and hardware costs are for bespoke command and control systems delivered to the U.S. Air Force, not commercial off-the-shelf solutions or even bespoke commercial systems.

Does bespoke software cost many times more than the hardware it runs on?

I don’t have any data that might be used to answer this questions, to any worthwhile degree of accuracy. I know of situations where I believe the bespoke software did cost a lot more than the hardware, and I know of some where the hardware cost more (I have never been privy to exact numbers on large projects).

Where did the pre-1973 data come from?

The USAF funded the creation of lots of source code, and the reports cite hardware and software figures from 1972.

To summarise: the above plot is for USAF spending on bespoke command and control hardware and software, and is extrapolated from 1973 into the future.

Unrelenting Technology (myfreeweb)

FreeBSD and custom firmware on the Google Pixelbook October 06, 2019 08:38 PM

Back in 2015, I jumped on the ThinkPad bandwagon by getting an X240 to run FreeBSD on. Unlike most people in the ThinkPad crowd, I actually liked the clickpad and didn’t use the trackpoint much. But this summer I’ve decided that it was time for something newer. I wanted something..

  • lighter and thinner (ha, turns out this is actually important, I got tired of carrying a T H I C C laptop - Apple was right all along);
  • with a 3:2 display (why is Lenovo making these Serious Work™ laptops 16:9 in the first place?? 16:9 is awful in below-13-inch sizes especially);
  • with a HiDPI display (and ideally with a good size for exact 2x scaling instead of fractional);
  • with USB-C ports;
  • without a dGPU, especially without an NVIDIA GPU;
  • assembled with screws and not glue (I don’t necessarily need expansion and stuff in a laptop all that much, but being able to replace the battery without dealing with a glued chassis is good);
  • supported by FreeBSD of course (“some development required” is okay but I’m not going to write big drivers);
  • how about something with open source firmware, that would be fun.

The Qualcomm aarch64 laptops were out because embedded GPU drivers like freedreno (and UFS storage drivers, and Qualcomm Wi-Fi..) are not ported to FreeBSD. And because Qualcomm firmware is very cursed.

Samsung’s RK3399 Chromebook or the new Pinebook Pro would’ve been awesome.. if I were a Linux user. No embedded GPU drivers on FreeBSD, again. No one has added the stuff needed for FDT/OFW attachment to LinuxKPI. It’s rather tedious work, so we only support PCIe right now. (Can someone please make an ARM laptop with a PCIe GPU, say with an MXM slot?)

So it’s still gonna be amd64 (x86) then.

I really liked the design of the Microsoft Surface Book, but the iFixit score of 1 (one) and especially the Marvell Wi-Fi chip that doesn’t have a driver in FreeBSD are dealbreakers.

I was considering a ThinkPad X1 Carbon from an old generation - the one from the same year as the X230 is corebootable, so that’s fun. But going back in processor generations just doesn’t feel great. I want something more efficient, not less!

And then I discovered the Pixelbook. Other than the big huge large bezels around the screen, I liked everything about it. Thin aluminum design, a 3:2 HiDPI screen, rubber palm rests (why isn’t every laptop ever doing that?!), the “convertibleness” (flip the screen around to turn it into.. something rather big for a tablet, but it is useful actually), a Wacom touchscreen that supports a pen, mostly reasonable hardware (Intel Wi-Fi), and that famous coreboot support (Chromebooks’ stock firmware is coreboot + depthcharge).

So here it is, my new laptop, a Google Pixelbook.

What is a Chromebook, even

The write protect screw is kind of a meme. All these years later, it’s That Thing everyone on various developer forums associates with Chromebooks. But times have moved on. As a reaction to glued devices and stuff, the Chrome firmware team has discovered a new innovative way of asserting physical presence: sitting around for a few minutes, pressing the power button when asked. Is actually pretty clever though, it is more secure than.. not doing that.

Wait, what was that about?

Let’s go back to the beginning and look at firmware security in Chromebooks and other laptops.

These devices are designed for the mass market first. Your average consumer trusts the vendor and (because they’ve read a lot of scary news) might be afraid of scary attackers out to install a stealthy rootkit right into their firmware. Businesses are even more afraid of that, and they push for boot security on company laptops even more. This is why Intel Boot Guard is a thing that the vast majority of laptops have these days. It’s a thing that makes sure only the vendor can update firmware. Evil rootkits are out. Unfortunately, the user is also out.

Google is not like most laptop vendors.

Yes, Google is kind of a surveillance capitalism / advertising monster, but that’s not what I’m talking about here. Large parts of Google are very much driven by FOSS enthusiasts. Or something. Anyway, the point is that Chromebooks are based on FOSS firmware and support user control as much as possible. (Without compromising regular-user security, but turns out these are not conflicting goals and we can all be happy.)

Instead of Boot Guard, Google has its own way of securing the boot process. The root of trust in modern (>=2017) devices is a special Google Security Chip, which in normal circumstances also ensures that only Google firmware runs on the machine, but:

  • if you sit through the aforementioned power-button-clicking procedure, you get into Developer Mode: OS verification is off, you have a warning screen at boot, and you can press Ctrl-D to boot into Chrome OS, or (if you enabled this via a command run as root) Ctrl-L to open SeaBIOS.

    • Here’s the fun part.. it doesn’t have to be SeaBIOS. You can flash any Coreboot payload into the RW_LEGACY slot right from Chrome OS, reboot, press a key and you’re booting that payload!
  • if you also buy or solder a special cable (“SuzyQable”) and do the procedure a couple times more, your laptop turns into the Ultimate Open Intel Firmware Development Machine. Seriously.

    • the security chip is a debug chip too! Case Closed Debugging gives you serial consoles for the security chip itself, the embedded controller (EC) and the application processor (AP, i.e. your main CPU), and it also gives you a flasher (via special flashrom for now, but I’m told there’s plans to upstream) that allows you to write AP and EC firmware;
    • some security is still preserved with all the debugging: you can (and should) set a CCD password, which lets you lock-unlock the debug capabilities and change write-protect whenever you want, so that only you can flash firmware (at least without opening the case and doing very invasive things, the flash chip is not even SOIC anymore I think);
    • and you can hack without fear: the security chip is not brickable! Yes, yes, that means the chip is only a “look but don’t touch” kind of open source, it will only boot Google-signed firmware. Some especially paranoid people think this is An NSA Backdoor™. I think this is an awesome way to allow FULL control of the main processor and the EC, over just a cable, with no way of bricking the device! And to solve the paranoia, reproducible builds would be great.

You mentioned something about FreeBSD in the title?

Okay, okay, let’s go. I didn’t even want to write an introduction to Chromebooks but here we are. Anyway, while waiting for the debug cable to arrive, I’ve done a lot of work on FreeBSD, using the first method above (RW_LEGACY).

SeaBIOS does not have display output working in OSes that don’t specifically support the Coreboot framebuffer (OpenBSD does, FreeBSD doesn’t), and I really just hate legacy BIOS, so I’ve had to install a UEFI implementation into RW_LEGACY since I didn’t have the cable yet. My own EDK2 build did not work (now I see that it’s probably because it was a debug build and that has failing assertions). So I’ve downloaded MrChromebox’s full ROM image, extracted the payload using cbfstool and flashed that. Boom. Here we go, press Ctrl-L for UEFI. Nice. Let’s install FreeBSD.

The live USB booted fine. With the EFI framebuffer, an NVMe SSD and a PS/2 keyboard it was a working basic system. I’ve resized the Chrome OS data partition (Chrome OS recovers from that fine, without touching custom partitions), found that there’s already an EFI system partition (with a GRUB2 setup to boot Chrome OS, which didn’t boot like that o_0), installed everything and went on with configuration and developing support for more hardware.

(note: I’m leaving out the desktop configuration part here, it’s mostly a development post; I use Wayfire as my display server if you’re curious.)

So how’s the hardware?

Wi-Fi and Bluetooth

Well, that was easy. The Pixelbook has an Intel 7265. The exact same wireless chip that was in my ThinkPad. So, Wi-Fi works great with iwm.

Bluetooth.. if this was the newer 8265, would’ve already just worked :D

These Intel devices present a “normal” ubt USB Bluetooth adapter, except it only becomes normal if you upload firmware into it, otherwise it’s kinda dead. (And in that dead state, it spews interrupts, raising the idle power consumption by preventing the system from going into package C7 state! So usbconfig -d 0.3 power_off that stuff.) FreeBSD now has a firmware uploader for the 8260/8265, but it does not support the older protocol used by the 7260/7265. It wouldn’t be that hard to add that, but no one has done it yet.

Input devices

Keyboard

Google kept the keyboard as good old PS/2, which is great for ensuring that you can get started with a custom OS with a for-sure working keyboard.

About the only interesting thing with the keyboard was the Google Assistant key, where the Win key usually is. It was not recognized as anything at all. I used DTrace to detect the scancode without adding prints into the kernel and rebooting:

dtrace -n 'fbt::*scancode2key:entry { printf("[st %x] %x?\n", *(int*)arg0, arg1); } \
  fbt::*scancode2key:return { printf("%x\n", arg1);  }'

And wrote a patch to interpret it as a useful key (right meta, couldn’t think of anything better).

Touch*

The touchpad and touchscreen are HID-over-I²C, like on many other modern laptops. I don’t know why this cursed bus from the 80s is gaining popularity, but it is. At least FreeBSD has a driver for Intel (Synopsys DesignWare really) I²C controllers.

(Meanwhile Apple MacBooks now use SPI for even the keyboard. FreeBSD has an Intel SPI driver but right now it only supports ACPI attachment for Atoms and such, not PCIe yet.)

The even better news is that there is a nice HID-over-I²C driver in development as well. (note: the corresponding patch for configuring the devices via ACPI is pretty much a requirement, uncomment -DHAVE_ACPI_IICBUS in the iichid makefile too to get that to work. Also, upcoming Intel I²C improvement patch.)

The touchscreen started working with that driver instantly. (The multi-touch part, not the pen support - that requires a pen tablet driver, which I might write when I have time.)

The touchpad was.. a lot more “fun”. The I²C bus it was on would just appear dead. After some debugging, it turned out that the in-progress iichid driver was sending a wrong extra out-of-spec command, which was causing Google’s touchpad firmware to throw up and lock up the whole bus.

But hey, nice bug discovery, if any other device turns out to be as strict in accepting input, no one else would have that problem.

Another touchpad thing: by default, you have to touch it with a lot of pressure. Easily fixed in libinput:

% cat /usr/local/etc/libinput/local-overrides.quirks
[Eve touchpad]
MatchUdevType=touchpad
AttrPressureRange=12:6

Display backlight brightness

This was another “fun” debugging experience. The intel_backlight console utility (which was still the thing to use on FreeBSD) did nothing.

I knew that the i915 driver on Chrome OS could adjust the brightness, so I made it work here too, and all it took is:

  • adding more things to LinuxKPI to allow uncommenting the brightness controls in i915kms;
  • (and naturally, uncommenting them);
  • finding out that this panel uses native DisplayPort brightness configured via DPCD (DisplayPort Configuration Data), enabling compat.linuxkpi.i915_enable_dpcd_backlight="1" in /boot/loader.conf;
  • finding out that there’s a fun bug in the.. hardware, sort of:

    • the panel reports that it supports both DPCD backlight and a direct PWM line (which is true);
    • Google/Quanta/whoever did not connect the PWM line;
    • (the panel is not aware of that);
    • the i915 driver prefers the PWM line when it’s reported as available.

Turns out there was a patch sent to Linux to add a “prefer DPCD” toggle, but for some reason it was not merged. The patch does not apply cleanly so I just did a simpler hack version:

--- i/drivers/gpu/drm/i915/intel_dp_aux_backlight.c
+++ w/drivers/gpu/drm/i915/intel_dp_aux_backlight.c
@@ -252,8 +252,12 @@ intel_dp_aux_display_control_capable(struct intel_connector *connector)
         * the panel can support backlight control over the aux channel
         */
        if (intel_dp->edp_dpcd[1] & DP_EDP_TCON_BACKLIGHT_ADJUSTMENT_CAP &&
-           (intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_AUX_SET_CAP) &&
-           !(intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_PWM_PIN_CAP)) {
+           (intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_AUX_SET_CAP)
+/* for Pixelbook (eve), simpler version of https://patchwork.kernel.org/patch/9618065/ */
+#if 0
+            && !(intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_PWM_PIN_CAP)
+#endif
+                       ) {
                DRM_DEBUG_KMS("AUX Backlight Control Supported!\n");
                return true;
        }

And with that, it works, with 65536 steps of brightness adjustment even.

Suspend/resume

The Pixelbook uses regular old ACPI S3 sleep, not the fancy new S0ix thing, so that’s good.

On every machine with a TPM though, you have to tell the TPM to save state before suspending, otherwise you get a reset on resume. I already knew this because I’ve experienced that on the ThinkPad.

The Google Security Chip runs an open-source TPM 2.0 implementation (fun fact, written by Microsoft) and it’s connected via… *drum roll* I²C. Big surprise (not).

FreeBSD already has TPM 2.0 support in the kernel, the userspace tool stack was recently added to Ports as well. But of course there was no support for connecting to the TPM over I²C, and especially not to the Cr50 (GSC) TPM specifically. (it has quirks!)

I wrote a driver (WIP) hooking up the I²C transport (relies on the aforementioned ACPI-discovery-of-I²C patch). It does not use the interrupt (I found it buggy: at first attachment, it fires continuously, and after a reattach it stops completely) and after attach (or after system resume) the first command errors out, but that can be fixed and other than that, it works. Resume is fixed, entropy can be harvested, it could be used for SSH keys too.

Another thing with resume: I’ve had to build the kernel with nodevice sdhci to prevent the Intel SD/MMC controller (which is not attached to anything here - I’ve heard that the 128GB model might be using eMMC instead of NVMe but that’s unclear) from hanging for a couple minutes on resume.

Dynamic CPU frequency

At least on the stock firmware, the old-school Intel SpeedStep did not work because the driver could not find some required ACPI nodes (perf or something).

Forget that, the new Intel Speed Shift (which lets the CPU adjust frequency on its own) works nicely with the linked patch.

Tablet mode switch

When the lid is flipped around, the keyboard is disabled (unless you turn the display brightness to zero, I’ve heard - which is fun because that means you can connect a montior and have a sort-of computer-in-a-keyboard look, like retro computers) and the system gets a notification (Chrome OS reacts to that by enabling tablet mode).

Looking at the DSDT table in ACPI, it was quite obvious how to support that notification:

Device (TBMC) {
    Name (_HID, "GOOG0006")  // _HID: Hardware ID
    Name (_UID, One)  // _UID: Unique ID
    Name (_DDN, "Tablet Motion Control")  // _DDN: DOS Device Name
    Method (TBMC, 0, NotSerialized) {
        If ((RCTM () == One)) { Return (One) }
        Else { Return (Zero) }
    }
}

On Linux, this is exposed as an evdev device with switch events. I was able to replicate that quite easily. My display server does not support doing anything with that yet, but I’d like to do something like enabling an on-screen keyboard to pop up automatically when tablet mode is active.

Keyboard backlight brightness

I generally leave it off because I don’t look at the keyboard, but this was a fun and easy driver to write.

Also obvious how it works when looking at ACPI:

Device (KBLT) {
    Name (_HID, "GOOG0002")  // _HID: Hardware ID
    Name (_UID, One)  // _UID: Unique ID
    Method (KBQC, 0, NotSerialized) {
        Return (^^PCI0.LPCB.EC0.KBLV) /* \_SB_.PCI0.LPCB.EC0_.KBLV */
    }
    Method (KBCM, 1, NotSerialized) {
        ^^PCI0.LPCB.EC0.KBLV = Arg0
    }
}

Using the debug cable on FreeBSD

The debug cable presents serial consoles as bulk endpoints without any configuration capabilities. On Linux, they are supported by the “simple” USB serial driver.

Adding the device to the “simple” FreeBSD driver ugensatook some debugging. The driver was clearing USB stalls when the port is opened. That’s allowed by the USB spec and quite necessary on some devices. Unfortunately, the debug interface throws up when it sees that request. The responsible code in the device has a /* Something we need to add support for? */ comment :D

Audio?

The only thing that’s unsupported is onboard audio. The usual HDA controller only exposes the DisplayPort audio-through-the-monitor thing. The speakers, mic and headphone jack are all connected to various codecs exposed via… yet again, I²C. I am not about to write the drivers for these codecs, since I’m not really interested in audio on laptops.

Firmware is Fun

After the debug cable arrived, I’ve spent some time debugging the console-on-FreeBSD thing mentioned above, and then started messing with coreboot and TianoCore EDK2.

My discoveries so far:

  • there’s nothing on the AP console on stock firmware because Google compiles release FW with serial output off, I think to save on power or something;
  • me_cleanerneeds to be run with -S -w MFS. As mentioned in the --help, the MFS partition contains PCIe related stuff. Removing it causes the NVMe drive to detach soon after boot;
  • upstream Coreboot (including MrChromebox’s builds) fails to initialize the TPM, just gets zero in response to the vendor ID request. Funnily enough, that would’ve solved the resume problem without me having to write the I²C TPM driver for FreeBSD - but now that I’ve written it, I’d prefer to actually have the ability to use the TPM;
  • EDK2’s recent UefiPayloadPkg doesn’t support PS/2 keyboard and NVMe out of the box, but they’re very easy to add (hopefully someone would add them upstream after seeing my bug reports);
  • UefiPayloadPkg supports getting the framebuffer from coreboot very well;
  • coreboot can run Intel’s GOP driver before the payload (it’s funny that we’re running a UEFI module before running the UEFI implementation) and that works well;
  • but libgfxinit - the nice FOSS, written-in-Ada, verified-with-SPARK implementation of Intel GPU initialization and framebuffer configuration - supports Kaby Lake now!

    • however, we have a DPCD thing again with this display panel here - it reports max lane bandwidth as 0x00, libgfxinit interprets that as the slowest speed and we end up not having enough bandwidth for the high-res screen;
    • I’ve been told that this is because there’s a new way of conveying this information that’s unsupported. I’ll dig around in the Linux i915 code and try to implement it properly here but for now, I just did a quick hack, hardcoding the faster bandwidth. Ta-da! My display is initialized with formally verified open source code! Minus one blob running at boot!
  • persistent storage of EFI variables needs some SMM magic. There’s a quick patch that changes EDK2’s emulated variable store to use coreboot’s SMM store. EDK2 has a proper SMM store of its own, I’d like to look into making that coreboot-compatible or at least just writing a separate coreboot-compatible store module.

An aside: why mess with firmware?

If you’re not the kind of person who’s made happy by just the fact that some more code during the boot process of their laptop is now open and verified, and you just want things to work, you might not be as excited about open source firmware development as I am.

But you can do cool things with firmware that give you practical benefit. The best example I’m seeing is better Hackintosh support. Instead of patching macOS to work on your machine, you could patch your machine to pretend to almost be a Mac:

Is this cool or what?

Conclusion

Pixelbook, FreeBSD, coreboot, EDK2 good.

Seriously, I have no big words to say, other than just recommending this laptop to FOSS enthusiasts :)

Bogdan Popa (bogdan)

Announcing redis-rkt October 06, 2019 04:00 PM

Another Racket thing! redis-rkt is a new Redis client for Racket that I’ve been working on these past few weeks. Compared to the existing redis and rackdis packages, it: is fully documented, is safer due to strict use of contracts, is faster, supports more commands and its API tries to be idiomatic, rather than being just a thin wrapper around Redis commands. Check it out!

Ponylang (SeanTAllen)

Last Week in Pony - October 6, 2019 October 06, 2019 03:57 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

Carlos Fenollosa (carlesfe)

October 05, 2019

Bit Cannon (wezm)

Ryzen 9 SFF PC October 05, 2019 11:11 PM

I built this machine for work use. I started a new job in March 2019 that involves working with two compiled languages: Mercury and Rust. I wanted a machine with lots of cores/threads to keep the edit-compile-test cycle as short as possible.

Specifications

Component Description
CPU AMD Ryzen 9 3900X (Base: 3.80GHz, Boost: 4.60GHz, Cores: 12, Threads: 24)
CPU Cooling Noctua NH-U9S
Motherboard Gigabyte GA-X570 I Aorus Pro WiFi
Memory Corsair Vengeance LPX 16GB (2x8GB), PC4-25600 (3200MHz) DDR4, 16-18-18-36
Storage Samsung 500GB SSD, 970 EVO Plus, M.2 NVMe
Case Streacom DA2
Power Supply Corsair 450W SF450 High Performance SFX
Graphics Card Gigabyte Radeon RX 560 16CU Gaming OC
Display Dell P2415Q 23.8inch 4K (3840 × 2160) LCD

José Padilla (jpadilla)

Richard Kallos (rkallos)

Presentation: Hybrid Logical Clocks @ PWLMTL October 05, 2019 03:22 PM

This past thursday (2019–10–03), I presented this paper at Papers We Love Montreal. I had a really fun time!

Here are my slides.

October 03, 2019

Chris Double (doublec)

Defining Types in Shen October 03, 2019 05:00 AM

The Shen programming language has an extensible type system. Types are defined using sequent calculus and the system is powerful enough to create a variety of exotic types but it can be difficult when first starting with Shen to know how to use that power. In this post I hope to go through some basic examples of defining types in Shen without needing to know too much sequent calculus details.

For an overview of Shen there is Shen in 15 minutes and the Shen OS Kernel Manual. An interactive JavaScript REPL exists to try examples in the browser or pick on one of the existing Shen language ports. For these examples I'm using my Wasp Lisp port of Shen.

Shen is optionally typed. The type checker can be turned off and on. By default it is off and this can be seen in the Shen prompt by the presence of a '-' character:

(0-) ...shen code...

Turning type checking on is done by using (tc +). The '-' in the prompt changes to a '+' to show type checking is active. It can be turned off again with (tc -):

(0-) (tc +)
(1+) (tc -)
(2-) ...

Types in Shen are defined using datatype. The body of the datatype definition contains a series of sequent calculus rules. These rules define how an object in Shen can be proved to belong to a particular type. Rather than go through a detailed description of sequent calculus, I'm going to present common examples of types in Shen to learn by example and dive into details as needed. There's the Shen Language Book for much more detail if needed.

Records

One way of storing collections of data in Shen is to use lists or vectors. For example, given an the concept of a 'person' that has a name and age, this can be stored in a list with functions to get the relevent data:

(tc -)

(define make-person
  Name Age -> [Name Age])

(define get-name
  [Name Age] -> Name)

(define get-age
  [Name Age] -> Age)

(get-age (make-person "Person1" 42))
 => 42

In the typed subset of Shen we can define a type for this person object using datatype:

(datatype person
  N : string; A : number;
  _______________________
  [N A] : person;)

This defines one sequent calculus rule. The way to read it is starting with the code below the underscore line, followed by the code above it. In this case the rule states that if an expression matching the pattern [N A] is encountered, where N is a string and A is a number, then type that expression as person. With that rule defined, we can ask Shen if lists are of the type person:

(0+) ["Person" 42] : person
["Person1" 42] : person

(1+) ["Person1" "Person1"] : person
[error shen "type error"]

(2+) ["Person 42"]
["Person1" 42] : person

Given this person type, we might write a get-age function that is typed such that it only works on person objects as follows (The { ...} syntax in function definitions provides the expected type of the function):

(define get-age
  { person --> number }
  [N A] -> A)
[error shen "type error in rule 1 of get-age"]

Shen rejects this definition as not being type safe. The reason for this is because our datatype definition only states that [N A] is a person if N is a string and A is a number. It does not state that a person object is constructed only of a string and number. For example, we could have an additional definition as follows:

(datatype person2
  N : string; A : string;
  _______________________
  [N A] : person;)

Now we can create different types of person objects:

(0+) ["Person" 42 ] : person
["Person" 42] : person

(1+) ["Person" "young"] : person
["Person" "young"] : person

get-age is obviously badly typed in the presence of this additional type of person which is why Shen rejected it originally. To resolve this we need to tell Shen that an [N A] is a person if and only if N is a string and A is a person. This is done with what is called a 'left rule'. Such a rule defines how a person object can be deconstructed. It looks like this:

(datatype person3
  N : string, A: number >> P;
  ___________________________
  [N A] : person >> P;)

The way to read this type of rule is that, if [N A] is a person then N is a string and A is a number. With that loaded into Shen, get-age type checks:

(define get-age
   { person --> number }
   [N A] -> A)
get-age : (person --> number)

(0+) (get-age ["Person" 42])
42 : number

The need to create a left rule, dual to the right rule, is common enough that Shen has a short method of defining both in one definition. It looks like this - note the use of '=' instead of '_' in the separator line:

(datatype person
   N : string; A : number;
   =======================
   [N A] : person;)

(define get-age
   { person --> number }
   [N A] -> A)
get-age : (person --> number)

(0+) (get-age ["Person" 42])
42 : number

The above datatype is equivalent to declaring the two rules:

(datatype person
  N : string; A : number;
  _______________________
  [N A] : person;

  N : string, A: number >> P;
  ___________________________
  [N A] : person >> P;)

Controlling type checking

When progamming at the REPL of Shen it's common to create datatype definitions that are no longer needed, or part of a line of thought you don't want to pursue. Shen provides ways of excluding or including rules in the typechecker as needed. When defining a set of rules in a datatype, that datatype is given a name:

(datatype this-is-the-name
   ...
)

The rules within that definition can be removed from selection by the typechecker using preclude, which takes a list of datatype names to ignore during type checking:

(preclude [this-is-the-name])

To re-add a dataype, use include:

(include [this-is-the-name])

There is also include-all-but and preclude-all-but to include or remove all but the listed names. These commands are useful for removing definitions you no longer want to use at the REPL, but also for speeding up type checking in a given file if you know the file only uses a particular set of datatypes.

Enumerations

An example of an enumeration type would be days of the week. In an ML style language this can be done like:

datatype days =   monday | tuesday | wednesday
                | thursday | friday | saturday | sunday

In Shen this would be done using multiple sequent calculus rules.

(datatype days
    ____________
    monday : day;

    ____________
    tuesday : day;

    ____________
    wednesday : day;

    ____________
    thursday : day;

    ____________
    friday : day;

    ____________
    saturday : day;

    ____________
    sunday : day;)

Here there are no rules above the dashed underscore line, meaning that the given symbol is of the type day. A function that uses this type would look like:

(define day-number
  { day --> number }
  monday    -> 0
  tuesday   -> 1
  wednesday -> 2
  thursday  -> 3
  friday    -> 4
  saturday  -> 5
  sunday    -> 6)

It's quite verbose to define a number of enumeration types like this. It's possible to add a test above the dashed underline which allows being more concise. The test is introduced using if:

(datatype days
  if (element? Day [monday tuesday wednesday thursday friday saturday sunday])
  ____________________________________________________________________________
  Day : day;)

(0+) monday : day
monday : day

Any Shen code can be used in these test conditions. Multiple tests can be combined:

(datatype more-tests
  if (number? X)
  if (>= X 5)
  if (<= X 10)
  ___________
  X : between-5-and-10;)

  (2+) 5 : between-5-and-10
  5 : between-5-and-10

  (3+) 4 : between-5-and-10
  [error shen "type error\n"]

Polymorphic types

To create types that are polymorphic (ie. generic), like the built-in list type, include a free variable representing the type. For example, something like the built in list where the list elements are stored as pairs can be approximated with:

(datatype my-list
   _____________________
   my-nil : (my-list A);

   X : A; Y : (my-list A);
   ========================
   (@p X Y) : (my-list A);)


(define my-cons
  { A --> (my-list A) --> (my-list A) }
  X Y -> (@p X Y))

(0+) (my-cons 1 my-nil)
(@p 1 my-nil) : (my-list number)

(1+) (my-cons 1 (my-cons 2 my-nil))
(@p 1 (@p 2 my-nil)) : (my-list number)

(2+) (my-cons "a" (my-cons "b" my-nil))
(@p "a" (@p "b" my-nil)) : (my-list string)

Notice the use of the '=====' rule to combine left and right rules. This is required to enable writing something like my-car which requires proving that the type of the car of the list is of type A:

(define my-car
   { (my-list A) --> A }
   (@p X Y) -> X)

List encoded with size

Using peano numbers we can create a list where the length of the list is part of the type:

(datatype list-n
  ______
  [] : (list-n zero A);

  X : A; Y : (list-n N A);
  ================================
  [ X | Y ] : (list-n (succ N) A);)      

(define my-tail
  { (list-n (succ N) A) --> (list-n N A) }
  [Hd | Tl] -> Tl)

(define my-head
  { (list-n (succ N) A) --> A }
  [Hd | Tl] -> Hd)

This gives a typesafe head and tail operation whereby they can't be called on an empty list:

(0+) [] : (list-n zero number)
[] : (list-n zero number)

(1+) [1] : (list-n (succ zero) number)
[1] : (list-n (succ zero) number)

(2+) (my-head [])
[error shen "type error\n"]

(3+) (my-head [1])
1 : number

(4+) (my-tail [1 2 3])
[2 3] : (list-n (succ (succ zero)) number)

(5+) (my-tail [])
[error shen "type error\n"]      

Power and Responsibility

Shen gives a lot of power in creating types, but trusts you to make those types consistent. For example, the following creates an inconsistent type:

(datatype person
  N : string; A : number;
  _______________________
  [N A] : person;

  N : string; A : string;
  _______________________
  [N A] : person;

  N : string, A: number >> P;
  ___________________________
  [N A] : person >> P;)

Here we are telling Shen that a string and number in a list is a person, and so too is a string and another string. But the third rule states that given a person, that is is composed of a string and a number only. This leads to:

(0+) (get-age ["Person" "Person"])
...

This will hang for a long time as Shen attempts to resolve the error we've created.

Conclusion

Shen provides a programmable type system, but the responsibility lies on the programmer for making sure the types are consisitent. The examples given here provide a brief overview. For much more see The Book of Shen. The Shen OS Kernel Manual also gives some examples. There are posts on the Shen Mailing List that have more advanced examples of Shen types. Mark Tarver has a case study showing converting a lisp interpreter in Shen to use types.

October 01, 2019

Simon Zelazny (pzel)

For focused reading, disconnect wifi October 01, 2019 10:00 PM

Today I had a 50-page PDF whitepaper to read. I didn't know how long it was going to take, and — wanting to conserve my laptop battery charge – I disabled my wifi. It took me suprisingly little time to skim the paper and dig into the more relevant parts in detail, taking some notes as I went along.

During the ~2 hours it took me to work through the document, I tried to access the Internet about ten times. Either by clicking a link in the whitepaper itself, or trying to follow up on a tangential thought with some info online. But! My wifi was switched off, and I'd need to click the network manager icon to re-connect to the net. I chose not to do that, and instead continue reading.

Each time my impulse to access information was frustrated, I realized that had I been online, I would have wasted precious minutes reading tangentially-related web pages, and then some more time again, trying to get back to reading the original PDF, reestablishing the reading context and exerting willpower to stay in the PDF reader.

Acting on these distractions would have defintely prevented me from ingesting the whitepaper in 2 hours.

After the fact, I realized that what I'd achieved accidentally is the productivity hack identified by Matt Might as crippling your technology. By removing functionality from our tools and keeping that which is strictly necessary for completing the task at hand, we remove the 'friction' that gets in the way of sustained attention. Yes, we do "lose" some capabilities, but we make up for it by making it easier for ourselves to focus on the goal.

I'll try to keep this technique in my tool-belt, especially when I need long periods of focus.

Apart from Matt Might's productivity writings, a lot more in this vein can be found in Cal Newport's books & blog posts.

Gustaf Erikson (gerikson)

Pete Corey (petecorey)

Animating a Canvas with Phoenix LiveView: An Update October 01, 2019 12:00 AM

In my previous post on animating an HTML5 canvas using Phoenix LiveView, we used both a phx-hook attribute and a phx-update="ignore" attribute simultaneously on a single DOM element. The goal was to ignore DOM updates (phx-update="ignore"), while still receiving updated data from our server (phx-hook) via our data-particles attribute.

Unfortunately, the technique of using both phx-hook and phx-update="ignore" on a single component no longer works as of phoenix_live_view version 0.2.0. The "ignore" update rule causes our hook’s updated callback to not be called with updates. In hindsight, the previous behavior doesn’t even make sense, and the new behavior seems much more consistent with the metaphors in play.

Joxy pointed this issue out to me, and helped me come up with a workaround. The solution we landed on is to wrap our canvas component in another DOM element, like a div. We leave our phx-update="ignore" on our canvas to preserve our computed width and height attributes, but move our phx-hook and data attributes to the wrapping div:


<div
  phx-hook="canvas"
  data-particles="<%= Jason.encode!(@particles) %>"
>
  <canvas phx-update="ignore">
    Canvas is not supported!
  </canvas>
</div>

In the mounted callback of our canvas hook, we need to look to the first child of our div to find our canvas element:


mounted() {
  let canvas = this.el.firstElementChild;
  ...
}

Finally, we need to pass a reference to a Phoenix Socket directly into our LiveSocket constructor to be compatible with our new version of phoenix_live_view:


import { Socket } from "phoenix";
let liveSocket = new LiveSocket("/live", Socket, { hooks });

And that’s all there is to it! Our LiveView-powered confetti generator is back up and running with the addition of a small layer of markup.

For more information on this update, be sure to check out this issue I filed to try to get clarity on the situation. And I’d like to give a huge thanks to Joxy for doing all the hard work in putting this fix together!

September 30, 2019

Pete Corey (petecorey)

All Hail Glorious Voice Leader! September 30, 2019 12:00 AM

I’ve been writing code that generates guitar chords for over a year now, in various languages and with varying degrees of success. My newest addition to this family of chord-creating programs is Glorious Voice Leader!

Glorious Voice Leader is an enigmatic leader tool who’s mission is to help you voice lead smoothly between chords. It does this by generating all possible (and some impossible) voicings of a given chord, and sorting them based on the chromatic distance from the previous chord in the progression.

Glorious Voice Leader says, “the less you move, the more you groove!”

Obviously, this robotic “rule” needs to be tempered by human taste and aesthetic, so the various choices are presented to you, the user, in the form of a heat map laid over a guitar fretboard. The notes in the voicings that Glorious Voice Leader think lead better from the previous chord are darkened, and notes that don’t lead as well are lightened.

To get a grasp on this, let’s consider an example.

Let’s pretend we’re trying to play a ii-V-I progression on the guitar in the key of C. When we tell Glorious Voice Leader that our first chord will be a Dm7, it gives us a heat map of the various initial voicings to choose from:

With this initial chord, darker notes in the heat map are used more frequently by the generated voicings, and lighter notes are used more rarely. Click on the notes of the Dm7 voicing you want to start with.

Once we’ve told Glorious Voice Leader where to start, we can tell it where we want to go next. In our case, our next chord will be a G7. Here’s where things get interesting. Glorious Voice Leader generates all possible G7 voicings, and ranks them according to how well they lead from the Dm7 we just picked out.

Pick out a G7 voicing with darkened notes:

Now we tell Glorious Voice Leader that we want to end our progression with a Cmaj7 chord.

Choose your Cmaj7 voicing:

That’s it! With Glorious Voice Leader’s help, we’ve come up with an entire ii-V-I chord progression. Grab yourself a guitar and play through the whole progression. I’m willing to bet it sounds pretty nice.

For this example, we’ve embedded a small, reluctant version of Glorious Voice Leader directly into this page. Check out the above example in its full-fledged glory at the Glorious Voice Leader website. If you’re eager for another example, here’s the entire series of diatonic seventh chords descending in fourths, as suggested by Glorious Voice Leader.

If you find this interesting, be sure to give Glorious Voice Leader a try and let me know what you think! Expect more features and write-ups in the near future.

#d, #g, #c { width: 100%; cursor: pointer; } #d .fretboard, #d canvas, #g .fretboard, #g canvas, #c .fretboard, #c canvas { width: 120% !important; margin-left: -10%; }

September 29, 2019

Carlos Fenollosa (carlesfe)

checkm8: What you need to know to keep your iPhone safe September 29, 2019 06:12 PM

A couple days ago, Twitter user axi0mX introduced checkm8, a permanent unpatchable bootrom exploit for iPhones 4S to X

The jailbreak community celebrated this great achievement, the netsec community was astounded at the scope of this exploit, and regular users worried what this meant for their phone's security.

Even though I've jailbroken my iPhone in the past, I have no interest to do it now. If you want to read the implications for the jailbreak community, join the party on /r/jailbreak

I have been reading articles on the topic to understand what are the implications for regular people's security and privacy. All my family has A9 iPhones which are exploitable, and I wanted to know whether our data was at risk and, if such, what could we do to mitigate attacks.

I think the best way to present the findings is with a FAQ so people can understand what's going on.

1-Line TL;DR

If you have an iPhone 4s, 5, or 5c, somebody who has physical access to your phone can get all the data inside it. If your phone is more modern and the attacker doesn't know your password, they can still install malware, but rebooting your phone makes it safe again.

What is Jailbreak?

Your iPhone is controlled by Apple. You own it, but you are limited in what you can do with it.

Some people like this approach, others prefer to have total control of their phone.

A jailbreak is a way of breaking these limitations so you can 100% control what's running on your phone.

The goal of jailbreaking is not necessarily malicious. In fact, the term "jailbreak" has the connotation that the user is doing it willingly.

However, the existence of a jailbreak method means that an attacker could use this same technique to compromise your phone. Therefore, you must understand what is going on and how to protect yourself from these attackers.

Jailbreaking has existed since the first iPhone. Why is this one different?

Typically, jailbreaking methods exploit a software bug. This means that Apple can (and does) fix that bug in the next software release, negating the method and any related security issues.

This method, however, exploits a hardware bug on the bootrom. The bootrom is a physical chip in your iPhone that has some commands literally hard-wired in the chip. Apple cannot fix the bug without replacing the chip, which is unfeasible.

Therefore, it is not possible to fix this bug, and it will live with your phone until you replace it

These kind of bugs are very rare. This exact one has been already patched on recent phones (XS and above) and it has been a long time since the last one was found.

☑ This bug is extremely rare and that is why it's important to know the consequences.

How can an attacker exploit this bug? Can I be affected by it without my knowledge?

This exploit requires an attacker to connect your phone to a computer via Lightning cable.

It cannot be triggered by visiting a website, receiving an email, installing an app, or any non-suspicious action.

☑ If your phone never leaves your sight, you are safe.

I left my phone somewhere out of sight. May it be compromised?

Yes. However, if you reboot your phone, it goes back to safety. Any exploit does not persist upon reboots, at least, at this point in time. If that changes, this text will be updated to reflect that.

Any virus or attack vector will be uninstalled or disabled by Apple's usual protections after a reboot.

If you feel that you are targeted by a resourceful attacker, read below "Is there a feasible way to persist the malware upon reboot?"

☑ If you are not sure about the safety of your phone, reboot it.

Can my personal data be accessed if an attacker gets physical access to my phone?

For iPhones 4S, 5 and 5c, your data may be accessed regardless of your password. For iPhones 5s and above (6, 6s, SE, 7, 8, X), your data is safe as long as you have a strong password.

If you have an iPhone 4s, 5, or 5c, anybody with physical access to your phone will have access to its contents if your password is weak (4 to 8 digit PIN code, or less than 8 characters alphanumeric code)

If your iPhone 4s-5-5c has a strong password, and the attacker does not know it and cannot guess it, they may need a long time (months to years) to extract the data. Therefore this attack cannot be run in the scenario where the phone leaves your sight for a few minutes, but you get it back quickly afterwards. However, if your phone 4s-5-5c is stolen, assume that your data is compromised.

It is unknown if this exploit allows the attacker to guess your password quicker than a "months to years" period on older iPhones.

iPhones 5s and above have a separate chip called the Secure Enclave which manages access to your personal data. Your data is encrypted on the device and can not be accessed. The Secure Enclave does not know your password, but uses some math to decrypt it with your password.

If you have an iPhone 5s and above, an attacker can only access your data if they know, or can easily guess, your password.

☑ Use a strong password (>8 alphanumeric characters) that an attacker can not guess

Can it be used to disable iCloud lock, and therefore re-use stolen phones?

It is unknown at this point.

Assuming the scenario where iCloud lock is not broken, and the Secure Enclave is not affected, what is the worst that can happen to my phone?

You may suffer a phishing attack: they install a fake login screen on your iPhone, or replace the OS with an exact copy that works as expected, but it also sends all your keystrokes and data to the attacker.

The fake environment may be indistinguishable from the real one. If you are not aware of this attack, you will fall for it.

Fortunately, this malware will be purged or disabled upon reboot.

All phones (4s to X) are vulnerable to this attack.

☑ Always reboot your phone if you think it may be compromised.

Is there a feasible way to persist the malware upon reboot?

Unlikely. The jailbreak is tethered, which means that the phone must be connected to a computer every time it boots.

However, somebody may develop a tiny device that connects to the Lightning port of the iPhone and conveniently injects code/malware every time it is rebooted.

This device may be used on purpose by jailbreakers, for convenience (i.e. a Lightning-USB key, or a small computer) or inadvertently installed by a sophisticated attacker (i.e. a phone case that by-passes the lightning port without the victim knowing)

In most cases, this external device will be easy to spot even to the untrained eye.

An extremely sophisticated attacker may develop a custom chip that is connected internally to the Lightning port of the iPhone and runs the malware automatically and invisibly. To do so, they would need physical access to your phone for around 10 minutes, the time it takes to open the phone, solder the new chip, and close it again.

☑ Watch out for unexpected devices connected to your Lightning port

Who are these "attackers" you talk about?

Three-letter agencies (NSA, FBI, KGB, Mossad...) and also private companies who research their own exploits (Cellebrite, Greyshift) to sell them to the former.

It is entirely possible that the above already knew about this exploit, however.

Other attackers may be regular thieves, crackers, pranksters, or anybody interested in developing a virus for the iPhone.

If you are a regular user who is not the target of a Government or Big Criminal, remember:

  1. Don't let people connect your iPhone to an untrusted device
  2. Otherwise, reboot it when you get it back
  3. Watch out for small devices on your Lightning port

References:

Tags: apple, security

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

Famous public figure in tech suffers the consequences for asshole-ish behavior September 29, 2019 12:05 PM

This last month, a very famous computer guy who regularly appears in public and has amassed a cultish-like following has been forced to step down due to pressure from journalists.

Let's make a list of all the unacceptable behaviours of Computer Guy:

Living in his office and disgusting smell

He is not really homeless, but Computer Guy used to sleep in his office.

Coworkers and friends reported that he reeked and would avoid contact with him.

Sexually harassing employees

It has been reported, even in video, that Computer Guy made inappropriate sexual utterances to their colleagues.

Of course, Computer Guy denied it.

Drug intake

Computer Guy is a known hippie, I mean, just look at his appearance.

He is not ashamed to admit that he has taken illegal drugs and that they are an important part of his life.

Psychological abuse to women

Not many people know this, but Computer Guy has a daughter which he denied for a long time.

Computer Guy basically abandoned his former partner who was pregnant with their daughter, denied her alimony, and even abused the child psychologically when she was 9.

Keeping payments from group projects for himself

In one of his projects, Computer Guy profited more that he had earned by lying to colleagues. Instead of fairly distributing the money from a project, he decided to take most of it for himself.

In a similar case, he denied fair compensation to an old friend of his.

Bad temper

All these examples can be summarized as: Computer Guy is an asshole who must be taken down.

Even though Computer Guy did nothing technically illegal, being such a big asshole must not be acceptable in our society and the right thing to do is to pressure him to resign from his public positions.

Since mobs don't read the news, only the headlines, and I don't want any association with any of the parties in this drama, I think I must write the non-snarky interpretation of the events.

Of course, the headlines above are about Steve Jobs, not Richard Stallman.

I only had one goal with this piece: to reflect on the double standards in society.

Being an asshole is acceptable if you are a respected powerful businessman. You are portrayed as a quirky millionaire. However, it is not acceptable if you're a contrarian weird hippie. You are portrayed as a disgusting creep.

I obviously have no interest or authority to defend or justify their actions. They're adults and their behavior is their own. Screw their asshole-ism. They should have been better people. Stallman is a stubborn asshole, Jobs was an even bigger stubborn asshole.

The truth is, there is a strong correlation between being a powerful public figure and being a stubborn asshole. This is because after some point, non-assholes quit the race because they are not willing to pay the toll it takes to be at the top. That is unfortunate, and we should definitely push for respecful leaders.

Why did two independent journalists take Stallman down, and not Jobs, or any of the other assholes in the world?

Probably, because they could.

It's their right to free speech, and ultimately it was a consequence of Stallman's actions. And I can't reflect on whether it's fair or good that Stallman is forced to step down, because I'm not smart enough to foresee the positive or negative consequences. So maybe after a few months we all realize it was the right thing to do, and end this discussion once and for all.

However, one thing is still true, again, the only point that should be taken from this article: to hell with double standards when representing public figures.

Not that it matters for this article, and it's outside the scope of my point, but I want to share my personal vision on Stallman and Jobs. The thing is, this was a difficult article to write. They are both people who I strongly admire and have had a great influence in my life.

Reading Stallman's essays are what got me into Free Software. I have attended his conferences twice and his brave stance on freedom and privacy is flawless and admirable. I have a small laptop that Stallman signed and many of his books. He has constantly fought for the rights of the people against corporations. I hope he keeps doing it.

The world is a better place thanks to Stallman.

Jobs was an inspiration. I own most books about him, an Apple "Think Different" poster is hanging at my office, and I treasure the issue Time released after his death. He was a genius, a visionary, he basically invented consumer computers and smartphones. I do not doubt that the contributions of Woz and other people at Apple were instrumental, but he was the mastermind behind the strategy. What Jobs achieved with his work is beyond belief and 100% worth of praise.

The world is a better place thanks to Jobs.

If you want more context about the actual facts, I wrote about the news a week ago.

Tags: news

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

September 28, 2019

Frederik Braun (freddyb)

Remote Code Execution in Firefox beyond memory corruptions September 28, 2019 10:00 PM

This is the blog post version of my presentation form OWASP Global AppSec in Amsterdam 2019. It was presented in the AllStars Track.

Abstract:

Browsers are complicated enough to have attack surface beyond memory safety issues. This talk will look into injection flaws in the user interface of Mozilla Firefox, which is implemented in JS, HTML, and an XML-dialect called XUL. With an Cross-Site Scripting (XSS) in the user interface attackers can execute arbitrary code in the context of the main browser application process. This allows for cross-platform exploits of high reliability. The talk discusses past vulnerabilities and will also suggest mitigations that benefit Single Page Applications and other platforms that may suffer from DOM-based XSS, like Electron.

Prologue

(This is the part, where we reduce the lighting and shine a flashlight into my face)

Listen, well, young folks. Old people, browser hackers or Mozilla fanboys, might use this as an opportunity to lean back and stroke their mighty neckbeard, as they have heard all of this before

It was the year 1997, and people thought XML was a great idea. In fact, it was so much better than its warty and unparseable predecessor HTML. While XHTML was the clear winner and successor for great web applications, it was obvious that XML would make a great user interface markup language to create a powerful cross-platform toolkit dialect. This folly marks the hour of birth for XUL. XUL was created as the XML User Interface Language at Netscape (the company that created the origins of the Mozilla source code. Long story. The younger folks might want to read upon Wikipedia or watch the amazing Movie "Code Rush", which is available on archive.org). Jokingly, XUL was also a reference to the classic 1984 movie Ghostbusters, in which an evil deity called Zuul (with a Z) possesses innocent people.

Time went by and XUL did not take off as a widely-recognized standard for cross-platform user interfaces. Firefox has almost moved from XUL and re-implemented many parts in HTML. Aptly named after an evil spirit, we will see that XUL still haunts us today.

Mapping the attack surface

Let's look into Firefox, to find some remnants of XUL, by visiting some internal pages. Let's take a look at some Firefox internal pages. By opening about:preferences in a new tab (I won't be able to link to it for various good reasons). Now either look at the source code using the Developer Tools (right-click "Inspect Element") or view the source code of Firefox Nightly using the source code search at searchfox.org.

We can also open the developer console and poke around with the obscure objects and functions that are available for JavaScript in privileged pages. As a proof-of-concept, we may alert(Components.stack), which gives us a stringified JavaScript call stack - notably this is a JavaScript object that is left undefined for normal web content.

Inspecting the source code we also already see some markup that screams both XML as well as XML-dialect. While still in our information gathering phase, we will not go too deep, but make note of two observations: - XUL is not HTML. To get a better understanding of elements like <command>, <colorpicker> or <toolbar>, we will be able to look at the XUL Reference on MDN - XUL is scriptable! A <script> tag exists and it may contain JavaScript.

There are also some newer pages like about:crashes, which holds previously submitted (or unsubmitted) crash reports. Whether those internal pages are written in (X)HTML or XUL. Most of the interacive parts are written in JavaScript. I suppose most of you will by now understand that we are looking for Cross-Site Scripting (XSS) vulnerabilities in the browser interface. What's notable here, is that this bypasses the sandbox.

As an aside the page behind about:cache is actually implemented using C++ that emits HTML-ish markup.

Let's start with search and grep

Being equipped with the right kind of knowledge and the crave for a critical Firefox bug under my name, I started using our code search more smartly. Behold:

Search: .innerHTML =

Number of results: 1000 (maximum is 1000)

Hm. Excluding test files.

Search: innerHTML =

Number of results: 414

That's still a lot. And that's not even all kinds of XSS sinks. I would also look for outerHTML, insertAdjacentHTML and friends.

Search (long and hairy regular expression that tries to find more than innerHTML)

Number of results: 997

That's bad. Let's try to be smarter!

JavaScript Parsing - Abstract Syntax Trees. ESLint to the rescue!

I've actually dabbled in this space for a long while before. This would be another talk, but a less interesting one. So I'll skip ahead and tell you that I wrote an eslint plugin, that will analyze JavaScript files to look for the following:

  1. Checking the right-hand side in assignments (+, +=) where the left part ends with either innerHTML or outerHTML.
  2. Checking the first argument in calls to document.write(), document.writeln(), eval and the second argument for insertAdjacentHTML.

For both, we'll check whether they contain a variable. String literals or empty strings are ignored. The plug-in is available at as eslint-plugin-no-unsanitized and allows for configuration to detect and ignore built-in escape and sanitize functions. If you're worried about DOM XSS, I recommend you check it out.

Discovered Vulnerabilities

Using this nice extension to scan all of Firefox yields us a handy amount of 32 matches. We create a spreadsheet and audit all of them by hand. Following around long calling chains, with unclear input values and patterns that either escape HTML close to the final innerHTML, upon creation or stuff that's extracted from databses (like the browsing history), which does its escaping upon insertion.


Many nights later


A first bug appears

Heureka! This sounds interesting:

  let html = `
    <div style="flex: 1;
                display: flex;
                padding: ${IMAGE_PADDING}px;
                align-items: center;
                justify-content: center;
                min-height: 1px;">
      <img class="${imageClass}"
           src="${imageUrl}"/>       <----- boing
    </div>`;
  // …
  div.innerHTML = html;

When hovering over an markup that points to an image in the web developer tools, they will helpfully create a tooltip that preloads and shows the image for the web developer to enjoy. Unfortunately, that URL is not escaped.

Firefox Developer Tools Inspector opening images in a tooltip when hovering an image element's source attribute

Writing the exploit

After spending a few sleepless nights on this, I didn't get anything beyond a XML-conformant proof of concept of <button>i</button>. At some point I filed the bug as sec-moderate, i.e., this is almost bad, but likely needs another bug to be actually terrible. I wrote:

I poked a bit again and I did not get further than <button>i</button> for various reasons … In summary: I'd be amazed to see if someone else gets any farther.

A few nights later, I actually came up with an exploit that breaks the existing syntax while staying XML conformant. We visit an evil web page that looks like this:

<img src='data:bb"/><button><img src="x" onerror="alert(Components.stack)" /></button><img src="x'>

The image URL that is used in the vulnerable code spans all the way from data: to the closing single quote at the end. Our injection alerts Components.stack, which indicates that we have left the realms of mortal humans.

This is Bug 1372112 (CVE-2017-7795). Further hikes through our spreadsheets of eslint violations lead to Bug 1371586 (CVE-2017-7798). Both were fixed in Firefox 56, which was released in the fall of 2017.

We find and fix some minor self-xss bugs (e.g., creating a custom preference in about:config with the name of <button>hi</button> lead to XUL injections. All of them are fixed and we're fearful that mistakes will be made again.

Critical bugs are a great way to impact coding style discussions and it is decided that the linter might as well be included in all of our linters. innerHTML and related badness is forbidden and we rub our hands in glee. Unfortunately, it turned out that lots of legacy code will not be rewritten and security engineers do not want to deal with the affairs of front end engineers (joke's on me in the end though, I promise). So, we allow some well-audited and finely escaped functions with a granular and exception, that gives us a confident feeling of absolute security (it's a trap!)

// eslint-disable-next-line no-unsanitized/property

A Dark Shadow

I feel like I have eradicated the bug class from the entirety of our codebase. We may now look for more complicated bugs and our days get more exciting.

Of course, I wander through the office bragging with my cleverness, warning young folks from the danger of XSS and proudly wearing my security t-shirts. There's lots of colorful war stories to be told and even more free snacks or fizzy drinks to be consumed.

Meanwhile: My great colleagues that contribute and actually develope useful stuff. On top of their good work, some of them even mentor aspiring students and enthusiastic open source fans. Having listened to my stories of secure and well-audited code that should eventually be replaced, they make an effort to get someone remove all of the danger, so we get to live in an exception-less world that truly disallows all without these pesky eslint-disable-next-line comments.

Naturally, code is being moved around, refactored and improved by lots of other people in the organization.

So, while I'm sitting there, enjoying my internet fame (just browsing memes, really), people show up at my desk asking me for a quick look at something suspicious:

// eslint-disable-next-line no-unsanitized/property
doc.getElementById("addon-webext-perm-header").innerHTML = strings.header;

// data coming *mostly* from localization-templates
    let strings = {
      header: gNavigatorBundle.getFormattedString("webextPerms.header", [data.name]),
      text: gNavigatorBundle.getFormattedString("lwthemeInstallRequest.message2",
                                                [uri.host]),
// ..
// but of course all goes through _sanitizeTheme(aData, aBaseURI, aLocal)
// (which does not actually sanitize HTML)

I feel massively stupid and re-create my spreadsheet. Setting eslint to ignore the disable-next-line stuff locally allows me to start all over. We build an easy exploit that pops calc. How funny! We also notice that a few more bugs like that have crept in, since the "safe" call sites were whitelisted. Yikes.

Having learned about XML namespaces, a simpler example payload (without the injection trigger) would like look this:

<html:img onerror='Components.utils.import("resource://gre/modules/Subprocess.jsm");Subprocess.call({ command: "/usr/bin/gnome-open" });' src='x'/>,

This is Bug 1432778.

Hope on the horizon

A good patch is made and circulated with a carefully selected group of senior engineers. We have various people working on the code and are concerned about this being noticed by bad actors. With the help of the aforementioned group, we convince engineering leadership that this warrants an unscheduled release of Firefox. We start a simplified briefing for Release Management and QA.

People point out that updates always take a while to apply to all of our release base and shipping a new version with a single commit that replaces .innerHTML with .textContent seems a bit careless. Anyone with a less-than sign on their keyboard could write a "1-day exploit" that would affect lots of users.

What can we do? We agree that DOM XSS deserves a heavier hammer and change our implementation for HTML parsing (which is being used in innerHTML, outerHTML, insertAdjacentHTML, etc.). Normally, this function parses the DOM tree and inserts where assigned. But now, for privileged JavaScript, we parse the DOM tree and omit all kinds of badness before insertion. Luckily, we have something like that in our source tree. In fact, I have tested it in 2013. We also use this to strip HTML email from <script> and its friends in Thunderbird, so it's even battle-tested. On top, we do some additional manual testing and identify some problems around leaving form elements in, which warrants follow-up patches in the future.

A nice benefit is that a commit which changes how DOM parsing works, doesn't allow reverse engineering our vulnerability from the patch. Neat.

In the next cycles, we've been able to make it stricter and removed more badness (e.g., form elements). This was Bug 1432966: Sanitize HTML fragments created for chrome-privileged documents (CVE-2018-5124)

Closing credits and Acknowledgements

Exploitation and Remediation were achieved with the support of various people. Thanks to, security folks, Firefox engineers, release engineers, QA testers. Especially to Johnathan Kingston (co-maintainer of the eslint plugin), Johann Hofman, who found the bad 0day in 2018 and helped testing, shaping of and arguing for an unscheduled release version of Firefox.

No real geckos were harmed in the making of this blog post.

September 27, 2019

Carlos Fenollosa (carlesfe)

The absolute best puzzle game for your phone September 27, 2019 01:23 PM

Sorry for the clickbaity title. I was slightly misleading.

Simon Tatham's Portable Puzzle Collection is not only the best puzzle game for your phone, it is actually a collection of the best puzzle games.

Wait! It is actually the best puzzle game collection for any device, since all games are playable via web (js and *cough* Java), and there are native binaries for Windows and UNIX.

In Simon's own words:

[This is] a collection of small computer programs which implement one-player puzzle games. All of them run natively on Unix (GTK), on Windows, and on Mac OS X. They can also be played on the web, as Java or Javascript applets.

I wrote this collection because I thought there should be more small desktop toys available: little games you can pop up in a window and play for two or three minutes while you take a break from whatever else you were doing.

Simon's collection consists of very popular single player puzzle games, like Sudoku, Minesweeper, Same Game, Pegs, and Master Mind, and some lesser known, at least for me, but extremely fun to play: Pattern, Signpost, Tents, Unequal.

All games are extremely configurable and can usually be learned by reading the instructions and trying to play on a small board where the solution is usually trivial. Then, when you are ready, start expanding the board size and enabling some of the higher difficulty board generators!

Greg Hegwill ported the games to iOS and Chris Boyle ported them to Android. Other people have ported it to more platforms, like the Palm, Symbian or Windows phone

The games can of course run in old devices, are 3x free (money free, free software, and free of ads) and, as context, each game takes around 300kb of space (yes, kb). The full collection weighs 3.5Mb on iOS. For reference, Simon's mines.exe is 295kb, where Windows 3.1's winmine.exe was 28kb.

Simon's last commit is from April 2019 and he is still adding improvements to make the games more fun.

I don't really know how this could not be the Best Puzzle Game Ever. Download it right now on your phone (iOS, Android) and you'll thank me later.

Tags: software, mobile, retro

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

Scott Sievert (stsievert)

Better and faster hyperparameter optimization with Dask September 27, 2019 05:00 AM

Dask’s machine learning package, Dask-ML now implements Hyperband, an advanced “hyperparameter optimization” algorithm that performs rather well. This post will

  • describe “hyperparameter optimization”, a common problem in machine learning
  • describe Hyperband’s benefits and why it works
  • show how to use Hyperband via example alongside performance comparisons

In this post, I’ll walk through a practical example and highlight key portions of the paper “Better and faster hyperparameter optimization with Dask”, which is also summarized in a ~25 minute SciPy 2019 talk.

Problem

Machine learning requires data, an untrained model and “hyperparameters”, parameters that are chosen before training begins that help with cohesion between the model and data. The user needs to specify values for these hyperparameters in order to use the model. A good example is adapting ridge regression or LASSO to the amount of noise in the data with the regularization parameter.1

Model performance strongly depends on the hyperparameters provided. A fairly complex example is with a particular visualization tool, t-SNE. This tool requires (at least) three hyperparameters and performance depends radically on the hyperparameters. In fact, the first section in “How to Use t-SNE Effectively” is titled “Those hyperparameters really matter”.

Finding good values for these hyperparameters is critical and has an entire Scikit-learn documentation page, “Tuning the hyperparameters of an estimator.” Briefly, finding decent values of hyperparameters is difficult and requires guessing or searching.

How can these hyperparameters be found quickly and efficiently with an advanced task scheduler like Dask? Parallelism will pose some challenges, but the Dask architecture enables some advanced algorithms.

Note: this post presumes knowledge of Dask basics. This material is covered in Dask’s documentation on Why Dask?, a ~15 minute video introduction to Dask, a video introduction to Dask-ML and a blog post I wrote on my first use of Dask.

Contributions

Dask-ML can quickly find high-performing hyperparameters. I will back this claim with intuition and experimental evidence.

Specifically, this is because Dask-ML now implements an algorithm introduced by Li et. al. in “Hyperband: A novel bandit-based approach to hyperparameter optimization”. Pairing of Dask and Hyperband enables some exciting new performance opportunities, especially because Hyperband has a simple implementation and Dask is an advanced task scheduler.2

Let’s go through the basics of Hyperband then illustrate its use and performance with an example. This will highlight some key points of the corresponding paper.

Hyperband basics

The motivation for Hyperband is to find high performing hyperparameters with minimal training. Given this goal, it makes sense to spend more time training high performing models – why waste more time training time a model if it’s done poorly in the past?

One method to spend more time on high performing models is to initialize many models, start training all of them, and then stop training low performing models before training is finished. That’s what Hyperband does. At the most basic level, Hyperband is a (principled) early-stopping scheme for RandomizedSearchCV.

Deciding when to stop the training of models depends on how strongly the training data effects the score. There are two extremes:

  1. when only the training data matter
    • i.e., when the hyperparameters don’t influence the score at all
  2. when only the hyperparameters matter
    • i.e., when the training data don’t influence the score at all

Hyperband balances these two extremes by sweeping over how frequently models are stopped. This sweep allows a mathematical proof that Hyperband will find the best model possible with minimal partial_fit calls3.

Hyperband has significant parallelism because it has two “embarrassingly parallel” for-loops – Dask can exploit this. Hyperband has been implemented in Dask, specifically in Dask’s machine library Dask-ML.

How well does it perform? Let’s illustrate via example. Some setup is required before the performance comparison in Performance.

Example

Note: want to try HyperbandSearchCV out yourself? Dask has an example use. It can even be run in-browser!

I’ll illustrate with a synthetic example. Let’s build a dataset with 4 classes:

>>> from experiment import make_circles
>>> X, y = make_circles(n_classes=4, n_features=6, n_informative=2)
>>> scatter(X[:, :2], color=y)

Note: this content is pulled from stsievert/dask-hyperband-comparison, or makes slight modifications.

Let’s build a fully connected neural net with 24 neurons for classification:

>>> from sklearn.neural_network import MLPClassifier
>>> model = MLPClassifier()

Building the neural net with PyTorch is also possible4 (and what I used in development).

This neural net’s behavior is dictated by 6 hyperparameters. Only one controls the model of the optimal architecture (hidden_layer_sizes, the number of neurons in each layer). The rest control finding the best model of that architecture. Details on the hyperparameters are in the Appendix.

>>> params = ...  # details in appendix
>>> params.keys()
dict_keys(['hidden_layer_sizes', 'alpha', 'batch_size', 'learning_rate'
           'learning_rate_init', 'power_t', 'momentum'])
>>> params["hidden_layer_sizes"]  # always 24 neurons
[(24, ), (12, 12), (6, 6, 6, 6), (4, 4, 4, 4, 4, 4), (12, 6, 3, 3)]

I choose these hyperparameters to have a complex search space that mimics the searches performed for most neural networks. These searches typically involve hyperparameters like “dropout”, “learning rate”, “momentum” and “weight decay”.5 End users don’t care hyperparameters like these; they don’t change the model architecture, only finding the best model of a particular architecture.

How can high performing hyperparameter values be found quickly?

Finding the best parameters

First, let’s look at the parameters required for Dask-ML’s implementation of Hyperband (which is in the class HyperbandSearchCV).

Hyperband parameters: rule-of-thumb

HyperbandSearchCV has two inputs:

  1. max_iter, which determines how many times to call partial_fit
  2. the chunk size of the Dask array, which determines how many data each partial_fit call receives.

These fall out pretty naturally once it’s known how long to train the best model and very approximately how many parameters to sample:

n_examples = 50 * len(X_train)  # 50 passes through dataset for best model
n_params = 299  # sample about 300 parameters

# inputs to hyperband
max_iter = n_params
chunk_size = n_examples // n_params

The inputs to this rule-of-thumb are exactly what the user cares about:

  • a measure of how complex the search space is (via n_params)
  • how long to train the best model (via n_examples)

Notably, there’s no tradeoff between n_examples and n_params like with Scikit-learn’s RandomizedSearchCV because n_examples is only for some models, not for all models. There’s more details on this rule-of-thumb in the “Notes” section of the HyperbandSearchCV docs.

With these inputs a HyperbandSearchCV object can easily be created.

Finding the best performing hyperparameters

This model selection algorithm Hyperband is implemented in the class HyperbandSearchCV. Let’s create an instance of that class:

>>> from dask_ml.model_selection import HyperbandSearchCV
>>>
>>> search = HyperbandSearchCV(
...     est, params, max_iter=max_iter, aggressiveness=4
... )

aggressiveness defaults to 3. aggressiveness=4 is chosen because this is an initial search; I know nothing about how this search space. Then, this search should be more aggressive in culling off bad models.

Hyperband hides some details from the user (which enables the mathematical guarantees), specifically the details on the amount of training and the number of models created. These details are available in the metadata attribute:

>>> search.metadata["n_models"]
378
>>> search.metadata["partial_fit_calls"]
5721

Now that we have some idea on how long the computation will take, let’s ask it to find the best set of hyperparameters:

>>> from dask_ml.model_selection import train_test_split
>>> X_train, y_train, X_test, y_test = train_test_split(X, y)
>>>
>>> X_train = X_train.rechunk(chunk_size)
>>> y_train = y_train.rechunk(chunk_size)
>>>
>>> search.fit(X_train, y_train)

The dashboard will be active during this time6:

Your browser does not support the video tag.

How well do these hyperparameters perform?

>>> search.best_score_
0.9019221418447483

HyperbandSearchCV mirrors Scikit-learn’s API for RandomizedSearchCV, so it has access to all the expected attributes and methods:

>>> search.best_params_
{"batch_size": 64, "hidden_layer_sizes": [6, 6, 6, 6], ...}
>>> search.score(X_test, y_test)
0.8989070100111217
>>> search.best_model_
MLPClassifier(...)

Details on the attributes and methods are in the HyperbandSearchCV documentation.

Performance

I ran this 200 times on my personal laptop with 4 cores. Let’s look at the distribution of final validation scores:

The “passive” comparison is really RandomizedSearchCV configured so it takes an equal amount of work as HyperbandSearchCV. Let’s see how this does over time:

This graph shows the mean score over the 200 runs with the solid line, and the shaded region represents the interquartile range. The dotted green line indicates the data required to train 4 models to completion. “Passes through the dataset” is a good proxy for “time to solution” because there are only 4 workers.

This graph shows that HyperbandSearchCV will find parameters at least 3 times quicker than RandomizedSearchCV.

Dask opportunities

What opportunities does combining Hyperband and Dask create? HyperbandSearchCV has a lot of internal parallelism and Dask is an advanced task scheduler.

The most obvious opportunity involves job prioritization. Hyperband fits many models in parallel and Dask might not have that workers available. This means some jobs have to wait for other jobs to finish. Of course, Dask can prioritize jobs7 and choose which models to fit first.

Let’s assign the priority for fitting a certain model to be the model’s most recent score. How does this prioritization scheme influence the score? Let’s compare the prioritization schemes in a single run of the 200 above:

These two lines are the same in every way except for the prioritization scheme. This graph compares the “high scores” prioritization scheme and the Dask’s default prioritization scheme (“fifo”).

This graph is certainly helped by the fact that is run with only 4 workers. Job priority does not matter if every job can be run right away (there’s nothing to assign priority too!).

Amenability to parallelism

How does Hyperband scale with the number of workers?

I ran another separate experiment to measure. This experiment is described more in the corresponding paper, but the relevant difference is that a PyTorch neural network is used through skorch instead of Scikit-learn’s MLPClassifier.

I ran the same experiment with a different number of Dask workers.8 Here’s how HyperbandSearchCV scales:

Training one model to completion requires 243 seconds (which is marked by the white line). This is a comparison with patience, which stops training models if their scores aren’t increasing enough. Functionally, this is very useful because the user might accidentally specify n_examples to be too large.

It looks like the speedups start to saturate somewhere between 16 and 24 workers, at least for this example. Of course, patience doesn’t work as well for a large number of workers.9

Future work

There’s some ongoing pull requests to improve HyperbandSearchCV. The most significant of these involves tweaking some Hyperband internals so HyperbandSearchCV works better with initial or very exploratory searches (dask/dask-ml #532).

The biggest improvement I see is treating dataset size as the scarce resource that needs to be preserved instead of training time. This would allow Hyperband to work with any model, instead of only models that implement partial_fit.

Serialization is an important part of the distributed Hyperband implementation in HyperbandSearchCV. Scikit-learn and PyTorch can easily handle this because they support the Pickle protocol10, but Keras/Tensorflow/MXNet present challenges. The use of HyperbandSearchCV could be increased by resolving this issue.

Appendix

I choose to tune 7 hyperparameters, which are

  • hidden_layer_sizes, which controls the activation function used at each neuron
  • alpha, which controls the amount of regularization

More hyperparameters control finding the best neural network:

  • batch_size, which controls the number of examples the optimizer uses to approximate the gradient
  • learning_rate, learning_rate_init, power_t, which control some basic hyperparameters for the SGD optimizer I’ll be using
  • momentum, a more advanced hyperparameter for SGD with Nesterov’s momentum.
  1. Which amounts to choosing alpha in Scikit-learn’s Ridge or LASSO 

  2. To the best of my knowledge, this is the first implementation of Hyperband with an advanced task scheduler 

  3. More accurately, Hyperband will find close to the best model possible with $N$ partial_fit calls in expected score with high probability, where “close” means “within log terms of the upper bound on score”. For details, see Corollary 1 of the corresponding paper or Theorem 5 of Hyperband’s paper

  4. through the Scikit-learn API wrapper skorch 

  5. There’s less tuning for adaptive step size methods like Adam or Adagrad, but they might under-perform on the test data (see “The Marginal Value of Adaptive Gradient Methods for Machine Learning”) 

  6. But it probably won’t be this fast: the video is sped up by a factor of 3. 

  7. See Dask’s documentation on Prioritizing Work 

  8. Everything is the same between different runs: the hyperparameters sampled, the model’s internal random state, the data passed for fitting. Only the number of workers varies. 

  9. There’s no time benefit to stopping jobs early if there are infinite workers; there’s never a queue of jobs waiting to be run 

  10. Pickle isn’t slow, it’s a protocol” by Matthew Rocklin 

September 26, 2019

Carlos Fenollosa (carlesfe)

Sourcehut, the free software development cloud September 26, 2019 01:00 PM

I've been following sourcehut for some time and realized that I hadn't talked about it yet.

Sourcehut, at a first glance, is a service that provides git repos and code/project management tools, but it's much more.

It runs CI through virtualised builds on various Linuxes and BSD, provides code review tools, tasks and third party integrations, and of course, mailing lists and wikis.

I think the landing page does a good job of explaining how it is different from other code hosting services. Especially:

  • Composable Unix-style mini-services
  • Powerful APIs and webhooks
  • Secure, reliable, and safe
  • Absolutely no tracking or advertising
  • All features work without JavaScript
  • 100% free and open source software

These bullet points are quite important: sourcehut is the web equivalent of piping UNIX commands for development and is built entirely on free software. The fact that it works without js is just a great bonus.

Sourcehut is the project of a single developer, Drew DeVault, better known as sir_cmpwn on the internet, and he's quite active on Mastodon in case you want to follow him. Amazing work!

Tags: software, unix

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

September 25, 2019

eta (eta)

Designing a new chat system - federation September 25, 2019 11:00 PM

[This is post 2 about designing a new chat system. See the previous post on chat systems for more context!]

Okay, so, the previous post on chat systems seemed to generate a fair deal of discussion (in that the Matrix lead developer showed up and started refuting all of my points1…!).

Partially since I said I’d try my own hand at making something new in that post, and partially because this project is also going to be my Computing A-level Non-Examined Assessment2, I now need to actually get on with it and try and make something! So, without further ado…

Key implementation goals

From the frustrations expressed in the previous post, and my own personal experience, I believe a working chat system probably wants to include some of the following points:

  • Reliability. First and foremost, the system needs to deliver the messages to the recipient. Notifications should work, and be as timely as possible. The system should avoid just dropping messages on the floor without any indication as to this happening.
    • If messages cannot be delivered, the system should make this abundantly clear to the user, so they know about it and can try again.
    • Read / delivery receipts, as well as presence3, should be taken into account.
  • Interoperability. As discussed previously, your chat system is useless if it doesn’t talk to the ones that are already out there, unless you’re really good at persuading people. There should be some provision for bridging to existing systems, and potentially some for federation as well.
    • More on this later; federation is arguably a questionable design decision, but we’ll discuss this.
  • Persistence. People nowdays have phones and computers that aren’t online 24/7. The server should account for this, and implement some sort of basic “store-and-forward” mechanism, so you can send messages to people when they’re offline.
  • Flexibility. Using an online chat system often means losing a degree of control about how you come across, and what information you send. Things like read receipts and presence should be configurable, to allow for users having different privacy preferences. Notifications should also be relatively granular, so you can avoid your phone vibrating every time someone says something in one of your many chatrooms.

Ideally I’d do some kind of research to prove that these goals are actually prized by your average user4 – who knows, maybe sending emoji or something is actually way more important, and users will sacrifice one or more of the above in order to get special features like that5 – but we’ll roll with this for now. (In fact, if you happen to be reading this blog and have strong opinions on the matter, please leave a comment; it’d be really helpful!) I suppose we’ll just say this blog post is a living document, and leave things at that for now.

The first one of these bullet-points I plan to tackle is interoperability – and specifically, whether this magical new ideal chat system should federate or not.

The question of federation

Federation refers to the practice of a bunch of people agreeing on a common standard, and designing software using this standard to create an open network of interconnected servers. The example cited in the previous post was Mastodon, which uses the ActivityPub standard to enable distributed social networking; the Matrix is another federated standard for chat6. Federated protocols, at least at a first glance, aren’t doing too badly; this site gives an overview of various federated protocols and their uptake in terms of users.

In fact, the biggest federated protocol in the world is email – you can set up your own mail server with your own domain (e.g. hi@theta.eu.org), and start emailing with people on any other server; email is based on a set of open standards that anyone can implement.

The spam problem

However, being federated can both be a blessing and a curse! Having your communications platform be wide open, so anybody can create an account, or federate their messages into it, or whatever, seems like a good idea at first. However, there’s another side to it: how do you control spam and abuse? Most people get spam emails nowadays, which is one of the unfortunate side effects of this openness; if anyone can connect to your mail server and send you email, without having to do anything first to confirm themselves as legitimate or trusted, you’re opening yourself up to receiving a whole bunch of junk.

There are ways to combat this – spam email checkers nowadays, like the venerable SpamAssassin7, tend to use a rule-based approach, essentially giving emails a ‘spam score’ based on various metrics of sketchiness: does the email pass SPF and DKIM checks? Does the subject line contain something like “get rich quick”? Is the sender using a commonly abused free email service, like Gmail or Yahoo Mail? (and so on and so forth). These work pretty well for the run-of-the-mill spam, but still can’t really protect against actively malicious people who conform to all the standards, and still send you spam.

In many chat protocols, we see a similar story. IRC has suffered greatly at the hands of robots that connect to any IRC servers they can find, join all the channels they can, and start flooding text until something notices and kicks them off (having still managed to cause a considerable amount of annoyance). The freenode IRC network suffered a recent spamwave that was actually targeted at them, with people spamming rude and unhelpful messages about the network administration until something was done about it8. Matrix looks like it’s also vulnerable to the same kind of thing as well910. Spam is a non-negotiable part of the internet, and efforts to fight it are almost as old as the internet itself!

Mastodon manages to avoid a lot of these problems (at least from what I can see) simply as a result of what it is – in something like Twitter, you don’t really get spam in your timeline unless you explicitly choose to follow the spammers, and posts only start federating from instance to instance once an actual human issues that follow request. This is in contrast to, say, a wide-open federated chatroom, where people can just join on as many anonymous clients as they want whenever they feel like flooding a channel with text.

The full-mesh problem

The other main issue with federation is somewhat more practical: if your chat system, or social network, or whatever, is now distributed across multiple servers, how do you actually shunt messages around between them in a vaguely efficient manner? So far, the answer seems to just be “give up and use a full mesh architecture” – that is, if we need to send messages out to users on 20 different servers, connect individually to each 20 and deliver the message. (‘what happens when you activity post’ is a good, short explainer for how this works in terms of ActivityPub, Mastodon’s federation protocol.)

This works fine for smaller use-cases, but can be somewhat problematic when it comes to either (a) large follower counts, or (b) sending big files like images. As Ted Unangst says in his honk 0.1 announcement:

I post a honk, with a picture of a delicious pickle, and it goes out to all my followers. These are small messages, great. Their instances will in turn immediately come back and download the attached picture, however, resulting in a flood of traffic. If a particularly popular follower shares the post, even bigger flood.

There’s only so much honk can do here, since even trickling out outbound notes doesn’t control what happens when another instance shares it. Ironically, I think some [Mastodon] instances are spared from overload because other instances are already overloaded and unable to immediately process inbound messages.

Essentially, the issue is twofold: firstly, when you’re sending something in a federated environment, you usually have to talk to each server in your chatroom, or with your followers on it, or whatever, to deliver your message, which takes time and OS resources (most OSes limit the number of outgoing TCP connections, for example, and constructing/destructign them isn’t free in any case). Secondly, as mentioned above, if you do something like join a new chatroom or have your popular message shared by another larger server, hundreds of interested servers might go and hammer yours asking for information of some kind, causing you to become somewhat overloaded. (For example, protocols often require users to have cryptographic keys, in order to be able to verify message authenticity, and getting these keys usually involves going to the server and asking for them – which is a problem if there are suddenly loads of new servers that have never heard of your user before!).

In contrast, non-openly-federated protocols like IRC don’t have this problem; IRC is as old as dirt and still realised that having everything be full-mesh isn’t the best of ideas. Instead, servers are linked in a spanning tree (see ircdocs.horse for a better explanation and pretty diagram), such that servers only need to broadcast messages to the servers they’re directly connected to, which will forward the messages on further, propagating them throughout the tree without any need for non-connected servers to ever talk to one another. This is far more efficient – but it does assume a high degree of trust in all the servers that make up the network, which wouldn’t work in a federated context where you can’t trust anyone.

The CAP theorem problem

The other big problem is that this thing called the CAP theorem exists, which says we can only have at most two of the following three guarantees, if trying to store state across a distributed system:

  • Consistency - all servers have a consistent view of the world
  • Availability - the system can be queried at all times, although asking two servers may return different results
  • Partition tolerance - the system doesn’t break if there’s a network split

Essentially, what this distils down to is the idea that you will have a network split or partition somewhere (where some servers are unreachable for whatever reason), and, when this happens, you will either have two sides of the network that have a different view of the world (thus violating consistency), or you will have to stop accepting new data in order to make sure that stuff doesn’t get out of sync (thus violating availability). This isn’t really something you can get around; it’s a fact of life when designing distributed systems.

Different systems deal with this in different ways:

  • If an IRC network suffers a [netsplit][ircns], the two sides of the network see all the users on the other side quitting, and they can’t talk to them any more.
  • ActivityPub doesn’t really have to care about this, because there’s no distributed state anywhere; if an AP server wants to find out something about a user, it asks that user’s server.
  • Matrix actually has this rather nifty “eventual consistency” property, where you sacrifice consistency in the short-term when netsplits occur, but the system eventually sorts itself out when everyone gets reconnected again.

Given that we’ve specified our ideal chat system wants to be persistent as well as interoperable, we’re going to have to consider this problem somewhere – unless we get rid of distributed state entirely, that is. However, chat systems usually do have some idea of state, like who has administrator permissions in a chatroom.

Conclusions

Federation has been the source of many good things (interoperability! no lock-in!) and many bad ones (spam! networking complications!). As such, it’s not immediately obvious that completely open federation works for chat – contrary to what was said in the previous post, wholesale copying ActivityPub might not really work for something like chat, due to spam and issues with distributed state. Instead, we’re going to need something else – and we’ll explore what that something else could perhaps be in further blog posts in this series!


  1. It’s worth reading through his comments, and coming to your own judgement as to whether you think what I’ve said is fair or not. I didn’t really respond, because I think we fundamentally disagree on whether the Matrix architecture is a good idea, and there wasn’t much point debating that further; the comments are there, and you’re welcome to form your own opinion! 

  2. (Blogging about this process is arguably a questionable plan.) 

  3. Presence is the feature that lets you know whether someone’s online or not, or when they were last online. WhatsApp calls it “last seen”; IRC has the /away command; multiple other examples exist. 

  4. The A-level examiners tend to like that sort of stuff – and for a good reason; it’s nice to know that you aren’t just blindly spending time implementing something nobody wants. 

  5. Snapchat is a pretty good example of people sacrificing both interoperability (you can’t bridge Snapchat to anything else) and flexibility (you have very little control over read receipts, presence, and things like that). However, the added functionality seems to be worth it! 

  6. …which I somewhat harshly criticised in the last blog post. 

  7. This is what theta.eu.org runs for spam detection, actually, and it mostly works! 

  8. Actually, I’m still not entirely sure why this spamwave stopped; I’d be tempted to believe that the spammers giving up and stopping it themselves was probably the main reason, although I know some additional filtering protections were put in place. 

  9. Citation: I was lurking in #matrix on Freenode yesterday (2019-09-25, ~20:00) when some random angry user started coming in and flooding the channel with random spam images. 

  10. (I swear, I don’t have a vendetta against them or anything!) 

Unrelenting Technology (myfreeweb)

Noticed something on dmesgd… looks like MIPS (64) isn’t that dea... September 25, 2019 03:34 PM

Noticed something on dmesgd… looks like MIPS (64) isn’t that dead: new(ish) Ubiquiti EdgeRouters have newer Octeon processors — quad-core 1GHz (and with an FPU). And 1GB RAM. That’s much better than the Lite’s dual-core 500MHz && 512MB RAM.

…wait, actually, there’s even big 16-cores (and 16GB RAM) in 10G routers!

September 23, 2019

Pete Corey (petecorey)

Apollo Quirks: Polling After Refetching with New Variables September 23, 2019 12:00 AM

While working on a recent client project, Estelle and I ran into a fun Apollo quirk. It turns out that an Apollo query with an active pollInterval won’t respect new variables provided by calls to refetch.

To demonstrate, imagine we’re rendering a paginated table filled with data pulled from the server:


const Table = () => {
    let { data } = useQuery(gql`
        query items($page: Int!) {
            items(page: $page) {
                pages
                results {
                    _id
                    result
                }
            }
        }
    `, {
        pollInterval: 5000
    });
    
    return (
        <>
            <table>
                {data.items.results.map(({ _id, result }) => (
                    <tr key={_id}>
                        <td>{result}</td>
                    </tr>
                ))}
            </table>
        </>
    );
};

The items in our table change over time, so we’re polling our query every five seconds.

We also want to give the user buttons to quickly navigate to a given page of results. Whenever a user presses the “Page 2” button, for example, we want to refetch our query with our variables set to { page: 2 }:


 const Table = () => {
-    let { data } = useQuery(gql`
+    let { data, refetch } = useQuery(gql`
         query items($page: Int!) {
             items(page: $page) {
                 pages
                 results {
                     _id
                     result
                 }
             }
         }
     `, {
         pollInterval: 5000
     });
     
+    const onClick = page => {
+        refetch({ variables: { page } });
+    };
     
     return (
         <>
             <table>
                 {data.items.results.map(({ _id, result }) => (
                     <tr key={_id}>
                         <td>{result}</td>
                     </tr>
                 ))}
             </table>
+            {_.chain(data.items.pages)
+                .map(page => (
+                    <Button onClick={() => onClick(page)}>
+                        Page {page + 1}
+                    </Button>
+                ))
+                .value()}
         </>
     );
 };

This works… for a few seconds. But then we’re unexpectedly brought back to the first page. What’s happening here?

It turns out that our polling query will always query the server with the variables it was given at the time polling was initialized. So in our case, even though the user advanced to page two, our polling query will fetch page one and render those results.

So how do we deal with this? This GitHub issue on the apollo-client project suggests calling stopPolling before changing the query’s variables, and startPolling to re-enable polling with those new variables.

In our case, that would look something like this:


 const Table = () => {
-    let { data, refetch } = useQuery(gql`
+    let { data, refetch, startPolling, stopPolling } = useQuery(gql`
         query items($page: Int!) {
             items(page: $page) {
                 pages
                 results {
                     _id
                     result
                 }
             }
         }
     `, {
         pollInterval: 5000
     });
     
     const onClick = page => {
+        stopPolling();
         refetch({ variables: { page } });
+        startPolling(5000);
     };
     
     return (
         <>
             <table>
                 {data.items.results.map(({ _id, result }) => (
                     <tr key={_id}>
                         <td>{result}</td>
                     </tr>
                 ))}
             </table>
             {_.chain(data.items.pages)
                 .map(page => (
                 <Button onClick={() => onClick(page)}>
                     Page {page + 1}
                 </Button>
                 ))
                 .value()}
         </>
     );
 };

And it works! Now our polling queries will fetch from the server with the correctly updated variables. When a user navigates to page two, they’ll stay on page two!

My best guess for why this is happening, and why the stopPolling/startPolling solution works is that when polling is started, the value of variables is trapped in a closure. When refetch is called, it changes the reference to the options.variables object, but not the referenced object. This means the value of options.variables doesn’t change within polling interval.

Calling stopPolling and startPolling forces our polling interval to restart under a new closure with our new variables values.

September 22, 2019

Carlos Fenollosa (carlesfe)

September 21, 2019

Gonçalo Valério (dethos)

kinspect – quickly look into PGP public key details September 21, 2019 07:59 PM

Sometimes I just need to look into the details of a PGP key that is provided in its “armored” form by some website (not everyone is publishing their keys to the keyservers).

Normally I would have to import that key to my keyring or save it into a file and use gnupg to visualize it (as it is described in this Stack Overflow answers).

To avoid this hassle I just created a simple page with a text area where you can paste the public key and it will display some basic information about it. Perhaps an extension would be a better approach, but for now this works for me.

You can use it on: https://kinspect.ovalerio.net

In case you would like to contribute in order to improve it or extend the information displayed about the keys, the source code is available on Github using a Free Software license: https://github.com/dethos/kinspect

Unrelenting Technology (myfreeweb)

“header” is a rather unfortunate word overloading in English. Searching for “tpm header” gives... September 21, 2019 12:49 PM

“header” is a rather unfortunate word overloading in English. Searching for “tpm header” gives you everything about… well, the physical header on mainboards :D Even with “tpm protocol header”, only Google finds something related to protocol specs, and in the lower half of the first page.

September 16, 2019

Unrelenting Technology (myfreeweb)

I’ve been rewriting the engine that runs this website in the past few months..... September 16, 2019 05:55 PM

I’ve been rewriting the engine that runs this website in the past few months..

and now, finally, unrelenting.technology runs on sweetroll2! Longer writeup coming, this is more of a test post.

Pete Corey (petecorey)

Elixir Style Conditions in Javascript September 16, 2019 12:00 AM

Elixir has a useful control flow structure called cond that lets you branch on arbitrary conditions. Unlike the more common switch control structure (case in Elixir), cond doesn’t match against a predetermined value. Instead, it evaluates each condition, in order, looking for the first one that evaluates to a truthy value (not nil or false).


numbers = [1, 2, 3]
result = cond do
  4 in numbers ->
    :four
  6 == Enum.sum(numbers) ->
    :sum
  true ->
    :default
end

This is all probably old hat to you.

As I mentioned, cond can be an incredibly useful control structure, and there are times when I’ve missed it while working in languages like Javascript that only have switch expressions.

A traditional Javascript implementation of the above (with a little help from Lodash) would look something like this:


let numbers = [1, 2, 3];
if (_.includes(numbers, 4)) {
  var result = "four";
} else if (6 === _.sum(numbers)) {
  var result = "sum";
} else {
  var result = "default";
}

However, I recently stumbled upon a trick that lets you implement a switch statement in Javascript that behaves very similarly to a cond expression in Elixir. The key is to switch on the value of true. The case expressions that evaluate to true will match, and their corresponding statements will be evaluated in order.


let numbers = [1, 2, 3];
switch (true) {
  case _.includes(numbers, 4):
    var result = "four";
    break;
  case 6 === _.sum(numbers):
    var result = "sum";
    break;
  default:
    var result = "default";
    break;
}

Whether or not this is any more useful or readable than a series of if/else blocks is debatable. That said, this is definitely an interesting example of perspective shifting and seeing old code in a new light. Hopefully you find it as interesting as I do.

September 15, 2019

Carlos Fenollosa (carlesfe)

September 14, 2019

Caius Durling (caius)

curl --resolve September 14, 2019 05:10 PM

Sometimes it's useful to be able to craft a request to one server, using a DNS name that's either not defined or currently pointed to a different IP. (Migrating webservers, testing a new webserver config out, etc.)

Historically for HTTP calls this was easy, just set the Host header as you make the http request to the IP directly:

 curl -H "Host: caiustheory.com" http://10.200.0.1/

However, HTTPS throws a bit of a spanner in the works, if we just try to connect using an overridden Host header, we get an error back from the server if it's not configured with a certificate for the IP address:

 $ curl -H "Host: caiustheory.com" https://10.200.0.1/
curl: (51) SSL: no alternative certificate subject name matches target host name '10.200.0.1'

Usually at this point I'd just start editing /etc/hosts to add 10.200.0.1 caius.name to it and carry on testing. This is a pain when you're testing more than one server, or you're on a machine where you don't have root access to edit /etc/hosts.

In later versions of curl there's a solution for this built into the binary, in the form of the --resolve flag. You can tell it to override the DNS lookup for a specific hostname/port combination. This in turn means that the correct host is forwarded to the server for the correct SSL certificate to be chosen to serve the request based on host.

It takes the form --resolve HOST:PORT:IP where HOST is the human-friendly host, PORT is the webserver's port (convention is 80 for HTTP, 443 for HTTPS) and IP being the destination IP you want to hit. (As opposed to the one in DNS currently.)

$ curl --silent --head --resolve caiustheory.com:443:10.200.0.1 https://caiustheory.com | head -n1
HTTP/2 200

And voila, you don't need to fiddle with editing /etc/hosts. Just use --resolve to hit a different IP for a given host.

September 13, 2019

Patrick Louis (venam)

September 2019 Projects September 13, 2019 09:00 PM

In the blink of an eye 6 months have gone by. Since then, I’ve written a single article about time on the internet and thus the blog needs an update on my latest endeavours.

Psychology, Philosophy & Books

Language: brains
Explanation:

The Stand

I’ve let go of my daily diary and happiness tracking application, the ones I hinted in my previous post, and it was apparent that I find genuine happiness in books. In short, I enjoy reading much more than I thought I was.

Here’s the reading list of the past few months.

  • Who moved my cheese - Spencer Johnson
  • Peaks and valeys - Spencer Johnson
  • Mature optimization handbook - Carlos Bueno
  • Leadership presence - HBR Emotional Intelligence Series
  • irobot - Isaac Asimov
  • The pleasure of finding things out - Richard Feynman
  • The stand - Stephen King (My longest read to date)
  • Smoke and Mirror - Neil Gaiman
  • Out of the Dark - Gregg Hurwitz
  • Childhood’s end - Arthur Clarke
  • Clean Architecture - Robert Martin

And the countless articles from the newsletter, books that aren’t physical, or anything that my memory is unkind to.

Furthermore, I’m currently in the process of reading “What If?” by Randall Munroe, “how to” by the same author, “The science of food” by Marty Jopson which I’m enjoying with my SO, and eagerly devouring my copy of “Software Architecture in Practice 3rd edition”.

As for podcasts, I’ve exhausted my list which means I’m keeping up with the dozen subscriptions I have. Let’s mention them.

  • The Joe Rogan Experience
  • Stuff You Should Know
  • Snap Judgement
  • Philosophy Bites
  • Planet Money
  • The Philosopher’s Zone
  • Radiolab
  • Making Sense
  • The Psychology Podcast
  • LKT - Let’s know things
  • You Are not so Smart
  • Hidden Brain
  • Team Human
  • Philosophize This!
  • Intercepted
  • Modern Love
  • The Food Chain
  • Lore
  • InfoQ podcast
  • All in the mind
  • Science Friday
  • Big Questions Podcast
  • Levar Burton Reads
  • The knowledge Project
  • Darknet Diaries
  • CMV - Change my view
  • 50 things that made the modern economy
  • Invisibilia
  • Hi-Phi Nation
  • 30 Animals that made us smarter
  • The history of GNOME
  • The end of the world with Josh Clark
  • A history of the world in a 100 object
  • General philosophy
  • The Rhine

I’ve given The Rhine series a second listen. History gives fantastic insights on where the shape of our world comes from. It draws the lines that aren’t apparent.
Giving such a long list wouldn’t matter if I wasn’t sharing my favorite picks. Obviously, this would include The Rhine, but also Planet Money, The Food Chain, Levar Burton Reads, Science Friday, Hidden Brain, and LKT. Those are the on-going podcasts that keeps me from dying on the road each week.

Learning, Growth, & Community

Language:growth
Explanation:

clean architecture

I’ve matured professionally. It’s been two consecutive years that I’ve taken part in a small team, initially an XP duo, later on a trio, and then virally creating new roles all throughout the company, building a highly secure and accredited eSIM solution from scratch.

The project is based on new specifications and thus uncharted territories. This has allowed me to get even more comfortable reading specifications, protocols, architectures, and RFCs. You got to have patience and reading skills in this field and I don’t think I’m missing any of those. It’s similar to what I used to do in the newsletter but having it as my day-to-day job has been an added benefice.

The XP methodology, though informal, is intellectually stimulating. I’ve learned extensively from my seniors not only through seeing them handle tasks but also via discussions (sometimes heated) about technical decisions that needed to be made. And there were a lot of those decisions, this is the kind of flexibility that let you experiment. Many of my misconceptions have been shattered.
If you know one thing about me, through my blog, newsletter, or podcast, it’s that I’ve got a knack for shinning light on software stacks and how they fall into place, I like to explain systems from above. I have a penchant for software architecture, drawing diagrams, understanding why things are built the way they are.
However, time is a costly parameter in the life’s equation and I had to give up something to get another in return.

Edward

Consequentially, I’ve put the newsletter on hiatus. It pains me to get a bit further away from Unix, already knowing I’ve discontinued the nixers podcast more than a year ago. Unix stays in my heart!
I’m thankful to Vermaden for his participation throughout the newsletter, he’s still continuing his valuable news on his blog.
Nevertheless, next February I’ll be in Belgium for Fosdem, the tickets are booked and I’m ecstatic about finally assisting in person. (See you there)

Now it’s been almost a month that I’ve begun my software architecture studies and I plan to keep track of the path I’m undertaking. I’ll write articles on the topic as soon as I’m experienced enough. Meanwhile, here’s what I have in mind as my future steps.
To help me remember important details as I learn them, I’m using a software called Anki. Apart from the two books I’ve previously listed, namely “Clean Architecture”, which I finished, and “Software Architecture in Practice”, I’m following a Youtube channel called “Software Architecture Mondays”, I’m training to draw diagrams the right way, I’m reading a couple of articles everyday analogous to what I was doing for the newsletter, and I’m planning to register for an official certification later on.
That’s the plan.

Apart from these, I’m reviewing my algorithm knowledge. Doing a couple of exercises a week on leetcode and geeksforgeeks. This helps keep my problem solving skills up to date, or at least prepares me for the modern and radically tough interview style.

And all of this happens on my redesigned workstation! First of all, I’m now on a standing desk. Secondly, I got a new mechanical keyboard with red switches for work. Thirdly, I got o-rings dampeners. Fourthly, I got a 22inch screen for my home-station. Fifthly, I got a wrist rocker for the new keyboard. And finally, I got a trackball.

workplace

Other Learning

Language: gray matter
Explanation:

curiosity stream

My SO influenced me to install a brain teaser application called Elevate. I’ve been playing it ever since, a 142 days streak. I should note too that I’ve bought the pro version.

Another of my girlfriend’s influence, is a subscription to a documentary streaming service named Curiosity Stream, which was initially intended for her but that I happen to use quite often.

To contrast those, I’ve haven’t continued the Spanish refreshment that I was intending. It simply isn’t useful at this point in time.

Ascii Art & Art

Language: ASCII
Explanation:

Abstract

I’ve pushed 4 pieces since the last post, that may not seem like an impressive number but quantity doesn’t mean quality. Of those pieces, one was the last dinosaur in the series, a triceratops, another is the amazing totem pole piece that I’ve added above this section, the next one is a piece dedicated to the now defunct Linux Journal that was featured in one of their issues (command line July 2019), and the last one is a piece christened Alamanni that reached third place at Evoke 2019.
Moreover, the Impure Ascii art group released two packs, 72, and 73.

Aside from ASCII art, I’ve painted a single canvas. Instead, what I’ve focused on the most is singing. I’ve been training since February, using an application called Sing Sharp, another one called Perfect Ear, and wrapping this with a small playlist to sing-along in the car while in traffic. I can affirm objectively, through the app rating, that I’ve progressed.

2bwm and Window management

Language: C
Explanation:

There hasn’t been much change to 2bwm. A user named netcrop has been sending pull requests and I’ve reviewed them.
I’ve tried to implement centered pop-ups but there is still one issue I didn’t catch yet.

Life and other hobbies

Language: life
Explanation:

miami

In June I spent 2 weeks in the USA, one week in South Beach Miami and the other in New York City. Even though the 12-15 hours in the plane were intense it was all worth it. After seeing those two cities I’ve formed a better opinion of urban planning.

During my stay in Miami I reignited my love of swimming. Therefore when I came back to Lebanon I had to give it another try. Still, in spite of all the love I have, my attempts were thwarted. The pollution of the beach in Lebanon is just incomparable to what I’ve experienced in Miami. In sum, I was discouraged.

Summer isn’t the time for mushrooms in Lebanon but I’ve nevertheless relished in the wonders of fresh King Oyster, fresh Shiitake, fresh Portobello, fresh White Button, dried morels, dried Porcini, and more. Adding to those, I’m testing taking Lion’s Mane and Cordyceps supplements, so far so good.
Aside from those, I’ve found ginseng tea to be fantastically savory and revitalizing.

On that topic, I’ve been cooking many new recipes. Trying to use the oven more. My cooking page though hasn’t been updated in a while and it’s not as presentable as I wish it was and thus I haven’t been adding pictures on it.

Finally, summer is all about flowers and plants. So I’ve spent a big amount of time investing in my garden, cutting branching, cleaning, planting pumpkins, growing from cuttings, etc..

Now

Which all leads to what’s in store for tomorrow. More of the same but upgraded.

The next months I’m going to put the emphasis on software architecture. That’s really the single new and most important activity I’m going to partake in.
Plus whatever I’m already doing but more of it!

This is it!

As usual… If you want something done, no one’s gonna do it for you, use your own hands.
And let’s go for a beer together sometime, or just chill.

Gustaf Erikson (gerikson)

July September 13, 2019 07:30 PM

Introducing HN&&LO; September 13, 2019 07:28 PM

HN&&LO is a web page that scrapes the APIs of Hacker News and Lobste.rs and collates those entries that share a URL.

This lets you easily follow the discussion on both sites for these links.

I wrote this tool because I’m much more active on Lobste.rs than on HN, but I would like to keep “tabs” on both sites. I’m especially interested in how the information is disseminated between the sites.

“Slow webapps”

I’ve long liked to grab stuff via a web API, stuff it into a DB, and output a web page based on the data. This project was a first attempt to use a templating engine in Perl, and I’m ashamed to say I’ve taken so long to understand the greatness of this approach.

The data updates hourly, there’s rarely more than a couple of new entries on Lobste.rs per hour.

In fact, this can illustrate the difference in scale between the two sites.

Time period: 5 Jul 2019 11:19 to 8 Aug 2019 23:59 (both times in UTC).

  • Number of submissions with URLS on Hacker News: 26,974
  • Number of submissions on Lobste.rs: 899

Average number of submissions per hour: 33 for HN, 1 for Lobste.rs.

Once updated, a static web page is generated. This keeps overhead low.

Differences between Hacker News and Lobste.rs

Coincidentally, while I was writing this post, an article was published in the New Yorker about the moderators (all two of them) of Hacker News. This sparked a discussion on HN (obviously) but also on Lobster.rs (see both linked here).

To me, what makes Lobste.rs better is that it’s much smaller (one can easily catch up with a day’s worth of comments), the editing tools are more sophisticated, and the tag system allows one to determine whether a post is interesting or not. HN is much harder to grasp.

I think the tendencies noted in the article are common to both sites - male, youngish software developers with a rationalist bent. But I think Lobste.rs is a bit more self-aware of this fact (or rather, critiques are more visible there). In some ways, Lobste.rs, being a reaction to HN, defines itself culturally both for and against the bigger site.

Source and TODO

If you’re interested in pedestrian Perl code, source is here.

TODO list is here

Pete Corey (petecorey)

Obverse and Under September 13, 2019 12:00 AM

I previously wrote about plotting an “amazing graph” using the J programming language. The solution I landed on looked something like this:


require 'plot'
f =: ] - [: #. [: |. #:
'type dot' plot f"0 p: i. 10000

Our verb, f, is taking a very explicit approach by making judicious use of “capped” ([:) verb trains. We’re essentially saying that f is (=:) the given number (]) minus (-) the base two (#.) of the reverse (|.) of the antibase two (#:) of the given number.

Several members of the J community pointed out to me that this verb could be simplified with the help of the “under” (&.) conjunction. Let’s dig into what “under” is, and how we can use it.

Under What?

The best way to think about “under” (&.), as explained by the NuVoc page on “under”, is to think in terms of domains and transformations in and out of those domains.

Verb v defines a transformation of the argument(s) (x and) y into the v-domain. Next, verb u operates on the transformed argument(s). Lastly the result is transformed back from the v-domain to the original domain.

In our example, the domain of our input is base ten, but the transformation we want to apply (reversal) needs to happen in the base two domain. “Under” (&.) can be used to transform our input into base two (#:), apply our reversal (|.), and transform the result of that reversal back to our original base ten domain with the obverse, or opposite, of our base two verb, anti base (#.):


f =: ] - |. &. #:

Notice that we’re not explicitly stating how to transform the result of our reversal back into our original domain. J knows that the obverse of #: is #., and automatically applies it for us.

Out of the box, J comes with many obverse pairings. “Open” (>), for example, is the obverse of “box” (<), and visa versa. This pairing is especially useful when applying transformations to boxed values:


   1+&.>1;2;3
┌─┬─┬─┐
│2│3│4│
└─┴─┴─┘

Check out a full listing of obverse pairs at the end of this Shades of J article.

Inferred Obverses

Even compound verbs built up of verbs with well-defined obverse pairings can be used with “under” (&.). J will correctly infer and apply the compound obverse without any intervention or instruction.

For example, if we wanted to unbox a list of values and then work with them in the “square root domain” (whatever that means), we could do something like this:


   1+&.([:%:>)1;2;3
┌────────────────┐
│4 5.82843 7.4641│
└────────────────┘

J takes each value, opens it and finds its square root ([:%:>), adds one to the result, and then squares and boxes up ([:*:<) the incremented value.

Explicit Obverses

Even more interestingly, if an obverse pairing isn’t defined or inferable for a given verb, J lets us define our own pairing using the “obverse” (:.) verb.

As an example, imagine that we have a JSON string holding an array of values. We want to parse our string, perform some operation on those values, and then serialize the resulting list back into JSON.

We can use the dec_json and enc_json verbs provided by the convert/json package, and tell J that the obverse of dec_json is enc_json:


   json =: dec_json :. enc_json

Running dec_json on a JSON array like '[1, 2, 3]' will return a list of boxed numbers, so we’ll want to open each of these boxes, perform our operation, and box the results back up. This sounds like another job for “under” (&.):


   transform =: 1&+&.>

All together, we can perform our transform “under” (&.) the json domain:


   transform &. json '[1, 2, 3]'
[2,3,4]

And our result is the JSON string '[2,3,4]'!

“Under” is definitely a very powerful conjunction, and I can see myself using it extensively in the future. Thanks to everyone in the J community who was kind enough to point it out and teach me something new!

September 11, 2019

Pages From The Fire (kghose)

Walk through of a UX decision September 11, 2019 07:44 PM

A software developer has to make decisions all the time about how to implement something. Decisions related to the user interface (UI) of a program strongly affect the user experience (UX) which is critical to the successful adoption of the software. In this post I will talk about the design of a particular aspect of some …

Carlos Fenollosa (carlesfe)

La predicción del tiempo en tu calendario September 11, 2019 05:21 PM

(Even though I write my blog in English, this post is in Spanish for obvious reasons. Click here to translate it with Google)

Si te pasas el día mirando el calendario, agendando reuniones y eventos, y echas en falta tener a mano el tiempo que va a hacer, he creado una utilidad que te puere resultar muy interesante.

Se trata de mostrarte la predicción del tiempo en tu municipio en el mismo calendario

Es muy sencillo: escribe el nombre de tu municipio y pulsa el botón. No hay que registrarse, ni dar tus datos, ni pagar nada. Es una herramienta simple y anónima.

Es compatible con todos los teléfonos y ordenadores ya que usa tecnologías estándar. Tu dispositivo se encargará de ir actualizando las predicciones de manera regular.

Creé esta utilidad para mi uso personal al descubrir que no existía nada similar, y tras usarla unas semanas pensé que podía ser útil dar acceso a los demás.

Los datos están sacados de AEMET por lo que sólo funciona en el territorio español.

Tienes el enlace aquí: el tiempo en tu calendario

Tags: software, projects, spanish

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

September 10, 2019

Carlos Fenollosa (carlesfe)

The iPhone 11 & co. September 10, 2019 09:10 PM

This year's phone keynote has delivered, according to Apple, all-new products from the top down

Quite boring hardware unfortunately, as was expected.

  • Better cameras, though for use cases I'm not sure are very useful
  • Better battery life thanks to the A13 chip
  • Marginally better screen on the Pro phone
  • Always-on screen on the Watch, which is nice
  • Simple update on the entry-level iPad

The landing page for each phone is 60% camera features and 40% other features. Not saying that is wrong, on the contrary, the marketing team is doing their job as in my experience most people use their phones as an Instagram device.

Where I think Apple nailed it is with the Watch. They are really, really good at the health and fitness message, and the product itself is fantastic.

However, I will criticise them for two things.

First, the fact that they are not even advertising the full price for the phone, but rather an installment plan first, then a discounted price with the trade of an old phone, and only when you say "no" to these options you get the actual price. Let me reiterate that. When you visit Apple.com the price you see for the phones is not the actual retail price.

They are aware that their hardware is not attractive at those price points, but at the same time they can't lower them because of positioning. Well; to be precise, the iPhone 11 is actually sliiightly cheaper than last year's but, in my opinion, not attractive enough to upgrade. And let's not even mention EU prices. On top of the 21% sales tax —nothing to do there— we are eating up a 1:1 USD:EUR ratio which is bullshit.

Second, Apple is advertising a "Pro" phone that can shoot incredible 4K movies, but stuffing it with only 64 GB of storage. The consumer experience is terrible when you are out of disk space.

My phone memory is full and every time I take a picture it is immediately uploaded to iCloud and deleted from the phone. If I want to show it to somebody later the same day, I have to wait for it to load from the network. My UX is that I have no pictures or videos stored locally, not even for pictures I took 15 minutes ago. That is definitely not a feature you want on a super advanced camera-phone.

The phone market is too mature

Regarding innovation, what can Apple really do? I honestly do not have an answer. The majority of the population is not renewing their phones on a yearly cycle, not even a two year cycle. I have an SE only because my 5S broke. I loved my 5S and there is no feature in current phones that would make me upgrade.

I commute every day and see what "normal people" (excuse me) do on their phones. It is 40% scrolling through Instagram, 30% Whatsapp, 20% watching shows, 5% taking pictures, 5% playing games.

If you want to read the best take on the keynote, read Ben Thompson's The iPhone and Apple’s Services Strategy.

The phone market and phone technology have crossed the chasm long ago and they're on diminishing returns. I stick by my reaction of last year's keynote:

  • Apple should seriously consider 2-year iPhone cycles.
  • People who want smaller phones, regardless of price, may move to Android, myself included, so an updated iPhone SE is strategic for Apple.
  • Hardware improvements are going to be mostly incremental from now on. Therefore...
  • Apple should focus on software, which they are doing very well, and keep coming up with really crazy innovative hardware, which they appeared to be doing but rumors say they scraped at the last minute like the U1 chip.

Apple is a company full of smart people that can reinvent boring products like beige PCs, Nokia phones, and even headphones and watches. I am hopeful for the next wave of hardware, whatever it is. AR glasses? Car stuff? TVs? We will see.

Personally, I am indifferent at this keynote. Since my main need is a laptop, I'm still waiting for the new wave of macbooks to renew my 2013 MBA. I simply refuse to buy any laptop from Apple's current lineup. The rumors are very promising, so let's check what they can come up with!

Tags: apple

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

September 09, 2019

eta (eta)

Thoughts on improving chat systems September 09, 2019 11:00 PM

I’m quite interested in instant messaging technology! I’ve been an avid user of internet relay chat (IRC) – probably the oldest chat protocol in existence – for years, I went through a brief period of using Matrix.org for everything, and I’ve of course sampled most of the smorgasbord of proprietary chat solutions, including WhatsApp, Discord, Facebook Messenger (eww!), and probably some others I’m forgetting. This blog post is essentially a data dump of opinions and thoughts on the various open chat standards and protocols that are out there (and if you get all the way to the very end, you get to hear me propose yet another one of them…!).

The dream of the universal standard

There have been countless blog posts on how it sucks that we have lots of things to choose from, and there isn’t any universal standard yet for everything. Of course, there’s even a relevant xkcd:

xkcd #1810: Chat Systems

Writing a universal standard is hard, and indeed there’s yet another relevant xkcd about that one as well1. Part of the problem is that, well, instant messaging provides various ways for people to express themselves, and so different people are likely to prefer different platforms – some people actually like IRC for what it is, a basic, purely textual form of communication, whereas others want features like avatars, images, and other modern niceties. Some people want encryption or privacy and would prefer that to take centre stage, whereas others want to use custom emoji, have native GIF support, and stuff like that. In some ways, the reason we have so many chat protocols and services available is due to this difference in preference and target market:

  • WhatsApp is, or was originally, “simple SMS equivalent that doesn’t cost as much and that lets you have group chats”
  • Discord is “chat for gamers”
  • Slack is “chat for Big Important Corporate Uses”
  • IRC is “basic text-mode chat for developers and people like SirCmpwn

While it would conceivably be possible to unify all of these competing applications under one common standard, doing so would seem a bit awkward. WhatsApp, for example, is the only one of the above platforms that supports end-to-end encryption, and it doesn’t store any chat history on the server either; conversely, Discord and Slack treat your chat history more like an append-only database, with the whole thing being easily searchable (indeed, Slack try and make money by charging you access to this database).

Bridging

The eventual fate of an open messaging standard is not generally one where everyone on the planet drops whatever tools they’re using and starts to use it. After all, standards are paper, and paper isn’t worth much on its own; you need to be able to talk to the friends you already have! To remedy this, most protocols worth their salt have an ecosystem of various bridging tools to go with them that let you do just that; for example, my own sms-irc project lets you talk to your WhatsApp contacts using your IRC client.

Of course, not many chat services open up their protocols and make this easy to do – the last mainstream one to really do so was Google Talk, which was based on the open XMPP standard, and even supported federation (i.e. you could spin up your own XMPP server and talk to people using Google Talk, which was nice). Services like WhatsApp are based on standards like XMPP, but they’ve heavily locked everything down so you can’t just go and connect with your XMPP client. Discord has an API, but using it on user accounts (instead of their officially sanctioned bot account option) is forbidden; I could extend this list on and on. The main point is, of course, it’s not beneficial for 99% of these chat companies to allow people to connect their own custom software; their business model usually heavily relies on you using their version, so they can either serve you with ads, perform some tracking, etc., or so they can pull a Slack and prevent you logging your chat history elsewhere. In fact, Slack even shut down their IRC gateway a while ago – the reason they gave was the above one of “we have custom features that don’t come through well via gateway”, but everyone secretly suspected it was to increase vendor lock-in.

Nevertheless, though, bridging tools do exist - bitlbee is one of the most popular ones for IRC, XMPP has Spectrum 2, and Matrix has their collection of bridges. As an example, I personally use IRC, but most of the people I talk to use other chat platforms; IRC is just a way to bring all of my communication together in one place. Indeed, this is one of the main selling points for open protocols and platforms: “in the future, people will realise our standard is better, but for now, you can still talk to your old friends”.

Competitor analysis

In terms of protocols (not services) that I could use to set up a personal messaging system, there are really only three competitors: IRC, XMPP, and Matrix. Of these three, they each have their advantages and disadvantages:

IRC

  • …is rock-solid reliable, perhaps to a fault
    • If you send a message to someone, the message is going to arrive within a few seconds of you sending it, maximum.
    • If it doesn’t, you will get some obvious error as to why.
  • …doesn’t handle mobile connections / Multiple Points of Presence (MPoP) well
    • People nowadays expect to be able to use their mobile device. IRC is still really tethered to the desktop.
    • Mobile IRC clients exist, but require usage of a bouncer or something to multiplex your desktop and mobile connections together, and maintain the connection when your phone goes offline.
    • On Android, the mobile story is a bit dire, if you exclude things like Weechat Android and Quasseldroid that are frontends for some bouncer or other desktop client.
  • …is incredibly simplistic at the protocol level
    • You can chat on IRC using just telnet. The protocol really isn’t very involved, and is quite easy to write parsers for.
    • IRC doesn’t use JSON, XML or any other sort of formal data format.
  • …is incredibly simplistic in terms of functionality
    • There’s no chat history, sending messages to people who are offline, or anything modern, really.
    • No typing or delivery notifications either.

XMPP

  • …uses XML
    • XML is crap2 and hard to write parsers for.
    • Older programming languages like Java and friends handle XML fine, but it doesn’t necessarily translate that well in a world where everything is JavaScript or Rust.
    • Even if you think XML is okay, it’s still arguably too complex for a messaging protocol.
  • …suffers from extreme fragmentation, both protocol-wise and app-wise
    • XMPP comes as a base standard that then gets extended by a bunch of XEPs (extensions to the spec). This is dangerous, since quite a few clients don’t bother implementing all the useful XEPs you actually need to make XMPP worthwhile.
    • There seems to be quite a large disparity between best-in-class XMPP clients like Conversations and the rest of the ecosystem.
  • …has built-in support for Multiple Points of Presence (MPoP)
    • At least they thought this one through; you can chat on multiple devices at once, and that’s natively supported by the protocol.
  • …doesn’t really seemed to be used (as an open protocol, that is) by many
    • Quality iOS XMPP clients, for example, don’t really exist. Sure, there are a few, but they’re quite hacky.
    • Weirdly popular in Germany, though (?)
  • …has some support for end-to-end encryption
    • See this website, which tracks implementation in multiple XMPP clients.
  • …subjectively never really worked for me
    • I did give XMPP a try once, and found it quite shoddy. Group chats would fail to work in mysterious ways or not sync across devices, with no real reason as to why; chat history didn’t always work as advertised; and I generally got a bad impression of the whole protocol and its ecosystem.
    • The fact that multi-user chats were an extension instead of being built in to the protocol is perhaps a cause of some of the jankiness.
    • I acknowledge that I’m probably not doing XMPP justice, but there you go; I didn’t see anything in it when I tried it!

Matrix

Oh, boy.

  • …uses JSON and HTTP
    • This is better than XMPP, since these standards are actually used nowadays, and are relatively lightweight; pretty much everything can speak JSON.
    • Some people do complain about HTTP being still a bit too heavyweight, in contrast to something like XMPP where you don’t need to pay the price of making a new HTTP request every time you want to send something.
    • However, Matrix’s long polling model for fetching new messages is something I actually think is quite clever; it’s clearly been designed thoughtfully to allow clients to perform well with patchy connections, which is a pain point in older protocols like IRC.
  • …has a growing amount of people, organizations and development effort rallying around it
  • …has a healthy bridging ecosystem
    • Taking a look at their bridging page shows as much; this is one thing they are pretty good at.
    • The specification even includes a separate part for Application Services (ASes), which are specifically designed to do things like bridging.
  • …has a somewhat problematic, slow reference implementation
    • Essentially, the problem is that the protocol requires a “state resolution” algorithm to verify permissions in a chatroom. This is the root of all sorts of performance issues, and also has been the source of security issues in the past (q.v.)
    • There are long-standing related issues like #1760 that still aren’t fixed and have lots of users complaining.
      • (although there’s now a somewhat hacky workaround for this that involves sending dummy events into the room)
    • I mean, just browsing issues flagged major is pretty enlightening, and hints at some real problems with the way the reference implementation is built.
    • In my personal experience, I ran a Matrix homeserver for the best part of the year until I got fed up with it; I had to enable zswap / zram on the servers I was running it on (since it had a habit of eating all the RAM available), and had to contend with 100% CPU usage spikes every time I logged on.
  • …is questionably reliable as a messaging platform
    • When I used to use it, I’d frequently encounter problems with messages just not being delivered, or push notifications breaking.
    • I once had an incident where a friend of mine, trying to send messages to me on my self-hosted server, was sending things for about a week without me getting any part of it, until I figured out something was amiss and rebooted it.
    • I personally value the messages actually being delivered above all else, and Matrix at least in my experience is pretty bad at this…
  • …has a questionable security story
    • The official matrix.org server has been hacked in the past, although this is nothing to do with the protocol.
    • The original version of the protocol had a bug known as “state resets”, where room state would be reset back to some earlier version. This caused all sorts of fun security issues – one user was even able to wrest control of the main Matrix HQ chatroom back in the day and ban the project developers from it – until they fixed it.
      • This also resulted in the hilariously named “Hotel California bug”, where people could leave a chatroom, but would end up being forcibly rejoined whenever a state reset occurred.
      • This was eventually solved with a new implementation of the “state resolution” algorithm – which required upgrading rooms to support the new version, essentially creating a new chatroom and trying to get everyone to join it.
    • Although they’ve cleaned a lot of things up for their Synapse 1.0 release, and it’s improving, looking at their list of open and closed security bugs isn’t exactly reassuring.
  • …has a questionable data model for chat purposes
    • Really, the root cause of a lot of Matrix’s problems seems to be that each chatroom is actually a distributed database, stored in the form of a directed acyclic graph (DAG) – that anyone can append to, given it’s all publicly available over an open federation.
    • Trying to do things in a way where the chatroom is completely independent of any of the servers it’s hosted on is cool, but also quite unwieldy.
    • Arguably, a more lightweight protocol based on message passing would be better suited to chat – but hey, it’s a free world, and different implementations can exist for a reason.

A gap in the market

This is getting quite long (especially with the rather dense bullet point list presented above!), so I’ll wrap things up: essentially, I think there’s a gap in the market somewhere between IRC and Matrix for a new standard, or at least an attempt at one. Matrix does a lot of things right, but is a bit too ambitious; they’re trying to tackle the problem of making a distributed (in such a way that it doesn’t depend on any single server), end-to-end-encrypted, fault-tolerant database, which is arguably a cool thing! However, I posit that something more like IRC or XMPP is better suited to chat, which seems like it could be implemented in a far more lightweight manner while retaining quite a lot of the functionality – like XMPP does, but with less XML and hacky standards.

Bonus round: Mastodon

The Mastodon decentralized social network is a pretty good example of what I mean, actually; it’s a relatively simple, open protocol (ActivityPub) that doesn’t try to implement the world (users are still tied to a single server, for example, and data is lost if that server dies) - but its simplicity has allowed for the development of other so-called “fediverse” servers, like Pleroma, and even ultra-minimalist ones like honk3, without too much hassle.

It’s simple, reliable, and still federated; it works well enough for non-geeks to use it, because the Mastodon UI essentially imitates Twitter. IRC is simple, reliable, but not federated and hard for non-geeks to use. XMPP is, well, XMPP. Matrix is not simple and not reliable, but it is federated and reasonably easy to use.

So, I suppose, why don’t we just build a Mastodon, but for chat?

(Stay tuned; this isn’t the only thing I’m going to post about this…!)


  1. Which I’m not going to include inline, otherwise this whole blog post is gonna be a string of xkcd comics. 

  2. “XML is crap. Really. There are no excuses. XML is nasty to parse for humans, and it’s a disaster to parse even for computers. There’s just no reason for that horrible crap to exist.” ~ Linus Torvalds 

  3. I use this! Check out honk.theta.eu.org (you can follow @eta@honk.theta.eu.org from your Mastodon if you like!). 

Gustaf Erikson (gerikson)

August September 09, 2019 08:00 PM

September 06, 2019

Mark Fischer (flyingfisch)

Sharpie Stainless Steel Pen Refill Hack September 06, 2019 04:00 AM

Backstory

If you have owned a Sharpie Stainless Steel Pen, you will know two things. The first thing you’ll know is that it is the most perfect pen ever invented and no other pen will ever compare. The second thing you’ll know, or find out, is that both the pen and the refills have been discontinued by Sharpie.

WHY SHARPIE?! WHY WOULD YOU DISCONTINUE SUCH AN EXCELLENT PRODUCT???? WHY DID YOU DO THIS TO US?!

Ahem. Anyway… after realizing that refills are now $812.91, I decided to try to figure out if I could find an alternative way to refill them. Below are instructions for creating what I call the Sharpie Stainless Steel Pen Frankenfill Mark I.

Stainless Steel Pen Frankenfill Mark I

Materials

  • Sharpie Stainless Steel Pen
  • Cross Selectip Porous Point Refill (Model number 8443)

Step 1: Remove the original Sharpie Pen refill

Remove the cap from your Sharpie pen. Grasp the stainless barrel with one hand and rubber grip in the other and twist to remove the original Sharpie refill.

Step 2: Disassemble the original Sharpie Pen refill

Grasp the rear cap of the Sharpie refill with a pair of pliers. Pull firmly to remove, it’s press fit and shouldn’t take much effort. Inside there is a plastic covered piece of foam with the nib sticking out of it, remove that. Also remove the metal nib by pressing the point down firmly on a hard surface.

Step 3: Insert the Cross Selectip refill into the original Sharpie

Insert the Cross Selectip refill into the carcass of the Sharpie refill. Press firmly until the nib is exposed. A little bit of the back of the Cross refill will be exposed, this is fine. The rear cap on the Sharpie refill can be discarded at this point.

Step 4: Reassemble the pen

At this point, all that’s left is to reassemble the pen.

Review

I’m satisfied with this as a first try and it does give me hope that my Sharpie Stainless Pens have a future. However, although the Cross refill is very pleasant to write with, it has a couple key differences from the original Sharpie pen. The first difference is that the ink is much darker and less fine. The second difference is that it bleeds more. Below is an image comparing the two from the front and back.

Future Plans

I’m currently searching for a felt tip refill that matches the properties of the original Sharpie pen a little more closely and will hopefully be posting an update in the near future. However, I am happy to say that I have found a way to use standard pen refills in the Sharpie Stainless Steel Grip Pen and I’m hoping to get several more years of use out of my current Sharpie pens. Maybe some day Sharpie will bring the pen back to production, but until then I hope this guide helps others wondering how to continue using their Sharpies.

September 04, 2019

Gokberk Yaltirakli (gkbrk)

Tampermonkey is not Open Source September 04, 2019 11:00 PM

This post is meant to be a short remark about something I noticed today. It is about Tampermonkey, a browser extension for managing User scripts.

Tampermonkey, like Greasemonkey, lets you create and run User scripts. These are small JavaScript snippets that can execute on any URL you want. You can consider them mini-extensions.

Tampermonkey came out as a Greasemonkey alternative for Chrome. Since writing User scripts is a “developer” thing to do, and since pretty much all software development tools are open source, so was Tampermonkey. This quickly allowed them to gather a user base and get contributions from the community.

At some point (around version 2.9), they switched their project to a proprietary license. They stopped taking contributions of course, and the old version is still on GitHub, so there is no license violation. This post isn’t meant to be a “grab your pitchforks there’s proprietary software” post against Tampermonkey either. I just know that some people don’t want to run non-free software, and their browser might’ve auto-updated to a non-free version.

This was certainly the case for me. I remembered Tampermonkey as an open-source user script manager and started using it, it took me a while to realize the license situation. While this information was available on the Firefox add-ons page, I think it should be more prominent in the install process.

After some time, and with developments like major browsers all implementing the same extension API, Tampermonkey took its place on most Add-On stores. I believe for Firefox at least this was after the license change, so people on FF shouldn’t have had unexpected non-free software.

September 02, 2019

Pete Corey (petecorey)

Animating a Canvas with Phoenix LiveView September 02, 2019 12:00 AM

Phoenix LiveView recently released a new feature called “hooks” that introduces Javascript interoperability into the LiveView lifecycle. Put simply, we can now run arbitrary Javascript every time a DOM node is changed by LiveView! LiveView hooks are a complete game changer, and open the doors to a whole new world of applications that can be built with this amazing technology.

As a proof of concept, let’s use LiveView hooks to animate an HTML5 canvas in real time using data provided by the server!

Getting Set Up

To keep this article short(er), we’ll skip the rigmarole of configuring your application to use LiveView. If you need help with this step, I highly recommend you check out Sophie DeBenedetto’s thorough walkthrough. Be sure to cross reference with the official documentation, as things are moving quickly in the LiveView world.

Moving forward, let’s assume that you have a bare-bones LiveView component attached to a route that looks something like this:


defmodule LiveCanvasWeb.PageLive do
  use Phoenix.LiveView
  
  def render(assigns) do
    ~L"""
    <canvas>
      Canvas is not supported!
    </canvas>
    """
  end
  
  def mount(_session, socket) do
    {:ok, socket}
  end
end

We’ll also assume that your assets/js/app.js file is creating a LiveView connection:


import LiveSocket from "phoenix_live_view";

let liveSocket = new LiveSocket("/live");

liveSocket.connect();

Now that we’re on the same page, let’s get started!

Generating Data to Animate

Before we start animating on the client, we should have some data to animate. We’ll start by storing a numeric value called i in our LiveView process’ assigns:


def mount(_session, socket) do
  {:ok, assign(socket, :i, 0)}
end

Next, we’ll increase i by instructing our LiveView process to send an :update message to itself after a delay of 16 milliseconds:


def mount(_session, socket) do
  Process.send_after(self(), :update, 16)
  {:ok, assign(socket, :i, 0)}
end

When we handle the :udpate message in our process, we’ll schedule another recursive call to :update and increment the value of i in our socket’s assigns:


def handle_info(:update, %{assigns: %{i: i}} = socket) do
  Process.send_after(self(), :update, 16)
  {:noreply, assign(socket, :i, i + 0.05)}
end

Our LiveView process now has an i value that’s slowly increasing by 0.05 approximately sixty times per second.

Now that we have some data to animate, let’s add a canvas to our LiveView’s template to hold our animation:


def render(assigns) do
  ~L"""
  <canvas data-i="<%= @i %>">
    Canvas is not supported!
  </canvas>
  """
end

Notice that we’re associating the value of i with our canvas by assigning it to a data attribute on the DOM element. Every time i changes in our process’ state, LiveView will update our canvas and set the value of data-i to the new value of i.

This is great, but to render an animation in our canvas, we need some way of executing client-side Javascript every time our canvas updates. Thankfully, LiveView’s new hook functionality lets us do exactly that!

Hooking Into LiveView

LiveView hooks lets us execute Javascript at various points in a DOM node’s lifecycle, such as when the node is first mounted, when it’s updated by LiveView, when it’s destroyed and removed from the DOM, and when it becomes disconnected or reconnected to our Phoenix server.

To hook into LiveView’s client-side lifecycle, we need to create a set of hooks and pass them into our LiveSocket constructor. Let’s create a hook that initializes our canvas’ rendering context when the element mounts, and renders a static circle every time the element updates:


let hooks = {
  canvas: {
    mounted() {
      let canvas = this.el;
      let context = canvas.getContext("2d");
      
      Object.assign(this, { canvas, context });
    },
    updated() {
      let { canvas, context } = this;
      
      let halfHeight = canvas.height / 2;
      let halfWidth = canvas.width / 2;
      let smallerHalf = Math.min(halfHeight, halfWidth);
      
      context.clearRect(0, 0, canvas.width, canvas.height);
      context.fillStyle = "rgba(128, 0, 255, 1)";
      context.beginPath();
      context.arc(
        halfWidth,
        halfHeight,
        smallerHalf / 16,
        0,
        2 * Math.PI
      );
      context.fill();
    }
  }
};

let liveSocket = new LiveSocket("/live", { hooks });

liveSocket.connect();

Notice that we’re storing a reference to our canvas and our newly created rendering context on this. When LiveView calls our lifecycle callbacks, this points to an instance of a ViewHook class. A ViewHook instance holds references to our provided lifecycle methods, a reference to the current DOM node in el, and various other pieces of data related to the current set of hooks. As long as we’re careful and we don’t overwrite these fields, we’re safe to store our own data in this.

Next, we need to instruct LiveView to attach this new set of canvas hooks to our canvas DOM element. We can do that with the phx-hook attribute:


<canvas
  data-i="<%= @i %>"
  phx-hook="canvas"
>
  Canvas is not supported!
</canvas>

When our page reloads, we should see our circle rendered gloriously in the center of our canvas.

Resizing the Canvas

On some displays, our glorious circle may appear to be fuzzy or distorted. This can be fixed by scaling our canvas to match the pixel density of our display. While we’re at it, we might want to resize our canvas to fill the entire available window space.

We can accomplish both of these in our mounted callback:


mounted() {
  let canvas = this.el;
  let context = canvas.getContext("2d");
  let ratio = getPixelRatio(context);
  
  resize(canvas, ratio);
  
  Object.assign(this, { canvas, context });
}

Where getPixelRatio is a helper function that determines the ratio of physical pixels in the current device’s screen to “CSS pixels” which are used within the rendering context of our canvas:


const getPixelRatio = context => {
  var backingStore =
    context.backingStorePixelRatio ||
    context.webkitBackingStorePixelRatio ||
    context.mozBackingStorePixelRatio ||
    context.msBackingStorePixelRatio ||
    context.oBackingStorePixelRatio ||
    context.backingStorePixelRatio ||
    1;
  
  return (window.devicePixelRatio || 1) / backingStore;
};

And resize is a helper function that modifies the canvas’ width and height attributes in order to resize our canvas to fit the current window, while fixing any pixel density issues we may be experiencing:


const resize = (canvas, ratio) => {
  canvas.width = window.innerWidth * ratio;
  canvas.height = window.innerHeight * ratio;
  canvas.style.width = `${window.innerWidth}px`;
  canvas.style.height = `${window.innerHeight}px`;
};

Unfortunately, our canvas doesn’t seem to be able to hold onto these changes. Subsequent calls to our updated callback seem to lose our resize changes, and the canvas reverts back to its original, blurry self. This is because when LiveView updates our canvas DOM node, it resets our width and height attributes. Not only does this revert our pixel density fix, it also forcefully clears the canvas’ rendering context.

LiveView has a quick fix for getting around this problem. By setting phx-update to "ignore" on our canvas element, we can instruct LiveView to leave our canvas element alone after its initial mount.


<canvas
  data-i="<%= @i %>"
  phx-hook="canvas" 
  phx-update="ignore"
>
  Canvas is not supported!
</canvas>

Now our circle should be rendered crisply in the center of our screen.

Animating Our Circle

We didn’t go all this way to render a static circle in our canvas. Let’s tie everything together and animate our circle based on the ever-changing values of i provided by the server!

The first thing we’ll need to do is update our updated callback to grab the current value of the data-i attribute:


let i = JSON.parse(canvas.dataset.i);

The value of canvas.dataset.i will reflect the contents of our data-i attribute. All data attributes are stored as strings, so a call to JSON.parse will convert a value of "0.05" to its numeric counterpart.

Next, we can update our rendering code to move our circle based on the value of i:


context.arc(
  halfWidth + (Math.cos(i) * smallerHalf) / 2,
  halfHeight + (Math.sin(i) * smallerHalf) / 2,
  smallerHalf / 16,
  0,
  2 * Math.PI
);

That’s it! With those two changes, our circle will rotate around the center of our canvas based entirely on real-time data provided by our server!

Requesting Animation Frames

Our solution works, but by forcing re-renders on the browser, we’re being bad net citizens. Our client may be forcing re-renders when its tab is out of focus, or it may be re-rendering more than sixty times per second, wasting CPU cycles.

Instead of telling the browser to re-render our canvas on every LiveView update, we should invert our control over rendering and request an animation frame from the browser on every update.

The process for this is straight forward. In our updated callback, we’ll wrap our rendering code in a lambda passed into requestAnimationFrame. We’ll save the resulting request reference to this.animationFrameRequest:


this.animationFrameRequest = requestAnimationFrame(() => {
  context.clearRect(0, 0, canvas.width, canvas.height);
  context.beginPath();
  context.arc(
    halfWidth + (Math.cos(i) * smallerHalf) / 2,
    halfHeight + (Math.sin(i) * smallerHalf) / 2,
    smallerHalf / 16,
    0,
    2 * Math.PI
  );
  context.fill();
});

It’s conceivable that our LiveView component may update multiple times before our browser is ready to re-render our canvas. In those situations, we’ll need to cancel any previously requested animation frames, and re-request a new frame. We can do this by placing a guard just above our call to requestAnimationFrame:


if (this.animationFrameRequest) {
  cancelAnimationFrame(this.animationFrameRequest);
}

With those two changes, our LiveView hooks will now politely request animation frames from the browser, resulting in a smoother experience for everyone involved.

Taking it Further

Using a canvas to animate a numeric value updated in real-time by a LiveView process running on the server demonstrates the huge potential power of LiveView hooks, but it’s not much to look at.

We can take things further by generating and animating a much larger set of data on the server. Check out this example project that simulates over two hundred simple particles, and renders them on the client at approximately sixty frames per second:

Is it a good idea to take this approach if your goal is to animate a bunch of particles on the client? Probably not. Is it amazing that LiveView gives us the tools to do this? Absolutely, yes! Be sure to check out the entire source for this example on Github!

Hooks have opened the doors to a world of new possibilities for LiveView-based applications. I hope this demonstration has given you a taste of those possibilities, and I hope you’re as eager as I am to explore what we can do with LiveView moving forward.

Update: 9/30/2019

The technique of using both phx-hook and phx-update="ignore" on a single component no longer works as of phoenix_live_view version 0.2.0. The "ignore" update rule causes our hook’s updated callback to not be called with updates.

Joxy pointed this issue out to me, and helped me come up with a workaround. The solution we landed on is to wrap our canvas component in another DOM element, like a div. We leave our phx-update="ignore" on our canvas to preserve our computed width and height attributes, but move our phx-hook and data attributes to the wrapping div:


<div
  phx-hook="canvas"
  data-particles="<%= Jason.encode!(@particles) %>"
>
  <canvas phx-update="ignore">
    Canvas is not supported!
  </canvas>
</div>

In the mounted callback of our canvas hook, we need to look to the first child of our div to find our canvas element:


mounted() {
  let canvas = this.el.firstElementChild;
  ...
}

Finally, we need to pass a reference to a Phoenix Socket directly into our LiveSocket constructor to be compatible with our new version of phoenix_live_view:


import { Socket } from "phoenix";
let liveSocket = new LiveSocket("/live", Socket, { hooks });

And that’s all there is to it! Our LiveView-powered confetti generator is back up and running with the addition of a small layer of markup. For more information on this update, be sure to check out this issue I filed to try to get clarity on the situation. And I’d like to give a huge thanks to Joxy for doing all the hard work in putting this fix together!

August 31, 2019

Jeff Carpenter (jeffcarp)

Things I Learned as a First-Time Intern Host August 31, 2019 06:13 PM

I hosted an intern for the first time this summer. It was my first time being somebody’s manager and it became a huge learning experience for me as well as a really fun time. My intern worked on adding many features to velocity-tracking charts, rewriting both of our ML models in TensorFlow 2.0, and a few other projects. Here are the biggest areas where I struggled as a host and the important lessons I took away from those experiences.

Frederic Cambus (fcambus)

My OpenBSD commits August 31, 2019 05:45 PM

Today marks my three year anniversary as an OpenBSD developer. I got my commit bit on August 31th 2016 during the g2k16 hackathon in Cambridge, UK.

A few months ago, I came across a Perl one-liner script to produce commit time distribution ASCII graphs from a Git repository, and I finally have a good pretext to run it :-)

As of this day, I have done 749 commits to OpenBSD, in the following repositories: src (127), ports(596), www (24), and xenocara (2).

Commits in the src repository:

00 -    0
01 -    0
02 -    0
03 -    0
04 -    0
05 -    0
06 -    1 ***
07 -    4 **************
08 -    8 ****************************
09 -    9 ********************************
10 -   13 **********************************************
11 -    9 ********************************
12 -   10 ***********************************
13 -   11 ***************************************
14 -   13 **********************************************
15 -    4 **************
16 -    5 *****************
17 -    6 *********************
18 -    4 **************
19 -    9 ********************************
20 -   14 **************************************************
21 -    4 **************
22 -    3 **********
23 -    0

Commits in the ports repository:

00 -    1
01 -    0
02 -    0
03 -    0
04 -    0
05 -    2 *
06 -   14 **********
07 -   32 ***********************
08 -   34 *************************
09 -   67 **************************************************
10 -   46 **********************************
11 -   53 ***************************************
12 -   40 *****************************
13 -   38 ****************************
14 -   34 *************************
15 -   34 *************************
16 -   35 **************************
17 -   20 **************
18 -   15 ***********
19 -   24 *****************
20 -   34 *************************
21 -   43 ********************************
22 -   19 **************
23 -   11 ********

Commits in the www repository:

00 -    0
01 -    0
02 -    0
03 -    0
04 -    0
05 -    0
06 -    0
07 -    1 ************
08 -    0
09 -    3 *************************************
10 -    2 *************************
11 -    4 **************************************************
12 -    0
13 -    2 *************************
14 -    1 ************
15 -    1 ************
16 -    1 ************
17 -    1 ************
18 -    1 ************
19 -    3 *************************************
20 -    3 *************************************
21 -    1 ************
22 -    0
23 -    0

Commits in the xenocara repository:

00 -    0
01 -    0
02 -    0
03 -    0
04 -    0
05 -    0
06 -    0
07 -    0
08 -    0
09 -    0
10 -    0
11 -    0
12 -    0
13 -    0
14 -    1 **************************************************
15 -    0
16 -    0
17 -    0
18 -    0
19 -    0
20 -    0
21 -    1 **************************************************
22 -    0
23 -    0

August 30, 2019

Andreas Zwinkau (qznc)

Mindstorms August 30, 2019 12:00 AM

A book about a revolution of education which sadly never happened.

Read full article!

Robin Schroer (sulami)

Why I like Clojure August 30, 2019 12:00 AM

This is somewhat of a response to Uncle Bob’s post of similar nature, which I would say has gotten a mixed to positive reception. I had planned a similar post a week or two before the release of his, but archived the idea upon reading his post. But after having read it over a couple of times I have now decided that I still have something meaningful to write. What follows are the purely subjective reasons for which I enjoy using Clojure. Some have criticised Bob for being very absolute and not giving up any screen estate for more nuanced viewpoints, something I will try to avoid.

Lisp

There is not much to say about this that has not already been said. The homoiconicity, meaning code can be represented as a data structure inside the same language, extensibility through macros which can modify both the evaluation order as well as the very syntax to the point where it looks more like LaTeX than Lisp.Racket is actually a great example of the flexibility of Lisp. A language designed to build other languages in it, call it a meta-language. Even package import can be wrapped up in a “language” trivially, meaning that you can essentially write a tiny DSL for every project. Not saying that is necessarily a good idea, but you can.

This also means that you are not stuck with a paradigm. While OO seems to be out, and FP the new hotness, Lisp can do them all, and historically often did before anyone else. Bob mentions dynamic typing in his signature retorts to (I am guessing) fictional counter-arguments, and he is right to mention clojure.spec, a library for gradual typing (omitting schema, an alternative). Racket has a fully typed variant, there is something that is basically Haskell in a Lisp-bun, and let us not forget that there is actually Typed Clojure, with static type checking and all.There are some issues with it, namely that the coverage is not all that great, but it exists and works, meaning if you really need static types, you can get them.

Being able to generate code without being stuck on a “dumb” level by generating strings and passing them into eval like for example in Python allows for sane hyper-dynamic programming, where the program adapts itself to the conditions. Being able to read and write code in a safe manner enables extremely powerful tooling in the Lisp world. Linters are very smart, because reading code into a data structure is trivial, usually a one-liner, and there are many more tools to automatically rewrite and refactor code than for other languages.

Now I do not want to discount the Lisp Curse, it is a real thing, and one of the reasons that while Lisp has stuck around for over half a century, it has not made it into the mainstream. The other main factor probably being the performance problems due to the gap between software and hardware architecture.There were Lisp machines, which had hardware tailored towards running Lisp, but they never took off either.

But with the advent of the internet, ecosystems like GitHub, and hardware that is fast enough that we consider running basically full web browsers for half of our applicationsLooking at you, Slack.

, I think that these issues have become surmountable.

I do not think I need to mention how incredibly useful the REPL and hot-loading code into a running system are?

Hosted

Clojure is explicitly designed as a hosted language, which I think was a very good move. If you are writing a new language today, it might be better than the established ones, but the cost of leaving an existing ecosystem of libraries and Stack Overflow answers just because the new language is 5% nicer is not a trade off many people will want to make. Clojure being hosted and having excellent interoperability with its host platform means it can benefit from existing ecosystem, let alone platform implementations.Do you really want to implement your runtime for FreeBSD on a smart toaster oven? Raspberry Pis are non x86, BSD is not Linux, and who knows what is up with Windows. This matrix is growing quickly.

While the primary platform is the JVM, superbly uncool but stable and relatively performant, there is a CLR (.NET) version which is “almost even on features” thanks to Davit Miller, as well as a very mature JavaScript version in the shape of ClojureScript. The JVM (and to some extent the CLR) have excellent support by big software vendors, if you buy some kind of software with an API, chances are there is a Java SDK which you can use easily from your Clojure code. The JavaScript ecosystem is the largest in numbersIn part due to left-pad-like five-line-packages, but still.

, and includes Electron and React-Native, both of which can be used with some, but not unreasonable, effort from ClojureScript code. One of the newest additions has been GraalVM, which while not 100% feature-complete yet, already allows compilation to native static binaries of many Clojure programsZprint is one of those CLI tools that takes advantage of the reduced startup time.

, running without the JVM at all, and doing away with the dreaded multi-second startup time.I am planning to write a piece about GraalVM some time later this year.

The platform split could have been one of the big, curse-like problems for Clojure, but there is Clojure Common, used by many popular libraries, which allows you to write platform-independent code by using conditional branching for all platform-specific code.

Community

Despite all the positive points I mentioned, Clojure is still a niche language, and in some way that is good as well. Sure, finding jobs is harderLarge companies like Walmart and CircleCI (my employer) are Clojure shops, so it is far less obscure than one might think.

, but not impossible. Clojure developers, like for example Haskell or Rust ones, tend to be more experienced, as it is not a typical first language, and requires a certain interest in the craft. Many Clojure developers have written widely used tools and libraries, not just in Clojure, but also for example for Emacs, which is understandably quite popular with Clojurists.

Rich Hickey himself, the BDFL of Clojure, is someone with decades of industry experience and a desire to get it right. I think he is doing a pretty good job. While there are some small inconsistencies in places, the bulk of the language, and all the important parts are very well thought out.We can also see right now how clojure.spec is being adapted after community feedback to the first alpha version, which has been available for about 1½ years.

Clojure is a very stable language, which means that smaller problems will stick around for a while, but also means you can trust that your code will not break every time you update your dependencies.

In the end, it comes down to enjoyment. I enjoy working with Clojure. I feel like there is a lot to learn, and the language is inviting me to explore beyond the current possibilities of software development. I feel good about the elegant and concise solutionsconcise ≠ obtuse

I can come up with. It has changed the way I think, in a good way.

August 26, 2019

Pete Corey (petecorey)

Prime Parallelograms August 26, 2019 12:00 AM

A few weeks ago I wrote about an interesting graph of numbers recently investigated by the Numberphile crew. We used it as an opportunity to journey into the world of agendas and gerunds by implementing the graph using the J programming language.

The second half of that same video outlines another interesting number series which has a similarly interesting implementing in J. Let’s try our hand at plotting it.


The basic idea behind calculating the series of numbers in question is to take any positive prime, represent it in base two, reverse the resulting sequence of bits, and subtract the reversed number from the original number in terms of base ten.

Implemented as a tacit, monadic verb in J, this would look something like:


f =: ] - [: #. [: |. #:

Our verb, f, is (=:) the given number (]) minus (-) the base two (#.) of the reverse (|.) of the antibase two (#:) of the given number.

We can plot the result of applying f to the first ten thousand primes (p: i. 10000) like so:


require 'plot'
'type dot' plot f"0 p: i. 10000

If we’re feeling especially terse, we could write this as an almost-one-liner by substituting f for our implementation of f:


require 'plot'
'type dot' plot (] - [: #. [: |. #:)"0 p: i. 10000

Our implementation of f is a “train of verbs”, which is to say, a collection of verbs that compose together into hooks and forks. We can visualize this composition by looking at the “boxed” representation of our train:


┌─┬─┬──────────────────┐
│]│-│┌──┬──┬──────────┐│
│ │ ││[:│#.│┌──┬──┬──┐││
│ │ ││  │  ││[:│|.│#:│││
│ │ ││  │  │└──┴──┴──┘││
│ │ │└──┴──┴──────────┘│
└─┴─┴──────────────────┘

From right to left, J greedily groups verbs into three verb forks potentially followed by a final two verb hook if the total number of verbs in the train is even.

We can see that the first fork, [: |. #:, is a capped fork, which means it’s roughly equivalent to |. @: #:. In the monadic case, this fork takes its argument, converts it to a base two list of ones and zeroes, and reverses that list. Let’s refer to this newly composed verb as a moving forward.

The next fork in our train, [: #. a, is another capped fork building off of our previous fork. Again, this could be expressed using the @: verb to compose #. and a together: #. @: a. In the monadic case, this fork takes its argument, converts it to a reversed binary representation with a, and then converts that reversed list of ones and zero back to base ten with #.. Let’s call this newly composed verb b.

Our final fork, ] - b, runs our monadic input through b to get the base ten representation of our reversed binary, and subtracts it from the original argument.


If we wanted to make J’s implicit verb training explicit, we could define a, b, and our final f ourselves:


a =: [: |. #:
b =: [: #. a
f =: ] - b

But why go through all that trouble? Going the explicit route feels like a natural tendency to me, coming from a background of more traditional programming languages, but J’s implicit composition opens up a world of interesting readability properties.

I’m really fascinated by this kind of composition, and I feel like it’s what makes J really unique. I’ll never pass up an opportunity to try implementing something as a train of verbs.

August 20, 2019

Pete Corey (petecorey)

TIL About Node.js’ REPL Module August 20, 2019 12:00 AM

Today I learned that Node.js ships with a repl module that can be used to spin up a full-featured REPL on any Node.js process. This can be a fantastic tool for debugging a running server, or manually triggering back-end events.

Let’s assume that we’ve built a Node.js server who’s entry point is a server.js file. Let’s also assume that we have a constant (maybe pulled from our environment, maybe elsewhere) called REPL who’s truthiness determines whether we should start our REPL instance on standard in. Spinning up our REPL is as easy as:


if (REPL) {
    require('repl').start();
}

Once our server starts up, we’ll be greeted by a familiar prompt:


Starting server...
Listening on localhost:8080!
> 

Fantastic! Normal REPL rules apply. Our server will continue to run and its output will continue to stream to standard out. Our REPL prompt will stick to the bottom of the tail, as expected.

More advanced options can be gleaned from the repl documentation. Happy REPLing!

August 19, 2019

Pete Corey (petecorey)

Animating a Canvas with React Hooks August 19, 2019 12:00 AM

A recent React-based client project of mine required an HTML5 canvas animation. Already knee-deep in the new React hooks API, I decided to forgo the “traditional” (can something be “traditional” after just five years?) technique of using componentDidMount and componentWillUnmount in a class-based component, and try my hand at rendering and animating a canvas using React’s new useEffect hook.

Let’s dig into it!


Let’s set the scene by creating a new React component that we want to add our canvas to. We’ll assume that we’re trying to render a circle, so we’ll call our new component, Circle:


import React from 'react';

const Circle = () => {
    return (
        <canvas
            style={{ width: '100px', height: '100px' }}
        />
    );
};

export default Circle;

So far so good.


Our Circle component renders a canvas onto the page, but we have no way of interacting with it. Typically, to interact with an element of the DOM from within a React component, you need a “ref”. The new React hooks API gives us a convenient way to create and use refs:


 import React from 'react';
+import { useRef } from 'react';

 const Circle = () => {
+    let ref = useRef();
     return (
         <canvas
+            ref={ref} 
             style={{ width: '100px', height: '100px' }}
         />
     );
 };

 export default Circle;

Now ref.current holds a reference to our canvas DOM node.


Interacting with our canvas produces “side effects”, which isn’t allowed from within the render phase of a React component. Thankfully, the new hooks API gives us a simple way to introduce side effects into our components via the useEffect hook.


 import React from 'react';
+import { useEffect } from 'react';
 import { useRef } from 'react';
 
 const Circle = () => {
     let ref = useRef();
     
+    useEffect(() => {
+        let canvas = ref.current;
+        let context = canvas.getContext('2d');
+        context.beginPath();
+        context.arc(50, 50, 50, 0, 2 * Math.PI);
+        context.fill();
+    });
     
     return (
         <canvas
             ref={ref} 
             style={{ width: '100px', height: '100px' }}
         />
     );
 };
 
 export default Circle;

Our useEffect callback is free to interact directly with our canvas living in the DOM. In this case, we’re drawing a circle to our canvas.


Unfortunately, our circle may look a little fuzzy or distorted, depending on the pixel density of the screen it’s being viewed on. Let’s fix that by adjusting the scaling of our canvas:


 import React from 'react';
 import { useEffect } from 'react';
 import { useRef } from 'react';
 
+const getPixelRatio = context => {
+    var backingStore =
+    context.backingStorePixelRatio ||
+    context.webkitBackingStorePixelRatio ||
+    context.mozBackingStorePixelRatio ||
+    context.msBackingStorePixelRatio ||
+    context.oBackingStorePixelRatio ||
+    context.backingStorePixelRatio ||
+    1;
+    
+    return (window.devicePixelRatio || 1) / backingStore;
+};
 
 const Circle = () => {
     let ref = useRef();
     
     useEffect(() => {
         let canvas = ref.current;
         let context = canvas.getContext('2d');
         
+        let ratio = getPixelRatio(context);
+        let width = getComputedStyle(canvas)
+            .getPropertyValue('width')
+            .slice(0, -2);
+        let height = getComputedStyle(canvas)
+            .getPropertyValue('height')
+            .slice(0, -2);
         
+        canvas.width = width * ratio;
+        canvas.height = height * ratio;
+        canvas.style.width = `${width}px`;
+        canvas.style.height = `${height}px`;
         
         context.beginPath();
         context.arc(
+            canvas.width / 2,
+            canvas.height / 2,
+            canvas.width / 2,
             0,
             2 * Math.PI
         );
         context.fill();
     });
     
     return (
         <canvas
             ref={ref} 
             style={{ width: '100px', height: '100px' }}
         />
     );
 };
 
 export default Circle;

And with that, our circle should be crystal clear.


Now let’s introduce some animation. The standard way of animating an HTML5 canvas is using the requestAnimationFrame function to repeatedly call a function that renders our scene. Before we do that, we need to refactor our circle drawing code into a render function:


 useEffect(() => {
     ...
+    const render = () => {
         context.beginPath();
         context.arc(
             canvas.width / 2,
             canvas.height / 2,
             canvas.width / 2,
             0,
             2 * Math.PI
         );
         context.fill();
+    };
     
+    render();
 });

Now that we have a render function, we can instruct the browser to recursively call it whenever its appropriate to render another frame:


 const render = () => {
     context.beginPath();
     context.arc(
         canvas.width / 2,
         canvas.height / 2,
         canvas.width / 2,
         0,
         2 * Math.PI
     );
     context.fill();
+    requestAnimationFrame(render);
 };

This works, but there are two problems. First, if our component unmounts after our call to requestAnimationFrame, but before our render function is called, it can lead to problems. We should really cancel any pending animiation frame requests any time our component unmounts. Thankfully, requestAnimationFrame returns a request identifier that can be passed to cancelAnimationFrame to cancel our request.

The useEffect hook optionally expects a function to be returned by our callback. This function will be called to handle any cleanup required by our effect. Let’s refactor our animation loop to properly clean up after itself:


 useEffect(() => {
     ...
+    let requestId;
     const render = () => {
         context.beginPath();
         context.arc(
             canvas.width / 2,
             canvas.height / 2,
             canvas.width / 2,
             0,
             2 * Math.PI
         );
         context.fill();
+        requestId = requestAnimationFrame(render);
     };
     
     render();
     
+    return () => {
+        cancelAnimationFrame(requestId);
+    };
 });

Perfect.


Our second problem is that our render function isn’t doing anything. We have no visual indicator that our animation is actually happening!

Let’s change that and have some fun with our circle:


 let requestId,
+    i = 0;
 const render = () => {
     context.clearRect(0, 0, canvas.width, canvas.height);
     context.beginPath();
     context.arc(
         canvas.width / 2,
         canvas.height / 2,
+        (canvas.width / 2) * Math.abs(Math.cos(i)),
         0,
         2 * Math.PI
     );
     context.fill();
+    i += 0.05;
     requestId = requestAnimationFrame(render);
 };

Beautiful. Now we can clearly see that our animation is running in full swing. This was obviously a fairly contrived example, but hopefully it serves as a helpful recipe for you in the future.

#root { margin: 4em auto; } #root canvas { height: 100%!important; width: 100%!important; }

August 18, 2019

David Wilson (dw)

Mitogen v0.2.8 released August 18, 2019 08:45 PM

Mitogen for Ansible v0.2.8 has been released. This version (finally) supports Ansible 2.8, comes with a supercharged replacement fetch module, and includes roughly 85% of what is needed to implemement fully asynchronous connect.

As usual a huge slew of fixes are included. This is a bumper release, running to over 20k lines of diff. Get it while it's hot, and as always, bug reports are welcome!

August 15, 2019

Pepijn de Vos (pepijndevos)

Open Source Formal Verification in VHDL August 15, 2019 12:00 AM

I believe in the importance of open source synthesis, and think it’s important that open source tools support both Verilog and VHDL. Even though my GSoC proposal to add VHDL support to Yosys was rejected, I’ve still been contributing small bits and pieces to GHDL and its Yosys plugin.

This week we reached what I think is an important milestone: I was able to synthesize my VHDL CPU and then formally verify the ALU of it using completely open source tools. (and then synthesize it to an FPGA which is not supported by open source tools yet) There is a lot to unpack here, so let’s jump in.

Yosys, Nextpnr, SymbiYosys, GHDL, ghdlsynth-beta

Yosys is an open source synthesis tool that is quickly gaining momentum and supporting more and more FPGAs. Yosys currently supports Verilog, and turns that into various low-level netlist representations.

Nextpnr is a place-and-rout tool, which takes a netlist and turns it into a bitstream for any of the supported FPGA types. These bitstream formats are not publicly documented, so this is a huge reverse-engineering effort.

SymbiYosys is a tool based around Yosys and various SAT solvers to let you do formal verification on your code. More on formal verification later. But important to know is that it works on the netlists generated by Yosys.

GHDL is an open source VHDL simulator, and as far as I know, one of its kind. VHDL is notoriously hard to parse, so many other open source attempts at VHDL simulation and synthesis have faltered. Work is underway to add synthesis to GHDL.

And last but not least, ghdlsynth-beta is a plugin for Yosys that converts the synthesis format of GHDL to the intermediate representation of Yosys, allowing it to be synthesized to various netlist formats and used for FPGA, ASIC, formal verification, and many other uses. It is currently a separate repository, but the goal is to eventually upstream it into Yosys.

Formal Verification

I think formal verification sounds harder and more scary than it is. An alternative description is property testing with a SAT solver. Think Quickcheck, not Coq. This is much simpler and less formal than using a proof assistent.

Basically you describe properties about your code, and SymbiYosys compiles your code an properties to a netlist and from a netlist to Boolean logic. A SAT solver is then used to find inputs to your code that (dis)satisfy the properties you described. This does not “prove” that your code is correct, but it proves that it satisfies the properties you defined.

In hardware description languages you describe properties by assertions and assumptions. An assumption constrains what the SAT solver can consider as valid inputs to your program, and assertions are things you believe to be true about your code.

The powerful thing about formal verification is that it considers all valid inputs at every step, and not just the happy case you might test in simulation. It will find so many edge cases it’s not even funny. Once you get the hang of it, it’s actually less work than writing a testbench. Just a few assertions in your code and the bugs come flying at you.

If you want to learn more about formal verification, Dan Gisselquist has a large number of articles and tutorials about it, mainly using Verilog.

Installation

To play along at home, you need to install a fair number of programs, so better get some of your favourite hot beverage.

At this point you should be able to run ghdl --synth foo.vhd -e foo which will output a VHDL representation of the synthesized netlist. You should be able to run yosys -m ghdl and use the Yosys command ghdl foo.vhd -e foo to obtain a Yosys netlist which you can then show, dump, synth, or even write_verilog.

Verifying a bit-serial ALU

To demonstrate how formal verification works and why it is so powerful, I want to walk you through the verification of the ALU of my CPU.

I’m implementing a bit-serial architecture, which means that my ALU operates on one bit at a time, producing one output bit and a carry. The carry out is then the carry in to the next bit. The logic that produces the output and the carry depends on a 3-bit opcode.

  process(opcode, a, b, ci)
  begin
    case opcode is
      when "000" => -- add
        y <= a xor b xor ci; -- output
        co <= (a and b) or (a and ci) or (b and ci); -- carry
        cr <= '0'; -- carry reset value
      -- [...]
    end case;
  end process;

  process(clk, rst_n)
  begin
    if(rising_edge(clk)) then
      if(rst_n = '0') then
        ci <= cr; -- reset the carry
      else
        ci <= co; -- copy carry out to carry in
      end if;
    end if;
  end process;

Important to note is the carry reset value. For addition, the first bit is added without carry, but for subtraction the carry is 1 because -a = (not a) + 1, and similarly for other different opcodes. So when in reset, the ALU sets the carry in to the reset value corresponding to the current opcode.

So now onward to the verification part. Since VHDL only has assert and none of the SystemVerilog goodies, Property Specification Language is used. (that link contains a good tutorial) PSL not only provides restrict, assume, and cover, but also allows you to express preconditions and sequences.

To make my life easier, I want to specify that I want to restrict valid sequences to those where the design starts in reset, processes 8 bits, back to reset, and repeat, so that the reset will look like 011111111011111111...

restrict { {rst_n = '0'; (rst_n = '1')[*8]}[+]};

Then, I want to specify that when the ALU is active, the opcode will stay constant. Else you’ll just get nonsense.

assume always {rst_n = '0'; rst_n = '1'} |=>
  opcode = last_op until rst_n = '0';

Note that I did not define any clock or inputs. Just limiting the reset and opcode is sufficient. With those assumptions in place, we can assert what the output should look like. I shift the inputs and outputs into 8-bit registers, and then when the ALU goes into reset, we can verify the output. For example, if the opcode is “000”, the output should be the sum of the two inputs.

assert always {opcode = "000" and rst_n = '1'; rst_n = '0'} |->
  y_sr = a_sr+b_sr;

After adding the other opcodes, I wrapped the whole thing in a generate block so I can turn it off with a generic parameter for synthesis

formal_gen : if formal generate
  signal last_op : std_logic_vector(2 downto 0);
  signal a_sr : unsigned(7 downto 0);
  signal b_sr : unsigned(7 downto 0);
  signal y_sr : unsigned(7 downto 0);
begin
-- [...]
end generate;

And now all that’s left to do is write the SymbiYosys script and run it. The script just specifies how to compile the files and the settings for the SAT solver. Note that -fpsl is required for reading --psl code in comments, or --std=08 to use VHDL-2008 which supports PSL as part of the core language.

[options]
mode bmc
depth 20

[engines]
smtbmc z3

[script]
ghdl --std=08 alu.vhd -e alu
prep -top alu

[files]
alu.vhd

To load the GHDL plugin, SymbiYosys has to be run as follows:

$ sby --yosys "yosys -m ghdl" -f alu.sby 
SBY 15:02:25 [alu] Removing direcory 'alu'.
SBY 15:02:25 [alu] Copy 'alu.vhd' to 'alu/src/alu.vhd'.
SBY 15:02:25 [alu] engine_0: smtbmc z3
SBY 15:02:25 [alu] base: starting process "cd alu/src; yosys -m ghdl -ql ../model/design.log ../model/design.ys"
SBY 15:02:25 [alu] base: finished (returncode=0)
SBY 15:02:25 [alu] smt2: starting process "cd alu/model; yosys -m ghdl -ql design_smt2.log design_smt2.ys"
SBY 15:02:25 [alu] smt2: finished (returncode=0)
SBY 15:02:25 [alu] engine_0: starting process "cd alu; yosys-smtbmc -s z3 --presat --noprogress -t 20 --append 0 --dump-vcd engine_0/trace.vcd --dump-vlogtb engine_0/trace_tb.v --dump-smtc engine_0/trace.smtc model/design_smt2.smt2"
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Solver: z3
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Checking assumptions in step 0..
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Checking assertions in step 0..
[...]
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Checking assumptions in step 9..
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Checking assertions in step 9..
SBY 15:02:25 [alu] engine_0: ##   0:00:00  BMC failed!
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Assert failed in alu: /179
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Writing trace to VCD file: engine_0/trace.vcd
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Writing trace to Verilog testbench: engine_0/trace_tb.v
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Writing trace to constraints file: engine_0/trace.smtc
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Status: FAILED (!)
SBY 15:02:25 [alu] engine_0: finished (returncode=1)
SBY 15:02:25 [alu] engine_0: Status returned by engine: FAIL
SBY 15:02:25 [alu] summary: Elapsed clock time [H:MM:SS (secs)]: 0:00:00 (0)
SBY 15:02:25 [alu] summary: Elapsed process time [H:MM:SS (secs)]: 0:00:00 (0)
SBY 15:02:25 [alu] summary: engine_0 (smtbmc z3) returned FAIL
SBY 15:02:25 [alu] summary: counterexample trace: alu/engine_0/trace.vcd
SBY 15:02:25 [alu] DONE (FAIL, rc=2)

Oh no! We have a bug! Let’s open the trace to see what went wrong.

gtkwave alu/engine_0/trace.vcd

gtkwave trace

So we’re doing a subtraction, and according to my math 29-150=-121 but the ALU output is -122, so we’re off by one. A little head-scratching later, we can see the problem: On the first cycle of the subtraction the carry in is zero rather than one! Why? Because on the previous clock cycle the instruction was exclusive or, which reset the carry in to zero.

Note that this bug would never show up if you did a test bench that executes a fixed instruction from reset. But the SAT solver managed to find a specific sequence of opcodes that cause the carry to be wrong. Awesome.

So how do we fix it? There are two ways. The first is to change the code to asynchronously determine the carry in. The second is to write you code so the opcode is stable before the ALU comes out of reset, which ended up using less logic. In this case we can change the opcode assumption to

assume always {rst_n = '0'; rst_n = '1'} |->
  opcode = last_op until rst_n = '0';

Note that we used the thin arrow |-> rather than the fat arrow |=> now. The fat arrow triggers after the precondition has been met, while the thin arrow overlaps with the end of the precondition. So now we’re saying that when reset became inactive, the opcode is the same as it was while the device was in reset. Let’s try again.

$ sby --yosys "yosys -m ghdl" -f alu.sby 
SBY 15:31:36 [alu] Removing direcory 'alu'.
SBY 15:31:36 [alu] Copy 'alu.vhd' to 'alu/src/alu.vhd'.
SBY 15:31:36 [alu] engine_0: smtbmc z3
SBY 15:31:36 [alu] base: starting process "cd alu/src; yosys -m ghdl -ql ../model/design.log ../model/design.ys"
SBY 15:31:36 [alu] base: finished (returncode=0)
SBY 15:31:36 [alu] smt2: starting process "cd alu/model; yosys -m ghdl -ql design_smt2.log design_smt2.ys"
SBY 15:31:36 [alu] smt2: finished (returncode=0)
SBY 15:31:36 [alu] engine_0: starting process "cd alu; yosys-smtbmc -s z3 --presat --noprogress -t 20 --append 0 --dump-vcd engine_0/trace.vcd --dump-vlogtb engine_0/trace_tb.v --dump-smtc engine_0/trace.smtc model/design_smt2.smt2"
SBY 15:31:36 [alu] engine_0: ##   0:00:00  Solver: z3
SBY 15:31:36 [alu] engine_0: ##   0:00:00  Checking assumptions in step 0..
SBY 15:31:36 [alu] engine_0: ##   0:00:00  Checking assertions in step 0..
[...]
SBY 15:31:37 [alu] engine_0: ##   0:00:01  Checking assumptions in step 19..
SBY 15:31:37 [alu] engine_0: ##   0:00:01  Checking assertions in step 19..
SBY 15:31:37 [alu] engine_0: ##   0:00:01  Status: PASSED
SBY 15:31:37 [alu] engine_0: finished (returncode=0)
SBY 15:31:37 [alu] engine_0: Status returned by engine: PASS
SBY 15:31:37 [alu] summary: Elapsed clock time [H:MM:SS (secs)]: 0:00:01 (1)
SBY 15:31:37 [alu] summary: Elapsed process time [H:MM:SS (secs)]: 0:00:01 (1)
SBY 15:31:37 [alu] summary: engine_0 (smtbmc z3) returned PASS
SBY 15:31:37 [alu] DONE (PASS, rc=0)

Yay!

Debugging tips

It should be said that all of this is very experimental and you are therefore likely to run into bugs and missing features. I would say that at this point it is feasible to write new code and work around GHDL’s current limitations (or fix them!), but running large existing codebases is unlikely to be successful. (but very much the goal!)

When you run into errors, the first step is to find out if it is a bug in the plugin or GHDL itself.

If you see Unsupported(1): instance X of Y. this means the plugin does not know how to translate a GHDL netlist item to Yosys. These are usually pretty easy to fix. See this pull request for an example. Good to know: Id_Sub is defined in ghdlsynth_gates.h which is generated from netlists-gates.ads. module->addSub is defined in rtlil.h.

If you just see ERROR: vhdl import failed. this likely means GHDL crashed. Run GHDL outside Yosys (ghdl --synth) to see the actual error. Usually it’ll show something like some_package: cannot handle IIR_KIND_SOMETHING (mycode.vhd:26:8) which means that some_pacakge in the src/synth part of GHDL can’t handle some language construct yet. This can be anything from a missing operator to whole language constructs, and the fix can be anything for copy-pasting another operator to a serious project. See this pull request for an example on how to add a missing operator.

If it’s not obvious what is going on, it’s time to break out gdb. It’s important to know that in the GHDL repo there is a .gdbinit that you can source inside gdb. This enables catching exceptions and includes utilities for printing IIR values. If you want to debug inside Yosys, it is helpful to first run the program without breakpoints so all the libraries are loaded and gdb understands there is Ada code involved. Then source .gdbinit, set breakpoints, and run again. (note: GHDL/Yosys command line arguments are passed to run and not gdb)

Happy debugging!

August 12, 2019

Marc Brooker (mjb)

Kindness, Wickedness and Safety August 12, 2019 12:00 AM

Kindness, Wickedness and Safety

We must build kind systems.

David Epstein's book Range: Why Generalists Triumph in a Specialized World turned me on to the idea of Kind and Wicked learning environments, and I've found the idea to be very useful in framing all kinds of problems.1 The idea comes from The Two Settings of Kind and Wicked Learning Environments. The abstract gets right to the point:

Inference involves two settings: In the first, information is acquired (learning); in the second, it is applied (predictions or choices). Kind learning environments involve close matches between the informational elements in the two settings and are a necessary condition for accurate inferences. Wicked learning environments involve mismatches.

The authors go on to describe the two environments in terms of the information that we can learn from L (for learning), and information that we use when we actually have to make predictions T (for target). They break environments down into kind or wicked depending on how L relates to T. In kind environments, L and T are closely related: if you learn a rule from L it applies at least approximately to T. In wicked environments, L is a subset or superset of T, or the sets intersect only partially, or are completely unrelated.

Simplifying this a bit more, in kind environments we can learn the right lessons from experience, in wicked environment we learn the wrong lessons (or at least incomplete lessons).

From the paper again:

If kind, we have the necessary conditions for accurate inference. Therefore, any errors must be attributed to the person (e.g., inappropriate information aggregation). If wicked, we can identify how error results from task features, although these can also be affected by human actions. In short, our framework facilitates pinpointing the sources of errors (task structure and/or person).

This has interesting implications for thinking about safety, and the role of operators (and builders) in ensuring safety. In kind environments, operator mistakes can be seen as human error, where the human learned the wrong lesson or did the wrong thing. In wicked environments, humans will always make errors, because there are risks that are not captured by their experience.

Going back to Anatoly Dyatlov's question to the IAEA:

How and why should the operators have compensated for design errors they did not know about?

Dyatlov is saying that operating Chernobyl was a wicked environment. Operators applying their best knowledge and experience, even flawlessly, weren't able to make the right inferences about the safety of the system.

Back to the paper:

Since kind environments are a necessary condition for accurate judgments, our framework suggests deliberately creating kind environments.

I found reading this to be something of a revelation. When building safe systems, we need to make those systems kind. We need to deliberately create kind evironments. If we build them so they are wicked, then we set our operators, tooling and automation up for failure.

Some parts of our field are inherently wicked. In large and complex systems the set of circumstances we learn from is always incomplete, because the system has so many states that there's no way to have seen them all before. In security, there's an active attacker who's trying very hard to make the environment wicked.

The role of the designer and builder of systems is to make the environment as kind as possible. Extract as much wickedness as possible, and try not to add any.

Footnotes

  1. The book is worth reading. It contains a lot of interesting ideas, but like all popular science books also contains a lot of extrapolation beyond what the research supports. If you're pressed for time, the EconTalk episode about the book covers a lot of the material.

August 10, 2019

Andrew Montalenti (amontalenti)

JavaScript: The Modern Parts August 10, 2019 07:00 AM

In the last few months, I have learned a lot about modern JavaScript and CSS development with a local toolchain powered by Node 8, Webpack 4, and Babel 7. As part of that, I am doing my second “re-introduction to JavaScript”. I first learned JS in 1998. Then relearned it from scratch in 2008, in the era of “The Good Parts”, Firebug, jQuery, IE6-compatibility, and eventually the then-fledgling Node ecosystem. In that era, I wrote one of the most widely deployed pieces of JavaScript on the web, and maintained a system powered by it.

Now I am re-learning it in the era of ECMAScript (ES6 / ES2017), transpilation, formal support for libraries and modularization, and, mobile web performance with things like PWAs, code splitting, and WebWorkers / ServiceWorkers. I am also pleasantly surprised that JS, via the ECMAScript standard and Babel, has evolved into a pretty good programming language, all things considered.

To solidify all this stuff, I am using webpack/babel to build all static assets for a simple Python/Flask web app, which ends up deployed as a multi-hundred-page static site.

One weekend, I ported everything from Flask-Assets to webpack, and to play around with ES2017 features, as well as explore the Sass CSS preprocessor and some D3.js examples. And boy, did that send me down a yak shaving rabbit hole. Let’s start from the beginning!

JavaScript in 1998

I first learned JavaScript in 1998. It’s hard to believe that this was 20 years — two decades! — ago. This post will chart the two decades since — covering JavaScript in 1998, 2008, and 2018. The focus of the article will be on “modern” JavaScript, as of my understanding in 2018/2019, and, in particular, what a non-JavaScript programmer should know about how the language — and its associated tooling and runtime — have dramatically evolved. If you’re the kind of programmer who thinks, “I code in Python/Java/Ruby/C/whatever, and thus I have no use for JavaScript and don’t need to know anything about it”, you’re wrong, and I’ll describe why. Incidentally, you were right in 1998, you could get by without it in 2008, and you are dead wrong in 2018.

Further, if you are the kind of programmer who thinks, “JavaScript is a tire fire I’d rather avoid because it lacks basic infrastructure we take for granted in ‘real’ programming languages”, then you are also wrong. I’ll be able to show you how “not taking JavaScript seriously” is the 2018 equivalent of the skeptical 2008-era programmer not taking Python or Ruby seriously. JavaScript is a language that is not only here to stay, but has already — and will continue to — take over the world in several important areas. To be a serious programmer, you’ll have to know JavaScript’s Modern and Good Parts — as well as some other server-side language, like Python, Ruby, Go, Elixir, Clojure, Java, and so on. But, though you can swap one backend language for the other, you can’t avoid JavaScript: it’s pervasive in every kind of web deployment scenario. And, the developer tooling has fully caught up to your expectations.

JavaScript during The Browser Wars

Browsers were a harsh environment to target for development; not only was Internet adoption low and not only were internet connections slow, but the browser wars — mainly between Netscape and Microsoft — were creating a lot of confusion.

Netscape Navigator 4 was released in 1997, and Internet Explorer 5 was released in 1998. The web was still trying to make sense of HTML and CSS; after all, CSS1 had only been released a year earlier.

In this environment, the definitive web development book of the era was “JavaScript: The Definitive Guide”, which weighed in at over 500 pages. Note that, in 1998, the most widely used programming languages were C, C++, and Java, as well as Microsoft Visual Basic for Windows programmers. So expectations about “what programming was” were framed mostly around these languages.

In this sense, JavaScript was quite, quite different. There was no compiler. There was no debugger (at least, not very good ones). There was no way to “run a JavaScript program”, except to write scripts in your browser, and see if they ran. Development tools for JavaScript were still primitive or inexistent. There was certainly not much of an open source community around JS; to figure out how to do things, you would typically “view source” on other people’s websites. Plus, much of the discussion in the programming community of web developers was how JavaScript represented a compatibility and security nightmare.

Not only differing implementations across browsers, but also many ways for you to compromise the security of your web application by relying upon JavaScript too directly. A common security bug in that era was to validate forms with JavaScript, but still allow invalid (and insecure) values to be passed to the server. Or, to password-protect a system, but in a way that inspection of JavaScript code could itself crack access to that system. Combined with the lack of a proper development environment, the “real web programmers” used JavaScript as nothing more than a last resort — a way to inject a little bit of client-side code and logic into pages where doing it server-side made no sense. I remember one of the most common use cases for JavaScript at the time was nothing more than changing an image upon hover, as a stylistic effect, or implementing a basic on-hover menu on a complex multi-tab form. These days, these tasks can be achieved with vanilla CSS, but, at the time, JavaScript DOM manipulation was your only option.

JavaScript in 2008

Fast forward 10 years. In 2008, Douglas Crockford released the book, “JavaScript: The Good Parts”. By using a language subsetting approach, Crockford pointed out that, not only was JavaScript not a bad language, it was actually a good language, well-designed, with certain key features that made it stand out vs competitors.

Around this time, several JavaScript libraries were becoming popular, notably jQuery, Prototype, YUI, and Dojo. These libraries attempted to provide JavaScript with something it was missing: a cross-browser compatibility layer and programming model for doing dynamic manipulation of pages inside the browser, and especially for a new model of JavaScript programming that was emerging, with the moniker AJAX. This was the beginning of the trend of rich internet applications, “dynamic” web apps, single-page applications, and the like.

JavaScript’s Tooling Leaps

The developer tooling for JavaScript also took some important leaps. In 2006, the Firefox team released Firebug, a JavaScript and DOM debugger for Firefox, which was then one of the world’s most popular web browsers, and open source. Two years later, Google would make the first release of Google Chrome, which bundled some developer tooling. Around the same time that Chrome was released, Google also released V8, the JavaScript engine that was embedded inside of Chrome. That marked the first time that the world had seen a full-fledged, performant open source implementation of the JavaScript language that was not completely tied to a browser. Firefox’s JS engine, SpiderMonkey, was part of its source tree, but was not necessarily marketed to be modularized and used outside the context of the Firefox browser.

I remember that aside from Crockford’s work on identifying the good parts of JavaScript, and aside from the new (and better) developer tooling, a specific essay on Mozilla’s website helped me re-appreciate the language, and throw away my 1998 conception. That article was called “A Reintroduction to JavaScript”. It showed how JavaScript was actually a real programming language, once you got past the tooling bumps. A little under-powered in its standard library, thus you had to rely upon frameworks (like jQuery) to give you some tools, and little micro-libraries beyond that.

A year after reading that essay, I wrote my own about JavaScript, which was called “Real, Functional Programs with JavaScript” (archived PDF here). It described how JavaScript was, quite surprisingly, more of a functional language than Java 8 or Python 2.7. And that with a little focus on understanding the functional core, really good programs could be written. I recently converted this essay into a set of instructional slides with the name, “Lambda JavaScript” (archived notes here), which I now use to teach new designers/developers the language from first principles.

But, let’s return to history. Only a year after the release of Chrome, in 2009, we saw the first release of NodeJS, which took the V8 JavaScript engine and embedded it into a server-side environment, which could be used to experiment with JavaScript on a REPL, to write scripts, and even to write HTTP servers on a performant event loop.

People began to experiment with command-line tools written in JavaScript, and with web frameworks written in JavaScript. It was at this point that the pace of development in the JavaScript community accelerated. In 2010, npm — the Node Package Manager — was released, and it and its package registry quickly grew to represent the full JavaScript open source community. Over the next few years, the browser vendors of Mozilla, Google, Apple, and Microsoft engaged in the “JavaScript Engine Wars”, with each developing SpiderMonkey, V8, Nitro, and Chakra to new heights.

Meanwhile, NodeJS and V8 became the “standard” JS engine running on developer’s machines from the command line. Though developers still had to target old “ECMAScript 3” browsers (such as IE6), and thus had to write restrained JavaScript code, the “evergreen” (auto-updating) browsers from Mozilla, Google, and Apple gained support for ECMAScript 5 and beyond, and mobile web browsing went into ascendancy, thus making Chrome and Safari dominant in market share especially on smartphones.

I remember in 2012, I gave a presentation at a local tech conference entitled, “Writing Real Programs… with JavaScript!?”. The “!?” punctuation was intentional. That was the general zeitgeist I remember in a room full of developers: that is, “is writing real programs with JavaScript… actually possible!?” It’s funny to review those slides as a historical relic. I spent the first half of the talk convincing the audience that JavaScript’s functional core was actually pretty good. And then I spent the second half convincing them that NodeJS might… it just might… create a developer tooling ecosystem and standard library for JavaScript. There are also a few funny “detour” slides in there around things like Comet vs Ajax, a debate that didn’t really amount to much (but it’s good to remind one of fashion trends in tech).

Zooming ahead a few years, in all of this noise of web 2.0, cloud, and mobile, we finally reached “mobilegeddon” in 2015, where mobile traffic surpassed desktop traffic, and we also saw several desktop operating systems move to a mostly-evergreen model, such as Windows 10, Mac OS X, and ChromeOS. As a result, as early as 2015 — but certainly by 2018 — JavaScript became the most widely deployed and performant programming language with “built-in support” on almost every desktop and mobile computer in the world.

In other words, if you wanted your code to be “write once, run everywhere” in 2015 or so (but even as far back as 2009), your best option was JavaScript. Well, that’s even more true today. The solid choice for widespread distribution of your code continues to be JavaScript. As Crockford predicted in 2008: “It is better to be lucky than smart.”

JavaScript in 2018-2019

In 2018-2019, several things have changed about the JavaScript community. Development tools are no longer fledgling, but are, instead, mature. There are built-in development tools in all of Safari, Firefox, and Chrome browsers (and the Firebug project is mostly deprecated). There are also ways to debug mobile web browsers using mobile development tools. NodeJS and npm are mature projects that are shared infrastructure for the whole JavaScript community.

What’s more, JavaScript, as a language, has evolved. It’s no longer just the kernel language we knew in 1998, nor the “good parts” we knew in 2008, but instead the “modern parts” of JavaScript include several new language features that go by the name “ES6” (ECMAScript v6) or “ES2017” (ECMAScript 2017 Edition), and beyond.

Some concepts in HTML have evolved, such as HTML5 video and audio elements. CSS, too, has evolved, with the CSS2 and CSS3 specifications being ratified and widely adopted. JSON has all but entirely replaced XML as an interchange format and is, of course, JavaScript-based.

The V8 engine has also gotten a ton of performance-oriented development. It is now a JIT compiled language with speedy startup times and speedy near-native performance for CPU-bound blocks. Modern web performance techniques are almost entirely based on a speedy JavaScript engine and the ability to script different elements of a web application’s loading approach.

The language itself has become comfortable with something akin to “compiler” and “command line” toolchains you might find in Python, Ruby, C, and Java communities. In lieu of a JavaScript “compiler”, we have node, JavaScript unit testing frameworks like Mocha/Jest, as well as eslint and babel for syntax checking. (More on this later.)

In lieu of a “debugger”, we have the devtools built into our favorite browser, like Chrome or Firefox. This includes rich debuggers, REPLs/consoles, and visual inspection tools. Scriptable remote connections to a node environment or a browser process (via new tools like Puppeteer) further close the development loop.

To use JavaScript in 2018/2019, therefore, is to adopt a system that has achieved 2008-era maturity that you would see in programming ecosystems like Python, Ruby, and Java. But, in many ways, JavaScript has surpassed those communities. For example, where Python 3’s reference implementation, CPython, is certainly fast as far as dynamic languages go, JavaScript’s reference implementation, V8, is optimized by JIT and hotspot optimization techniques that are only found in much more mature programming communities, such as Java’s (which received millions of dollars of commercial support in applied/advanced compiler techniques in the Sun era). That means that unmodified, hotspot JavaScript code can be optimized into native code automatically by the Node runtime and by browsers such as Chrome.

Whereas Java and C users may still have debates about where, exactly, open source projects should publish their releases, that issue is settled in the JavaScript community: it’s npm, which operates similarly to PyPI and pip in the Python community.

Some essential developer tooling issues were only recently settled. For example, because modern JavaScript (such as code written using ES2017 features) needs to target older browsers, a “transpilation” toolchain is necessary, to compile ES2017 code into ES3 or ES5 JavaScript code, suitable for older browsers. Because “old JavaScript” is a Turing complete, functional programming language, we know we can translate almost any new “syntactic sugar” to the old language, and, indeed, the designers of the new language features are being careful to only introduce syntax that can be safely transpiled.

What this means, however, is that to do JavaScript development “The Modern Way”, while adopting its new features, you simply must use a local transpiler toolchain. The community standard for this at the moment is known as babel, and it’s likely to remain the community standard well into the future.

Another issue that plagued 2008-era JavaScript was build tooling and modularization. In the 2008-2012 era, ad-hoc tools like make were used to concatenate JavaScript modules together, and often Java-based tools such as Google’s Closure Compiler or UglifyJS were used to assemble JavaScript projects into modules that could be included onto pages. In 2012, the Grunt tool was released as a JavaScript build tool, written atop NodeJS, runnable from the command-line, and configurable using a JavaScript “Gruntfile”. A whole slew of build tools similar to this were released in the period, creating a whole lot of code churn and confusion.

Thankfully, today, Single Page Application frameworks like React have largely solved this problem, with the ascendancy of webpack and the reliance on npm run-script. Today, the webpack community has come up with a sane approach to JavaScript modularization that relies upon the modern JS support for modules, and then development-time tooling, provided mainly through the webpack CLI tool, allow for local development and production builds. This can all be scripted and wired together with simple npm run-script commands. And since webpack can be itself installed by npm, this keeps the entire development stack self-contained in a way that doesn’t feel too dissimilar from what you might enjoy with lein in Clojure or python/pip in Python.

Yes, it has taken 20 years, but JavaScript is now just as viable a choice for your backend and CLI tooling projects as Python was in the past. And, for web frontends, it’s your only choice. So, if you are a programmer who cares, at all, about distribution of your code to users, it’s time to care about JavaScript!

In a future post, I plan to go even deeper on JavaScript, covering:

  • How to structure your first “modern” JavaScript project
  • Using Modern JS with Python’s Flask web framework for simple static sites
  • Understanding webpack, and why it’s important
  • Modules, modules, modules. Why JS modules matter.
  • Understanding babel, and why it’s important
  • Transpilation, and how to think about evolving JS/ES features and “compilation”
  • Using eslint for bonus points
  • Using sourcemaps for debugging
  • Using console.assert and console for debugging
  • Production minification with uglify
  • Building automated tests with jest
  • Understanding the value of Chrome scripting and puppeteer

Want me to keep going? Let me know via @amontalenti on Twitter.

August 09, 2019

Oliver Charles (ocharles)

Who Authorized These Ghosts!? August 09, 2019 12:00 AM

Recently at CircuitHub we’ve been making some changes to how we develop our APIs. We previously used Yesod with a custom router, but we’re currently exploring Servant for API modelling, in part due to it’s potential for code generation for other clients (e.g., our Elm frontend). Along the way, this is requiring us to rethink and reinvent previously established code, and one of those areas is authorization.

To recap, authorization is

the function of specifying access rights/privileges to resources related to information security and computer security in general and to access control in particular.

This is in contrast to authentication, which is the act of showing that someone is who they claim to be.

Authorization is a very important process, especially in a business like CircuitHub where we host many confidential projects. Accidentally exposing this data could be catastrophic to both our business and customers, so we take it very seriously.

Out of the box, Servant has experimental support for authorization, which is a good start. servant-server gives us Servant.Server.Experimental.Auth which makes it a doddle to plug in our existing authorization mechanism (cookies & Redis). But that only shows that we know who is asking for resources, how do we check that they are allowed to access the resources?

As a case study, I want to have a look at a particular end-point, /projects/:id/price. This endpoint calculates the pricing options CircuitHub can offer a project, and there are few important points to how this endpoint works:

  1. The pricing for a project depends on the user viewing it. This is because some users can consign parts so CircuitHub won’t order them. Naturally, this affects the price, so pricing is viewer dependent.
  2. Some projects are owned by organizations, and should be priced by the organization as a whole. If a user is a member of the organization that owns the project pricing has been requested for, return the pricing for the organization. If the user is not in the organization, return their own custom pricing.
  3. Private projects should only expose their pricing to superusers, the owner of the project, and any members of the project’s organization (if it’s owned by an organization).

This specification is messy and complicated, but that’s just reality doing it’s thing.

Our first approach was to try and represent this in Servant’s API type. We start with the “vanilla” route, with no authentication or authorization:

Next, we add authorization:

At this point, we’re on our own - Servant offers no authorization primitives (though there are discussions on this topic).

My first attempt to add authorization to this was:

There are two new routing combinators here: AuthorizeWith and CanView. The idea is AuthorizeWith somehow captures the result of authenticating, and provides that information to CanView. CanView itself does some kind of authorization using a type class based on its argument - here Capture "id" ProjectId. The result is certainly something that worked, but I was unhappy with both the complexity to implement it (which is scope to get it wrong), and the lack of actual evidence of authorization.

The latter point needs some expanding. What I mean by “lacking evidence” is that with the current approach, the authorization is essentially like writing the following code:

If I later add more resource access into doThings, what will hold me accountable to checking authorization on those resources? The answer is… nothing! This is similar to boolean blindless - we performed logical check, only to throw all the resulting evidence away immediately.

At this point I wanted to start exploring some different options. While playing around with ideas, I was reminded of the wonderful paper “Ghosts of Departed Proofs”, and it got me thinking… can we use these techniques for authorization?

Ghosts of Departed Proofs

The basic idea of GDP is to name values using higher-rank quantification, and then - in trusted modules - produce proofs that refer to these names. To name values, we introduce a Named type, and the higher-ranked function name to name things:

Note that the only way to construct a Named value outside of this module is to use name, which introduces a completely distinct name for a limited scope. Within this scope, we can construct proofs that refer to these names. As a basic example, we could use GDP to prove that a number is prime:

Here we have our first proof witness - IsPrime. We can witness whether or not a named Int is prime using checkPrime - like the boolean value isPrime this determines if a number is or isn’t prime, but we get evidence that we’ve checked a specific value for primality.

This is the whirlwind tour of GDP, I highly recommend reading the paper for a more thorough explanation. Also, the library justified-containers explores these ideas in the context of maps, where we have proofs that specific items are in the map (giving us total lookups, rather than partial lookups).

GDP and Authorization

This is all well and good, but how does this help with authorization? The basic idea is that authorization is itself a proof - a proof that we can view or interact with resources in a particular way. First, we have to decide which functions need authorization - these functions will be modified to require proof values the refer to the function arguments. In this example, we’ll assume our Servant handler is going to itself make a call to the price :: ProjectId -> UserId -> m Price function. However, given the specification above, we need to make sure that user and project are compatible. To do this, we’ll name the arguments, and then introduce a proof that the user in question can view the project:

But what is this CanViewProject proof?

A first approximation is to treat it as some kind of primitive or axiom. A blessed function can postulate this proof with no further evidence:

This is a good start! Our price function can only be called with a CanViewProject that matches the named arguments, and the only way to construct such a value is to use canViewProject. Of course we could get the implementation of this wrong, so we should focus our testing efforts to make sure it’s doing the right thing.

However, the Agda programmer in me is a little unhappy about just blindly postulating CanViewProject at the end. We’ve got a bit of vision back from our boolean blindness, but the landscape is still blurry. Fortunately, all we have to do is recruit more of the same machinery so far to subdivide this proof into smaller ones:

Armed with these smaller authorization primitives, we can build up our richer authorization scheme:

Now canViewProject just calls out to the other authorization routines to build it’s proof. Furthermore, there’s something interesting here. CanViewProject doesn’t postulate anything - everything is attached with a proof of the particular authorization case. This means that we can actually open up the whole CanViewProject module to the world - there’s no need to keep anything private. By doing this and allowing people to pattern match on CanViewProject, authorization results become reusable - if something else only cares that a user is a super user, we might be able to pull this directly out of CanViewProject - no need for any redundant database checks!

In fact, this very idea can help us implement the final part of our original specification:

Some projects are owned by organizations, and should be priced by the organization as a whole. If a user is a member of the organization that owns the project pricing has been requested for, return the pricing for the organization. If the user is not in the organization, return their own custom pricing.

If we refine our UserBelongsToProjectOrganization proof, we can actually maintain a bit of extra evidence:

Now whenever we have a proof UserBelongsToProjectOrganization, we can pluck out the actual organization that we’re talking about. We also have evidence that the organization owns the project, so we can easily construct a new CanViewProject proof - proofs generate more proofs!

Relationship to Servant

At the start of this post, I mentioned that the goal was to integrate this with Servant. So far, we’ve looked at adding authorization to a single function, so how does this interact with Servant? Fortunately, it requires very little to change. The Servant API type is authorization free, but does mention authentication.

It’s only when we need to call our price function do we need to have performed some authorization, and this happens in the server-side handler. We do this by naming the respective arguments, witnessing the authorization proof, and then calling price:

Conclusion

That’s where I’ve got so far. It’s early days so far, but the approach is promising. What I really like is there is almost a virtual slider between ease and rigour. It can be easy to get carried away, naming absolutely everything and trying to find the most fundamental proofs possible. I’ve found so far that it’s better to back off a little bit - are you really going to get some set membership checks wrong? Maybe. But a property check is probably gonig to be enough to keep that function in check. We’re not in a formal proof engine setting, pretending we are just makes things harder than they need to be.

August 05, 2019

eta (eta)

Presuming mental health issues considered harmful August 05, 2019 11:00 PM

This post isn’t at all technical, unlike the other ones. Here be dragons! Feel free to go and read something more sane if posts about non-technical interpersonal relations aren’t your cup of tea.

It’s generally acknowledged that talking to other people is a hard problem. Generally, though, in order to make it less of one, we have this amazing thing called ‘courtesy’ that gets wheeled out to deal with it, and make everything alright again. The point of being polite and courteous toward people, really, is to acknowledge that not everyone sees things the way you do, and so you need to account for that by giving everyone a bit of leeway. In a way, it shows respect for the other person’s way of thinking – being polite allows you to consider that your opinion may not be better than theirs. And, by and large, it works (except for the examples in that Wait But Why post there, but oh well).

I don’t want to talk about the situations where it works, though, because those are largely uninteresting! Where things get interesting (read: harmful) is when people, for one reason or another, decide that this respect for others is just not something they want to bother with.

Criticism

Of course, you don’t always agree with what other people are saying or doing; in fact, you might believe that, according to the way you see things, they’re being downright horrible! When this happens, there are usually a number of options available to you:

  1. Bring the issue up with the person (i.e. directly give them some constructive criticism)
  2. Tell your friends about the situation, and see what they think
  3. Start bad-mouthing the person when they’re not around

Option (1) might seem like the ‘best’ thing to do, but often it’s not; delivering criticism to people only works if they’re actually willing to receive it. Otherwise, all they hear is someone who thinks they know how to run their life better than them, which they perceive is disrespectful and somewhat rude – exactly the opposite of what you might have intended. Even if they don’t mind criticism, they may disagree with it and refuse to act on it, without getting angry at the fact that you delivered it, which is also fair enough.

Nonviolent Communication

When delivering criticism in this way, it’s also important to emphasise how your opinion is highly subjective; this is kind of what Nonviolent Communication (NVC) is trying to get at. Instead of saying things like:

You were very hurtful when you did X.

(which can come across as absolutism - you were very hurtful, according to some globally defined definition of ‘hurtful’), NVC advocates for statements along the lines of:

When you did X, I felt very hurt by that.

Here, this can’t possibly be disagreed with, as you’re only stating your subjective experience of what happened, instead of making a seemingly objective judgement - you’re not saying the person is bad, or that they necessarily “did anything wrong”; rather, you’re giving them feedback on what effect their actions have had, which they can use to be a better person in future. (Or not, if they don’t think your experience is worth caring about.)

Phoning a friend

Because option (1) is such a minefield sometimes – you have to try and phrase things the right way, avoid getting angry or upset yourself, and even then the person might hate you a bit for the criticism – most people opt for option (2) in everyday life, which is to discuss the problem with some friends.

This is distinct from option (3), in which you actively start spouting abuse about how terrible the person is; in (2), you’re trying to neutrally share your view with only a few friends of yours, with the expectation that they may well turn around and tell you that you’re actually the person at fault here. That is to say, the idea of (2) is not to presume that you’re in the right automatically, and for the people you tell to be blind yes-men who go along with it (!). Rather, the point is to get some crowd-sourced feedback on the problem at hand, in order to give you some perspective. It may even then lead to (1), or maybe one of your friends letting this person know, or whatever – the point is, option (2) is usually a pretty good solution, which is why it’s probably quite popular.

Giving up, or worse

I don’t need to say something like “option (3) [i.e. bad-mouthing other people] is bad”, because you probably knew that already. It’s also not as if every person in the world is a paragon of virtue who would never do anything so shocking, either; people share their negative judgements on other people quite widely all the time, and that’s just part of life. Obviously, it has interesting implications for your relationship with the person you’re bad-mouthing, but that’s your own problem – and is also something you’re unlikely to care about that much, or else you wouldn’t spread your opinion.

No, what I’m more interested in discussing is where this option turns more toxic than it usually is, which is usually helped by the context present in certain institutions nowadays.

Mental health, in 2019

In universities, schools, and other such educational institutions, a rather large focus has recently been placed on the idea of dealing with students’ mental health, for one reason or another. Reports like this 2018 NUS blog post about how Oxbridge may be ‘in decline’, due to how taxing going there can be for your mental health, abound; in many places, safeguarding children is the new buzzword. I’m not wishing to in any way dismiss or trivialize this focus; it’s arguably a good thing, and, at any rate, that’d be for another day and another blog post.

Rather, I want to discuss a rather interesting side-effect of this focus, and how it might interact with the things we’ve just been discussing in the rest of the post.

A not-entirely-fictitious example

Suppose someone has a problem with you. You might not necessarily have much of an idea as to why, or you might do; it doesn’t really matter. Clearly, they think option (1) (telling you about it directly) is right out; you have no way of doing whether or not they did (2), but you decide you don’t care that much. “They have a problem with me, but I don’t think that’s really anything I need to worry about; rather, I think they’re as responsible for this situation as I am.” In a normal situation, this would be alright; everyone’s entitled to their own standpoint, after all, and people generally tend to respect one another.

However, this person really doesn’t like you. As part of that, they’re building their own internal list of things you’ve done that support this standpoint; everything you say, or do, that can be in some way interpreted as validating their view gets added, and their opinion of you gets worse and worse. Which would also be fine, as long as you didn’t actually need to rely on this person for anything important; they can sit over there with their extreme judgements, and it doesn’t really bother you at all: what are they going to do about it?

Well, but, remember the context I was talking about. In a lot of institutions, the management are constantly on the lookout for possible ‘mental health issues’, which isn’t a bad thing at all. However, this can give your fun new friend some extra leverage; all they need to do is find something that would seem to suggest you’re not 100% alright, or that you might be in need of some help. (Either that, or they cherry-pick examples of you supposedly being nasty, and claim that their mental health is being affected. For bonus points, one could even do both!)

Now, they can go and deliver a long spiel about how you’re clearly not right in the head, or something to that effect, and how it’s caused great problems for them. They don’t really have to worry that much about it being traced back to them, of course; usually in these kinds of systems people reporting problems are granted a degree of anonymity, in order to prevent people being put off from reporting anything at all.

And, if you haven’t been very carefully controlling everything you do or say to ensure none of it could possibly be used against you, you might now find yourself a bit stuck; how, after all, do you counter the allegations laid against you? You often don’t even know who raised them, and you’re lucky if you’re even told anything about what they actually are; rather, you now have to contend with the completely fabricated idea that there’s something wrong with you, which tends to be awfully sticky and hard to get rid of once people get wind of it.

Even though that’s not at all true, and never was.

August 03, 2019

Alex Wilson (mrwilson)

Notes from the Week #28 August 03, 2019 12:00 AM

Notes from the Week #28

Here’s a non-exhaustive summary of what happened this week.

One

The work I mentioned last week to run our smoke-tests from a different data-centre is effectively complete.

We’re running the new system in parallel with the existing system of checks for a bit to ensure that they behave the same before switching off the existing system.

I’m consistently impressed by the thought that’s clearly gone into Concourse CI’s domain concepts and API.

Two

I’ve almost finished the write up for the “Mental Health in Software Development” session that I ran last month.

It turns out that writing up an hour and a half of post-its into a structured blogpost takes a while, but I’m pleased with how it’s coming together.

I’ve had a couple of realisations during the reflection on the session, especially around intersectionality with regards to mental health and other kinds of diversity, but more on that in the post.

Three

Wednesday was a bit of a rough day — for whatever reason, I was off my game and wasn’t as well equipped as I would have liked to contribute during a 2hour mapping session.

I was annoyed at myself about this for a while, but going for a brief walk helped clear my head. In future I need to be better about just stepping out of the room if I need some air, much as I would be okay with it if someone else asked to.

Four

I have to sign some of my commits at work to verify that they’ve come from me. It’s been a while since I’ve needed to do this, and it’s got me thinking about a problem I ran into while pairing/mobbing.

git supports signing commits and tags, but with a single key. How can you verify (a) that a single commit was paired/mobbed on and (b) the people whose names are on the commit were actually involved?

I’m playing with a rough idea of “primary committer signs the commit, secondary committers create and sign unique tags”, with something like a pre-receive-hook on the git server, which has a list of allowed public keys and rejects pushes where there are <2 verified authors.

The disadvantage of this is that tags are deliberately lightweight and easy to create, and so just as easy to delete. I suppose that this could be enforced on the server-side too (forbid deletions of tags).

The tag itself would have to have a unique id, something that uses the short-hash of the commit like a8b7c6-co-author-1. If tags were being used for release management, this might "pollute" your tag history to an unacceptable extent.

However, these might be acceptable trade-offs for being able to verify that everyone who claims to have worked on the commit did actually do so (assuming decent protections of each person’s signing key).

Originally published at https://blog.probablyfine.co.uk on August 3, 2019.

July 30, 2019

Gustaf Erikson (gerikson)

Fotografiska, July 2019 July 30, 2019 03:41 PM

Mandy Barker - Sea of Artifacts

An earnest exploration of plastic in seas and waterways. For some reason this exhibit was in a room even more dimly lit than usual at Fotografiska - and that’s saying something. Sadly nothing that’s not been said or done many times before.

Scarlett Hooft Graafland - Vanishing Traces

This grew on me. Staged images in different environments with surreal components (yet of course it’s all “analog” and there’s no manipulation). Impressive for the visuals and conceptions alone.

Vincent Peters - Light Within

Peters photographs our modern Hollywood idols like they where 1940s Hollywood idols.

James Nachtwey - Memoria

A visual onslaught of images from famines, wars, catastrophes and field hospitals. Almost unbearable in its intensity. I became numb after just a few walls. It depresses me that future epigones will have no shortage of similar subjects, all without having to travel that far.

July 29, 2019

Jeremy Morgan (JeremyMorgan)

How to Nail Your Next Coding Interview July 29, 2019 04:45 PM

The room is silent except for the buzzing of the fluorescent lights. The judges across the table are staring at you, expressionless. Some have pen and paper, some don't. They're all staring at you. Your mouth is so dry it feels like you've been eating sawdust all day. You grab the marker and head for the whiteboard. One judge is staring at a laptop. It's time to show them a quicksort.

I've been on both sides of the coding interview. I've interviewed for jobs, sometimes I got the offer, sometimes I didn't. Sometimes I nailed the whiteboard tests and didn't get a call, and vice versa. I've interviewed probably hundreds of people in my career and I did my best to make candidates feel comfortable, but many managers won't. They will try to trip you up, make you choke. We can argue about the effectiveness of a whiteboard interview later, but they will happen.

Let's look at some tips to rock your next coding interview. You can set yourself up for success and nail it. Don't be mistaken, this isn't a set of "hacks", "tricks", or some kind of brain dump. Don't use sneaky tactics or tricks to crack your way into a position you aren't qualified for. They'll just fire you later. Follow this path and you will nail the interview because you'll be a better developer.

What They're Looking for

Here's what most of the interviewers will be looking for, in no particular order:

  • Problem Solving - How well can you solve problems, and more importantly: what is your process. Many tests will look for this. 

  • Coding Skills - You'll need to be ruthlessly good here. I've handed people laptops and told them to write something for me. You can tell how much actual coding people have done by watching them do it. 

  • Technical Knowledge - This is where the trivia questions come in, but your interviewer wants to know just how technical you are. 

  • Experience - This is where you talk about your past projects. Interviewers want to hear your war stories, and what you learned from them. 

  • Culture Fit - I can't help you much with this one. This is where they see how you'll fit into a team. There are ways you can improve in this area..

So what do you need to do to get prepared? Let's dive in. 

Stage 1: Laying the Foundation

So here's what you need to be doing way before any interview. Days, months or weeks before you need to build some core skills. 

Learn Computer Science Basics

You need to get the basics of Computer Science down. You don't have to be Donald Knuth here, but you need to know the theory, language, and idioms. This is the bare minimum for an interview. If an interviewer starts casually mentioning a binary tree in the interview you better know what they're talking about. 

  1. Review The Basics of Computer Science Tutorial - some of this may be pretty basic to you but it gives you a framework of things to pay attention to. Where do your strengths lie? If you find any areas where you are weak at, work to build your knowledge and skills in that area. 

  2. Review Teach Yourself Computer Science - Again we're talking basics foundational stuff that you may be lacking in. You need to know the basics and speak the language. 

  3. You can even Take a course on Computer Science 101 from Stanford, for free.

Learn Different Algorithms

Algorithms run the world and if you're a developer you'll need to know them. So how do you get good at algorithms? It's not black magic or a secret art. 

According to Geeks for Geeks these are the top 10 algorithms in interview questions:

  • Graph
  • Linked List
  • Dynamic Programming
  • Sorting and Searching
  • Tree / Binary Search Tree
  • Number Theory
  • BIT Manipulation
  • String / Array 

It seems pretty accurate to me. It's super helpful to know this. What are you great at in this list? What are you weak at? This link is excellent for getting a high-level view of each and some examples. 

These courses will help you absolutely dominate in this area:

In about 6 hours you'll be able to understand and really talk the talk when it comes to algorithms. 

Action: Study this stuff. Learn it. Know it.

Stage 2: Practice Practice Practice

Here's another thing you need to make a part of your routine: practice. You need to practice this stuff a few times a week or more to really get good. 

The more you practice the better you'll do in any whiteboard situation. Most of the time they ask for pseudocode but if you really put your practice in you can write real compilable code on a whiteboard without blinking. 

  1. Check out the HackerRank Interview Preparation Kit - This is where the rubber meets the road with all that algorithm knowledge you now have. Put it to work with real examples. 

  2. Start participating in HackerRank Challenges - This is how you can really apply your knowledge in different areas. This is the ultimate whiteboard practice area.      
    It's broken down like this:

    Do an exercise a day when you can. Work your way through many of the challenges. If you just do a challenge a day for 30 days you can nail a whiteboard interview and will become a better coder. I promise you. 

  1. Sign up for LeetCode and start participating in challenges - Nothing sharpens your skills like competing with others. There are tons of challenges and fun stuff here. 

  2. Check out Project Euler and start writing code to solve the problems there. Project Euler is a set of math and programming problems that really challenge your problem-solving abilities. Use code to solve your problems here and your skills will grow. 

Action: Write some code. A lot of it. Get ruthlessly good at it.

Stage 3: Build Your Public Profile

You need to have work you've done online and accessible. Every recruiter, manager, or someone interested in you will Google you. Make sure they find your work. 

  1. Put your stuff Online and start a GitHub account if you don't already have one. Put all of your code up there. Your personal projects, stuff you write for tutorials, all of it.

  2. Create accounts on sites like JSFiddle and CodePen - This is mostly for web developers but it creates a good spot where people can find your work. 

  3. Create a Blog - It doesn't matter if it's WordPress, GitHub pages, or a custom hosted server setup like my blog you should create a blog that talks about what you're working on, what you're learning, and what you can teach. 

Note: I have thrown away resumes of people when I couldn't find their GitHub or any public work. This shows that they aren't enthusiastic and managers who look for passionate people will look for what you've posted publicly. Put your work out there no matter how good it is. 

Action: Sign up and start making your code and projects public.

Stage 4: Prepare for Interviews

Preparation ultimately determines your success. Nobody succeeds without preparation. Here's how you can prepare for your interview and kick the ball through the uprights. 

  1. Pick up a copy of Cracking the Code Interview - This book is the bible of coding interviews. Like this article it doesn't tell you how to cheat or shortcut your way into getting hired - it gives you a great framework to upgrade your skills. Practice questions are included and it gives you all the tools you need to really sharpen your skillset. 

  2. Take this Course on preparing for a job interview - This gives you the information you need to really prepare yourself. 

    This course covers:

    • Job Interview Basics
    • Algorithm Based Questions
    • Typical Questions
    • Computer Science Questions
    • Getting Experience  It really covers what you need to know to succeed in 2 1/2 hours. Well worth it. 
  3. Watch this video about how to prepare yourself for Developer Job Interviews - This is created by John Sonmez, who you may already know. There's no question that John is a successful developer and he's more than willing to share what has worked for him. It's well worth checking out. 

  4. Exercise - Ok I know this will sound silly, but here's something that will give you an extra edge. No matter what time of day your interview is, hit the gym or do some cardio an hour or two before the interview. This will ensure:

    • You are refreshed and energetic
    • You have oxygen flowing through your blood
    • Your muscles will be relaxed

    A good hard workout will make sure you are charged and ready to go for your interview. You don't want to seem tired or lethargic in your interview. You want to be at your physical and mental BEST. 

Preparation is everything. The more you prepare the better you'll feel on the interview day.

Action: Start training like a boxer. Get fight ready.

Conclusion

Coding interviews can be brutal. You can take the sting off them by doing the following:

  • Learning 
  • Sharpening your skills
  • Practicing

These things will ensure your success. Trust me, after working on crazy problems in LeetCode and HackerRank the whiteboard tests are so much easier. If you do this alone, you'll be successful. 

If you have comments feel free to share or yell at me on Twitter and we can discuss it. 

Sevan Janiyan (sevan)

Something blogged (on pkgsrcCon 2019) July 29, 2019 02:55 PM

pkgsrcCon 2019 was held in Cambridge this year. The routine as usual was social on the Friday evening, day of talks on the Saturday, hacking on things on the Sunday.Armed with a bag of bits to record the talks, I headed up to Cambridge on Friday evening to meet up with sborrill before heading to …

July 28, 2019

Andreas Zwinkau (qznc)

Sacrifice of Scar July 28, 2019 12:00 AM

Short Lion King fanfic

Read full article!

July 24, 2019

Jeremy Morgan (JeremyMorgan)

Creating Trimmed Self Contained Executables in .NET Core July 24, 2019 02:09 PM

I'm going to show you a cool new feature in .NET Core 3.0.

Let's say you want to create a simple, lean executable you can build and drop on to a server.

For an example we'll create a console app that opens a line of text, reads it and displays the output.

First, let's create a new .NET Core app:

dotnet new console

This will scaffold a new console application.

Now run it:

dotnet run

It should look something like this:

Creating Trimmed Self Contained Executables in Net Core

I'm on a Mac here, but it doesn't matter as long as your development box has the .NET Core CLI Installed.

This displays "Hello World" on the console.

Now, lets create a file called file.txt:

this is a file! With some lines whatever

Doesn't matter what you put in here, as long as it has some text in it.

Next we'll create a something that will read those lines and display them. Remove the "Hello World!" code and replace it with this:

``` string[] lines = System.IO.File.ReadAllLines(@"test.txt");

    foreach (string line in lines)
    {
        Console.WriteLine("\t" + line);
    }

Console.WriteLine("Press any key to exit."); System.Console.ReadKey(); ```

This is pretty much your basic cookie cutter code for:

  • opening up a file
  • reading it into a string array
  • loop through the array line by line
  • print each line
  • exit

Pretty simple stuff. When I run it on my machine it looks like this:

Creating Trimmed Self Contained Executables in Net Core

And that's great. But I'm on a Mac here, what if I want it to run on a Windows Machine? Linux? No problem, this is .NET Core right? We'll just publish it to multiple targets.

But what if .NET Core isn't installed on the machine?

What if I just want a simple executable I can run to read this file without a pile of files or .Net core installed?

Publishing in .Net Core

Let's back up a little. .NET Core has had publishing profiles for a long time. The idea behind "target" publishing is one of the biggest selling points of the platform. Build your app, then publish it for a specific target, Windows, OSX, or Linux.

You can publish it a few different ways

  • Framework Dependent Deployment - This means relies on a shared version of .NET Core that's installed on the Computer/Server.
  • Self Contained Deployment - This doesn't rely on .Net Core being installed on the server. All components are included with the package (tons of files usually).
  • Framework Dependent Executables - This is very similar to a framework dependent deployment, but it creates executables that are platform specific, but require the .NET Core libraries.

Ok, so what's this cool new feature I'm going to show?

Well, when you do a self contained deployment it's cool because you don't need the runtime installed, but it ends up looking something like this:

Creating Trimmed Self Contained Executables in Net Core

This is the application we just built published as a Self Contained Deployment for Windows. Yikes.

Let's say you wanted to share this file reader application, and asking someone to copy all these files into a folder to run something to read a text file. It's silly.

New Feature: Self Contained Executables

So to build that self contained executable I ran the following command:

dotnet publish -c release -r win10-x64

This should look pretty familiar to you if you've done it before. But .NET Core 3.0 has a cool new feature:

dotnet publish -r win-x64 -c Release /p:PublishSingleFile=true Using this flag, it will build something a little different:

Creating Trimmed Self Contained Executables in Net Core

That's MUCH better. It's now a single exe and .pdb. If we look at the info, it's not super small:

Creating Trimmed Self Contained Executables in Net Core

But it includes the .NET Core runtime with it as well. And here's another cool feature.

dotnet publish -r win-x64 -c Release /p:PublishSingleFile=true /p:PublishTrimmed=true

So in our example it doesn't change the size, but if you have a large complex application with a lot of libraries, if you just publish it to a single file, it can get HUGE. By adding the PublishTrimmed flag it will only extract the libraries you need to run the application.

So when we copy the files to a Windows 10 machine, we have a nice small package:

Creating Trimmed Self Contained Executables in Net Core

And we run it and it works! Without .NET Core!

Creating Trimmed Self Contained Executables in Net Core

and if I change my target:

dotnet publish -r linux-x64 -c Release /p:PublishSingleFile=true /p:PublishTrimmed=true

I can run it on a Linux server without .NET Core just as easily:

Creating Trimmed Self Contained Executables in Net Core

Just remember on a Linux machine you won't need the .NET Core runtime, but you will need the Prerequisites for .NET Core on Linux installed.

Conclusion

So this is a cool feature of .NET Core 3.0. If you want to build trimmed down self contained executables for any of the platforms, you can do it easily with a couple of flags.

This is great for those stupid simple things: console apps, data readers, microservices, or whatever you want to build and just drop on a machine with no hassle. I thought it was a cool feature to show.

If you want to master .NET Core, check out the ASP.Net Core Path on Pluralsight. The courses go pretty in depth and are a great way to ramp up.

Yell at me on Twitter with questions or comments!



What is your .NET Core IQ?


My ASP.NET Core IQ is 200. Can you beat it? Take the test now to find out your score!!

Benjamin Pollack (gecko)

Falsehoods Programmers Believe About Cats July 24, 2019 12:43 PM

Inspired by Falsehoods Programmers Believe About Dogs, I thought it would be great to offer you falsehoods programmers believe about mankind’s other best friend. But since I don’t know what that is, here’s instead a version about cats.

  1. Cats would never eat your face.
  2. Cats would never eat your face while you were alive.1
  3. Okay, cats would sometimes eat your face while you’re alive, but my cat absolutely would not.
  4. Okay, fine. At least I will never run out of cat food.
  5. You’re kidding me.
  6. There will be a time when your cat knows enough not to vomit on your computer.
  7. There will be a time when your cat cares enough not to vomit on your computer.
  8. At the very least, if your cat begins to vomit on your computer and you try to move it to another location, your cat will allow you to do so.
  9. When your cat refuses to move, it will at least not manage to claw your arm surprisingly severely while actively vomiting.
  10. Okay, but at least they won’t attempt to chew the power cord while vomiting and clawing your hand, resulting in both of you getting an electric shock.
  11. …how the hell are you even alive?2
  12. Cats enjoy belly rubs.
  13. Some cats enjoy belly rubs.
  14. Cats reliably enjoy being petted.
  15. Cats will reliably tell you when they no longer enjoying being petted.
  16. Cats who trust their owners will leave suddenly when they’re done being petted, but at least never cause you massive blood loss.
  17. Given all of the above, you should never adopt cats.
  18. You are insane.

Happy ten years in your forever home, my two scruffy kitties. Here’s to ten more.


  1. Here, ask Dewey, he knows more about it than I do. [return]
  2. Because, while my cat has absolutely eaten through a power cord, this is an exaggeration. The getting scratched while trying to get my cat not to puke on a computer I was actively using happened at a different time from the power cord incident. Although this doesn’t answer the question how she is alive. [return]

July 20, 2019

Awn Umar (awn)

memory retention attacks July 20, 2019 12:00 AM

In my post on implementing an in-memory encryption scheme to protect sensitive information, I referenced a mitigation strategy called a Boojum. It is described by Bruce Schneier, Niels Ferguson and Tadayoshi Kohno in their book, Cryptography Engineering: Design Principles and Practical Applications.

A number of people asked me about the mechanisms of the attack and the scheme, so I am including the relevant parts here. It is an excellent resource and I recommend that you go and buy it.

page one page two page three

[32] Giovanni Di Crescenzo, Niels Ferguson, Russel Impagliazzo, and Markus Jakobsson. How to Forget a Secret. In Christoph Meinel and Sophie Tison, editors, STACS 99, volume 1563 of Lecture Notes in Computer Science, pages 500-509. Springer-Verlag, 1999.

[57] Peter Gutmann. Secure Deletion of Data from Magnetic and Solid-State Memory. In USENIX Security Symposium Proceedings, 1996. Available from http://www.cs.auckland.ac.nz/~pgut001/pubs/secure_del.html.

[59] J. Alex Halderman, Seth D. Schoen, Nadia Heninger, William Clarkson, William Paul, Joseph A. Calandrino, Ariel J. Feldman, Jacob Appelbaum, and Edward W. Felten. Lest We Remember: Cold Boot Attacks on Encryption Keys. In USENIX Security Symposium Proceedings, pages 45-60, 2008.

July 19, 2019

Jeremy Morgan (JeremyMorgan)

You Can Get the Source Code for Apollo 11 and Take a Course on It July 19, 2019 04:47 AM

In software development you'll hear the term "moon shot". If something is a "moon shot" it's something that's extraordinarily difficult, like landing on the moon. We say this about some app doing something cool, but what about the software that... landed us on the moon? What was the original "moon shot" all about? 

The Software That Put Us on the Moon

Margaret Hamilton

Meet Margaret Hamilton ). She was the director of Software engineering at MIT Instrumentation Laboratory, which was contracted to build the onboard software for the Apollo Space Program. This is her standing with the stack of source code used to launch us to the moon. Today we complain when Visual Studio runs slow. 

If you're a developer of any kind, you owe a lot of thanks to her, since "Software Engineer" wasn't even a term before she came along. 

So imagine having to write software that had to work the first time it ran, and run perfectly, or people would die. Sounds like a lot of pressure right? Margaret and her team did it. They built it, and it worked. It succeeded. 

They made a piece of machinery land on the moon with a computer that had about .08 percent of the processing power of an iPhone. 

You can get a lot more background and details here.

The Source Code

A few years ago, Ron Burkey released the source code for Apollo 11. He put a ton of work into this. This is the source code for the Apollo Guidance Computer, or AGC. 

Not only can you download the source code, but he created a simulator for the AGC, at the Virtual AGC Page. You can dig deep into the systems and how they work, trust me it's a rabbit hole for geeks. Tons of awesome stuff here. There's even a kinder, gentler introduction to the AGC you can check out to get familiar.

The AGC source code is written in assembler, which is foreign to most of us. I've played around enough with x86 assembler to know it's not my calling, but perusing through a lot of this source code, you can piece together how some of this stuff works.  

Comanche and Luminary

If you dig into the code, you'll see it's divided into two parts, Comanche and Luminary. 

Comanche is the software for the Command Module and Luminary is the Lunar Module

The Command Module was the cone that contained the crew and vital equipment, and was the vessel returned back to earth. 

Apollo 11 Source Code

The Lunar Module well, it was the module that landed on the moon. 

Apollo 11 Source Code

It's very interesting to see how these systems interact. When you look through the source code you can see a lot of cool hints how everything works. 

The DSKY 

The DSKY was the user interface for the computer. You could enter commands through a calculator-like interface and it was connected directly to the AGC. The commands contained a verb and a noun for each command. 

Apollo 11 Source Code

If you dig deep enough into the source code you'll see a lot of references to the DSKY and commands related to it. It was a marvel of engineering for its time. Of course, there is a DSKY simulator if you want to play around. 

How Do I Know so Much About This? 

I may sound like a seasoned expert here, but I just took this free course on the code of the Apollo 11, then started digging in the code and researching stuff. 

Apollo 11 Source Code

Click Here to Take This Course for FREE

For the 50th anniversary of the Apollo 11 Moon Landing, Simon Allardice created this awesome course on Pluralsight exploring the AGC. It's a quick course, and it won't teach you how to write AGC specific assembler code, but it will give you a good idea of how things worked back then and how the whole thing fits together. After watching this you'll get a better understanding of the source code, and start looking for easter eggs. 

Conclusion

Ok if you're a nerd like me, you love stuff like this. Obviously this isn't some kind of actively developed code where you can do pull requests and add features. Nobody will hire you because you put AGC knowledge on your resume. But it sure is fun to go through this code and figure out how things worked, and look at the comical comments in the code. It's fascinating and well worth the time to explore if you're genuinely curious about this kind of stuff.

You can always yell at me on Twitter if you'd like to discuss it further. 

Enjoy! 



What is your DevOps IQ?

what's your devops score


My Devops Skill IQ is 232. Can you beat it? Take the test now to find out your Devops IQ score!!

Joe Nelson (begriffs)

History and effective use of Vim July 19, 2019 12:00 AM

This article is based on historical research and on simply reading the Vim user manual cover to cover. Hopefully these notes will help you (re?)discover core functionality of the editor, so you can abandon pre-packaged vimrc files and use plugins more thoughtfully.

physical books

physical books

To go beyond the topics in this blog post, I’d recommend getting a paper copy of the manual and a good pocket reference. I couldn’t find any hard copy of the official Vim manual, and ended up printing this PDF using printme1.com. The PDF is a printer-friendly version of the files $VIMRUNTIME/doc/usr_??.txt distributed with the editor. For a convenient list of commands, I’d recommend the vi and Vim Editors Pocket Reference.

Table of Contents

History

Birth of vi

Vi commands and features go back more than fifty years, starting with the QED editor. Here is the lineage:

  • 1966 : QED (“Quick EDitor”) in Berkeley Timesharing System
  • 1969 Jul: moon landing (just for reference)
  • 1969 Aug: QED -> ed at AT&T
  • 1976 Feb: ed -> em (“Editor for Mortals”) at Queen Mary College
  • 1976 : em -> ex (“EXtended”) at UC Berkeley
  • 1977 Oct: ex gets visual mode, vi

hard copy terminal

You can discover the similarities all the way between QED and ex by reading the QED manual and ex manual. Both editors use a similar grammar to specify and operate on line ranges.

Editors like QED, ed, and em were designed for hard-copy terminals, which are basically electric typewriters with a modem attached. Hard-copy terminals print system output on paper. Output could not be changed once printed, obviously, so the editing process consisted of user commands to update and manually print ranges of text.

video terminal

By 1976 video terminals such as the ADM-3A started to be available. The Ex editor added an “open mode” which allowed intraline editing on video terminals, and a visual mode for screen oriented editing on cursor-addressible terminals. The visual mode (activated with the command “vi”) kept an up-to-date view of part of the file on screen, while preserving an ex command line at the bottom of the screen. (Fun fact: the h,j,k,l keys on the ADM-3A had arrows drawn on them, so that choice of motion keys in vi was simply to match the keyboard.)

Learn more about the journey from ed to ex/vi in this interview with Bill Joy. He talks about how he made ex/vi, and some things that disappointed him about it.

Classic vi is truly just an alter-ego of ex – they are the same binary, which decides to start in ex mode or vi mode based on the name of the executable invoked. The legacy of all this history is that ex/vi is refined by use, requires scant system resources, and can operate under limited bandwidth communication. It is also available on most systems and fully specified in POSIX.

From vi to vim

Being a derivative of ed, the ex/vi editor was intellectual property of AT&T. To use vi on platforms other than Unix, people had to write clones that did not share in the original codebase.

Some of the clones:

  • nvi - 1980 for 4BSD
  • calvin - 1987 for DOS
  • vile - 1990 for DOS
  • stevie - 1987 for Atari ST
  • elvis - 1990 for Minix and 386BSD
  • vim - 1991 for Amiga
  • viper - 1995 for Emacs
  • elwin - 1995 for Windows
  • lemmy - 2002 for Windows

We’ll be focusing on that little one in the middle: vim. Bram Moolenaar wanted to use vi on the Amiga. He began porting Stevie from the Atari and evolving it. He called his port “Vi IMitation.” For a full first-hand account, see Bram’s interview with Free Software Magazine.

By version 1.22 Vim was rechristened “Vi IMproved,” matching and surpassing features of the original. Here is the timeline of the next major versions, with some of their big features:

1991 Nov 2 Vim 1.14: First release (on Fred Fish disk #591).
1992 Vim 1.22: Port to Unix. Vim now competes with Vi.
1994 Aug 12 Vim 3.0: Support for multiple buffers and windows.
1996 May 29 Vim 4.0: Graphical User Interface (largely by Robert Webb).
1998 Feb 19 Vim 5.0: Syntax coloring/highlighting.
2001 Sep 26 Vim 6.0: Folding, plugins, vertical split.
2006 May 8 Vim 7.0: Spell check, omni completion, undo branches, tabs.
2016 Sep 12 Vim 8.0: Jobs, async I/O, native packages.

For more info about each version, see e.g. :help vim8. To see plans for the future, including known bugs, see :help todo.txt.

Version 8 included some async job support due to peer pressure from NeoVim, whose developers wanted to run debuggers and REPLs for their web scripting languages inside the editor.

Vim is super portable. By adapting over time to work on a wide variety of platforms, the editor was forced to keep portable coding habits. It runs on OS/390, Amiga, BeOS and BeBox, Macintosh classic, Atari MiNT, MS-DOS, OS/2, QNX, RISC-OS, BSD, Linux, OS X, VMS, and MS-Windows. You can rely on Vim being there no matter what computer you’re using.

In a final twist in the vi saga, the original ex/vi source code was finally released in 2002 under a BSD free software license. It is available at ex-vi.sourceforge.net.

Let’s get down to business. Before getting to odds, ends, and intermediate tricks, it helps to understand how Vim organizes and reads its configuration files.

Configuration hierarchy

I used to think, incorrectly, that Vim reads all its settings and scripts from the ~/.vimrc file alone. Browsing random “dotfiles” repositories can reinforce this notion. Quite often people publish monstrous single .vimrc files that try to control every aspect of the editor. These big configs are sometimes called “vim distros.”

In reality Vim has a tidy structure, where .vimrc is just one of several inputs. In fact you can ask Vim exactly which scripts it has loaded. Try this: edit a source file from a random programming project on your computer. Once loaded, run

:scriptnames

Take time to read the list. Try to guess what the scripts might do, and note the directories where they live.

Was the list longer than you expected? If you have installed loads of plugins the editor has a lot to do. Check what slows down the editor most at startup by running the following and look at the start.log it creates:

vim --startuptime start.log name-of-your-file

Just for comparison, see how quickly Vim starts without your existing configuration:

vim --clean --startuptime clean.log name-of-your-file

To determine which scripts to run at startup or buffer load time, Vim traverses a “runtime path.” The path is a comma-separated list of directories that each contain a common structure. Vim inspects the structure of each directory to find scripts to run. Directories are processed in the order they appear in the list.

Check the runtimepath on your system by running:

:set runtimepath

My system contains the following directories in the default value for runtimepath. Not all of them even exist in the filesystem, but they would be consulted if they did.

~/.vim
The home directory, for personal preferences.
/usr/local/share/vim/vimfiles
A system-wide Vim directory, for preferences from the system administrator.
/usr/local/share/vim/vim81
Aka $VIMRUNTIME, for files distributed with Vim.
/usr/local/share/vim/vimfiles/after
The “after” directory in the system-wide Vim directory. This is for the system administrator to overrule or add to the distributed defaults.
~/.vim/after
The “after” directory in the home directory. This is for personal preferences to overrule or add to the distributed defaults or system-wide settings.

Because directories are processed by their order in line, the only thing that is special about the “after” directories is that they are at the end of the list. There is nothing magical about the word “after.”

When processing each directory, Vim looks for subfolders with specific names. To learn more about them, see :help runtimepath. Here is a selection of those we will be covering, with brief descriptions.

plugin/
Vim script files that are loaded automatically when editing any kind of file. Called “global plugins.”
autoload/
(Not to be confused with “plugin.”) Scripts in autoload contain functions that are loaded only when requested by other scripts.
ftdetect/
Scripts to detect filetypes. They can base their decision on filename extension, location, or internal file contents.
ftplugin/
Scripts that are executed when editing files with known type.
compiler/
Definitions of how to run various compilers or linters, and of how to parse their output. Can be shared between multiple ftplugins. Also not applied automatically, must be called with :compiler
pack/
Container for Vim 8 native packages, the successor to “Pathogen” style package management. The native packaging system does not require any third-party code.

Finally, ~/.vimrc is the catchall for general editor settings. Use it for setting defaults that can be overridden for particular file types. For a comprehensive overview of settings you can choose in .vimrc, run :options.

Third-party plugins

Plugins are simply Vim scripts that must be put into the correct places in the runtimepath in order to execute. Installing them is conceptually easy: download the file(s) into place. The challenge is that it’s hard to remove or update some plugins because they litter subdirectories in the runtimepath with their scripts, and it can be hard to tell which plugin is responsible for which files.

“Plugin managers” evolved to address this need. Vim.org has had a plugin registry going back at least as far as 2003 (as identified by the Internet Archive). However it wasn’t until about 2008 that the notion of a plugin manager really came into vogue.

These tools add plugins’ separate directories to Vim’s runtimepath, and compile help tags for plugin documentation. Most plugin managers also install and update plugin code from the internet, sometimes in parallel or with colorful progress bars.

In chronological order, here is the parade of plugin managers. I based the date ranges on earliest and latest releases of each, or when no official releases are identified, on the earliest and latest commit dates.

  • Mar 2006 - Jul 2014 : Vimball (A distribution format and associated Vim commands)
  • Oct 2008 - Dec 2015 : Pathogen (Deprecated in favor of native vim packages)
  • Aug 2009 - Dec 2009 : Vimana
  • Dec 2009 - Dec 2014 : VAM
  • Aug 2010 - Nov 2010 : Jolt
  • Oct 2010 - Nov 2012 : tplugin
  • Oct 2010 - Feb 2014 : Vundle (Discontinued after NeoBundle ripped off code)
  • Mar 2012 - Mar 2018 : vim-flavor
  • Apr 2012 - Mar 2016 : NeoBundle (Deprecated in favor of dein)
  • Jan 2013 - Aug 2017 : infect
  • Feb 2013 - Aug 2016 : vimogen
  • Oct 2013 - Jan 2015 : vim-unbundle
  • Dec 2013 - Jul 2015 : Vizardry
  • Feb 2014 - Oct 2018 : vim-plug
  • Jan 2015 - Oct 2015 : enabler
  • Aug 2015 - Apr 2016 : Vizardry 2
  • Jan 2016 - Jun 2018 : dein.vim
  • Sep 2016 - Present : native in Vim 8
  • Feb 2017 - Sep 2018 : minpac
  • Mar 2018 - Mar 2018 : autopac
  • Feb 2017 - Jun 2018 : pack
  • Mar 2017 - Sep 2017 : vim-pck
  • Sep 2017 - Sep 2017 : vim8-pack
  • Sep 2017 - May 2019 : volt
  • Sep 2018 - Feb 2019 : vim-packager
  • Feb 2019 - Feb 2019 : plugpac.vim

The first thing to note is the overwhelming variety of these tools, and the second is that each is typically active for about four years before presumably going out of fashion.

The most stable way to manage plugins is to simply use Vim 8’s built-in functionality, which requires no third-party code. Let’s walk through how to do it.

First create two directories, opt and start, within a pack directory in your runtimepath.

mkdir -p ~/.vim/pack/foobar/{opt,start}

Note the placeholder “foobar.” This name is entirely up to you. It classifies the packages that will go inside. Most people throw all their plugins into a single nondescript category, which is fine. Pick whatever name you like; I’ll continue to use foobar here. You could theoretically create multiple categories too, like ~/.vim/pack/navigation and ~/.vim/pack/linting. Note that Vim does not detect duplication between categories and will double-load duplicates if they exist.

Packages in “start” get loaded automatically, whereas those in “opt” won’t load until specifically requested in Vim with the :packadd command. Opt is good for lesser-used packages, and keeps Vim fast by not running scripts unnecessarily. Note that there isn’t a counterpart to :packadd to unload a package.

For this example we’ll add the “ctrlp” fuzzy find plugin to opt. Download and extract the latest release into place:

curl -L https://github.com/kien/ctrlp.vim/archive/1.79.tar.gz \
	| tar zx -C ~/.vim/pack/foobar/opt

That command creates a ~/.vim/pack/foobar/opt/ctrlp.vim-1.79 folder, and the package is ready to use. Back in vim, create a helptags index for the new package:

:helptags ~/.vim/pack/foobar/opt/ctrlp.vim-1.79/doc

That creates a file called “tags” in the package’s doc folder, which makes the topics available for browsing in Vim’s internal help system. (Alternately you can run :helptags ALL once the package has been loaded, which takes care of all docs in the runtimepath.)

When you want to use the package, load it (and know that tab completion works for plugin names, so you don’t have to type the whole name):

:packadd ctrlp.vim-1.79

Packadd includes the package’s base directory in the runtimepath, and sources its plugin and ftdetect scripts. After loading ctrlp, you can press CTRL-P to pop up a fuzzy find file matcher.

Some people keep their ~/.vim directory under version control and use git submodules for each package. For my part, I simply extract packages from tarballs and track them in my own repository. If you use mature packages you don’t need to upgrade them often, plus the scripts are generally small and don’t clutter git history much.

Backups and undo

Depending on user settings, Vim can protect against four types of loss:

  1. A crash during editing (between saves). Vim can protect against this one by periodically saving unwritten changes to a swap file.
  2. Editing the same file with two instances of Vim, overwriting changes from one or both instances. Swap files protect against this too.
  3. A crash during the save process itself, after the destination file is truncated but before the new contents have been fully written. Vim can protect against this with a “writebackup.” To do this, it writes to a new file and swaps it with the original on success, in a way that depends on the “backupcopy” setting.
  4. Saving new file contents but wanting the original back. Vim can protect against this by persisting the backup copy of the file after writing changes.

Before examining sensible settings, how about some comic relief? Here are just a sampling of comments from vimrc files on GitHub:

  • “Do not create swap file. Manage this in version control”
  • “Backups are for pussies. Use version control”
  • “use version control FFS!”
  • “We live in a world with version control, so get rid of swaps and backups”
  • “don’t write backup files, version control is enough backup”
  • “I’ve never actually used the VIM backup files… Use version control”
  • “Since most stuff is on version control anyway”
  • “Disable backup files, you are using a version control system anyway :)”
  • “version control has arrived, git will save us”
  • “disable swap and backup files (Always use version control! ALWAYS!)”
  • “Turn backup off, since I version control everything”

The comments reflect awareness of only the fourth case above (and the third by accident), whereas the authors generally go on to disable the swap file too, leaving one and two unprotected.

Here is the configuration I recommend to keep your edits safe:

" Protect changes between writes. Default values of
" updatecount (200 keystrokes) and updatetime
" (4 seconds) are fine
set swapfile
set directory^=~/.vim/swap//

" protect against crash-during-write
set writebackup
" but do not persist backup after successful write
set nobackup
" use rename-and-write-new method whenever safe
set backupcopy=auto
" patch required to honor double slash at end
if has("patch-8.1.0251")
	" consolidate the writebackups -- not a big
	" deal either way, since they usually get deleted
	set backupdir^=~/.vim/backup//
end

" persist the undo tree for each file
set undofile
set undodir^=~/.vim/undo//

These settings enable backups for writes-in-progress, but do not persist them after successful write because version control etc etc. Note that you’ll need to mkdir ~/.vim/{swap,undodir,backup} or else Vim will fall back to the next available folder in the preference list. You should also probably chmod the folders to keep the contents private, because the swap files and undo history might contain sensitive information.

One thing to note about the paths in our config is that they end in a double slash. That ending enables a feature to disambiguate swaps and backups for files with the same name that live in different directories. For instance the swap file for /foo/bar will be saved in ~/.vim/swap/%foo%bar.swp (slashes escaped as percent signs). Vim had a bug until a fairly recent patch where the double slash was not honored for backupdir, and we guard against that above.

We also have Vim persist the history of undos for each file, so that you can apply them even after quitting and editing the file again. While it may sound redundant with the swap file, the undo history is complementary because it is written only when the file is written. (If it were written more frequently it might not match the state of the file on disk after a crash, so Vim doesn’t do that.)

Speaking of undo, Vim maintains a full tree of edit history. This means you can make a change, undo it, then redo it differently and all three states are recoverable. You can see the times and magnitude of changes with the :undolist command, but it’s hard to visualize the tree structure from it. You can navigate to specific changes in that list, or move in time with :earlier and :later which take a time argument like 5m, or the count of file saves, like 3f. However navigating the undo tree is an instance when I think a plugin – like undotreeis warranted.

Enabling these disaster recovery settings can bring you peace of mind. I used to save compulsively after most edits or when stepping away from the computer, but now I’ve made an effort to leave documents unsaved for hours at a time. I know how the swap file works now.

Some final notes: keep an eye on all these disaster recovery files, they can pile up in your .vim folder and use space over time. Also setting nowritebackup might be necessary when saving a huge file with low disk space, because Vim must otherwise make an entire copy of the file temporarily. By default the “backupskip” setting disables backups for anything in the system temp directory.

Vim’s “patchmode” is related to backups. You can use it in directories that aren’t under version control. For instance if you want to download a source tarball, make an edit and send a patch over a mailing list without bringing git into the picture. Run :set patchmod=.orig and any file ‘foo’ Vim is about to write will be backed up to ‘foo.orig’. You can then create a patch on the command line between the .orig files and the new ones.

Include and path

Most programming languages allow you to include one module or file from another. Vim knows how to track program identifiers in included files using the configuration settings path, include, suffixesadd, and includeexpr. The identifier search (see :help include-search) is an alternative to maintaining a tags file with ctags for system headers.

The settings for C programs work out of the box. Other languages are supported too, but require tweaking. That’s outside the scope of this article, see :help include.

If everything is configured right, you can press [i on an identifier to display its definition, or [d for a macro constant. Also when you press gf with the cursor on a filename, Vim searches the path to find it and jump there. Because the path also affects the :find command, some people have the tendency to add ‘**/*’ or commonly accessed directories to the path in order to use :find like a poor man’s fuzzy finder. Doing this slows down the identifier search with directories which aren’t relevant to that task.

A way to get the same level of crappy find capability, without polluting the path, is to just make another mapping. You can then press <Leader><space> (which is typically backslash space) then start typing a filename and use tab or CTRL-D completion to find the file.

" fuzzy-find lite
nmap <Leader><space> :e ./**/

Just to reiterate: the path parameter was designed for header files. If you want more proof, there is even a :checkpath command to see whether the path is functioning. Load a C file and run :checkpath. It will display filenames it was unable to find that are included transitively by the current file. Also :checkpath! with a bang dumps the whole hierarchy of files included from the current file.

By default path has the value “.,/usr/include,,” meaning the working directory, /usr/include, and files that are siblings of the active buffer. The directory specifiers and globs are pretty powerful, see :help file-searching for the details.

In my C ftplugin (more on that later), I also have the path search for include files within the current project, like ./src/include or ./include .

setlocal path=.,,*/include/**3,./*/include/**3
setlocal path+=/usr/include

The ** with a number like **3 bounds the depth of the search in subdirectories. It’s wise to add depth bounds where you can to avoid identifier searches that lock up.

Here are other patterns you might consider adding to your path if :checkpath identifies that files can’t be found in your project. It depends on your system of course.

  • More system includes: /usr/include/**4,/usr/local/include/**3
  • Homebrew library headers: /usr/local/Cellar/**2/include/**2
  • Macports library headers: /opt/local/include/**
  • OpenBSD library headers: /usr/local/lib/\*/include,/usr/X11R6/include/\*\*3

See also: :he [, :he gf, :he :find.

Edit ⇄ compile cycle

The :make command runs a program of the user’s choice to build a project, and collects the output in the quickfix buffer. Each item in the quickfix records the filename, line, column, type (warning/error) and message of each output item. A fairly idomatic mapping uses bracket commands to move through quickfix items:

" quickfix shortcuts
nmap ]q :cnext<cr>
nmap ]Q :clast<cr>
nmap [q :cprev<cr>
nmap [Q :cfirst<cr>

If, after updating the program and rebuilding, you are curious what the error messages said last time, use :colder (and :cnewer to return). To see more information about the currently selected error use :cc, and use :copen to see the full quickfix buffer. You can populate the quickfix yourself without running :make with :cfile, :caddfile, or :cexpr.

Vim parses output from the build process according to the errorformat string, which contains scanf-like escape sequences. It’s typical to set this in a “compiler file.” For instance, Vim ships with one for gcc in $VIMRUNTIME/compiler/gcc.vim, but has no compiler file for clang. I created the following definition for ~/.vim/compiler/clang.vim:

" formatting variations documented at
" https://clang.llvm.org/docs/UsersManual.html#formatting-of-diagnostics
"
" It should be possible to make this work for the combination of
" -fno-show-column and -fcaret-diagnostics as well with multiline
" and %p, but I was too lazy to figure it out.
"
" The %D and %X patterns are not clang per se. They capture the
" directory change messages from (GNU) 'make -w'. I needed this
" for building a project which used recursive Makefiles.

CompilerSet errorformat=
	\%f:%l%c:{%*[^}]}{%*[^}]}:\ %trror:\ %m,
	\%f:%l%c:{%*[^}]}{%*[^}]}:\ %tarning:\ %m,
	\%f:%l:%c:\ %trror:\ %m,
	\%f:%l:%c:\ %tarning:\ %m,
	\%f(%l,%c)\ :\ %trror:\ %m,
	\%f(%l,%c)\ :\ %tarning:\ %m,
	\%f\ +%l%c:\ %trror:\ %m,
	\%f\ +%l%c:\ %tarning:\ %m,
	\%f:%l:\ %trror:\ %m,
	\%f:%l:\ %tarning:\ %m,
	\%D%*\\a[%*\\d]:\ Entering\ directory\ %*[`']%f',
	\%D%*\\a:\ Entering\ directory\ %*[`']%f',
	\%X%*\\a[%*\\d]:\ Leaving\ directory\ %*[`']%f',
	\%X%*\\a:\ Leaving\ directory\ %*[`']%f',
	\%DMaking\ %*\\a\ in\ %f

CompilerSet makeprg=make

To activate this compiler profile, run :compiler clang. This is typically done in an ftplugin file.

Another example is running GNU Diction on a text document to identify wordy and commonly misused phrases in sentences. Create a “compiler” called diction.vim:

CompilerSet errorformat=%f:%l:\ %m
CompilerSet makeprg=diction\ -s\ %

After you run :compiler diction you can use the normal :make command to run it and populate the quickfix. The final mild convenience in my .vimrc is a mapping to run make:

" real make
map <silent> <F5> :make<cr><cr><cr>
" GNUism, for building recursively
map <silent> <s-F5> :make -w<cr><cr><cr>

Diffs and patches

Vim’s internal diffing is powerful, but it can be daunting, especially the three-way merge view. In reality it’s not so bad once you take time to study it. The main idea is that every window is either in or out of “diff mode.” All windows put in diffmode (with :difft[his]) get compared with all other windows already in diff mode.

For example, let’s start simple. Create two files:

echo "hello, world" > h1
echo "goodbye, world" > h2

vim h1 h2

In vim, split the arguments into their own windows with :all. In the top window, for h1, run :difft. You’ll see a gutter appear, but no difference detected. Move to the other window with CTWL-W CTRL-W and run :difft again. Now hello and goobye are identified as different in the current chunk. Continuing in the bottom window, you can run :diffg[et] to get “hello” from the top window, or :diffp[ut] to send “goodbye” into the top window. Pressing ]c or [c would move between chunks if there were more than one.

A shortcut would be running vim -d h1 h2 instead (or its alias, vimdiff h1 h2) which applies :difft to all windows. Alternatively, load just h1 with vim h1 and then :diffsplit h2. Remember that fundamentally these commands just load files into windows and set the diff mode.

With these basics in mind, let’s learn to use Vim as a three-way mergetool for git. First configure git:

git config merge.tool vimdiff
git config merge.conflictstyle diff3
git config mergetool.prompt false

Now, when you hit a merge conflict, run git mergetool. It will bring Vim up with four windows. This part looks scary, and is where I used to flail around and often quit in frustration.

+-----------+------------+------------+
|           |            |            |
|           |            |            |
|   LOCAL   |    BASE    |   REMOTE   |
+-----------+------------+------------+
|                                     |
|                                     |
|             (edit me)               |
+-------------------------------------+

Here’s the trick: do all the editing in the bottom window. The top three windows simply provide context about how the file differs on either side of the merge (local / remote), and how it looked prior to either side doing any work (base).

Move within the bottom window with ]c, and for each chunk choose whether to replace it with text from local, base, or remote – or whether to write in your own change which might combine parts from several.

To make it easier to pull changes from the top windows, I set some mappings in my vimrc:

" shortcuts for 3-way merge
map <Leader>1 :diffget LOCAL<CR>
map <Leader>2 :diffget BASE<CR>
map <Leader>3 :diffget REMOTE<CR>

We’ve already seen :diffget, and here our bindings pass an argument of the buffer name that identifies which window to pull from.

Once done with the merge, run :wqa to save all the windows and quit. If you want to abandon the merge instead, run :cq to abort all changes and return an error code to the shell. This will signal to git that it should ignore your changes.

Diffget can also accept a range. If you want to pull in all changes from one of the top windows rather than working chunk by chunk, just run :1,$+1diffget {LOCAL,BASE,REMOTE}. The “+1” is required because there can be deleted lines “below” the last line of a buffer.

The three-way marge is fairly easy after all. There’s no need for plugins like Fugitive, at least for presenting a simplified view for resolving merge conflicts.

Finally, as of patch 8.1.0360, Vim is bundled with the xdiff library and can create diffs internally. This can be more efficient than shelling out to an external program, and allows for a choice of diff algorithms. The “patience” algorithm often produces more human-readable output than the default, “myers.” Set it in your .vimrc like so:

if has("patch-8.1.0360")
	set diffopt+=internal,algorithm:patience
endif

Buffer I/O

See if this sounds familiar: you’re editing a buffer and want to save it as a new file, so you :w newname. After editing some more, you :w, but it writes over the original file. What you want for this scenario is :saveas newname, which does the write but also changes the filename of the buffer for future writes. Alternately, the :file newname command will change the filename without doing a write.

It also pays off to learn more about the read and write commands. Becuase r and w are Ex commands, they work with ranges. Here are some variations you might not know about:

:w >>foo append the whole buffer to a file
:.w >>foo append current line to a file
:$r foo read foo into the end of the buffer
:0r foo read foo into the start, moving existing lines down
:.,$w foo write current line and below to a file
:r !ls read ls output into cursor position
:w !wc send buffer to wc and display output
:.!tr ‘A-Za-z’ ‘N-ZA-Mn-za-m’ apply ROT-13 to current line
:w|so % chain commands: write and then source buffer
:e! throw away unsaved changes, reload buffer
:hide edit foo edit foo, hide current buffer if dirty

Useless fun fact: we piped a line to tr in an example above to apply a ROT-13 cypher, but Vim has that functionality built in with the the g? command. Apply it to a motion, like g?$.

Filetypes

Filetypes are a way to change settings based on the type of file detected in a buffer. They don’t need to be automatically detected though, we can manually enable them to interesting effect. An example is doing hex editing. Any file can be viewed as raw hexadecimal values. GitHub user the9ball created a clever ftplugin script that filters a buffer back and forth through the xxd utility for hex editing.

The xxd utility was bundled as part of Vim 5 for convenience. The Vim todo.txt file mentions they want to make it more seamless to edit binary files, but xxd can take us pretty far.

Here is code you can put in ~/.vim/ftplugin/xxd.vim. Its presence in ftplugin means Vim will execute the script when filetype (aka “ft”) becomes xxd. I added some basic comments to the script.

" without the xxd command this is all pointless
if !executable('xxd')
	finish
endif

" don't insert a newline in the final line if it
" doesn't already exist, and don't insert linebreaks
setlocal binary noendofline
silent %!xxd -g 1
%s/\r$//e

" put the autocmds into a group for easy removal later
augroup ftplugin-xxd
	" erase any existing autocmds on buffer
	autocmd! * <buffer>

	" before writing, translate back to binary
	autocmd BufWritePre <buffer> let b:xxd_cursor = getpos('.')
	autocmd BufWritePre <buffer> silent %!xxd -r

	" after writing, restore hex view and mark unmodified
	autocmd BufWritePost <buffer> silent %!xxd -g 1
	autocmd BufWritePost <buffer> %s/\r$//e
	autocmd BufWritePost <buffer> setlocal nomodified
	autocmd BufWritePost <buffer> call setpos('.', b:xxd_cursor) | unlet b:xxd_cursor

	" update text column after changing hex values
	autocmd TextChanged,InsertLeave <buffer> let b:xxd_cursor = getpos('.')
	autocmd TextChanged,InsertLeave <buffer> silent %!xxd -r
	autocmd TextChanged,InsertLeave <buffer> silent %!xxd -g 1
	autocmd TextChanged,InsertLeave <buffer> call setpos('.', b:xxd_cursor) | unlet b:xxd_cursor
augroup END

" when filetype is set to no longer be "xxd," put the binary
" and endofline settings back to what they were before, remove
" the autocmds, and replace buffer with its binary value
let b:undo_ftplugin = 'setl bin< eol< | execute "au! ftplugin-xxd * <buffer>" | execute "silent %!xxd -r"'

Try opening a file, then running :set ft. Note what type it is. Then:set ft=xxd. Vim will turn into a hex editor. To restore your view, :set ft=foo where foo was the original type. Note that in hex view you even get syntax highlighting because $VIMRUNTIME/syntax/xxd.vim ships with Vim by default.

Notice the nice use of “b:undo_ftplugin” which is an opportunity for filetypes to clean up after themselves when the user or ftdetect mechanism switches away from them to another filetype. (The example above could use a little work because if you :set ft=xxd then set it back, the buffer is marked as modified even if you never changed anything.)

Ftplugins also allow you to refine an existing filetype. For instance, Vim already has some good defaults for C programming in $VIMRUNTIME/ftplugin/c.vim. I put these extra options in ~/.vim/after/ftplugin/c.vim to add my own settings on top:

" the smartest indent engine for C
setlocal cindent
" my preferred "Allman" style indentation
setlocal cino="Ls,:0,l1,t0,(s,U1,W4"

" for quickfix errorformat
compiler clang
" shows long build messages better
setlocal ch=2

" auto-create folds per grammar
setlocal foldmethod=syntax
setlocal foldlevel=10

" local project headers
setlocal path=.,,*/include/**3,./*/include/**3
" basic system headers
setlocal path+=/usr/include

setlocal tags=./tags,tags;~
"                      ^ in working dir, or parents
"                ^ sibling of open file

" the default is menu,preview but the preview window is annoying
setlocal completeopt=menu

iabbrev #i #include
iabbrev #d #define
iabbrev main() int main(int argc, char **argv)

" add #include guard
iabbrev #g _<c-r>=expand("%:t:r")<cr><esc>VgUV:s/[^A-Z]/_/g<cr>A_H<esc>yypki#ifndef <esc>j0i#define <esc>o<cr><cr>#endif<esc>2ki

Notice how the script uses “setlocal” rather than “set.” This applies the changes to just the current buffer rather than the whole Vim instance.

This script also enables some light abbreviations. Like I can type #g and press enter and it adds an include guard with the current filename:

#ifndef _FILENAME_H
#define _FILENAME_H

/* <-- cursor here */

#endif

You can also mix filetypes by using a dot (“.”). Here is one application. Different projects have different coding conventions, so you can combine your default C settings with those for a particular project. The OpenBSD source code follows the style(9) format, so let’s make a special openbsd filetype. Combine the two filetypes with :set ft=c.openbsd on relevant files.

To detect the openbsd filetype we can look at the contents of buffers rather than just their extensions or locations on disk. The telltale sign is that C files in the OpenBSD source contain /* $OpenBSD: in the first line.

To detect them, create ~/.vim/after/ftdetect/openbsd.vim:

augroup filetypedetect
        au BufRead,BufNewFile *.[ch]
                \  if getline(1) =~ 'OpenBSD;'
                \|   setl ft=c.openbsd
                \| endif
augroup END

The Vim port for OpenBSD already includes a special syntax file for this filetype: /usr/local/share/vim/vimfiles/syntax/openbsd.vim. If you recall, the /usr/local/share/vim/vimfiles directory is in the runtimepath and is set aside for files from the system administrator. The provided openbsd.vim script includes a function:

function! OpenBSD_Style()
	setlocal cindent
	setlocal cinoptions=(4200,u4200,+0.5s,*500,:0,t0,U4200
	setlocal indentexpr=IgnoreParenIndent()
	setlocal indentkeys=0{,0},0),:,0#,!^F,o,O,e
	setlocal noexpandtab
	setlocal shiftwidth=8
	setlocal tabstop=8
	setlocal textwidth=80
endfun

We simply need to call the function at the appropriate time. Create ~/.vim/after/ftplugin/openbsd.vim:

call OpenBSD_Style()

Now opening any C or header file with the characteristic comment at the top will be recognized as type c.openbsd and will use indenting options that conform with the style(9) man page.

Don’t forget the mouse

This is a friendly reminder that despite our command-line machismo, the mouse is in fact supported in Vim, and can do some things more easily than the keyboard. Mouse events work even over SSH thanks to xterm turning mouse events into stdin escape codes.

To enable mouse support, set mouse=n. Many people use mouse=a to make it work in all modes, but I prefer to enable it only in normal mode. This avoids creating visual selections when I click links with a keyboard modifier to open them in my browser.

Here are things the mouse can do:

  • Open or close folds (when foldcolumn > 0).
  • Select tabs (beats gt gt gt…)
  • Click to complete a motion, like d<click!>. Similar to the easymotion plugin but without any plugin.
  • Jump to help topics with double click.
  • Drag the status line at the bottom to change cmdheight.
  • Drag edge of window to resize.
  • Scroll wheel.

Misc editing

This section could be enormous, but I’ll stick to a few tricks I learned. The first one that blew me away was :set virtualedit=all. It allows you to move the cursor anywhere in the window. If you enter characters or insert a visual block, Vim will add whatever spaces are required to the left of the inserted characters to keep them in place. Virtual edit mode makes it simple to edit tabular data. Turn it off with :set virtualedit=.

Next are some movement commands. I used to rely a lot on } to jump by paragraphs, and just muscle my way down the page. However the ] character makes more precise motions: by function ]], scope ]}, paren ‘])’, comment ]/, diff block ]c. This series is why the quickfix mapping ]q mentioned earlier fits the pattern so well.

For big jumps I used to try things like 1000j, but in normal mode you can actually just type a percentage and Vim will go there, like 50%. Speaking of scroll percentage, you can see it at any time with CTRL-G. Thus I now do :set noruler and ask to see the info as needed. It’s less cluttered. Kind of the opposite of the trend of colorful patched font powerlines.

After jumping around between tags, files, or within a file, there are some commands to get your bearings. Try :ls, :tags, :jumps, and :marks. Jumping through tags actually creates a stack, and you can press CTRL-T to pop one back. I used to always press CTRL-O to back out of jumps, but it is not as direct as popping the tag stack.

In a project directory that has been indexed with ctags, you can open the editor directly to a tag with -t, like vim -t main. To find tags files more flexibly, set the tags configuration variable. Note the semicolon in the example below that allows Vim to search the current directory upward to the home directory. This way you could have a more general system tags file outside the project folder.

set tags=./tags,**5/tags,tags;~
"                          ^ in working dir, or parents
"                   ^ in any subfolder of working dir
"           ^ sibling of open file

There are some buffer tricks too. Switching to a buffer with :bu can take a fragment of the buffer name, not just a number. Sometimes it’s harder to memorize those numbers than remember the name of a source file. You can navigate buffers with marks too. If you use a capital letter as the name of a mark, you can jump to it across buffers. You could set a mark H in a header, C in a source file, and M in a Makefile to go from one buffer to another.

Do you ever get mad after yanking a word, deleting a word somewhere else, trying paste the first word in, and then discovering your original yank is overwritten? The Vim registers are underappreciated for this. Inspect their contents with :reg. As you yank text, previous yanks are rotated into the registers "0 - "9. So "0p pastes the next-to-last yank/deletion. The special registers "+ and "* can copy/paste from/to the system clipboard. They usually mean the same thing, except in some X11 setups that distinguish primary and secondary selection.

Another handy hidden feature is the command line window. It it’s a buffer that contains your previous commands and searches. Bring it up with q: or q/. Once inside you can move to any line and press enter to run it. However you can also edit any of the lines before pressing enter. Your changes won’t affect the line (the new command will merely be added to the bottom of the list).

This article could go on and on, so I’m going to call it here. For more great topics, see these help sections: views-sessions, viminfo, TOhtml, ins-completion, cmdline-completion, multi-repeat, scroll-cursor, text-objects, grep, netrw-contents.

vim logo

vim logo

July 18, 2019

Pepijn de Vos (pepijndevos)

VHDL to PCB July 18, 2019 12:00 AM

When learning to program FPGAs using VHDL or Verilog, you also learn that these hardware description languages can be used to design ASICs (application specific integrated circuit). But this is only something big corporations with millions of dollars can afford, right? Even though I later learned it only costs thousands, not millions, to make an ASIC on an older process, it is still far away from hobby budgets.

I had been keeping an eye on Yosys, the open source HDL synthesis tool, which can apparently do ASIC by giving it a liberty file that specifies the logic cells your foundry supports. Meanwhile I also toyed with the idea of making a 7400 series computer, and I wondered if you could write a liberty file for 7400 chips. I had kind of dismissed the idea, but then ZirconiumX came along and did it.

It suffices to say this revived my interest in the idea and a lively discussion and many pull requests followed. First some small changes, then simulations to verify the synthesized result is still correct, and finally a KiCad netlist generator.

You see, generating a Yosys netlist is nice, but eventually these 7400 chips have to end up on a PCB somehow. Normally you draw your schematic in Eeschema, generate a netlist, and import that to Pcbnew. But instead I used skidl to generate the netlist directly. Then all there is to do is add the inputs and outputs and run the autorouter (or do it manually of course).

I decided to do a proof-of-concept “application specific interconnected circuit”, with the goal of making something fun in under 10 chips. (a Risc-V CPU currently sits at about 850) I settled on a fading PWM circuit to drive an LED. I manually added a 555-based clock, and ordered a PCB for a few bucks. A few weeks later, this was the result. It worked on the first try! This feeling is even more amazing than with software, and shows that as long as there are no compiler/library bugs or DRC errors, logic simulations are a good way to prove your PCB design.

To follow along at home you need to install Yosys. A recent release might work, but it’s getting better every day, so building from source is recommended. Then you can just git clone 74xx-liberty and go. There are a number of Verilog programs in benchmarks in case you’d rather make PicoRV32 in 7400 chips.

cd stat
make pwmled.stat # synthesize and run stat
../ic_count.py pwmled.stat # count number of chips used
cd ../sim
make pwmled.vcd # synth to low-level verilog and simulate
gtkwave pwmled.vcd # show test bench results
cd ../kicad
make pwmled.net # generate kicad netlist

But this was all done in Verilog, so where is the VHDL, you might wonder. Well, Yosys does not really support VHDL yet, but Tristan Gingold is hard at work making GHDL synthesize VHDL as a Yosys plugin. I think this is very important work, so I’ve been contributing there as well. After some pull requests I was able to port the breathing LED to VHDL.

Getting VHDL to work in Yosys is a bit of effort. First you need to compile GHDL, which requires installing a recent version of GNAT. Then you need to install ghdlsynth-beta as a plugin, allowing you to run yosys -m ghdl. My fork of 74xx-liberty contains additional make rules for doing the above synthesization for VHDL files, which does something like this before calling the 7400 synthesis script.

cd stat
ghdl -a ../benchmarks/pwmled.vhd # analyse VHDL file
yosys -m ghdl -p "ghdl pwmled; show" # load pwmled entity, show graph

yosys dot graph

A huge thank you to all the people working tirelessly to make open source hardware design a reality. You’re awesome!

Awn Umar (awn)

encrypting secrets in memory July 18, 2019 12:00 AM

In a previous post I talked about designing and implementing an in-memory data structure for storing sensitive information in Go. The latest version of memguard adds something new: encryption.

But why? Well, there are limitations to the old solution of using guarded heap allocations for everything.

  1. A minimum of three memory pages have to be allocated for each value: two guard pages sandwiching n≥1 n \geq 1 n1 data pages.
  2. Some systems impose an upper limit on the amount of memory that an individual process is able to prevent from being swapped out to disk.

Memory layout of guarded heap allocation.

Typical layout of a 32 byte guarded heap allocation.

So it is worth looking into the use of encryption to protect information. After all, ciphertext does not have to be treated with much care and authentication guarantees immutability for free. The problem of recovering a secret is shifted to recovering the key that protects it.

But there is the obvious problem. Where and how do you store the encryption key?

We use a scheme described by Bruce Schneier in Cryptography Engineering. The procedure is sometimes referred to as a Boojum. I will formally define it below for convenience.

Define B={i:0≤i≤255} B = \{i : 0 \leq i \leq 255\} B={i:0i255} to be the set of values that a byte may take. Define h:Bn→B32 h : B^n \to B^{32} h:BnB32 for n∈N0 n \in \mathbb{N}_0 nN0 to be a cryptographically-secure hash function. Define the binary XOR operator ⊕:Bn×Bn→Bn \oplus : B^n \times B^n \to B^n :Bn×BnBn.

Suppose k∈B32 k \in B^{32} kB32 is the key we want to protect and Rn∈B32 R_n \in B^{32} RnB32 is some random bytes sourced from a suitable CSPRNG. We initialise the two partitions:

x1=R1y1=h(x1)⊕k \begin{aligned} x_1 &= R_1\\ y_1 &= h(x_1) \oplus k\\ \end{aligned} x1y1=R1=h(x1)k

storing each inside its own guarded heap allocation. Then every m m m milliseconds we overwrite each value with:

xn+1=xn⊕Rn+1=R1⊕R2⊕⋯⊕Rn+1yn+1=yn⊕h(xn)⊕h(xn+1)=h(xn)⊕h(xn)⊕h(xn+1)⊕k=h(xn+1)⊕k \begin{aligned} x_{n+1} &= x_n \oplus R_{n+1}\\ &= R_1 \oplus R_2 \oplus \cdots \oplus R_{n+1}\\ y_{n+1} &= y_n \oplus h(x_n) \oplus h(x_{n+1})\\ &= h(x_n) \oplus h(x_n) \oplus h(x_{n+1}) \oplus k\\ &= h(x_{n+1}) \oplus k\\ \end{aligned} xn+1yn+1=xnRn+1=R1R2Rn+1=ynh(xn)h(xn+1)=h(xn)h(xn)h(xn+1)k=h(xn+1)k

and so on. Then by the properties of XOR,

k=h(xn)⊕yn=h(xn)⊕h(xn)⊕k=0⊕k=k \begin{aligned} k &= h(x_n) \oplus y_n\\ &= h(x_n) \oplus h(x_n) \oplus k\\ &= 0 \oplus k\\ &= k\\ \end{aligned} k=h(xn)yn=h(xn)h(xn)k=0k=k

It is clear from this that our iterative overwriting steps do not affect the value of k k k, and the proof also gives us a way of retrieving k k k. My own implementation of the protocol in fairly idiomatic Go code is available here.

An issue with the Boojum scheme is that it has a relatively high overhead from two guarded allocations using six memory pages in total, and we have to compute and write 64 bytes every m m m milliseconds. However we only store a single global key, and the overhead can be tweaked by scaling m m m as needed. Its value at the time of writing is 8 milliseconds.

The authors of the Boojum claim that it defends against cold boot attacks, and I would speculate that there is also some defence against side-channel attacks due to the fact that k k k is split across two different locations in memory and each partition is constantly changing. Those attacks usually have an error rate and are relatively slow.

OpenBSD added a somewhat related mitigation to their SSH implementation that stores a 16 KiB (static) “pre-key” that is hashed to derive the final key when it is needed. I investigated incorporating it somehow but decided against it. Both schemes have a weak point when the key is in “unlocked” form so mimimising this window of opportunity is ideal.

In memguard the key is initialised when the program starts and then hangs around in the background—constantly flickering—until it is needed. When some data needs to be encrypted or decrypted, the key is unlocked and used for the operation and then it is destroyed.

Diagram showing the high-level structure of the scheme.

High-level overview of the encryption scheme.

The documentation provides a relatively intuitive guide to the package’s functionality. The Enclave stores ciphertext, the LockedBuffer stores plaintext, and core.Coffer implements the Boojum. Examples are available in the examples sub-package.

The most pressing issue at the moment is that the package relies on cryptographic primitives implemented by the Go standard library which does not secure its own memory and may leak values that are passed to it. There has been some discussion about this but for now it seems as though rewriting crucial security APIs to use specially allocated memory is the only feasible solution.

If you have any ideas and wish to contribute, please do get in touch or open a pull request.

July 16, 2019

Jeremy Morgan (JeremyMorgan)

Forget What You've Heard, Now Is the Best Time to Become a Coder July 16, 2019 04:15 PM

Do you want to be a coder? Are you on the fence about trying it? Nervous to get started?

The time is now. Time to pull the trigger. 

There has never been a better time to become a coder.

And I'm going to tell you how to get started.  

My History as a Coder

I started writing code professionally in 2002. Before that, I was building (terrible) websites for myself and friends. I even ran a business for a few years crapping out HTML/Perl sites in the late 90s for good money. I built shoddy software in Visual Basic for businesses.

To learn how to do this, I bought expensive books on HTML, Perl, Visual Basic and Unix. I read them cover to cover and marked them up. Google didn’t exist yet. I consulted Usenet news groups where they insulted you for asking "stupid questions" and figured out most of the stuff the hard way. I learned Unix so I could host my customer's sites and destroyed a lot of stuff before figuring it out. 

I was living in a rural area and had very few friends that were even in to this kind of thing so I had few people to bounce ideas and questions off of. I hadn't yet gone to college and met other technical folks. 

I'm not telling you this so you'll think I'm awesome or some kind of grizzled veteran who is better than you. It was hard, and I wasn't very good for many years, but I still got the job done. I spent countless nights chugging Surge and hacking away trying to figure out things. I got good. I figured things out. But it sucked.

 

Your History

Your story doesn't have to go that way. The world is at your fingertips. You have all the information to get where I am right in front of you. Some grizzled veterans like myself say things like "new developers don't even have to work for it" and pretend like we're so much better because of our struggles. Not true. In reality because you have this information readily available, you'll get better faster. The developer I was at the 3 year mark will be a rookie compared to you.


Today's beginning developers are positioned to be the best generation of developers yet.

That's part of the reason that NOW is the best time to start. You have Google, Stack Overflow, Dev.to, Social Media, Pluralsight, YouTube, you name it. NOW is exponentially better than 1995 as far a starting point goes. 

The Market: We need developers! 

Another great reason now is the time: we are in a talent shortage. Everyone I talk to is looking for developers. Junior, Senior, mid-level, whatever. Can you code? You'll get a job. It doesn't even have to be in the language they're using. 

In the mid 2000s being a developer I had to know everything under the sun to get in the door to interview. They had crowds lined up fighting for every job. In 2019 if you learn the basics, throw some projects on GitHub and start sending out resumes you'll get that call. 

According to code.org, there are 504,199 open computing jobs nationwide. There were 63,744 computer science graduates entering the workforce last year. 

There are more jobs available than coders to fill them.

The numbers are in your favor. 

What do you need? 

  • Do you need a Computer Science Degree? : No
  • Do you need expensive training? No
  • Do you need a MacBook Pro?: No 

You don't need any of these things. If you want to get started, you can do it with a ChromeBook. 

For instance: 

Want to learn JavaScript?

  1. Go to JavaScript.Info and start from the beginning. 
  2. Create an account at JSFiddle and start hacking away. 

Don't worry about details like getting your own website, server, etc. Just do it.   Want to learn some Python? You can install it on your old Dell laptop and start going. CSS? PHP? C#? These are available to download free and will run on nearly any machine built in the last 10 years. 

At some point you will want a faster computer and more in depth training. But to get started this is all you'll need. 

Next Steps

Disclaimer: So, remember how I talked about how hard it was to learn tech? It's what drives me now. This stage of my career is dedicated towards teaching people, and improving their tech skills. In the interest of transparency I work for a company called Pluralsight which is the leading tech skills platform. This is not the only reason I recommend them, I was a customer long before becoming and employee and it's boosted my career tremendously.

So for next steps after taking some tutorials and getting your feet wet, do the following to break into this industry. 

Phase 1

  • Determine what you want to develop (Web, Mobile, Desktop)
  • Find as many tutorials as you can on the subject.  (You can contact me if you need help with this)
  • Create a GitHub account
  • Start uploading code samples you build with tutorials

Phase 2

  • Start a project -No matter how stupid it seems. Make an app to store your favorite jokes, or todo lists.
  • Create a HackerRank account - Start tackling some puzzles. 
  • Pick problems from Project Euler and write code to solve them. 
  • Get to know Stack Overflow - Search it when you have a problem and answer questions if you know it!

Once you get to this point, you'll start feeling comfortable. Keep working and improving your craft. Dig deeper on the tech you've chosen. Build things, even small things. Then think about getting your first job. 

Phase 3

And many more. If you're targeting a specific role, this is a great way to get your skills up. 

  • Build something useful - Build something that solves a problem for you, a friend or your employer. Nothing teaches like you like building something real.
  • Start teaching - Nothing helps you learn a subject like teaching it. When you get comfortable with your knowledge, share it.

Conclusion

Stop making excuses and don't listen to people who tell you you can't or shouldn't learn to code. If you want it bad enough you can do it and get paid for it. Teaching others is a deep passion of mine so if you get stuck on something or need some advice, hit me up I'd be glad to help. 

Now start learning!  



What is your DevOps IQ?

what's your devops score


My Devops Skill IQ is 232. Can you beat it? Take the test now to find out your Devops IQ score!!

July 13, 2019

eta (eta)

Building the Physics Penitentiary July 13, 2019 11:00 PM

(Edited 2019-08-06: added links to download the source code, and made a few minor language changes.)

I “recently”1 took part in the Weizmann Institute’s international “safe-cracking” tournament, as part of a team of five people2 who placed highly enough in the UK version of the tournament to get to fly over to Israel and have a shot it it over there. Since I did most/all of the electronics and control system for our entry, I thought I’d write a blog post about it!

Essentially, the idea behind the competition is to build ‘safes’ - essentially boxes with physics-related puzzles and challenges that must be completed in order to ‘crack’ the safe, with the point being that only an understanding of the physics involved would let you unlock it. Our safe, the Physics Penitentiary3, looked a bit like this:

Picture of the safe

Safe background

So, I’m not a physicist by any means (which led to some fun discussions about what a random CS student was doing taking part in a physics competition, but we’ll gloss over that…), which means I have only a loose understanding of the physics. That aside, what was in the safe roughly looked like:

  • one Crookes radiometer / light mill (the weird glass thing in the very centre, surrounded by a cage)
  • a pump, hooked up to a container full of ethanol, arranged to deposit said ethanol on the radiometer
  • a soundproof(-ish) section to the right, with two speakers facing one another, plus a microphone
  • a control panel with lots of fun buttons for the user to press
  • a tube with two servo-controlled flags blocking it, which was used to dispense marbles (‘prisoners’) upon successful completion of the challenges

In addition to this, you were allowed:

  • two torches, one with a tungsten bulb, one LED
  • a canister of compressed air

With this, you had to ‘crack’ the safe by figuring out how the Crookes radiometer worked - essentially, due to some physicsey explanation I don’t quite comprehend4, the radiometer would spin if it was heated (e.g. by shining an inefficient tungsten bulb on it), and would also spin, but in the opposite direction, if you cooled it again (e.g. by inverting the canister of compressed air to get at the incredibly cold gas and spraying that on the surface of the radiometer). We also added the sound-based challenge seen to the right, where you had to get the two sound waves to match frequency and wave type (e.g. sine, square, or triangle) in order to cancel each other out5 - to make it a bit harder, and also to fit with the ‘prison’ theme.

Radiometer spinning

Above: nifty GIF of the radiometer doing its thing. Image courtesy of Nevit Dilmen on Wikimedia.

And that was all well and good; the thing spun one way if you used the light, and spun the other way if you used the inverted gas canister.

However, one important detail is still missing: how did the safe actually detect which way the radiometer was spinning, if at all? In planning, we sort of hand-waved this issue away with vague talk of ‘lasers’, ‘infrared’ and ‘proximity sensor hackery’, but it turned out that detecting the spin direction was one of the hardest challenges we had to solve - and the solution was nearly all in code as well, which makes it interesting to talk about!

You spin me right round, baby, right round

So, we had a problem. The problem was made harder by a number of restrictions:

  • The radiometer was a completely sealed partial vacuum chamber, so we couldn’t get in there to add sensors or machinery of any sort. Anything we’d want to use to detect the spin would have to be outside.
  • It’s curved glass, so using a laser, IR beam or something like that probably wouldn’t work too well (though we didn’t test it) due to the light being bent (i.e. refraction).
  • The method used to cool the radiometer (shooting freezing cold liquid at the glass through the grating at the top) caused it to frost over with an opaque white layer of ice. If any sensors were in the way of this blast, they probably wouldn’t have fared too well.
    • To remedy this issue, the ethanol dispenser would let you spray ethanol on the glass to defrost it, making it clear again - although, of course, this also meant the sensors had random extra liquids to contend with…

In the first revision of the safe (the one that we presented at the UK finals), this was accomplished using a generic chinesium proximity sensor we managed to procure from eBay pointed at the radiometer, and some very very questionable heuristics - to give you an idea, we only managed to get the heuristics to work semi-reliably on the very day of the competition itself, and even then, the output of the sensor detection algorithm was fed into an array of 5 recent readings, and we’d take the majority value out of these five to smooth out the inconsistencies (!)6. Very little science was involved in getting this to work; we7 just threw code at the problem until something semi-worked.

Needless to say, such a solution was not going to fly (quite literally8) at the international competition, so, after our miraculous success at the national finals, we needed to think of something else.

The solution we eventually settled on involved using a Vishay VCNL4010 proximity sensor (and SV-Zanshin’s wonderful Arduino library). This sensor had a number of nice features that made us choose it:

  • The actual sensor block (the small black square in the linked image) was very tiny - in fact, just smaller than one of the blades of the radiometer. This meant we could position it such that the block lined up exactly with the blades, so we’d get actually decent readings.
  • It had a built-in ambient light sensor, and combined this data with the IR proximity data to avoid ambient light messing up the sensor readings, which was an issue with the old sensor. As a bonus, this feature actually worked - shining a light on the radiometer didn’t seem to affect the sensor at all!

Sensor closeup

Above: one of the pictures we took of the sensor alignment during testing. Note the neat use of braided sheathing to make cable management not painful…

It’s graph time!

With the actual hardware sorted out, as best we could (positioning the sensor was painstakingly hard, and required multiple attempts to get it to work properly), the next remaining challenge was actually making sense of the proximity data.

Of course, a proximity sensor only gives you a reading indicating how near or far something is, in arbitrary units; you still need to do some processing to figure out spin direction (spin frequency is relatively easy to determine, but we didn’t really care about that much here). In theory, this should be relatively easy: if the sensor is spinning in one direction, the proximity should be increasing (modulo the spike down every now and then), and vice versa for the opposite direction. Indeed, if you plot the readings on the graph, you get something looking like:

Plot of radiometer proximity data

Above: a plot of the radiometer proximity data, taken from the sensor. The first half of the graph shows it spinning in one direction, then it changes at around the 4000 mark on the x-axis to spin in the other direction.

Looking at this (probably somewhat tiny) plot, a pattern is somewhat distinguishable: the troughs of each of the spikes have a different gradient, depending on which direction the radiometer is spinning in - somewhat as we’d expect, from the description we gave above. This reduces the problem to:

  • How do we filter out just the bit at the bottom of each spike that we’re interested in from all the rest of the stuff?
  • How do we figure out the gradient of said interesting points?

The first bullet point was solved with some hacky application of elementary statistics - we kept a buffer of the last 256 data points received from the sensor, calculated their mean and standard deviation, and excluded points that were above mean - (stddev / 4) (with the divisor determined experimentally; 4 seemed to work well enough). This isolated the points to just the few points at the bottom of the trough (or something approximating that).

Radiometer proximity data with gradient lines drawn

For the second, it was time for…some more elementary stats! As might be obvious, a simple linear regression sufficed, giving us a regression line and its gradient to work with. To prevent data points from being calculated from not enough data, we also prevented regressions from being carried out on data with an insufficiently high range, and rejected results with a low R-value. This finally gave us something vaguely capable of determining the spin direction - and we’d also require 3 or so concordant readings in order to actually come to a judgement, to minimize the chance of random errors. (Remember, a false positive would result in the challenge instantly unlocking, so we wanted to be sure; better to have the cracker wait for 5 seconds or so than to have it randomly unlock out of nowhere!)

Oh, and the issues with the user being able to spray frost (or ethanol) at the sensor were resolved by…judicious use of tape covering the sensor, so nothing could get in at it (!). We also added some tissue paper, so that the ethanol wouldn’t run everywhere and ruin the rest of the safe.

Radiometer closeup

Above: If you look closely, you can just about make out the tape on the other side of the radiometer…

What about the rest of the electronics?

This whole thing was done using STM32F103 microcontrollers, actually - the first revision of the safe used a Blue Pill (see linked blog post for more), while the second revision used the somewhat more reliable Maple Mini. The STM32s were definitely a big plus; they were pretty small, for one, which meant we could stuff the electronics underneath the radiometer (an Arduino Uno would definitely not have fit in!), and their speed (and 32-bit address bus…) made doing calculations like the above regression stuff actually feasible. I used the STM32duino IDE and tooling, primarily because writing drivers (for things like the VCNL4010) didn’t seem like my idea of a fun time (!).

Electronics stuffed into the radiometer well

Above: most of the electronics and cabling, stuffed into a well below the radiometer during construction. Connecting all the wires was pretty difficult!

To put things into context, the original revision didn’t use a PCB at all, and instead had tons of flying wires connecting everything on a piece of protoboard; we decided that making a proper PCB for the second revision might be a good idea, so we did:

Artsy shot of the PCB, with theta.eu.org logo in focus

Above: artsy shot of the PCB made for the project, with gratutitous theta.eu.org logo in focus.

In total, the electronics ended up being made up of:

  • 1 Maple Mini STM32F103 microcontroller
  • 1 VCNL4010 proximity sensor breakout
  • 2 AD9833 waveform generator modules, for the frequency challenge
    • The first iteration of the safe initially used digital potentiometers (!) to control some hacked up tone generator kits we had lying around; using the AD9833s was definitely a better idea.
    • However, finding breakout boards with this IC on them wasn’t an easy task9.
  • 1 SparkFun microphone breakout
  • 2 amplifiers, to drive the speakers
  • 2 servo motors, to control release of the marbles
  • 1 Tracopower TSR-2450 voltage regulator, for power input
    • I love these modules (for their efficiency); we used one to step down the 9V barrel jack input to a more manageable 5V.
  • Some well-placed debug (UART) and programming (SWDIO) headers, to get at the Maple Mini
  • Assorted push switches and LEDs for the control panel
  • Probably a few passives and other things like that

Source code

As with the Systems CAT, this code is intended to be looked at, rather than used (if you consider yourself patient enough to spend time untangling it, that is!). However, you might find it interesting:

(All rights reserved, no warranty provided, etc.)

Conclusion

Right, that’s probably about enough information for one blog post; it’s been upwards of 2000 words, after all! Hopefully some of it was vaguely informative / interesting; it’s also worth keeping in mind that the actual process of building this thing was significantly more chaotic and panicked than it might seem (c.f. the footnotes below). However, it was definitely a fun thing to build (despite not doing well in the international finals, sadly!)

Do let me know using the comment thing below if you have any more questions about the safe, or feedback; it’s always nice to receive comments!


  1. (read: a couple of months ago) 

  2. This means your odds of figuring out who the hell I actually am in real life are now about 1 in 5. 

  3. Not my choice of name. 

  4. The explanations behind the radiometer are apparently still disputed; if you do a search for it, there are a number of conflicting reasons why it behaves like this! 

  5. If you’re wondering how that actually worked in practice given differences in phase, etc., you’d be onto something. We may or may not have bodged this part a bit to get it to work… 

  6. It also stopped giving any readings at all if you shone a light at the sensor, which caused some fun issues when it came to cracking the safe. 

  7. When I say ‘we’, I really mean ‘I’, but it sounds nicer this way. 

  8. Transporting the safe to Israel was done, of course, on a plane; the sensor positioning was so sensitive that we thought it’d definitely get screwed up by being bumped around… 

  9. When one of them broke in the week before the international final, trying to hunt around Amazon for another one wasn’t as easy as we would have hoped! 

July 12, 2019

Gustaf Erikson (gerikson)

June July 12, 2019 06:57 PM

Kungsträdgården

Stensund

Jun 2018 | Jun 2017 | Jun 2016 | Jun 2015 | Jun 2014 | Jun 2013 | Jun 2012 | Jun 2011 | Jun 2010 | Jun 2009

Dan Luu (dl)

Files are fraught with peril July 12, 2019 12:00 AM

This is a psuedo-transcript for a talk given at Deconstruct 2019. To make this accessible for people on slow connections as well as people using screen readers, the slides have been replaced by in-line text (the talk has ~120 slides; at an average of 20 kB per slide, that's 2.4 MB. If you think that's trivial, consider that half of Americans still aren't on broadband and the situation is much worse in developing countries.

Let's talk about files! Most developers seem to think that files are easy. Just for example, let's take a look at the top reddit r/programming comments from when Dropbox announced that they were only going to support ext4 on Linux (the most widely used Linux filesystem). For people not familiar with reddit r/programming, I suspect r/programming is the most widely read English langauge programming forum in the world.

The top comment reads:

I'm a bit confused, why do these applications have to support these file systems directly? Doesn't the kernel itself abstract away from having to know the lower level details of how the files themselves are stored?

The only differences I could possibly see between different file systems are file size limitations and permissions, but aren't most modern file systems about on par with each other?

The #2 comment (and the top replies going two levels down) are:

#2: Why does an application care what the filesystem is?

#2: Shouldn't that be abstracted as far as "normal apps" are concerned by the OS?

Reply: It's a leaky abstraction. I'm willing to bet each different FS has its own bugs and its own FS specific fixes in the dropbox codebase. More FS's means more testing to make sure everything works right . . .

2nd level reply: What are you talking about? This is a dropbox, what the hell does it need from the FS? There are dozenz of fssync tools, data transfer tools, distributed storage software, and everything works fine with inotify. What the hell does not work for dropbox exactly?

another 2nd level reply: Sure, but any bugs resulting from should be fixed in the respective abstraction layer, not by re-implementing the whole stack yourself. You shouldn't re-implement unless you don't get the data you need from the abstraction. . . . DropBox implementing FS-specific workarounds and quirks is way overkill. That's like vim providing keyboard-specific workarounds to avoid faulty keypresses. All abstractions are leaky - but if no one those abstractions, nothing will ever get done (and we'd have billions of "operating systems").

In this talk, we're going to look at how file systems differ from each other and other issues we might encounter when writing to files. We're going to look at the file "stack" starting at the top with the file API, which we'll see is nearly impossible to use correctly and that supporting multiple filesystems without corrupting data is much harder than supporting a single filesystem; move down to the filesystem, which we'll see has serious bugs that cause data loss and data corruption; and then we'll look at disks and see that disks can easily corrupt data at a rate five million times greater than claimed in vendor datasheets.

File API

Writing one file

Let's say we want to write a file safely, so that we don't want to get data corruption. For the purposes of this talk, this means we'd like our write to be "atomic" -- our write should either fully complete, or we should be able to undo the write and end up back where we started. Let's look at an example from Pillai et al., OSDI’14.

We have a file that contains the text a foo and we want to overwrite foo with bar so we end up with a bar. We're going to make a number of simplifications. For example, you should probably think of each character we're writing as a sector on disk (or, if you prefer, you can imagine we're using a hypothetical advanced NVM drive). Don't worry if you don't know what that means, I'm just pointing this out to note that this talk is going to contain many simplifications, which I'm not going to call out because we only have twenty-five minutes and the unsimplified version of this talk would probably take about three hours.

To write, we might use the pwrite syscall. This is a function provided by the operating system to let us interact with the filesystem. Our invocation of this syscall looks like:

pwrite(
  [file], 
  “bar”, // data to write
  3,     // write 3 bytes
  2)     // at offset 2

pwrite takes the file we're going to write, the data we want to write, bar, the number of bytes we want to write, 3, and the offset where we're going to start writing, 2. If you're used to using a high-level language, like Python, you might be used to an interface that looks different, but underneath the hood, when you write to a file, it's eventually going to result in a syscall like this one, which is what will actually write the data into a file.

If we just call pwrite like this, we might succeed and get a bar in the output, or we might end up doing nothing and getting a foo, or we might end up with something in between, like a boo, a bor, etc.

What's happening here is that we might crash or lose power when we write. Since pwrite isn't guaranteed to be atomic, if we crash, we can end up with some fraction of the write completing, causing data corruption. One way to avoid this problem is to store an "undo log" that will let us restore corrupted data. Before we're modify the file, we'll make a copy of the data that's going to be modified (into the undo log), then we'll modify the file as normal, and if nothing goes wrong, we'll delete the undo log.

If we crash while we're writing the undo log, that's fine -- we'll see that the undo log isn't complete and we know that we won't have to restore because we won't have started modifying the file yet. If we crash while we're modifying the file, that's also ok. When we try to restore from the crash, we'll see that the undo log is complete and we can use it to recover from data corruption:

creat(/d/log) // Create undo log
write(/d/log, "2,3,foo", 7) // To undo, at offset 2, write 3 bytes, "foo"
pwrite(/d/orig, “bar", 3, 2) // Modify original file as before
unlink(/d/log) // Delete log file

If we're using ext3 or ext4, widely used Linux filesystems, and we're using the mode data=journal (we'll talk about what these modes mean later), here are some possible outcomes we could get:

d/log: "2,3,f"
d/orig: "a foo"

d/log: ""
d/orig: "a foo"

It's possible we'll crash while the log file write is in progress and we'll have an incomplete log file. In the first case above, we know that the log file isn't complete because the file says we should start at offset 2 and write 3 bytes, but only one byte, f, is specified, so the log file must be incomplete. In the second case above, we can tell the log file is incomplete because the undo log format should start with an offset and a length, but we have neither. Either way, since we know that the log file isn't complete, we know that we don't need to restore.

Another possible outcome is something like:

d/log: "2,3,foo"
d/orig: "a boo"

d/log: "2,3,foo"
d/orig: "a bar"

In the first case, the log file is complete we crashed while writing the file. This is fine, since the log file tells us how to restore to a known good state. In the second case, the write completed, but since the log file hasn't been deleted yet, we'll restore from the log file.

If we're using ext3 or ext4 with data=ordered, we might see something like:

d/log: "2,3,fo"
d/orig: "a boo"

d/log: ""
d/orig: "a bor"

With data=ordered, there's no guarantee that the write to the log file and the pwrite that modifies the original file will execute in program order. Instesad, we could get

creat(/d/log) // Create undo log
pwrite(/d/orig, “bar", 3, 2) // Modify file before writing undo log!
write(/d/log, "2,3,foo", 7) // Write undo log
unlink(/d/log) // Delete log file

To prevent this re-ordering, we can use another syscall, fsync. fsync is a barrier (prevents re-ordering) and it flushes caches (which we'll talk about later).

creat(/d/log)
write(/d/log, “2,3,foo”, 7)
fsync(/d/log) // Add fsync to prevent re-ordering
pwrite(/d/orig, “bar”, 3, 2)
fsync(/d/orig) // Add fsync to prevent re-ordering
unlink(/d/log)

This works with ext3 or ext4, data=ordered, but if we use data=writeback, we might see something like:

d/log: "2,3,WAT"
d/orig: "a boo"

Unfortunately, with data=writeback, the write to the log file isn't guaranteed to be atomic and the filesystem metadata that tracks the file length can get updated before we've finished writing the log file, which will make it look like the log file contains whatever bits happened to be on disk where the log file was created. Since the log file exists, when we try to restore after a crash, we may end up "restoring" random garbage into the original file. To prevent this, we can add a checksum (a way of making sure the file is actually valid) to the log file.

creat(/d/log)
write(/d/log,“…[✓∑],foo”,7) // Add checksum to log file to detect incomplete log file
fsync(/d/log)
pwrite(/d/orig, “bar”, 3, 2)
fsync(/d/orig)
unlink(/d/log)

This should work with data=writeback, but we could still see the following:

d/orig: "a boo"

There's no log file! Although we created a file, wrote to it, and then fsync'd it. Unfortunately, there's no guarantee that the directory will actually store the location of the file if we crash. In order to make sure we can easily find the file when we restore from a crash, we need to fsync the parent of the newly created log.

creat(/d/log)
write(/d/log,“…[✓∑],foo”,7)
fsync(/d/log)
fsync(/d) /// fsync parent directory
pwrite(/d/orig, “bar”, 3, 2)
fsync(/d/orig)
unlink(/d/log)

There are a couple more things we should do. We shoud also fsync after we're done (not shown), and we also need to check for errors. These syscalls can return errors and those errors need to be handled appropriately. There's at least one filesystem issue that makes this very difficult, but since that's not an API usage thing per se, we'll look at this again in the Filesystems section.

We've now seen what we have to do to write a file safely. It might be more complicated than we like, but it seems doable -- if someone asks you to write a file in a self-contained way, like an interview question, and you know the appropriate rules, you can probably do it correctly. But what happens if we have to do this as a day-to-day part of our job, where we'd like to write to files safely every time to write to files in a large codebase.

API in practice

Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope write to files safely, like databases and version control systems: Leveldb, LMDB, GDBM, HSQLDB, Sqlite, PostgreSQL, Git, Mercurial, HDFS, Zookeeper. They then wrote a static analysis tool that can find incorrect usage of the file API, things like incorrectly assuming that operations that aren't atomic are actually atomic, incorrectly assuming that operations that can be re-ordered will execute in program order, etc.

When they did this, they found that every single piece of software they tested except for SQLite in one particular mode had at least one bug. This isn't a knock on the developers of this software or the software -- the programmers who work on things like Leveldb, LBDM, etc., know more about filesystems than the vast majority programmers and the software has more rigorous tests than most software. But they still can't use files safely every time! A natural follow-up to this is the question: why the file API so hard to use that even experts make mistakes?

Concurrent programming is hard

There are a number of reasons for this. If you ask people "what are hard problems in programming?", you'll get answers like distributed systems, concurrent programming, security, aligning things with CSS, dates, etc.

And if we look at what mistakes cause bugs when people do concurrent programming, we see bugs come from things like "incorrectly assuming operations are atomic" and "incorrectly assuming operations will execute in program order". These things that make concurrent programming hard also make writing files safely hard -- we saw examples of both of these kinds of bugs in our first example. More generally, many of the same things that make concurrent programming hard are the same things that make writing to files safely hard, so of course we should expect that writing to files is hard!

Another property writing to files safely shares with concurrent programming is that it's easy to write code that has infrequent, non-deterministc failures. With respect to files, people will sometimes say this makes things easier ("I've never noticed data corruption", "your data is still mostly there most of the time", etc.), but if you want to write files safely because you're working on software that shouldn't corrupt data, this makes things more difficult by making it more difficult to tell if your code is really correct.

API inconsistent

As we saw in our first example, even when using one filesystem, different modes may have significantly different behavior. Large parts of the file API look like this, where behavior varies across filesystems or across different modes of the same filesystem. For example, if we look at mainstream filesystems, appends are atomic, except when using ext3 or ext4 with data=writeback, or ext2 in any mode and directory operations can't be re-ordered w.r.t. any other operations, except on btrfs. In theory, we should all read the POSIX spec carefully and make sure all our code is valid according to POSIX, but if they check filesystem behavior at all, people tend to code to what their filesystem does and not some abtract spec.

If we look at one particular mode of one filesystem (ext4 with data=journal), that seems relatively possible to handle safely, but when writing for a variety of filesystems, especially when handling filesystems that are very different from ext3 and ext4, like btrfs, it becomes very difficult for people to write correct code.

Docs unclear

In our first example, we saw that we can get different behavior from using different data= modes. If we look at the manpage (manual) on what these modes mean in ext3 or ext4, we get:

journal: All data is committed into the journal prior to being written into the main filesystem.

ordered: This is the default mode. All data is forced directly out to the main file system prior to its metadata being committed to the journal.

writeback: Data ordering is not preserved – data may be written into the main filesystem after its metadata has been committed to the journal. This is rumoured to be the highest-throughput option. It guarantees internal filesystem integrity, however it can allow old data to appear in files after a crash and journal recovery.

If you want to know how to use your filesystem safely, and you don't already know what a journaling filesystem is, this definitely isn't going to help you. If you know what a journaling filesystem is, this will give you some hints but it's still not sufficient. It's theoretically possible to figure everything out from reading the source code, but this is pretty impractical for most people who don't already know how the filesystem works.

For English-language documentation, there's lwn.net and the Linux kernel mailing list (LKML). LWN is great, but they can't keep up with everything, so you LKML is the place to go if you want something comprehensive. Here's an example of an exchange on LKML about filesystems:

Dev 1: Personally, I care about metadata consistency, and ext3 documentation suggests that journal protects its integrity. Except that it does not on broken storage devices, and you still need to run fsck there.
Dev 2: as the ext3 authors have stated many times over the years, you still need to run fsck periodically anyway.
Dev 1: Where is that documented?
Dev 2: linux-kernel mailing list archives.
FS dev: Probably from some 6-8 years ago, in e-mail postings that I made.

While the filesystem developers tend to be helpful and they write up informative responses, most people probably don't keep up with the past 6-8 years of LKML.

Performance / correctness conflict

Another issue is that the file API has an inherent conflict between performance and correctness. We noted before that fsync is a barrier (which we can use to enforce ordering) and that it flushes caches. If you've ever worked on the design of a high-performance cache, like a microprocessor cache, you'll probably find the bundling of these two things into a single primitive to be unusual. A reason this is unusual is that flushing caches has a significant performance cost and there are many cases where we want to enforce ordering without paying this performance cost. Bundling these two things into a single primitive forces us to pay the cache flush cost when we only care about ordering.

Chidambaram et al., SOSP’13 looked at the performance cost of this by modifying ext4 to add a barrier mechanism that doesn't flush caches and they found that, if they modified software appropriately and used their barrier operation where a full fsync wasn't necessary, they were able to achieve performance roughly equivalent to ext4 with cache flushing entirely disabled (which is unsafe and can lead to data corruption) without sacrificing safety. However, making your own filesystem and getting it adopted is impractical for most people writing user-level software. Some databases will bypass the filesystem entirely or almost entirely, but this is also impractical for most software.

That's the file API. Now that we've seen that it's extraordinarily difficult to use, let's look at filesystems.

Filesystem

If we want to make sure that filessystems work, one of the most basic tests we could do is to inject errors are the layer below the filesystem to see if the filesystem handles them properly. For example, on a write, we could have the disk fail to write the data and return the appropriate error. If the filesystem drops this error or doesn't handle ths properly, that means we have data loss or data corruption. This is analogous to the kinds of distributed systems faults Kyle Kingsbury talked about in his distributed systems testing talk yesterday (although these kinds of errors are much more straightforward to test).

Prabhakaran et al., SOSP’05 did this and found that, for most filesystems tested, almost all write errors were dropped. The major exception to this was on ReiserFS, which did a pretty good job with all types of errors tested, but ReiserFS isn't really used today for reasons beyond the scope of this talk.

We (Wesley Aptekar-Cassels and I) looked at this again in 2017 and found that things had improved significantly. Most filesystems (other than JFS) could pass these very basic tests on error handling.

Another way to look for errors is to look at filesystems code to see if it handles internal errors correctly. Gunawai et al., FAST’08 did this and found that internal errors were dropped a significant percentage of the time. The technique they used made it difficult to tell if functions that could return many different errors were correctly handling each error, so they also looked at calls to functions that can only return a single error. In those cases, depending on the function, errors were dropped roughly 2/3 to 3/4 of the time, depending on the function.

Wesley and I also looked at this again in 2017 and found significant improvement -- errors for the same functions Gunawi et al. looked at were "only" ignored 1/3 to 2/3 of the time, depending on the function.

Gunawai et al. also looked at comments near these dropped errors and found comments like "Just ignore errors at this point. There is nothing we can do except to try to keep going." (XFS) and "Error, skip block and hope for the best." (ext3).

Now we've seen that while filesystems used to drop even the most basic errors, they now handle then correctly, but there are some code paths where errors can get dropped. For a concrete example of a case where this happens, let's look back at our first example. If we get an error on fsync, unless we have a pretty recent Linux kernel (Q2 2018-ish), there's a pretty good chance that the error will be dropped and it may even get reported to the wrong process!

On recent Linux kernels, there's a good chance the error will be reported (to the correct process, even). Wilcox, PGCon’18 notes that an error on fsync is basically unrecoverable. The details for depending on filesystem -- on XFS and btrfs, modified data that's in the filesystem will get thrown away and there's no way to recover. On ext4, the data isn't thrown away, but it's marked as unmodified, so the filesystem won't try to write it back to disk later, and if there's memory pressure, the data can be thrown out at any time. If you're feeling adventurous, you can try to recover the data before it gets thrown out with various tricks (e.g., by forcing the filesystem to mark it as modified again, or by writing it out to another device, which will force the filesystem to write the data out even though it's marked as unmodified), but there's no guarantee you'll be able to recover the data before it's thrown out. On Linux ZFS, it appears that there's a code path designed to do the right thing, but CPU usage spikes and the system may hang or become unusable.

In general, there isn't a good way to recover from this on Linux. Postgres, MySQL, and MongoDB (widely used databases) will crash themselves and the user is expected to restore from the last checkpoint. Most software will probably just silently lose or corrupt data. And fsync is a relatively good case -- for example, syncfs simply doesn't return errors on Linux at all, leading to silent data loss and data corruption.

BTW, when Craig Ringer first proposed that Postgres should crash on fsync error, the first response on the Postgres dev mailing list was:

Surely you jest . . . If [current behavior of fsync] is actually the case, we need to push back on this kernel brain damage

But after talking through the details, everyone agreed that crashing was the only good option. One of the many unfortunate things is that most disk errors are transient. Since the filesystem discards critical information that's necessary to proceed without data corruption on any error, transient errors that could be retried instead force software to take drastic measures.

And while we've talked about Linux, this isn't unique to Linux. Fsync error handling (and error handling in general) is broken on many different operating systems. At the time Postgres "discovered" the behavior of fsync on Linux, FreeBSD had arguably correct behavior, but OpenBSD and NetBSD behaved the same as Linux (true error status dropped, retrying causes success response, data lost). This has been fixed on OpenBSD and probably some other BSDs, but Linux still basically has the same behavior and you don't have good guarantees that this will work on any random UNIX-like OS.

Now that we've seen that, for many years, filesystems failed to handle errors in some of the most straightforward and simple cases and that there are cases that still aren't handled correctly today, let's look at disks.

Disk

Flushing

We've seen that it's easy to not realize we have to call fsync when we have to call fsync, and that even if we call fsync appropriately, bugs may prevent fsync from actually working. Rajimwale et al., DSN’11 into whether or not disks actually flush when you ask them to flush, assuming everything above the disk works correctly (their paper is actually mostly about something else, they just discuss this briefly at the beginning). Someone from Microsoft anonymously told them "[Some disks] do not allow the file system to force writes to disk properly" and someone from Seagate, a disk manufacturer, told them "[Some disks (though none from us)] do not allow the file system to force writes to disk properly". Bairavasundaram et al., FAST’07 also found the same thing when they looked into disk reliability.

Error rates

We've seen that filessystems sometimes don't handle disk errors correctly. If we want to know how serious this issue is, we should look at the rate at which disks emit errors. Disk datasheets will usually an uncorrectable bit error rate of 1e-14 for consumer HDDs (often called spinning metal or spinning rust disks), 1e-15 for enterprise HDDs, 1e-15 for consumer SSDs, and 1e-16 for enterprise SSDs. This means that, on average, we expect to see one unrecoverable data error every 1e14 bits we read on an HDD.

To get an intuition for what this means in practice, 1TB is now a pretty normal disk size. If we read a full drive once, that's 1e12 bytes, or almost 1e13 bits (technically 8e12 bits), which means we should see, in expectation, one unrecoverable if we buy a 1TB HDD and read the entire disk ten-ish times. Nowadays, we can buy 10TB HDDs, in which case we'd expect to see an error (technically, 8/10th errors) on every read of an entire consumer HDD.

In practice, observed data rates are are significantly higher. Narayanan et al., SYSTOR’16 (Microsoft) observed SSD error rates from 1e-11 to 6e-14, depending on the drive model. Meza et al., SIGMETRICS’15 (FB) observed even worse SSD error rates, 2e-9 to 6e-11 depending on the model of drive. Depending on the type of drive, 2e-9 is 2 gigabits, or 250 MB, 500 thousand to 5 million times worse than stated on datasheets depending on the class of drive.

Bit error rate is arguably a bad metric for disk drives, but this is the metric disk vendors claim, so that's what we have to compare against if we want an apples-to-apples comparison. See Bairavasundaram et al., SIGMETRICS'07, Schroeder et al., FAST'16, and others for other kinds of error rates.

One thing to note is that it's often claimed that SSDs don't have problems with corruption because they use error correcting codes (ECC), which can fix data corruption issues. "Flash banishes the specter of the unrecoverable data error", etc. The thing this misses is that modern high-density flash devices are very unreliable and need ECC to be usable at all. Grupp et al., FAST’12 looked at error rates of the kind of flash the underlies SSDs and found errors rates from 1e-1 to 1e-8. 1e-1 is one error every ten bits, 1e-8 is one error every 100 megabits.

Power loss

Another claim you'll hear is that SSDs are safe against power loss and some types of crashes because they now have "power loss protection" -- there's some mechanism in the SSDs that can hold power for long enough during an outage that the internal SSD cache can be written out safely.

Luke Leighton tested this by buying 6 SSDs that claim to have power loss protection and found that four out of the six models of drive he tested failed (every drive that wasn't an Intel drive). If we look at the details of the tests, when drives fail, it appears to be because they were used in a way that the implementor of power loss protection didn't expect (writing "too fast", although well under the rate at which the drive is capable of writing, or writing "too many" files in parallel). When a drive advertises that it has power loss protection, this appears to mean that someone spent some amount of effort implementing something that will, under some circumstances, prevent data loss or data corruption under power loss. But, as we saw in Kyle's talk yesterday on distributed systems, if you want to make sure that the mechanism actually works, you can't rely on the vendor to do rigorous or perhaps even any semi-serious testing and you have to test it yourself.

Retention

If we look at SSD datasheets, a young-ish drive (one with 90% of its write cycles remaining) will usually be specced to hold data for about ten years after a write. If we look at a worn out drive, one very close to end-of-life, it's specced to retain data for one year to three months, depending on the class of drive. I think people are often surprised to find that it's within spec for a drive to lose data three months after the data is written.

These numbers all come from datasheets and specs, as we've seen, datasheets can be a bit optimistic. On many early SSDs, using up most or all of a drives write cycles would cause the drive to brick itself, so you wouldn't even get the spec'd three month data retention.

Corollaries

Now that we've seen that there are significant problems at every level of the file stack, let's look at a couple things that follow from this.

What to do?

What we should do about this is a big topic, in the time we have left, one thing we can do instead of writing to files is to use databases. If you want something lightweight and simple that you can use in most places you'd use a file, SQLite is pretty good. I'm not saying you should never use files. There is a tradeoff here. But if you have an application where you'd like to reduce the rate of data corruption, considering using a database to store data instead of using files.

FS support

At the start of this talk, we looked at this Dropbox example, where most people thought that there was no reason to remove support for most Linux filesystems because filesystems are all the same. I believe their hand was forced by the way they want to store/use data, which they can only do with ext given how they're doing things (which is arguably a mis-feature), but even if that wasn't the case, perhaps you can see why software that's attempting to sync data to disk reliably and with decent performance might not want to support every single filesystem in the universe for an OS that, for their product, is relatively niche. Maybe it's worth supporting every filesystem for PR reasons and then going through the contortions necessary to avoid data corruption on a per-filesystem basis (you can try coding straight to your reading of the POSIX spec, but as we've seen, that won't save you on Linux), but the PR problem is caused by a misunderstanding.

The other comment we looked at on reddit, and also a common sentiment, is that it's not a program's job to work around bugs in libraries or the OS. But user data gets corrupted regardless of who's "fault" the bug is, and as we've seen, bugs can persist in the filesystem layer for many years. In the case of Linux, most filesystems other than ZFS seem to have decided it's correct behavior to throw away data on fsync error and also not report that the data can't be written (as opposed to FreeBSD or OpenBSD, where most filesystems will at least report an error on subsequent fsyncs if the error isn't resolved). This is arguably a bug and also arguably correct behavior, but either way, if your software doesn't take this into account, you're going to lose or corrupt data. If you want to take the stance that it's not your fault that the filesystem is corrupting data, your users are going to pay the cost for that.

FAQ

While putting this talk to together, I read a bunch of different online discussions about how to write to files safely. For discussions outside of specialized communities (e.g., LKML, the Postgres mailing list, etc.), many people will drop by to say something like "why is everyone making this so complicated? You can do this very easily and completely safely with this one weird trick". Let's look at the most common "one weird trick"s from two thousand internet comments on how to write to disk safely.

Rename

The most frequently mentioned trick is to rename instead of overwriting. If you remember our single-file write example, we made a copy of the data that we wanted to overwrite before modifying the file. The trick here is to do the opposite:

  1. Make a copy of the entire file
  2. Modify the copy
  3. Rename the copy on top of the original file

This trick doesn't work. People seem to think that this is safe becaus the POSIX spec says that rename is atomic, but that only means rename is atomic with respect to normal operation, that doesn't mean it's atomic on crash. This isn't just a theoretical problem; if we look at mainstream Linux filesystems, most have at least one mode where rename isn't atomic on crash. Rename also isn't guaranteed to execute in program order, as people sometimes expect.

The most mainstream exception where rename is atomic on crash is probably btrfs, but even there, it's a bit subtle -- as noted in Bornholt et al., ASPLOS’16, rename is only atomic on crash when renaming to replace an existing file, not when renaming to create a new file. Also, Mohan et al., OSDI’18 found numerous rename atomicity bugs on btrfs, some quite old and some introduced the same year as the paper, so you want not want to rely on this without extensive testing, even if you're writing btrfs specific code.

And even if this worked, the performance of this technique is quite poor.

Append

The second most frequently mentioned trick is to only ever append (instead of sometimes overwriting). This also doesn't work. As noted in Pillai et al., OSDI’14 and Bornholt et al., ASPLOS’16, appends don't guarantee ordering or atomicity and believing that appends are safe is the cause of some bugs.

One weird tricks

We've seen that the most commonly cited simple tricks don't work. Something I find interesting is that, in these discussions, people will drop into a discussion where it's already been explained, often in great detail, why writing to files is harder than someone might naively think, ignore all warnings and explanations and still proceed with their explanation for why it's, in fact, really easy. Even when warned that files are harder than people think, people still think they're easy!

Conclusion

In conclusion, computers don't work (but you probably already know this if you're here at Gary-conf). This talk happened to be about files, but there are many areas we could've looked into where we would've seen similar things.

One thing I'd like to note before we finish is that, IMO, the underlying problem isn't technical. If you look at what huge tech companies do (companies like FB, Amazon, MS, Google, etc.), they often handle writes to disk pretty safely. They'll make sure that they have disks where power loss protection actually work, they'll have patches into the OS and/or other instrumentation to make sure that errors get reported correctly, there will be large distributed storage groups to make sure data is replicated safely, etc. We know how to make this stuff pretty reliable. It's hard, and it takes a lot of time and effort, i.e., a lot of money, but it can be done.

If you ask someone who works on that kind of thing why they spend mind boggling sums of money to ensure (or really, increase the probability of) correctness, you'll often get an answer like "we have a zillion machines and if you do the math on the rate of data corruption, if we didn't do all of this, we'd have data corruption every minute of every day. It would be totally untenable". A huge tech company might have, what, order of ten million machines? The funny thing is, if you do the math for how many consumer machines there are out there and much consumer software runs on unreliable disks, the math is similar. There are many more consumer machines; they're typically operated at much lighter load, but there are enough of them that, if you own a widely used piece of desktop/laptop/workstation software, the math on data corruption is pretty similar. Without "extreme" protections, we should expect to see data corruption all the time.

But if we look at how consumer software works, it's usually quite unsafe with respect to handling data. IMO, the key difference here is that when a huge tech company loses data, whether that's data on who's likely to click on which ads or user emails, the company pays the cost, directly or indirectly and the cost is large enough that it's obviously correct to spend a lot of effort to avoid data loss. But when consumers have data corruption on their own machines, they're mostly not sophisticated enough to know who's at fault, so the company can avoid taking the brunt of the blame. If we have a global optimization function, the math is the same -- of course we should put more effort into protecting data on consumer machines. But if we're a company that's locally optimizing for our own benefit, the math works out differently and maybe it's not worth it to spend a lot of effort on avoiding data corruption.

Yesterday, Ramsey Nasser gave a talk where he made a very compelling case that something was a serious problem, which was followed up by a comment that his proposed solution will have a hard time getting adoption. I agree with both parts -- he discussed an important problem, and it's not clear how solving that problem will make anyone a lot of money, so the problem is likely to go unsolved.

With GDPR, we've seen that regulation can force tech companies to protect people's privacy in a way they're not naturally inclined to do, but regulation is a very big hammer and the unintended consequences can often negate or more than negative the benefits of regulation. When we look at the history of regulations that are designed to force companies to do the right thing, we can see that it's often many years, sometimes decades, before the full impact of the regulation is understood. Designing good regulations is hard, much harder than any of the technical problems we've discussed today.

Acknowledgements

Thanks to Leah Hanson, Gary Bernhardt, Kamal Marhubi, Rebecca Isaacs, Jesse Luehrs, Tom Crayford, Wesley Aptekar-Cassels, Rose Ames, and Benjamin Gilbert for their help with this talk!

Sorry we went so fast. If there's anything you missed you can catch it in the pseudo-transcript at danluu.com/deconstruct-files.

This "transcript" is pretty rough since I wrote it up very quickly this morning before the talk. I'll try to clean it within a few weeks, which will include adding material that was missed, inserting links, fixing typos, adding references that were missed, etc.

Thanks to Anatole Shaw, Jernej Simoncic, @junh1024, and Josh Duff for comments/corrections/discussion on this transcript.

July 11, 2019

Frederic Cambus (fcambus)

Fuzzing DNS zone parsers July 11, 2019 01:00 PM

In my never-ending quest to improve the quality of my C codebases, I've been using AFL to fuzz statzone, the zone parser I use to generate monthly statistics on StatDNS. It helped me to find and fix a NULL pointer dereference.

I initially used the .arpa zone file as input, but then remembered that OpenDNSSEC bundles a special zone for testing purposes, containing a lot of seldom used resource records types, and decided to use this one too.

Out of curiosity, I decided to try fuzzing other DNS zone parsers. I started with validns 0.8, and within seconds the fuzzer found multiple NULL pointer dereferences.

validns

The first occurrence happens in the name2findable_name() function, and can be triggered with the following input:

arpa                    86400   IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2019021500 1800 900 604800 86400
arpa.                   86400   IN      RRSIG   SOA 8 1 86400 20190228000000 20190214230000 49906 arpa. Qot7qHAA2QhNmAz3oJUIGmxGJrKnWsIzEvZ92R+LV03K7YTFozio2U7Z534RZBhc0UJvlF1YenrbM6ugmF0z55CJD9JY7cFicalFPOkIuWslSl62vuIWHLwN5sA7VZ0ooVN2ptQpPHDa3W/9OPJRF0YqjBBBwD7IiL7V560rbXM=

With the above input, the following call to strlen(3) in rr.c results in a NULL pointer dereference because 's' ends up being NULL:

static unsigned char *name2findable_name(char *s)
{
    int l = strlen(s);

The second occurrence happens in the nsec_validate_pass2() function, and can be triggered with the following input:

arpa.                   86400   IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2019021500 1800 900 604800 86400
arpa.                   86400   IN      NSEC    a

With the above input, the following call to strcasecmp(3) in rr.c results in a NULL pointer dereference because 'rr->next_domain' ends up being NULL:

if (strcasecmp(rr->next_domain, zone_apex) == 0) {

Given those encouraging results, I went on to fuzz BIND, NSD and Knot zone parsers, using their zone validation tools named-checkzone, nsd-checkzone, and kzonecheck respectively.

While the fuzzers didn't produce any crash for BIND and Knot after running for 3 days and 11 hours, they did produce some valid ones for NSD, and I decided to continue on nsd-checkzone and stop the other fuzzers.

nsd-checkzone

I let AFL complete one cycle, and as I didn't need the box for anything else at this time, I decided to let it run for a few more days. I ended the process after 16 days and 19 hours, completing 2 cycles with 167 unique crashes.

nsd-checkzone

After sorting and analyzing the crashes, I had two valid issues to report.

The first one is an out-of-bounds read caused by improper validation of array index, in the rdata_maximum_wireformat_size() function, in rdata.c.

The second one is a stack-based buffer overflow in the dname_concatenate() function in dname.c, which got assigned CVE-2019-13207.

=================================================================
==7395==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffcd6a9763f at pc 0x0000004dadbc bp 0x7ffcd6a97510 sp 0x7ffcd6a96cc0
WRITE of size 8 at 0x7ffcd6a9763f thread T0
    #0 0x4dadbb in __asan_memcpy (/home/fcambus/nsd/nsd-checkzone+0x4dadbb)
    #1 0x534251 in dname_concatenate /home/fcambus/nsd/dname.c:464:2
    #2 0x69e61f in yyparse /home/fcambus/nsd/./zparser.y:1024:12
    #3 0x689fd1 in zonec_read /home/fcambus/nsd/zonec.c:1623:2
    #4 0x6aedd1 in check_zone /home/fcambus/nsd/nsd-checkzone.c:61:11
    #5 0x6aea07 in main /home/fcambus/nsd/nsd-checkzone.c:127:2
    #6 0x7fa60ece6b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
    #7 0x41c1d9 in _start (/home/fcambus/nsd/nsd-checkzone+0x41c1d9)

Address 0x7ffcd6a9763f is located in stack of thread T0 at offset 287 in frame
    #0 0x533f8f in dname_concatenate /home/fcambus/nsd/dname.c:458

  This frame has 1 object(s):
    [32, 287) 'temp' (line 459) <== Memory access at offset 287 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow (/home/fcambus/nsd/nsd-checkzone+0x4dadbb) in __asan_memcpy
Shadow bytes around the buggy address:
  0x10001ad4ae70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001ad4ae80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001ad4ae90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001ad4aea0: 00 00 00 00 f1 f1 f1 f1 00 00 00 00 00 00 00 00
  0x10001ad4aeb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x10001ad4aec0: 00 00 00 00 00 00 00[07]f3 f3 f3 f3 f3 f3 f3 f3
  0x10001ad4aed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001ad4aee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001ad4aef0: 00 00 00 00 f1 f1 f1 f1 00 00 00 00 00 00 00 00
  0x10001ad4af00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001ad4af10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==7395==ABORTING

Both issues have been fixed and will be part of the NSD 4.2.2 release.

I also fuzzed ldns using ldns-read-zone for 12 days and 7 hours but the only crashes it produced were in fact only triggering assertions.

It's been an interesting journey so far, and while finding issues is still relatively easy, time required to sort crashes and distinguish between valid, duplicates, and false positives takes a lot of time. Nonetheless, reading 3rd party source code and analyzing what is going on and why the program crashes is both very instructing and rewarding.

For the time being, I plan to continue fuzzing stuff and will write more about my findings.

July 09, 2019

Mark J. Nelson (mjn)

C.A.R. Hoare's distant connection to the C. Hoare & Co private bank July 09, 2019 12:00 PM

A recent article on UK private bank C. Hoare & Co. made the rounds, noting that it has been run for more than 300 years by the Hoare family. Somehow this found its way into tech discussion boards, which led some people to wonder whether pioneering British computer scientist C.A.R. "Tony" Hoare is from the same Hoare family.

I investigated briefly, and as it turns out the answer appears to be: yes, but with only a distant connection to the branch of the family running the bank, now five generations removed.

The hint for how to track this down came from a mention in the book Reflections on the Work of C.A.R. Hoare (Springer, 2010) that C.A.R. Hoare was of a "somewhat upper-class" background, with a footnote pointing to thepeerage.com, a site primarily focusing on genealogy of the British upper classes.

From this, I was able to trace his family history back along the paternal line as follows:

At this point the genealogy seems to connect to someone who has additional biographical details in Wikipedia, assuming that the Charles James Hoare is the same one described in this article (besides the same name, the approximate timeframe and occupations match). Wikipedia describes Charles James Hoare as "third son of Henry ('Harry') Hoare, banker, of Fleet Street, London, a Partner in C. Hoare & Co", so there's the connection to the banking family, at a great-great-great grandfather (five generations back) distance, around the late 1700s or early 1800s.

The generations after that appear unrelated to banking though: clergyman, clergyman, military officer, colonial civil servant, and finally professor. (The colonial-civil-servant occupation explains the somewhat well known fact that C.A.R. Hoare was born in Sri Lanka.)

Not that this distant connection is necessarily very important biographical information, but there you have it, assuming I haven't made an error in reading the sources, and assuming that they also haven't made any errors.

July 02, 2019

Andreas Zwinkau (qznc)

Definitions of Software Architecture July 02, 2019 12:00 AM

Software architecture documents the shared understanding of a software system.

Read full article!

July 01, 2019

Kevin Burke (kb)

Using a Bernzomatic TS8000 Kitchen Torch to Sear Meat July 01, 2019 04:43 AM

For a company trying to sell and explain a product, a lot of this information was amazingly difficult to find so I wrote this.

For the rest of this we'll assume you're cooking a steak but the same advice applies to most other meats/cuts.

Why Do You Want A Torch to Sear Meat

To get a good texture on a steak, you need to get the outside of the steak really hot. If you keep the steak on the heat source for too long, though, it will start to cook the inside too much, and you'll end up with a well done steak.

A torch helps with this problem by making it possible to sear the outside of the steak very quickly which means the outside can get a nice sear while the inside stays at medium rare.

What to Get

You want a torch that can char the surface of a steak in under a minute. Most of the torches can't actually get that hot!

I used a butane crème brûlée torch for years and would get frustrated my steaks didn't look like the ones on Serious Eats, unless I torched them for 5-6 minutes. The torch wasn't hot enough. Using a cast iron pan by itself, even on the highest setting is also not likely to get hot enough. If your steak does not look as black as the one in Serious Eats the outside is not getting hot enough.

I read the Wirecutter guide and got the Bernzomatic TS8000. This torch is definitely hot enough; with propane it's about 3x as hot as my crème brûlée torch. I used it tonight to sear a tri tip. It is so much better.

Is that it? (No, you need fuel)

No. The Bernzomatic torch is just a torch; you still need to buy fuel. The TS8000 works with either butane or propane. You don't want butane though, you want propane, because it gets 50% hotter than butane does.

Every photo on the Internet of both the TS8000 and propane fuel tanks either shows them connected or with the caps on. You need a propane tank that can connect to this, on the bottom of the TS8000.

The top of your propane canister should look like this, with the cap off:

Bernzomatic sells a compatible 14 oz propane tank. You can also use the green Coleman camping gas tanks. The advantage of the Coleman tank is it is cheap and your grocery/hardware store probably has them. It weighs 16 ounces full, which sounds heavy but, it's fine, it's what I used.

I tried to find a small tank - 5-10 ounces - but didn't find anything promising. You'll probably get something in the 12-16 ounce range. Just put it on the end of the torch and sear with the whole tank on the end of the torch.

Summary

  • Get the Bernzomatic TS8000, it is one of the few torches that gets hot enough

  • Use propane, not butane, with it

  • Coleman camping gas tanks are fine

That's it. Here is the manual for the TS8000. Happy searing!

Robin Schroer (sulami)

My Thoughts on spec July 01, 2019 12:00 AM

In the beginning of this year I started a new job, and I am now fortunate enough to be writing Clojure full-time. I believe that Clojure is a very well crafted language and enables developers like few others, but I have also some grievances to report. I want to prefix this by saying that I love Clojure despite its faults, and this is more of a constructive criticism piece than anything else.

What Is spec?

What I want to discuss today is clojure.spec, the gradual typing solution shipped with Clojure since 1.9.I am of course aware that spec is officially in alpha, and that there is a second alpha version which might address some of my points here. But as I have not tried alpha 2 yet, and a lot of people are using alpha 1, we will be mostly looking at alpha 1 in this post. Still, feel free to contact me and tell me how alpha 2 will improve things.

In case you are not familiar with spec, here is a quick run-down of how it works:

Clojure is a dynamically typed language, meaning no (or very few) type annotations in the source code, unlike say Haskell or most curl-brace languages. While this removes a lot of visual clutter, and the language is designed in such a way that most functions can operate on most types for maximum flexibility, this also means that sometimes things break at run time due to type errors. To make dealing with types easier, spec allows you to place type definitions in strategic places, say at entry- and exit points of an app or a module to ensure that your idea of structure and type of your data still lines up with reality.

The benefit of this approach over static typing is that you can encapsulate parts of your software with type safety without having to deal with the encumbrance in the internals. You define a contract to the outside, but stay flexible in your implementation.

In practice it looks like this:

(ns spec-demo
  (:require [clojure.spec.alpha :as s]
            [clojure.spec.gen.alpha :as gen]))
            
(s/def ::name string?)
(s/def ::age pos-int?)

(s/def ::person
  (s/keys :req-un [::name
                   ::age]))
                
(s/valid? ::person
          {:name "John"
           :age 35})
;; => true

(s/explain ::person
           {:name 35})
;; => 35 - failed: string? in: [name] at: [:name] spec: :spec-demo/name
;;    {:name 35} - failed: (contains? % :age) spec: :spec-demo/person

(gen/generate
  (s/gen ::person))
;; => {:name "3o311eSXA4Dm9cLkENIt3J5Gb", :age 3318}

This snippet demonstrates creating a spec person which has sub-specs name and age, validates some data against the spec, explains why some data does not validate, and also generates some random data compliant with the spec. The last bit is incredibly useful for testing, as you can imagine. There are some more advanced features than what I have shown here, like specs for in- and output types of functions, but this is the gist of it.

So What Is Wrong with spec?

specs as Macros

spec is implemented using macros, which are compile-time expansions, think template meta-programming in C++. This brings some limitations, which are especially noticeable in Clojure where, like in most other Lisps, code is treated as data and many parts of your software are introspectable at run time. specs are, once defined, not easy to extract data from.

Contrast this with schema, a Clojure library which does essentially the same thing, and got started before spec. schema “schemas” are just hash maps, which are easily constructed and introspected at run time. schema has other problems, but the community seems to generally flock to spec as the officially blessed version.

The situation I have now repeatedly seen is one where I have a spec defining a specific representation of an entity, where for example one spec extends another one. A practical example here is when augmenting an entity with additional data, so that the output representation is a modified version of the input type.

You can define a spec as a superset of another spec, and only in that direction, using s/or, but at that point you are already stuck in the spec an cannot get the list of fields easily out of your specs. It would be nice to be able to define one spec in terms of flat data, which would also be available at run time. This would be incredibly helpful for use with something like select-keys.

There are ways of working around this, for example by using eval, or wrapping your spec macros in another layer of macros, but both of these are more hacks than solutions. spec alpha 2 promises to solve this particular problem by separating symbolic specs from spec objects, the former being just flat data which can also be generated programmatically.

If spec alpha 2 works out the way I hope, this will be solved soon, and the documentation looks promising so far.

Namespacing

spec is very intent on you using namespaces for your keywords, which in general is a good idea. Many developers, including myself, have been using the namespace of a key to give context about its use, for example :person/name is a different spec from :company/name. The problem with this is the overloading of namespaced keywords, which are an existing Clojure feature. These namespaces person and company do not actually exist, and I do not want to clutter all my keys with a long prefix like :myapp.entities.person/name to make it match a real namespace.

Now namespaced keywords can have any namespace, and the namespace does not actually have to exist for everything related to keywords to work well, except when it does. If you want to use a spec from another namespace, but alias that namespace to shorten it, you need to require that namespace, for which it needs to exist. As a workaround to this I have created “fake” namespaces in the past using a helper.

This actually leads me to another problem, which is the question of where to place specs in the first place. spec comes with a global, centralised registry for specs, which in alpha 1 you cannot opt out of. In theory this allows you to define/register your spec once in any place you like, and then access it from anywhere without having to know where it comes from or requiring it. This, while having the potential for being too opaque and magical, is actually a very good feature in my opinion. It is trivial to override specs when reloading them, and I have not experienced any issues with evaluation order yet. Due to specs referring to other specs by their name, you can define dependencies of a spec after the fact, and the system will pick them up accordingly.

My current solution for this is having a file & namespace for every entity which can be required and aliased normally, the only problem with this being relationships between specs. As soon as one spec needs to include another spec, dependencies get muddled, so I have experimented with having a special namespace for specs which are shared across many other specs, but this is far from ideal. I wish there was a cleaner way to do this, especially leveraging the global registry.

Function spec Definitions

I mentioned above that spec also allows instrumenting functions, but the semantics for this are a bit wonky in my opinion. See for yourself:

(defn double [x]
  (* 2 x))
  
(s/fdef double
  :args (s/cat :x int?)
  :ret int?
  :fn #(= (:ret %)
          (-> % :args :x (* 2))))

This naive spec restricts in- and output types to integers, which is okay in this case. The :fn key describes the relationship between in- and output, and is in this case actually absurdly strict, but this is just an example. There are two issues I have with this:

First the :fn definition tends to be very elaborate and hard to understand at a glance. Even in this simple case, there is a lot going on in there, and the use of anonymous functions does not help it. In practice, this key is optional and I omit it almost always, because I cannot think of any formal assertions I want to make about the output which are also reasonably simple to encode. And if you want to make several separate assertions about the output, you almost cannot avoid breaking up the spec into pieces, at which point you have predicates which exist purely for spec.

The other issue I have is that this is decoupled from the actual function definition. In theory you can place the spec in a different namespace and refer to the function using its fully qualified name, and this is tempting, especially when looking at my previous point about these specs having the potential to be far more longer than the actual function definitions. But then your function exists conceptually in two places, and these two places have to be kept in sync. If you move, rename, or modify the function in almost any way, you have to modify the spec, too, but first you have to find the spec.

The problem I can see with this is that :fn can be used to make assertions about the functions, which can in almost all cases be made in unit tests as well. In fact, unit tests are meant for exactly this, asserting single assumptions about units at a time. The condition above could just as well be a test called the result equals twice the input value. Functions are usually only instrumented locally and/or during testing, as they incur non-negligible overhead, and I would argue that they do not provide significant assurances over unit tests.

I would much rather drop the :fn key, and include type declarations in the actual function definition, which is incidentally how schema works:

(s/defn double :- s/Int
  [x :- s/Int]
  (* 2 x))

In the end, I am looking forward to the next iteration of spec, and hope it addresses as many issues as possible, and I am already blown away by how much better it is than alternatives available in other languages.

Andreas Zwinkau (qznc)

One Letter Programming Languages July 01, 2019 12:00 AM

If you are looking for a free name, there is none.

Read full article!

June 29, 2019

jfb (jfb)

Movie Review - A Simple Favor (2019) June 29, 2019 12:00 AM

Untitled code{white-space: pre-wrap;} span.smallcaps{font-variant: small-caps;} span.underline{text-decoration: underline;} div.column{display: inline-block; vertical-align: top; width: 50%;}

I didn’t really know anything about this, going in. It was tonally all over the map; the performances were good, it was competently put together. The pure aesthetics of watching Henry Golding and Blake Lively interact was pretty great; but at the end of the day, the plot was stupid, the pacing weird, and the tone veered too radically between farce and thriller to work for me.

Any time you “solve” the big mystery with a twin, you have failed, screenwriter. Sure, that might be the plot failing of the source material, but it’s still dumb of hell.

I give this movie two stars: one for the aforementioned aesthetics, and one for Anna Kendrick’s teeth.

Rating: ★ ★

June 28, 2019

Grzegorz Antoniak (dark_grimoire)

Enabling lldb on Mojave June 28, 2019 06:00 AM

I have an impression that each version of macOS system introduces additional obstacles for developer. From one point of view I understand that they just want to make the system more secure, and that should be the biggest priority above all (well, maybe not above common sense). But it's hard …

June 27, 2019

Awn Umar (awn)

mutable strings in go June 27, 2019 12:00 AM

According to the Go language specification:

Strings are immutable: once created, it is impossible to change the contents of a string.

We’ll see about that.

data := []byte("yellow submarine")
str := *(*string)(unsafe.Pointer(&data))

for i := range data {
    data[i] = 'x'
    fmt.Printf("%s\n%s\n\n", string(data), str)
}

Try it out yourself.

June 24, 2019

Chris Allen (bitemyapp)

Abusing instance local functional dependencies for nicer number literals June 24, 2019 12:00 AM

I demonstrated how to make kilobyte/megabyte constants in Haskell and a Lobsters user asked how it worked. This is a bloggification of my reply to them explaining the trick.

June 23, 2019

Chris Double (doublec)

Getting Started with Mercury June 23, 2019 11:00 AM

Mercury is a logic programming language, similar to Prolog, but with static types. It feels like a combination of SML and Prolog at times. It was designed to help with programming large systems - that is large programs, large teams and better reliability, etc. The commercial product Prince XML is written in Mercury.

I've played around with Mercury in the past but haven't done anything substantial with it. Recently I picked it up again. This post is a short introduction to building Mercury, and some example "Hello World" style programs to test the install.

Build

Mercury is written in the Mercury language itself. This means it needs a Mercury compiler to bootstrap from. The way I got a build going from source was to download the source for a release of the day version, build that, then use that build to build the Mercury source from github. The steps are outlined in the README.bootstrap file, but the following commands are the basic steps:

$ wget http://dl.mercurylang.org/rotd/mercury-srcdist-rotd-2019-06-22.tar.gz
$ tar xvf mercury-srcdist-rotd-2019-06-22.tar.gz
$ cd mercury-srcdist-rotd-2019-06-22
$ ./configure --enable-minimal-install --prefix=/tmp/mercury
$ make
$ make install
$ cd ..
$ export PATH=/tmp/mercury/bin:$PATH

With this minimal compiler the main source can be built. Mercury has a number of backends, called 'grades' in the documentation. Each of these grades makes a number of tradeoffs in terms of generated code. They define the platform (C, assembler, Java, etc), whether GC is used, what type of threading model is available (if any), etc. The Adventures in Mercury blog has an article on some of the different grades. Building all of them can take a long time - multiple hours - so it pays to limit it if you don't need some of the backends.

For my purposes I didn't need the CSharp backend, but wanted to explore the others. I was ok with the time tradeoff of building the system. To build from the master branch of the github repository I did the following steps:

$ git clone https://github.com/Mercury-Language/mercury
$ cd mercury
$ ./prepare.sh
$ ./configure --enable-nogc-grades --disable-csharp-grade \
              --prefix=/home/myuser/mercury
$ make PARALLEL=-j4
$ make install PARALLEL=-j4
$ export PATH=/home/myuser/mercury/bin:$PATH

Change the prefix to where you want Mercury installed. Add the relevant directories to the PATH as specified by the end of the build process.

Hello World

A basic "Hello World" program in Mercury looks like the following:

:- module hello.

:- interface.
:- import_module io.
:- pred main(io, io).
:- mode main(di, uo) is det.

:- implementation.
main(IO0, IO1) :-
    io.write_string("Hello World!\n", IO0, IO1).

With this code in a hello.m file, it can be built and run with:

$ mmc --make hello
Making Mercury/int3s/hello.int3
Making Mercury/ints/hello.int
Making Mercury/cs/hello.c
Making Mercury/os/hello.o
Making hello    
$ ./hello
Hello World!

The first line defines the name of the module:

:- module hello.

Following that is the definitions of the public interface of the module:

:- interface.
:- import_module io.
:- pred main(io, io).
:- mode main(di, uo) is det.

We publically import the io module, as we use io definitions in the main predicate. This is followed by a declaration of the interface of main - like C this is the user function called by the runtime to execute the program. The definition here declares that main is a predicate, it takes two arguments, of type io. This is a special type that represents the "state of the world" and is how I/O is handled in Mercury. The first argument is the "input world state" and the second argument is the "output world state". All I/O functions take these two arguments - the state of the world before the function and the state of the world after.

The mode line declares aspects of a predicate related to the logic programming side of things. In this case we declare that the two arguments passed to main have the "destructive input" mode and the "unique output" mode respectively. These modes operate similar to how linear types work in other languages, and the reference manual has a section describing them. For now the details can be ignored. The is det portion identifies the function as being deterministic. It always succeeds, doesn't backtrack and only has one result.

The remaining code is the implementation. In this case it's just the implementation of the main function:

main(IO0, IO1) :-
    io.write_string("Hello World!\n", IO0, IO1).

The two arguments to main, are the io types representing the before and after representation of the world. We call write_string to display a string, passing it the input world state, IO0 and receiving the new world state in IO1. If we wanted to call an additional output function we'd need to thread these variables, passing the obtained output state as the input to the new function, and receiving a new output state. For example:

main(IO0, IO1) :-
    io.write_string("Hello World!\n", IO0, IO1),
    io.write_string("Hello Again!\n", IO1, IO2).

This state threading can be tedious, especially when refactoring - the need to renumber or rename variables is a pain point. Mercury has syntactic sugar for this called state variables, enabling this function to be written like this:

main(!IO) :-
    io.write_string("Hello World!\n", !IO),
    io.write_string("Hello Again!\n", !IO).

When the compiler sees !Variable_name in an argument list it creates two arguments with automatically generated names as needed.

Another syntactic short cut can be done in the pred and mode lines. They can be combined into one line that looks like:

:- pred main(io::di, io::uo) is det.

Here the modes di and uo are appended to the type prefixed with a ::. The resulting program looks like:

- module hello.

:- interface.
:- import_module io.
:- pred main(io::di, io::uo) is det.

:- implementation.
main(!IO) :-
    io.write_string("Hello World!\n", !IO),
    io.write_string("Hello Again!\n", !IO).

Factorial

The following is an implementation of factorial:

:- module fact.

:- interface.
:- import_module io.
:- pred main(io::di, io::uo) is det.

:- implementation.
:- import_module int.
:- pred fact(int::in, int::out) is det.

fact(N, X) :-
  ( N = 1 -> X = 1 ; fact(N - 1, X0), X = N * X0 ).

main(!IO) :-
  fact(5, X),
  io.print("fact(5, ", !IO),
  io.print(X, !IO),
  io.print(")\n", !IO).

In the implementation section here we import the int module to access functions across machine integers. The fact predicate is declared to take two arguments, both of type int, the first an input argument and the second an output argument.

The definition of fact uses Prolog syntax for an if/then statement. It states that if N is 1 then (the -> token) the output variable, X is 1. Otherwise (the ; token), calculate the factorial recursively using an intermediate variable X0 to hold the temporary result.

There's a few other ways this could be written. Instead of the Prolog style if/then, we can use an if/then syntax that Mercury has:

fact(N, X) :-
  ( if N = 1 then X = 1 else fact(N - 1, X0), X = N * X0 ).

Instead of using predicates we can declare fact to be a function. A function has no output variables, instead it returns a result just like functions in standard functional programming languages. The changes for this are to declare it as a function:

:- func fact(int) = int.
fact(N) = X :-
  ( if N = 1 then X = 1 else X = N * fact(N - 1) ).

main(!IO) :-
  io.print("fact(5, ", !IO),
  io.print(fact(5), !IO),
  io.print(")\n", !IO)

Notice now that the call to fact looks like a standard function call and is inlined into the print call in main. A final syntactic shortening of function implementations enables removing the X return variable name and returning directly:

fact(N) = (if N = 1 then 1 else N * fact(N - 1)).

Because this implementation uses machine integers it won't work for values that can overflow. Mercury comes with an arbitrary precision integer module, integer, that allows larger factorials. Replacing the use of the int module with integer and converting the static integer numbers is all that is needed:

:- module fact.

:- interface.
:- import_module io.
:- pred main(io::di, io::uo) is det.

:- implementation.
:- import_module integer.
:- func fact(integer) = integer.

fact(N) = (if N = one then one else N * fact(N - one)).

main(!IO) :-
  io.print("fact(1000, ", !IO),
  io.print(fact(integer(1000)), !IO),
  io.print(")\n", !IO).

Conclusion

There's a lot more to Mercury. These are just first steps to test the system works. I'll write more about it in later posts. Some further reading:

June 20, 2019

Marc Brooker (mjb)

When Redundancy Actually Helps June 20, 2019 12:00 AM

When Redundancy Actually Helps

Redundancy can harm more than it helps.

Just after I joined the EBS team at AWS in 2011, the service suffered a major disruption lasting more than two days to full recovery. Recently, on Twitter, Andrew Certain said:

We were super dependent on having a highly available network to make the replication work, so having two NICs and a second network fabric seemed to be a way to improve availability. But the lesson of this event is that only some forms of redundancy improve availability.

I've been thinking about the second part of that a lot recently, as my team starts building a new replicated system. When does redundancy actually help availability? I've been breaking that down into four rules:

  1. The complexity added by introducing redundancy mustn't cost more availability than it adds.
  2. The system must be able to run in degraded mode.
  3. The system must reliably detect which of the redundant components are healthy and which are unhealthy.
  4. The system must be able to return to fully redundant mode.

This might seem like obvious, even tautological, but each serves as the trigger of deeper thinking and conversation.

Don't add more risk than you take away

Andrew (or Kerry Lee, I'm not sure which) introduced this to the EBS team as don't be weird.

This isn't a comment on people (who are more than welcome to be weird), but on systems. Weirdness and complexity add risk, both risk that we don't understand the system that we're building, and risk that we don't understand the system that we are operating. When adding redundancy to a system, it's easy to fall into the mistake of adding too much complexity, and underestimating the ways in which that complexity adds risk.

You must be able to run in degraded mode

Once you've failed over to the redundant component, are you sure it's going to be able to take the load? Even in one of the simplest cases, active-passive database failover, this is a complex question. You're going from warm caches and full buffers to cold caches and empty buffers. Performance can differ significantly.

As systems get larger and more complex, the problem gets more difficult. What components do you expect to fail? How many at a time? How much traffic can each component handle? How do we stop our cost reduction and efficiency efforts from taking away the capacity needed to handle failures? How do we continuously test that the failover works? What mechanism do we have to make sure there's enough failover capacity? There's typically at least as much investment in answering these questions as building the redundant system in the first place.

Chaos testing, gamedays, and other similar approaches are very useful here, but typically can't test the biggest failure cases in a continuous way.

You've got to fail over in the right direction

When systems suffer partial failure, it's often hard to tell what's healthy and what's unhealthy. In fact, different systems in different parts of the network often completely disagree on health. If your system sees partial failure and fails over towards the truly unhealthy side, you're in trouble. The complexity here comes from the distributed systems fog of war: telling the difference between bad networks, bad software, bad disks, and bad NICs can be surprisingly hard. Often, systems flap a bit before falling over.

The system must be able to return to fully redundant mode

If your redundancy is a single shot, it's not going to add much availability in the long term. So you need to make sure the system can safely get from one to two, or N to N+1, or N to 2N. This is relatively easy in some kinds of systems, but anything with a non-zero RPO or asynchronous replication or periodic backups can make it extremely difficult. In small systems, human judgement can help. In larger systems, you need an automated plan. Most likely, you're going to make a better automated plan during daylight in the middle of the week during your design phase than at 3AM on a Saturday while trying to fix the outage.

June 17, 2019

Marc Brooker (mjb)

Is Anatoly Dyatlov to blame? June 17, 2019 12:00 AM

Is Anatoly Dyatlov to blame?

Without a good safety culture, operators are bound to fail.

(Spoiler warning: containers spoilers for the HBO series Chernobyl, and for history).

Recently, I enjoyed watching HBO's new series Chernobyl. Like everybody else on the internet, I have some thoughts about it. I'm not a nuclear physicist or engineer, but I do think a lot about safety and the role of operators.

The show tells the story of the accident at Chernobyl in April 1986, the tragic human impact, and the cleanup and investigation in the years that followed. One of the villains in the show is Anatoly Dyatlov, the deputy chief engineer of the plant. Dyatlov was present in the control room of reactor 4 when it exploded, and received a huge dose of radiation (the second, or perhaps third, large dose in his storied life of being near reactor accidents). HBO's portrayal of Dyatlov is of an arrogant and aggressive man whose refusal to listen to operators was a major cause of the accident. Some first-hand accounts agree2, 3, 6, and others disagree1. Either way, Dyatlov spent over three years in prison for his role in the accident.

There's little debate that the reactor's design was deeply flawed. The International Nuclear Safety Advisory Group (INSAG) found4 that certain features of the reactor "had a primary influence on the course of the accident and its consequences". During the time before the accident, operators had put the reactor into a mode where it was unstable, with reactivity increases leading to higher temperatures, and further reactivity increases. The IAEA (and Russian scientists) also found that the design of the control rods was flawed, both in that they initially increased (rather than decreasing) reactivity when first inserted, and in that they machinery to insert them moved too slowly. They also found issues with the control systems, cooling systems, and the fact that some critical safety measures could be manually disabled. Authorities had been aware of many of these issues since an accident at the Ignalina plant in 19834, page 13, but no major design or operational practice changes had been made by the time of the explosion in 1986.

In the HBO series' telling of the last few minutes before the event, Dyatlov was shown to dismiss concerns from his team that the reactor shouldn't be run for long periods of time at low power. Initially, Soviet authorities claimed that the dangers of doing this was made clear to operators (and Dyatlov ignored procedures). Later investigations by IAEA found no evidence that running the reactor in this dangerous mode was forbidden 4, page 11. The same is true of other flaws in the plant. Operators weren't clearly told that pushing the emergency shutdown (aka SCRAM, aka AZ-5) button could temporarily increase the reaction rate in some parts of the reactor. The IAEA also found that the reactors were safe in "steady state", and the accident would not have occurred without the actions of operators.

Who is to blame for the 1986 explosion at Chernobyl?

In 1995, Dyatlov wrote an article in which he criticized both the Soviet and IAEA investigations5, and asked a powerful question:

How and why should the operators have compensated for design errors they did not know about?

If operators make mistakes while operating systems which have flaws they don't know about, is that "human error"? Does it matter if their ignorance of those flaws is because of their own inexperience, bureaucratic incompetence, or some vast KGB-lead conspiracy? Did Dyatlov deserve death for his role in the accident, as the series suggests? As Richard Cook says in "How Complex Systems Fail"7:

Catastrophe requires multiple failures – single point failures are not enough. ... Each of these small failures is necessary to cause catastrophe but only the combination is sufficient to permit failure.

And

After accidents, the overt failure often appears to have been inevitable and the practitioner’s actions as blunders or deliberate willful disregard of certain impending failure. ... That practitioner actions are gambles appears clear after accidents; in general, post hoc analysis regards these gambles as poor ones. But the converse: that successful outcomes are also the result of gambles; is not widely appreciated.

If Dyatlov and the other operators of the plant had known about the design issues with the reactor that had been investigated following the accident at Ignalina in 1983, would they have made the same mistake? It's hard to believe they would have. If the reactor design had been improved following the same accident, would the catastrophe had occurred? The consensus seems to be that it wouldn't have, and if it did then it would have taken a different form.

From "How Complex Systems Fail":

Most initial failure trajectories are blocked by designed system safety components. Trajectories that reach the operational level are mostly blocked, usually by practitioners.

The show's focus on the failures of practitioners to block the catastrophe, and maybe on their unintentional triggering of the catastrophe seems unfortunate to me. The operators - despite their personal failings - had not been set up for success, either by arming them with the right knowledge, or by giving them the right incentives.

From my perspective, the show is spot-on in it's treatment of the "cost of lies". Lies, and the incentive to lie, almost make it impossible to build a good safety culture. But not lying is not enough. A successful culture needs to find the truth, and then actively use it to both improve the system and empower operators. Until the culture can do that, we shouldn't be surprised when operators blunder or even bluster their way into disaster.

Footnotes

  1. BBC, Chernobyl survivors assess fact and fiction in TV series, 2019
  2. Svetlana Alexievich, "Voices from Chernobyl".
  3. Serhii Plokhy, "Chernobyl: The History of a Nuclear Catastrophe". This is my favorite book about the disaster (I've probably read over 20 books on it), covering a good breadth of history and people without being too dramatic. There are a couple of minor errors in the book (like confusing GW and GWh in multiple places), but those can be overlooked.
  4. INSAG-7 The Chernobyl Accident: Updating of INSAG-1, IAEA, 1992
  5. Anatoly Dyatlov, Why INSAG has still got it wrong, NEI, 1995
  6. Adam Higginbotham, "Midnight in Chernobyl: The Untold Story of the World's Greatest Nuclear Disaster"
  7. Richard Cook, How Complex Systems Fail

June 13, 2019

Nikita Voloboev (nikivi)

June 03, 2019

Jeremy Morgan (JeremyMorgan)

Thinking About Reusable Code June 03, 2019 06:09 AM

The mythical "reusable code" idea has existed for decades. It showed up shortly after the first lines of code were written. We preach re-usability and sometimes strive for it but it rarely becomes a reality. I've seen various levels of success with this over the years. Everything from "we have a reusable library that 75% of us use" to "we have shared code libraries here, but never use them in your projects".

A recent discussion led me to think about this. Why don't more software development organizations have shared code libraries? It is the pointy-haired bosses preventing it? Team conflicts? Or is it the developers themselves? 

We can't blame the pointy-haired bosses for this one. If you explain the basic concepts of reusable code to management, most would agree it's a great idea. Building something once, so you don't have to build it repeatedly? Easy to find the upside here.

Team conflicts can also contribute to it, which is usually people disagreeing about who gets to determine what code is shared. Developers themselves can also be opposed to it, due not having enough time to build them.

All of these are contributing factors to lack of adoption, but the question you should ask is do we need reusable code libraries at all?


What Are Shared Libraries and Why Do We Care?

If you have tasks your developers are building that contain code you can use for something else, you put that code in its own "library" to use later. This can be a DLL, a folder of snippets, a Node module, whatever. Connecting to a database? There's no reason to write that code for every piece of software that accesses a database. Create a DB_Connect class, and put that in a file you can copy to another project later.

It's easy. You take a function and if it's abstract enough, parameterize it and make it available for other projects to use. When you start your project, you don't have to write code to connect to the database, pull the library and enter your parameters. Here are some upsides:

  • You write code once and use it multiple times (saving cycle times)
  • If tested thoroughly, it cuts regression errors
  • It enforces standards other groups can follow

These are all great reasons to use shared libraries. Countless articles and books have been written about code reuse and most of you are familiar with them. The biggest selling point for this is not having to code "boring stuff" over and over and have wild variations of the same methods in the wild. This frees up time to work on exciting features.


How Are They Used?

Here are some ways shared libraries are used in business:

  • Internal Use Code that shared with internal groups and used for projects
  • External Use Code that designed for programmers outside the organization to use
  • Forked - Users can take your initial code and "fork" it to make it work for their specific needs

This falls in line with the DRY principle of software development. Don't repeat yourself.


Do We Want Shared Code Libraries?

Why isn't everyone doing this? Your organization may avoid shared libraries for a good reason. Not every project or team benefits from this and it's not the magic bullet to solve all development problems. Here are some reasons not to use code libraries.

  • Distributed Monolith Problem - If you put something into a library and everyone uses it, that doesn't mean it isn't bad. An inefficient database connector design means everyone is using an inefficient database connector. Changes to your implementation can still have negative cascading effects. A small change to an API could mean every single system is now broken.

  • Dependency Hell - If every piece of software is relying on your library, there's a tight coupling in place. There will be a myriad of dependencies in place, for instance, relying on a framework like Angular. If you have software built to use a specific version, upgrading it could produce a chain reaction amongst your applications.

  • Inflexible Design - Sometimes you're asked to use a design in a certain way that's different from what you actually need. Because "everyone else is doing it" you're forced to make your code work around it, which can then increase development time. Wait, why did we build shared libraries in the first place?

  • Programming Language Lock-In - Too many times I've heard "well our library is written in ____ so you have to use that" which means you're locked into a single programming language. This can lead to using the wrong tool for the job. So don't assume because your team doesn't use shared libraries means they're unenlightened and don't want better production.


What to Examine if You Want to Use Reusable Code Libraries

If you're considering creating reusable code libraries, you should first see if it makes sense to do so.

  • Take a hard look at the design - Then look at it again. Will you really benefit from it? Do you have repeated code now? Are there any issues that may encourage tight coupling? 
  • Closely examine your dependencies - Are you creating dependencies others will have to work around? How often do they change? Can they bypass the ones you create? 
  • Start abstract as possible - You want your libraries to be highly abstract so that they perform expected functions, but can be overridden or extended to meet specialized use cases. Give them the basics but let them evolve it how they'd like.

How to Get Developers to Implement Them

So you've decided you want a shared library. Here are some ways you can help teams adopt your library:

  • Steal it from an existing project - This is the easiest way to start a new library. Take an existing project and remove everything that's specialized and keep everything that can be re-used and feels solid.

  • Thoroughly test it - The last thing you want to do is introduce problems into the software. After building your abstract library, test it thoroughly so you can rely on it, and help developers find regressions. Don't hand them garbage and expect them to use it.

  • Examine all current use cases - By digging in and understanding how people will use your library you can determine whether it will benefit them, and whether to make changes. Do this on a method by method basis to ensure that you're asking them to use something that will work for them with minimum effort.  Like most anything in business, success will come from great communication. Top down design is not your friend here, you'll want to examine the needs of the group and design a library from those needs.


Conclusion

Most of the organizations I've been involved with do not use a shared code library. They're usually in some form of working towards one, and only two where they were implemented and working well.

At one end of the spectrum you have a pattern where all repeated code is in a library. On the other you'll have no code library, and everyone builds software for their own projects. You'll find success somewhere between the two. How far you lean toward each extreme will depend on your needs.

There are abstract functions that should always be shared. Some that come to mind are authentication/authorization, database connections and logging. It should be either a shared library you developed, or one that's open sourced and built by someone else. Everything else should be examined, considering the overall design.

Don't jump to conclusions and take on a large project to build a giant code library that will make developers lives worse. But don't exclude the idea altogether. Be purposeful and observant to find which strategy will work best for your projects. 



What is your DevOps IQ?

what's your devops score


My Devops Skill IQ is 232. Can you beat it? Take the test now to find out your Devops IQ score!!

Andreas Zwinkau (qznc)

Thinking in Systems by Donella Meadows June 03, 2019 12:00 AM

Book review: A shallow introduction to Systems Thinking.

Read full article!