Planet Crustaceans

This is a Planet instance for lobste.rs community feeds. To add/update an entry or otherwise improve things, fork this repo.

January 14, 2019

Ponylang (SeanTAllen)

Last Week in Pony - January 14, 2019 January 14, 2019 08:29 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, our users’ mailing list or join us on IRC.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

Jan van den Berg (j11g)

The Sun Also Rises – Ernest Hemingway January 14, 2019 06:47 AM

On my honeymoon in 2009 I asked the store clerk at the Hemingway house on Key West: “what’s a good book to start with?”. She recommended The Sun also Rises.

The Sun also Rises – Ernest Hemingway (1926) – 222 pages

And I understand why. Because this novel is probably the most accessible summary of the typical Hemingway writing style. Very much set in the 1920s — yet timeless — it is a story about classic themes such as friendship, love, morality and searching for meaning, told through sparse dialogue and with minimal background. And where the main characters drink, a lot. A whole lot.

At the surface this book is, at most, an enjoyable adventure about a group of restless friends, who do a bit of travelling and go to see the bullfights. But when you dig a bit deeper: a lot more is going on. Which underscores the iceberg theory for which Hemingway is famous. There are many sources where you can read about what Hemingway supposedly meant. And while these can be a welcome addition to reading, I am sceptical about too much analysis. Sometimes maybe it just is what it is. Either way, one of the features of great art is that there is no definitive meaning and it is what it is to you.

I read a book about flawed and memorable characters fleeing and navigating through the complexities of life and love. And they drank. A lot.

The post The Sun Also Rises – Ernest Hemingway appeared first on Jan van den Berg.

January 13, 2019

Wesley Moore (wezm)

Goals and Directions for Rust in 2019 January 13, 2019 09:00 PM

This is my response to the call for 2019 roadmap blog posts proposing goals and directions for 2019 and future editions. See also Read Rust, where I've collected all the #Rust2019 posts.

2018 was a very busy year for the Rust project. A new edition was released, progress on stabilising Rust's asynchronous I/O story was made, a new website was launched, and so much more! In 2019 I'd like to see the language and wider crates community become more sustainable, address some common gothca's that newcomers experience, and promote more platforms/architectures to tier 1 status.

2018 Retrospective

Before diving into 2019 goals I think it's worth tracking how the project went on my ideas from last year:

  1. Become a better option for building network daemons and HTTP micro-services.
  2. Continue to improve the discoverability and approachability of crates and Rust's web presence in general.
  3. Gain wider, more diverse tier-1 platform support (especially on servers).
  4. Start delivering on the prospect of safer system components, with fewer security holes.

Network Services

A lot of progress was made on Futures async/await in 2018. The keywords were reserved in the 2018 edition but are not yet usable on a stable release. Are we async yet? shows there's still some work to do. Hyper saw more major changes to keep up with the Futures work and added HTTP/2 support!

Improve Rust's Web Presence

The Rust web presence was improved with the release of the new website and blog. The rest of the ecosystem remains much the same. Content from crates.io is still largely invisible to non-Google search engines such as DuckDuckGo (my primary web search tool), and Bing. The Rust Cookbook remains in the nursery.

Platform Support

Tier 1 platform support remains unchanged from last year. There were a number Tier 2 additions/promotions.

System Components and Increased Safety

The oxidisation of librsvg continued in 2018 to the point where almost all the public API is done in terms of Rust. I'm not aware of many other projects following this path at the moment:


Rust 2019

In 2019 I'd like to see the Rust community focus on three areas:

  1. Sustainable development
  2. Make is easier for newcomers to write fast code / don't surprise people
  3. More portability

Sustainable Development

Recently Murphy Randle and Jared Forsyth were discussing the event-stream compromise on the Reason Town podcast. Jared commented:

The problems of having infrastructure that’s based on unpaid labour that has a high degree of burnout.

Reason Town podcast episiode 13 @ 19:29

This is a pretty succinct summary of the problem with our industry. Rust hasn't shied away from tackling hard problems before and taking on the sustainability of open source doesn't feel out of the question. There's evidence that many of the folks deeply involved with the Rust project are already feeling the pressure and we don't want to lose them to burnout. Such as these posts:

Part of this revolves around culture. The Rust community generally values quality, correctness, performance, and treating each other with respect. I think it would be possible to make it normal to contribute financially, or other means (equipment, education) to Rust language and crate developers (where people are in a position to do so). A simple first step might be allowing for a donate badge, akin to CI badges to be added to crate meta data and have this shown on the Crate page.

Michael Gattozzi covered some similar thoughts in his, Rust in 2019: The next year and edition, post.

Naïve Code Is Fast Code

People hear that Rust is fast and lean, they try it out converting something from a language they already know and are surprised to find that it's slower and/or a much larger binary.

There are many many many many examples of this in the Rust Reddit. Things that frequently seem to trip newcomers up are:

  1. Not compiling with --release
  2. Stdio locking
  3. Binary size

It would be good to apply the principle of least surprise here. I think the current defaults are inspired by the behaviours expected from C/C++ developers, Rust's original target audience. However the Rust audience is now much wider than that. With that in mind it might worth reevaluating some of these things in terms of the current audience. These need not require API changes, perhaps they could be clippy lints. Perhaps they could be slight changes to language. For example, cargo currently says:

Finished dev [unoptimized + debuginfo] target(s) in 0.11s

Perhaps a second line could be added that says:

Compile with --release for an optimized build

to help guide people.

More Tier 1 Platforms

This one is inherited from last year. I do all my server hosting with FreeBSD and up until recently used it as my desktop OS as well. It's not uncommon to uncover portability bugs or assumptions when using such a platform. Portability is type of diversity of software. It makes it stronger and useful to more people.

Rust is already appealing to people because of its portability. I was recently talking to a veteran developer at the Melbourne Rust meetup's hack night about what got them into Rust. It was the combination of modern language features, native binary and portability that drew them to Rust.

To this end I'd like to see more platforms and CPU architectures promoted to tier 1 status. Up until recently one thing that made this difficult was the lack of hosted CI services with support for anything other than Linux, macOS, and Windows. Recently two options have become available that make it possible to test on other systems, such as FreeBSD. There is Cirrus CI which includes a FreeBSD image in their hosted option, as well as the ability to create custom images. Secondly there is sr.ht, a completely open source (but hosted) option that supports a variety of Linux distributions, and FreeBSD, with more planned.

Pietro Albini suggested in his post that the Rust infrastructure team is already planning to start the discussion on CI options. I think this would be a perfect opportunity to integrate more platforms into the CI infrastructure:

One of the biggest one is switching away from Travis CI for the compiler repository. In the past year we had countless issues with them (both small and big), and that's not acceptable when we're paying (a lot) for it. The infra team is already planning to start the discussion on where we should migrate in the coming weeks, and we'll involve the whole community in the discussion when it happens.

Conclusion

After an intense 2018 it sounds like the Rust project needs to focus on making the project sustainable over time. I'd love to see some improvements to address issues newcomers often experience and push more platforms to tier 1 status. Rust is still very exciting to me. I can't wait to see what 2019 brings!

For more great #Rust2019 posts check out readrust.net.

January 11, 2019

Geoff Wozniak (GeoffWozniak)

Timestamps, GNU autotools, and repositories January 11, 2019 04:07 AM

If you include the distributed source for software that uses the GNU autotools in a source repository, beware of timestamps on the files when you checkout the source.

Consider, for example, the source distribution of the GNU MPFR library. The source distribution includes the files aclocal.m4 and Makefile.in. In the autotools world, Makefile.in is dependent on aclocal.m4. When you run configure, the Makefile is derived from Makefile.in.

This all fine and not surprising. What will be surprising is that when you run make, you may immediately get an error like this.

> gmake
/home/woz/tmp/testing/mpfr/missing: automake-1.15: not found
WARNING: 'automake-1.15' is missing on your system.
         You should only need it if you modified 'Makefile.am' or
         'configure.ac' or m4 files included by 'configure.ac'.
         The 'automake' program is part of the GNU Automake package:
         <http://www.gnu.org/software/automake>
         It also requires GNU Autoconf, GNU m4 and Perl in order to run:
         <http://www.gnu.org/software/autoconf>
         <http://www.gnu.org/software/m4/>
         <http://www.perl.org/>
gmake: *** [Makefile:404: ../mpfr/Makefile.in] Error 1

The distribution may create a Makefile whose first course of action is to check that the build files are up-to-date. This means checking files such as aclocal.m4 and Makefile.in. If the timestamps on those files are such that Makefile.in is ever-so-slightly older than aclocal.m4, for example, the build will attempt to regenerate Makefile.in, which requires automake to be installed. It may even complain about having a specific version, such as in the example above. Perhaps more insidiously, if you do have the correct version installed, it will regenerate the file in the source directory. You may be really confused to see a modified Makefile.in (or .info files, or perhaps machine descriptions) after doing a clean checkout and build. (It’s also bad because, technically, the build is based on different source than what is in the repository.)

I’ve run into this where the distributed source for libraries such as MPFR are included in large source repositories. When cloning them, you never know what order the files will be written and when. If you try to do this with a small repository, there’s a very good chance the timestamps of the files will be the same.

This likely won’t happen when the source is unpacked from the distribution because TAR files preserve the timestamps.

If there’s an option to configure to prevent this, such as --disable-maintainer-mode, you may want to use that. (Why would “maintainer mode” be the default in the first place?) You could also touch the troublesome files in the correct order, although that will probably bring about its own maintenance headaches.

January 10, 2019

Alex Wilson (mrwilson)

Notes from the Week #14 January 11, 2019 12:00 AM

Getting back into the habit of blogging after stopping for almost a month is a bit of a wrench, but it means there’s more juicy stuff for me to talk about!

new year, new me

I’ve been digging really hard into reliability and SLXs this year — we’re already well down the road to better understanding the existing reliability for our Graphite metrics-collection system, and are hoping to take these learnings into building SLXs and Error Budgets for our other core systems.

We’ve already gained a lot of insight into how our customers use the metrics collection API to build dashboards:

  • Setting harder timeouts at the load balancer level has exposed just how many dashboards have panels with too many metrics (as Grafana bunches them into a single request)
  • Measuring our error rates highlighted the number of dashboards relying on metrics that are no longer being collected and cause 5xx responses at the backend.

I want Shift to set a good example for ProDev as a whole — as we scale out a service-oriented approach to architecture being able to have concrete discussions for reliability is going to become paramount.

Keeping records of our decisions

Shift built out a new Puppet module under our open-source repository. The nrpe_custom_check module wraps several different configuration files to provide a clean interface to build NRPE checks for production machines.

We designed it to have as few configurable parts as possible (and indeed only has 3 inputs — name, content, and whether the script needs sudo privileges) but hit a major snag on an architectural point.

There’s an existing module base which has some default NRPE plugins. Given base is designed to be completely standalone and “batteries included”, should it depend on and use the nrpe_custom_check module rather than using static files?

There was a lot of back-and-forth debate about the merits composing small well-defined modules together versus jumping into an abstraction too quickly rather than letting the design evolve “naturally”.

In the end, we decided that the points made were too valuable to lose in the mists of time and resolved to adopt Architecture Decision Records. The implementation we chose was the Nygard format supported by Nat Pryce’s excellent adr-tools toolchain.

You can check out our record here: 2. Standalone NRPE Custom Check module

Historically, we’ve been not as good as we could be at recording not just the decision but the context in which it was made. A key part of Norm Kerth’s Prime Directive is

Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.

Code, documentation, and domain models all succumb to rot over time and it’s important to remember that choices made are probably correct given the information available at the time.

I’m hopeful that we continue to adopt this, especially for shared projects like our monolithic codebases, so that knowledge about why we do certain things is not purely contained in the anecdotes of people who were there at the time.

What does value actually mean?

New year, and I’m having a bit of a new think on this topic. For a team like Shift whose role is primarily that of support:

  1. What does ‘value’ mean?
  2. How do we measure it effectively?

A (paraphrased) aphorism from our CTO has been rattling around in my head for a while:

At any given point in time, there is a good argument to not do X (usually in favour of feature development). Not doing X causes more problems, and will take longer to heal, the longer we leave it.

Where X can be anything from dependency upgrades, platform investments, even refactoring.

Shift are fundamentally an enabling team — the systems we build support the other teams in ProDev and we are also a source of extra hands for doing work related to our area of expertise.

We frequently pair with other teams to share the knowledge and help guide them around pitfalls that we have experienced, all while improving the state of our own infrastructure by learning about the needs of our teammates.

We don’t often get the quick highs of rapid feature delivery like a focused product development team but we are steadily building out our own tooling and systems to aid ProDev.

2019 is a year full of potential and Shift are going to seize as much of it as possible.

Originally published at blog.probablyfine.co.uk on January 11, 2019.

January 10, 2019

Derek Jones (derek-jones)

Wanted: 99 effort estimation datasets January 10, 2019 01:24 AM

Every now and again, I stumble upon a really interesting dataset. Previously, when this happened I wrote an extensive blog post; but the SiP dataset was just too big and too detailed, it called out for a more expansive treatment.

How big is the SiP effort estimation dataset? It contains 10,100 unique task estimates, from ten years of commercial development using Agile. That’s around two orders of magnitude larger than other, current, public effort datasets.

How detailed is the SiP effort estimation dataset? It contains the (anonymized) identity of the 22 developers making the estimates, for one of 20 project codes, dates, plus various associated items. Other effort estimation datasets usually just contain values for estimated effort and actual effort.

Data analysis is a conversation between the person doing the analysis and the person(s) with knowledge of the application domain from which the data came. The aim is to discover information that is of practical use to the people working in the application domain.

I suggested to Stephen Cullum (the person I got talking to at a workshop, a director of Software in Partnership Ltd, and supplier of data) that we write a paper having the form of a conversation about the data; he bravely agreed.

The result is now available: A conversation around the analysis of the SiP effort estimation dataset.

What next?

I’m looking forward to seeing what other people do with the SiP dataset. There are surely other patterns waiting to be discovered, and what about building a simulation model based on the charcteristics of this data?

Turning software engineering into an evidence-based disciple requires a lot more data; I plan to go looking for more large datasets.

Software engineering researchers are a remarkable unambitious bunch of people. The SiP dataset should be viewed as the first of 100 such datasets. With 100 datasets we can start to draw general, believable conclusions about the processes involved in software effort estimation.

Readers, today is the day you start asking managers to make any software engineering data they have publicly available. Yes, it can be anonymized (I am willing to do that for people who are looking to release data). Yes, ‘old’ data is useful (data from the 1980s could have an interesting story to tell; SiP runs from 2004-2014). Yes, I will analyze any interesting data that is made public for free.

Ask, and you shall receive.

January 09, 2019

Jan van den Berg (j11g)

Joy Division and the making of Unknown Pleasures – Jake Kennedy January 09, 2019 09:36 PM

I picked up this book in the bargain bin of a HMV in Manchester in 2006, when I was on a — sort of — pilgrimage. But I left it on my bookshelf for 12 years, thinking I probably knew most of it already. But books are meant to be read, so I had to get to it eventually.

Joy Division and the making of Unknown Pleasures – Jake Kennedy (2006) – 214 pages

I think I understand why this book was already in the bargain bin the same year it was released. Not that it is a terrible book, but it is just a really hard sell. In depth reviews of every Warsaw/Joy Division song, demo or recording session are something only hard hardcore fans have a thirst for. But if you do, you will find a chronological and remarkable tale of four guys who went from sloppy generic punk music to producing a timeless, unique and legendary album within two short years. With each page turn, they improve and get closer to reaching that point. Which was a natural conclusion of everything prior, and didn’t drop out of the sky. But even so, the progression and quality of a debut album is still unmatched (maybe only by that other band).

The book falls somewhere between a biography and an encyclopedia. In other words, just my cup of tea! I certainly picked up a few new things. Mick Middles gets quoted often, he is another authority on Joy Division because of his book. I also own this book, and I definitely need to read it now!

The post Joy Division and the making of Unknown Pleasures – Jake Kennedy appeared first on Jan van den Berg.

Indrek Lasn (indreklasn)

Thanks. January 09, 2019 11:49 AM

Thanks.

Gergely Nagy (algernon)

One hat less January 09, 2019 11:45 AM

Almost a week ago, on a snowy night of January 3, I hung up my Debian Developer hat, and sent my retirement letter in. This has been a long time coming, and I contemplated doing this many times over the past year or two, but never got the courage and the willpower to retire. It wasn't easy. Debian has been part of my life for the past two decades, and I've been a developer for about eighteen years. I considered - and to some extent, still do - Debian my second family. There have been times when Debian was my escape hatch, something I could focus on to help me pass the days by. I've made friends, I've met people, I learned humility, I learned empathy. If it weren't for Debian, I wouldn't be the person I am today. Retiring felt like admitting defeat for a long time. But it isn't.

It's not defeat when you admit that your circumstances changed. It's not defeat when you let others take better care of the stuff you've been neglecting for way too long. It's not defeat when you're taking a break. It's not defeat when you're reducing the amount of stress that keeps you awake at night. It's not defeat when you do the right thing. It's not defeat when you gracefully retire: rather, it is a start of something new.

I originally joined Debian for two reasons: I was a Debian user for a while by then, and felt that I should be giving back, as I received so much from Debian. The second reason wasn't this noble, but after eighteen years, I guess I could admit it: the other reason I applied was vanity. Having a @debian.org email-address was something.

At the time, I was... a different person. I was full of myself at times, I was angry, perhaps a tad too elitist at times too. I'm not proud of old me, but it's part of me. I grew, and became a better person, there's no shame in being able to grow - quite the contrary. And Debian helped immensely. I've had role models in the project, who I look up to even to this day, who helped shape me one way or the other.

There are two people I need to mention specifically: Martin Michlmayr and Rhonda D'Vine.

Martin was my Application Manager when I applied, I considered him a mentor, a friend. The example he set were instrumental in shaping me too. Rhonda helped me get to my first ever conference: I got on a train to Vienna, and she took me to LinuxTag in her car from there, and then back again. That first LinuxTag, the path that led there, the conference itself, was formative, and both Martin and Rhonda had a big part in it being so. Thank you again - so many years later, I still smile when I think back. Those years we were in touch, meant a lot to me.

I often feel nostalgic about the times way back then, when I was active on IRC. I miss the old bunch (oh, so many people I haven't talked to in years!). I miss being active in Debian. I miss talking to fellow developers. But life took me elsewhere, and there's only so many hours in a day. Over the years Debian and me, we grew apart. There was always something more important, so my Debian contributions dropped (though, I had some ups too, while I was ftp-master assistant for example). By 2018, I barely did anything, and one of my resolutions for the new year was that I'll be taking better care of my health, mental and physical alike. Part of this is reducing my workload, reducing the stress levels, and things I feel guilty about.

It's a strange feeling. A mix of sadness and relief. On the flip side, as Lars put it, I'm now part of the retired DD club, and that's one illustrious club to be part of. First I joined the Debian Parents Club, now the Retired Debian Developers Club. This is a good start of a new chapter.

At some point though, I will be back. Debian is part of my life, and always will be. One way or the other, there will be a way back. It may take years, or a decade, but I'll put on that swirly hat again.

January 07, 2019

Indrek Lasn (indreklasn)

Thanks for the kind words. January 07, 2019 09:35 PM

Thanks for the kind words. I don’t have any monetization plans yet but I’m thankful for every reader I have.

Jan van den Berg (j11g)

Faith – Jimmy Carter January 07, 2019 06:22 PM

I literally received this book from the hands of Jimmy Carter himself at a book signing in New York. I never met a president before — even if only for a few seconds — so I was keen to read his book!

Carter always struck me as an interesting person. A multifaceted outlier: deeply religious but pro science and evolution, anti-NRA but pro guns, peanut farmer by trade and a nuclear submarine operator by education. But apart from all that, any 93 year old who has been married for over 70 years, probably has some interesting experiences to share.

Faith – Jimmy Carter (2018) – 174 pages

But this is not a biography. This is a personal creed about his definition of faith in Jesus Christ, interwoven with experiences about engaging with world leaders or dealing with personal or global crises of the past. In the last chapter, Carter, without naming names, demonstrates awareness by exactly pinpointing current problems. And you learn that his general approach to past and current problems are often the same, which I think come down to one powerful word: inclusion.

At first I was confused who this book was for. I expected it to be for either Christians or deliberately for non-religious people. But none of that. Carter just writes from his own personal experiences and deeply held beliefs, for anyone who wants to listen and learn something. And he does so with authenticity, honesty and integrity.

Bonus!

I was also lucky enough to be at the taping of this interview on the same day as the book signing. He was a surprise extra guest. The interview offers a great recap of what to expect of the book.

The post Faith – Jimmy Carter appeared first on Jan van den Berg.

Mark J. Nelson (mjn)

New city, new job January 07, 2019 12:00 PM

A few days ago I moved to Washington, DC, to start a new job as Assistant Professor in the Computer Science department of American University. The CS department here is small but growing, and I'm looking forward to being a part of that. Thanks especially to profs. Mike Treanor and Nathalie Japkowicz for their part in hiring me and helping me to get oriented here.

For the past 2½ years I was Senior Research Fellow at the MetaMakers Institute, a grant-funded European Union research initiative attached to the Games Academy of Falmouth University, which is located in the remote but pretty region of Cornwall, in the far southwestern United Kingdom. Moving from a 20,000-person seaside village to the capital of the United States is a big change! A change for both good and bad naturally, but I'm optimistic overall.

In addition to my home in Computer Science, I'll be affiliated with the AU Game Lab. I'll teach a mixture of game-oriented courses and more standard computer science courses as a result. For Spring 2019 I'm teaching one of each: a special-topics course on AI & Games, and a core CS course, Programming Languages. I'm designing both with mostly new syllabi, which means things are pretty busy at the moment, but it promises to be interesting. I'll try to share some more on those courses once everything is worked out (classes start next week). It's nice that American University has small classes (~20 students), which allows for a little more experimentation with these things – I've got some more ideas for the coming years if things go well.

January 06, 2019

Indrek Lasn (indreklasn)

How I got average 200k monthly views on Medium January 06, 2019 10:34 PM

This came as a total surprise.

My Medium stats (As of 2019, January)

I never expected so many smart people to read my articles. At the time when I started blogging — I was happy enough to let [at that time] my girlfriend read my articles. I never sent her the link, but she managed to read them anyway. I thought it was cute.

Times have moved on, and the audience has grown. With great audience comes great responsibility and scrutiny. I’ve been writing consistently for about 2 years and this is what I’ve learned.

Photo by Nick de Partee on Unsplash

So here’s my giving back to you. My mission has always been to share as much knowledge as possible. I believe in no tricks and shortcuts; only in hard work and becoming the best as you can.

So without further ado, here are some of my personal habits to keep writing articles people enjoy reading;

Assume your audience is smarter than you

It’s true. I’m not the smartest person in the room. But before you close this tab, let me explain why I’m not the smartest person in the room.

I’m aware of what I know, and I’m aware of what I don’t know; I’ve met some extraordinarily smart people in my life, I’m thankful for this. Unfortunately, this is a two-sided blade. The smarter people you hang out with, the less dumb you feel. I’ve been hanging around with some of the smartest guys.

Put yourself in a situation where you can always nourish and flourish. Being in the teacher shoes should never discourage you from learning new stuff. It’s expected that at some point, the master becomes the student, and the student becomes the master.

Humility is key, keep your head low and your hunger for new information high.

Me taking a cheerful selfi

Persistence, Persistence, Persistence!

Nothing great will happen if you won’t stick to it. There are people who train martial arts from pre-teens to late senior years. That is true dedication to the art.

Photo by Leslie Jones on Unsplash

How many have you started doing something and realized you were terrible at it and quit right after?

Everyone starts at the same point and has the same feelings and questions.

“Is this for me?”

Sure, you might find it hard, that’s perfectly sane and reasonable. Quitting is not. Never give up on your dreams and goals. If your dream is to have a big audience reading your articles — so be it. But you got to but the effort, sweat and, work in.

I have a habit of minimum one article per month and if I can — one article per week. This habit keeps the consistency in check on my terms.

Love what you do with passion burning in your heart

I love it so much when people come to me saying how useful and how much my article helped them. I find that extremely rewarding and satisfying. Thank you, everyone, for the kind words!

Writing has always been my passion. I guess I inherited it from my father, who is also a passionate novel writer.

Writing definitely helps me clear my mind, put my thoughts on paper and connect with people around the globe.

Photo by Nick Fewings on Unsplash

Pick topics which interest you the most but also have a broader audience

Picking topics are tricky — pick something too niche and not many people will read and pick something too specific and many people have written about it already.

Finding the balance is key, and I believe passion is the most important ingredient.

If you’d like to write about coding, I’d recommend building a couple of apps and putting some experience under the belt. This way you can be sure you feel confident on what you’re writing about and make it fun.

Readers are only interested in interesting and useful content. It’s your job to find out which content and how to make it sound interesting.

Photo by Sophie Elvis on Unsplash

Thanks for reading and I hope you succeed! I hope you found this article useful.

If you have any questions, I’d be more than happy to discuss on Twitter.

Indrek Lasn (@lasnindrek) | Twitter

Here are some of my previous article you might enjoy;

This story is published in The Startup, Medium’s largest entrepreneurship publication followed by +411,714 people.

Subscribe to receive our top stories here.


How I got average 200k monthly views on Medium was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

Patrick Louis (venam)

Adding Glue To a Desktop Environment January 06, 2019 10:00 PM

glue

In this article we will put some light on a lot of tools used in the world of Unix desktop environment customization, particularly regarding wmctrl, wmutils, xev, xtruss, xwininfo, xprop, xdotools, xdo, sxhkd, xbindkeys, speckeysd, xchainkeys, alttab, triggerhappy, gTile, gidmgr, keynav, and more. If those don’t make sense then this article will help. Let’s hope this can open your mind to new possibilities.

NB: A lot of the questions regarding those tools and what they are usually come from people that don’t actually grasp what the purpose of a window manager is or what desktop environments are. If this is the case please read the following before starting:

However, even without reading the previous content, this article will try to explain concepts as simply as possible.

With that in mind we can wonder if what’s actually needed from a window manager, presentation and operation, can be split up and complemented with other tools. We can also start thinking laterally, the communication and interaction between the different components of the environment. We have the freedom to do so because the X protocol is transparent and components usually implement many standards for interfacing between windows. It’s like gluing parts together to create a desktop environment.

The tools we’ll talk about fall into one of those categories:

  • Debugging
  • Window manipulation
  • Simulation of interaction
  • Extended manipulation
  • Hotkey daemon
  • Layout manager

overview

Window Manipulation

Windows are objects living in the X server that have a visual representation when mapped on the screen (viewable and visible). Windows, like any other objects, have attributes and properties attached to them. Another interesting object within the X Server is the pointer aka cursor.

The tools in this section manipulate those bare objects using the X11 protocol to speak to the X server. They do things such as list the identifiers of the window objects within the X server and their basic properties, they let you do basic tasks on them such as moving them, resizing them, changing their stack order (which one is above the other), labeling them, hiding/unmapping them (or the opposite by mapping them), changing the focus from one to the other, and as an extra some also allow to manipulate the pointer position and cursor shape.

Two popular tools that offer those functionality are wmutils core and the window utility from jessies x11-extras softwares. Along with those come some other utilities used for window listing such as winselect also from jessies, xlsw from baskerville, and the one included within wmutils core lsw. We could also mention single usage tools such as xkill.

The philosophy behind wmutils is more Unix-like, portrayed as the “Coreutils for Xorg” it’s less monolithic than a lot of similar tools we’ll see which renders it useful on the pipeline. Moreover it’s written using the XCB which is more of a direct correspondence with the X protocol.

We can manipulate the windows basic object but how do we actually verify that the change has been done properly. This can be done using debugging utilities such as xwininfo or xwinfo.

We should also mention a particular one that is p9-like, the x11fs that does the manipulation though a virtual filesystem API.

Simulation of interaction

So far so good, we’re able to do some basic actions on windows but what about simulating inputs such as keyboard or mouse, in the same way as winmacro on Windows platform or headless browser for the javascript world.

This is possible if the X Server implementation comes with the XTEST extension which as you would’ve guessed was principally added to test the server without having user interaction.

Most popular implementations builds (X.org Server) nowadays come with this extension however you can make sure that it’s there by querying the list:

xdpyinfo -display :0 -queryExtensions | awk '/opcode/'

Which should return a line similar to this:

...
    XFree86-VidModeExtension  (opcode: 153, base error: 172)
    XINERAMA  (opcode: 141)
    XInputExtension  (opcode: 131, base event: 66, base error: 129)
    XKEYBOARD  (opcode: 135, base event: 85, base error: 137)
    XTEST  (opcode: 132)
    XVideo  (opcode: 151, base event: 93, base error: 155)

Thus a set of utilities such as xdotool and xdo use this extension to simulate inputs. Most of them also include the behavior of the previous tools, especially the listing of window identifiers, this is a bit redundant.

Another interesting one is keynav, the “retire the mouse” tool that let you control all mouse functions from the keyboard in an efficient way, splitting the screen into sections.

Extended manipulation

A standard called icccm (Inter-Client Communication Conventions Manual) was devised a long time ago between 1988 and 1989, and another one named EWMH (Extended Window Manager Hints) not long after builds upon it. Both are used for lateral communication between windows. Their main purpose, which you can guess by taking a glance at them here and here , is that they emphasize on giving information about components and creating the interface between the standard components that usually make up a desktop environment such as bars, panels, desktop and all that is related (managed windows for instance), viewports, state of the window, title of window, the current focus, and much more.
A window manager or a client can choose to implement all or parts of those standard (respond to events), for example you can find the list that Openbox supports here.

Naturally, there’s yet again a set of tools letting you manipulate those properties however keep in mind that they only have an effect if the EWM/NetWM atoms (it’s the name of the properties) are respected. wmctrl accesses almost all the EWMH features, xtitle as the name implies accesses only the title, xdotool previously seen can also access some of the features (moving between desktops).

Lastly, the debug command line utility that is useful to display such window properties is xprop.

Hotkey daemon

What if instead of having the window manager catching the keys you press you had another application listening to your keys and performing actions whenever it encounters certain combinations. This is the job of the hotkey daemon. The explanation couldn’t be simpler, assign keys to actions.
Examples of such are xbindkeys, speckeysd again from jessies x11-extras, sxhkd a popular one and yet another tool from baskerville, triggerhappy, xchainkeys which works by creating a chain of keys, devilspie2 which takes action on certain EWMH events, and alttab which is for task switching (keybind mixed with the window manipulation).

But how do you make sure, if there’s an issue, that your hotkey manager responds to the right keys or which keys to set in its configuration. The xev let you debug X input events in general.

Moreover, it’s good to note that you can debug anything seen previously in more depth using xscope, xtruss, or x11vis. However most of them are a hassle to set up, xtruss is the easiest to compile and use in my opinion.

Layout manager

In normal cases it’s the job of the window manager to choose where to place windows on the screen and their layout but there are separate programs that can be used for this. Generally the layout manager program add tiling feature to a window manager that doesn’t have tiling.
This can be achieved using a software that specializes in this such as gTile for GNOME or gridmgr for others. Or it could be achieved by combining a hotkey daemon along with a window manipulation tool, so let’s say sxhkd with wmutils for instance.

This puts into question the role of the window manager if you can have all of its behavior into separate entities.

Summary

  • Debugging the protocol in general
    • Action: xtruss, xscope, x11vis
  • Window manipulation
    • Action: wmutils, window, x11fs, xkill, ..
    • Listing: xlsw, lsw, ..
    • Debug: xwininfo, xwinfo
  • Simulation of interaction
    • Server Extensions: xdpyinfo
    • Action: xdotool, xdo, keynav
  • Extended manipulation
    • Action: wmctrl, xtitle, xdotool
    • Debug: xprop
  • Hotkey daemon
    • Action: xbindkeys, speckeysd, sxhkd, triggerhappy, xchainkeys, alttab, devilspie2
    • Debug: xev
  • Layout manager
    • Action: gTile, gridmgr, any combination of the above ..

Examples

What can be done with all this, let’s push some ideas.

  • Starting your graphical session with a specific layout everytime regardless of the window manager
  • Automate a user action inside your program, in test units for example
  • Beautify your desktop with conky and make it clickable like a panel
  • Automate captcha
  • Dump your window manager entirely and have pieces in separate programs instead

… And much more.

There’s really nothing you can’t do that a window manager can do.

Conclusion

We went over a lot of tools in this post, we covered basic manipulation, to extensions, to simulation of inputs, to listening for your own inputs to take action. The glue thinking should make sense now, the freedom to build and customize your environment the way you want.

Let’s close this post with something that might be confusing for some. In the reddit community called /r/unixporn users tag their posts with the window manager they are using. Some of the entries are tagged as wmutils but after reading this article you’ll realize that in fact wmutils is not a window manager in itself. Those post should really be tagged “No WM” however that would be confusing as the line is thin between the definition of a window manager and the glue that could possibly be created using the tools in this post.
Another window manager that is missing the operation part is bspwm and it’s usually accompanied by sxhkd, but that sxhkd (hotkey daemon) can be used with any other lightweight window manager such as monsterwm, dwm, catwm, or no window manager at all.

There’s no limit to what you can do, enjoy playing with glue.

Indrek Lasn (indreklasn)

How to improve your asynchronous Javascript code with async and await January 06, 2019 09:37 PM

Photo by Max Nelson on Unsplash

If you’ve had the chance to observe modern Javascript code — chance are high you have seen the async and await syntax somewhere.

Async/await is arguably one of the best-received new additions to the language. Async/await makes asynchronous code appear and behave like synchronous code. Async/await are promise-based.

Before we jump into async/await, we must understand what are promises and the role they play.

Promises

A Promise is an object representing the eventual completion or failure of an asynchronous operation.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Using_promises

Why do we need asynchronous code, anyway?

Javascript is a single threaded language — this means Javascript can only do one thing at once. Imagine calling our API synchronously and blocking the entire thread for the API call duration — our users would have to wait 30 seconds or as long it takes for the network request to resolve — a big no-no!

In case you’re interested to learn more — here’s a more in-depth explanation about asynchronous programming with Javascript.

Asynchronous Programming :: Eloquent JavaScript

The way we used to handle asynchronous Javascript code was via callbacks. Chances are high you’ve come across callbacks.

What are callbacks?

A callback function, also known as a higher-order function, is a function that is passed to another function. Functions are first class citizens in Javascript — this means they can be passed as arguments to functions.

You might have seen jQuery specific code like this.

The following code attaches an event listener on our button and calls the alert once it’s triggered.

Where is the callback? Can you tell?

You probably can — it’s the anonymous function inside the click function parentheses.

https://javascriptissexy.com/understand-javascript-callback-functions-and-use-them/

Callbacks in a nutshell: we trigger a function, do some stuff, then call another function.

Callbacks are not all bad, they worked — they still do. But, what happens if we have a callback inside a callback inside a callback — you get the point. It gets really messy and unmaintainable really quick.

This problem above is named “callback hell.”

http://callbackhell.com/

One more example to seal the case for callbacks.

We have a melon!

We check if we have a melon inside the array — if we do, we chew the melon, after chewing we throw away the melon. We’re also handling exceptions with the err callback.

Hey — we have a banana instead!

Note: the err callback is the first argument always in the Node world — best practices!

Just to throw you off a little — I made the previous code as readable as I can, here’s how it might look more often:

mini callback hell; technically, having anonymous arrow functions would be the remedy but still is not the perfect solution.

You can imagine, couple more callbacks and we’re on the highway to [callback] hell — pun intended!

Promises to the rescue!

Promises are a clean way to write asynchronous code. The promise has a single argument, which is a callback.

The callback has two arguments, the reject and resolve conditions.

And if we use the arrow function to shorten the code:

Inside the promise block, we can decide when to resolve and when to reject the promise.

Inside the promise we check if we have a melon. If we do, let’s resolve the function with the value we pass inside resolve — we can pass literally any value to the resolve.

Promises are immediately invoked/resolved, thus why we see the console.log without calling the promise.

Promises are called and chained with the then and catch methods. Think of it like this: the promise has resolved with a value — what do we do with it?

This is where the then and catch syntax comes in. They are both callbacks which accept one argument, which is the return value passed inside the promise.

Here’s how we could handle a promise rejection:

Handling a promise rejection with catch()

Right, now our promise is not very dynamic — why?

Well, because we have an if statement that isn’t dynamic. Promises are extremely powerful when wrapped inside a function. We call them higher order functions.

Wrapping our promise inside a function

Do you notice the small change? We wrapped our promise inside a function which takes one argument in this case. This gives our promises huge flexibility. We can pass any condition to our promise and based on that condition, it will get either rejected or resolved.

Here’s the fully working promise which gets resolved.

Resolved promise

And the rejected promise.

Rejected promise

Starts to look familiar? You might have seen Axios API calls like the one below.

Axios + React API call example (promise based)

https://medium.com/media/dccf0bed60056c668286137318baffcc/href

Or the fetch API call;

Fetch + react API call example (promise based)

https://medium.com/media/1b85a617edc829c5356df6342a49be57/href

What do they have in common?

Well, for starters they’re both a promise. They’re fundamentally using promises under the “hood”. Just like we wrapped our promise inside a function, so do both of these examples.

Secondly, they’re both asynchronous code. Promises are naturally asynchronous.

If you find promises confusing — please read this article and come back ❤

https://gist.github.com/domenic/3889970

Here’s how an asynchronous API call looks like:

It’s safe to say promises are much better than callbacks. Promises have their own flaws although — promises can get out of hand quite quickly.

What if there’s a better way, an even cleaner way. Async/wait to the rescue!

https://codesandbox.io/s/p9mr3jzwp0?autoresize=1&expanddevtools=1&hidenavigation=1

We mark the function as async — inside the function we mark the asynchronous code with await — Javascript will resolve the promise and then move on to the next line. In a nutshell, we change the asynchronous code to read like synchronous code while still running asynchronously.

Notice how we don’t call the promise constructor anymore and how there are much less then() and catch() methods.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function

What about error handling?

Our current async/await doesn’t handle exceptions. This can lead to catastrophic bugs which might crash your app.

Try…catch to the rescue!

Wrapping our async function inside try catch block

The try catch block attempts to execute the code and if it runs into any exceptions/problems, it passes the error object to the catch block and executes the code inside the catch block.

try...catch

Here’s how our async/await fetch would look with error handling.

This is just one of many examples, there are many ways to leverage async/wait. If you’re curious — here’s a pretty good example StackOverflow answer exploring the alternatives.

try/catch blocks with async/await

Bonus: Check out this great article on misconceptions about async/await.

A common misconception about async/await in JavaScript

Thanks for reading and I hope you found this useful!

I’d love to connect with you directly on Twitter and pass on more knowledge in the future. ❤

Indrek Lasn (@lasnindrek) | Twitter

About the author:

Indrek Lasn - Medium

Here are some of my previous articles:


How to improve your asynchronous Javascript code with async and await was originally published in freeCodeCamp.org on Medium, where people are continuing the conversation by highlighting and responding to this story.

Gustaf Erikson (gerikson)

December January 06, 2019 09:33 AM

January 05, 2019

Gokberk Yaltirakli (gkbrk)

Reverse Engineering the Godot File Format January 05, 2019 04:09 PM

I’ve been messing around with the Godot game engine recently. After writing some examples that load the assets and map data from files, I exported it and noticed that Godot bundled all the resources into a single .pck file. It was packing all the game resources and providing them during runtime as some sort of virtual file system.

Of course; after I was finished for the day with learning gamedev, I was curious about that file. I decided to give myself a small challenge and parse that file using only hexdump and Python. I opened the pack file with my hex editor and I was met with a mixture of binary and ASCII data. Here’s the beginning of the file from hexdump -C game.pck.

00000000  47 44 50 43 01 00 00 00  03 00 00 00 01 00 00 00  |GDPC............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000050  00 00 00 00 1e 00 00 00  40 00 00 00 72 65 73 3a  |........@...res:|
00000060  2f 2f 2e 69 6d 70 6f 72  74 2f 61 6d 67 31 5f 66  |//.import/amg1_f|
00000070  72 31 2e 70 6e 67 2d 32  32 66 30 32 33 32 34 65  |r1.png-22f02324e|
00000080  63 62 39 34 32 34 39 37  62 61 61 31 38 30 37 33  |cb942497baa18073|
00000090  37 36 62 37 31 64 30 2e  73 74 65 78 00 3d 00 00  |76b71d0.stex.=..|
000000a0  00 00 00 00 28 04 00 00  00 00 00 00 de a0 23 30  |....(.........#0|
000000b0  cf 7c 59 5c fb 73 5d f6  a7 f8 12 a7 40 00 00 00  |.|Y\.s].....@...|
000000c0  72 65 73 3a 2f 2f 2e 69  6d 70 6f 72 74 2f 61 6d  |res://.import/am|
000000d0  67 31 5f 6c 66 31 2e 70  6e 67 2d 62 33 35 35 35  |g1_lf1.png-b3555|
000000e0  34 66 64 31 39 36 37 64  65 31 65 62 32 63 64 31  |4fd1967de1eb2cd1|
000000f0  32 33 32 65 32 31 38 33  33 30 32 2e 73 74 65 78  |232e2183302.stex|

Immediately we can see that the file starts with some magic bytes, and before our ASCII filename begins there’s a lot of zeros. They might be keeping that space as a padding or for future extensions, or maybe it can even contain data depending on your settings, but for our immediate goal it doesn’t matter.

What looks interesting here is the two integers right before our path. 1e 00 00 00 and 40 00 00 00, probably lengths or counts since they are before real data. Saying these two are little-endian unsigned integers would be a good assumption, because otherwise they would be huge numbers that have no business being the length of anything.

The first number is 30 and the second one is 64. Now, what’s the name of that file? res://.import/amg1_fr1.png-22f02324ecb942497baa1807376b71d0.stex. Exactly 64 bytes. That means we now know that the paths are prefixed by their length.

If we look at the next path, we can see that a similar pattern of being length prefixed still applies. The first integer we found, 30, is most likely the number of files we have. And a rough eyeballing of the file contents reveals that to be the case.

Let’s get a little Python here and try to read the file with our knowledge so far. We’ll read the first integer and loop through all the files, trying to print their names.

pack = open('game.pck', 'rb')
pack.read(0x54) # Skip the empty padding

file_count = struct.unpack('<I', pack.read(4))[0]

name_len = struct.unpack('<I', pack.read(4))[0]
name = pack.read(name_len).decode('utf-8')

print(f'The first file is {name}')

Running this code produces the following output, success!

The first file is res://.import/amg1_fr1.png-22f02324ecb942497baa1807376b71d0.stex

Now let’s try to loop file_count times and see if our results are good. One thing we should notice is the data following the ASCII text, before the new one begins. If we miss that, we will read the rest of the data wrong and end up with garbage. Let’s go back to our hexdump and count how many bytes we need to skip until the next length. Looks like we have 32 extra bytes. Let’s account for those and print everything.

for i in range(file_count):
  name_len = struct.unpack('<I', pack.read(4))[0]
  name = pack.read(name_len)
  pack.read(32)
  print(name)
...
b'res://MapLoader.gd.remap'
b'res://MapLoader.gdc\x00'
b'res://MapLoader.tscn'
b'res://Player.gd.remap\x00\x00\x00'
b'res://Player.gdc'
...

Much better, the only issue is the trailing null bytes on some file names. This shouldn’t be a huge issue though, these are probably random padding and they don’t even matter if we consider the strings null-terminated. Let’s just get rid of trailing null bytes.

name = pack.read(name_len).rstrip(b'\x00').decode('utf-8')

After this change, we can get a list of all the resource files in a Godot pack file.

Getting the file contents

Sure, getting the list of files contained in the file is useful. But it’s not super helpful if our tool can’t get the file contents as well.

The file contents are stored separately from the file names. The thing we parsed so far is only the file index, like a table of contents. It’s useful to have it that way so when the game needs a resource at the end of the file, the game engine won’t have to scan the whole file to get there.

But how can we get find where the contents are without going through the whole thing? With offsets of course. Every entry we read from the index, along with the file name, contains the offset and the size of the file. It’s not the easiest thing to explain how you discover something after the fact, but it’s a combination of being familiar with other file formats and a bunch of guesswork. Now I’d like to direct your attention to the 32 bytes we skipped earlier.

Since we already have the file names, we can make assumptions like text files being smaller than other resources like images and sprites. This can be made even easier by putting files with known lengths there, but just for the sake of a challenge let’s pretend that we can’t create these files.

After each guess, we can easily verify this by checking with hexdump or plugging the new logic in to our Python script.

The 4 byte integer that follows the file name is our offset. If we go to the beginning of the file and count that many bytes, we should end up where our file contents begin.

offset = struct.unpack('<I', pack.read(4))[0]

This is followed by 4 empty bytes and then another integer which is our size. Those 4 bytes might be used for other purposes, but again they are irrelevant for our goal.

pack.read(4)
size = struct.unpack('<I', pack.read(4))[0]

The offset is where the file contents began, and the size is how many bytes it is. So the range between offset and offset + size is the file contents. And because we ended up reading 12 bytes from the file, we should make our padding 20 instead of 32.

Reading a File

To finish up, let’s read the map file I put in my game. It’s a plaintext file with JSON contents, so it should be easy to see if everything looks complete.

pack = open('game.pck', 'rb')
pack.read(0x54) # Skip the empty padding

file_count = struct.unpack('<I', pack.read(4))[0]

for i in range(file_count):
  name_len = struct.unpack('<I', pack.read(4))[0]
  name = pack.read(name_len).rstrip(b'\x00').decode('utf-8')

  offset = struct.unpack('<I', pack.read(4))[0]
  pack.read(4)
  size = struct.unpack('<I', pack.read(4))[0]
  pack.read(20)

  print(name)

  if name == 'res://maps/map01.tres':
    pack.seek(offset)
    content = pack.read(size).decode('utf-8')
    print(content)
    break
...
res://Player.gd.remap
res://Player.gdc
res://Player.tscn
res://block.gd.remap
res://block.gdc
res://block.tscn
res://default_env.tres
res://icon.png
res://icon.png.import
res://maps/map01.tres
{
    "spawn_point": {"x": 5, "y": 3},
    "blocks": [
        {"x": 5, "y": 5, "msg": "Welcome to Mr Jumpy Man", "jumpLimit": 0},
        {"x": 7, "y": 5, "msg": "We're still waiting on the trademark"},
        {"x": 9, "y": 5, "texture": "platformIndustrial_001.png"},
        {"x": 11, "y": 6, "msg": "If you fall down,\nyou will be teleported to the start point"},
...

Everything looks good! Using this code, we should be able to extract the resources of Godot games.

Finishing words

Now; before the angry comments section gets all disappointed, let me explain. I am fully aware that Godot is open source. Yes, I could’ve looked at the code to know exactly how it works. No, that wouldn’t be as fun. Kthxbye.

Stig Brautaset (stig)

My Musical Goals for 2019 January 05, 2019 04:00 PM

I list my musical goals for 2019, and touch briefly on achievements in 2018.

Geoff Wozniak (GeoffWozniak)

Book review: Retro debugging January 05, 2019 01:00 AM

This is the fifth in a series of reviews of debugging books.

  • The Frozen Keyboard: Living With Bad Software (Boris Beizer, TAB Books, 275 pp., 1988)
  • Secrets of Software Debugging (Truck Smith, TAB Books, 276 pp., 1984)
  • How to Debug Your Personal Computer (Jim Huffman and Robert C. Bruce, Reston Publishing, 157 pp., 1980)
  • Software Debugging for Microcomputers (Robert C. Bruce, Reston Publishing, 351 pp., 1980)
  • Program Style, Design, Efficiency, Debugging, and Testing, 2nd Ed. (Dennie Van Tassel, Prentice-Hall, 323 pp., 1978)
  • Program Debugging (A.R. Brown and W.A. Sampson, American Elsevier Computer, 166 pp., 1973)
  • Debugging Techniques in Large Systems (edited by Randall Rustin, Prentice-Hall, 148 pp., 1971)

Part of the reason I started this series of book reviews was to get a feel for how approaches to debugging have changed over the years. Similarly, I’m interested in seeing what has stayed the same, perhaps parading around in different guises.

To that end, I collected some debugging books that were published before 1990. I used the following methodology, that is totally and completely based on rigorous techniques practised in academic settings.

  1. Search for books published before 1990 with “debug” in the title.
  2. If the combination of title, cover, and author seemed interesting—and the price was not ridiculous—obtain a copy and read it.

There isn’t any theme to the content aside from that, save for Beizer’s book, which was a recommendation.

As it happens, the books fall into two different computing eras: mainframe computing and personal/home computing. These also happen to line up with the publication dates: 1970s for mainframe, 1980s for personal. You won’t glean much about debugging for the current day and age if you read them, but you will find some enlightening contrasts and parallels.

(A note about Beizer’s book: it’s really a general book for the home computer user and not a book about the process of software development or debugging. It should not be considered part of the home computing books. Although it does discuss how software is developed, particularly how it is (or should be) tested, it is more about the state of software and computers at the time. I’ve included it in this review because it is both a good book and helps provide context for the era.)

Aside from the books’ specific content—which is amusingly out of date in almost all respects—the primary difference they demonstrate from today’s world is speed. Every single aspect of development comes across as slower. Making a change in the code and just running it, let alone deploying it, could take on the order of days. Brown and Sampson’s book describes a case study, run in 1971, that contains the specification for a program that today would be, at best, an assignment for a first year student. Competent developers would probably be able to write it in under a day, with unit tests. In the case study it takes two professional programmers, each with 6-7 years experience, nearly two weeks of effort to get a working program.

As ludicrous as that may seem, a big reason for it is the lack of machine time. The case study contains a detailed program development log for one of the programmers. That programmer logged 13.75 days of development time and only 37 minutes of machine time. It takes him days to get to the point where he even starts writing code, which in all likelihood ends up on cards or some other tedious, error-prone input mechanism. All the mainframe era books allude to the fact that CPU time cost money and that “the computer” is often used for many other things. In other words, developers almost never had a computer to work on and initially wrote most of the code at their desk on paper. There were certainly terminals and time sharing operating systems in the 1970s so it is not accurate to assume that all programmers were using punch cards to submit programs in batch to the computer. Nevertheless, there was significantly less computing power and storage space than today and it was much more expensive. Hence, the mainframe era books consistently push the notion that you should aim to “get it right the first time” so as to avoid wasting CPU cycles.

The books for home computing don’t go to this extent for the obvious reason that you had a computer right in front of you, although it would be a stretch to say that development was a speedy affair. Truck Smith’s book contains the programming journals he recorded while writing three different programs on an Apple II. One of them is for a simple diff-like program written in BASIC that ends up being a little under 300 lines, including comments. Here he is reflecting on the effort.

Reading the blow by blow descriptions [about 50 pages worth] gives the impression that it took a long time to write this program and get it running. Actually it took me about a week, working nights (but not every night of the week).

The home computing books are devoid of discussion on software tools for development and debugging, including something as rudimentary as a text editor. They assume the use of the BASIC interpreter and primitive (that is, non-multitasking) operating system that comes with the computer to input and manage a program. Smith also has journals for a Pascal and assembly language program, but there is no hint as to how to go about actually using those languages on the computer. The assumption is that the user has the knowledge and can obtain the necessary software to use a different programming language. Bruce’s books centre entirely on the limited capabilities of BASIC on the Sol 20 microcomputer. The home computing world comes with more immediate computing power and very little in the way of taking advantage of it.

This lack of speed goes hand-in-hand with debugging being talked about as a distinct stage of software development. It is almost exclusively portrayed as an offline activity. This is explicitly stated in the mainframe books. It is also true in the home computing books, although the definition shifts slightly.

When you look at the debugging tools and techniques the books describe there is one item that rules them all: printed output. Nearly everything is geared toward making program listings or diagnostic output more efficient and useful. Like most things in these books, this will strike readers in this day and age as ridiculous. Still, it is entirely reasonable given the conditions of the time. Recall that in the mainframe domain CPU time was precious: running automated tests or using tools to query output (such as something like grep) may have been difficult or impossible. In the world of home computing merely inspecting your program on a monitor, without obtaining extra software, was likely a chore. There was very limited memory (be it disk or RAM) for holding output so even if there was a pager program, only so much could be stored. The cheapest, most abundant source of “memory” for storing output for the purposes of inspection was paper.

Given that paper is characterized as the preferred method for debugging it’s not surprising that debugging is talked about as something separate from programming. This is probably the starkest difference from today’s methods, where program output is easily stored and examined making debugging practically synonymous with development. Bruce talks about the “program development and shakedown” stages. Van Tassel says that one should plan for debugging to take four times as long as any of planning, writing, and testing. Brown and Sampson’s book starts out decrying the fact that so much time is spent on debugging. Practically every paper in the collection edited by Rustin assumes debugging is independent from the act of programming. This is not surprising: when output is removed from the runtime environment, the development process feels more delineated and less fluid.

The debugging techniques, however, are not markedly different. They’re just smaller in scope to the point where in modern environments we don’t really think of them as techniques. Everything that is prescribed for instrumenting a program to tease out information about its state is still applicable, although it may not be necessary.

The first is desk checking, that is, reading the program listing, possibly making a flow chart, and running through examples manually. Exactly when this step is recommended depends on whether you are reading the mainframe books (before you write the code) or the home computing books (after you run the code). The need to make a flow chart is pretty rare these days since we can quickly navigate code online to understand its control flow. Due to an abundance of computing power and storage space it’s also easy to execute example code and store it (automated testing, anyone?). Desk checking is a form of last resort today and a primary technique of the past.

The other technique, talked about at great length, is careful and judicious placement of print statements. When it comes to tools used in the online computing environment, print statements are the only one mentioned in the home computing books. There is no online debugger: no breakpoints, no program halting, and no memory inspection. There are also no system tools since the whole computer is devoted to running the program you’re writing. The mainframe books do talk about system-level monitors and online debugging, but it is done in a way that suggests it is an exotic luxury. It’s more or less “use ’em if you got ’em.” Any such tools are described as system specific, so a treatment of it would be out of place in a book about general debugging. The use of memory dumps is talked about by the mainframe books, saying that they are hard to work with (you probably wouldn’t examine them online) and that you’ll probably want to avoid them. In the end, the only reliable technique that you can use anywhere is a well-placed print statement. The advice on how to place them is largely the same as today, with more attention paid to avoiding extensive output since this would use a lot of paper.

As mentioned, the mainframe books see debugging as a stage of software development. The home computing books, by contrast, barely discuss the notion of process at all. Bruce’s books completely ignore how a program got written and dive into how you can fix an existing one; any programs provided come into existence through a contrived narrative for the sake of presentation. Smith’s book spends limited space on any development formalities and buries all the advice in enjoyable, haphazard experience reports. Debugging BASIC programs is something that gets done in fast edit-run cycles, the caveat being that the paucity of features in the environment makes it less appealing than it sounds. This is probably why the home computing books push offline debugging so much.

The lack of rigour in the home computing books carries over into testing. While testing is not the same as debugging, they are closely related and are frequently discussed together. So it’s a bit surprising that there is no serious mention of it in the home computing book set. Smith does touch on it, but doesn’t talk much about how to test; Bruce and Huffman completely ignore the subject. It’s embarrassing that Beizer provides more insight on how to test software, even superficially, than books directly aimed at programmers. Even more shameful, Beizer goes into some detail—with wonderful, witty diatribes—and demonstrates a deep knowledge of the subject. The lack of testing discussion in the home computing books relegates them to the realm of amateurs.

The mainframe books, on the other hand, are serious about testing and how it acts as a “debugging aid”. There is a clear emphasis on isolating parts of the program and testing them on a regular basis, especially after a change has been made. The practice of “testing units” or testing modules is mentioned often. This process is very drawn out in the description from Brown and Sampson as it involves multiple people: those who write the program, those who type it, and those who run it. Van Tassel says that using automated testing is the only way to handle large code bases and that storing test input so it can be reused is a necessity. Keeping a log of tests and test results is recommended if you desk check a routine. Testing plays a key role in managing how one debugs a program. (Included in the testing discussions is a lot of hopefulness on one day proving programs correct, complete with a healthy dose of skepticism and no serious expectation of it happening soon.)

Again, what differs from today is the scope. For example, merely checking types at module/function boundaries, let alone within a function, is referred to as “runtime overhead” that may require multiple compilers since optimizing compilers probably wouldn’t verify types. There is an emphasis on getting the compiler to check as much as possible if the resources are available (such as having a compiler that checks those things in the first place!). As usual this is because the resources to run tests are constrained and compilation itself took a significant amount of time. Descriptions of what is tested by the programmer is generally limited to a single module, which is something on the order of implementing a few equations or file operations.

All in all, debugging on mainframes in the 1970s and home computers in the early 1980s comes across a tedious exercise. In the mainframe books you can practically feel the authors pining for more power. There is all this potential and nothing to realize it. It’s mostly today’s development process played out in super-slow motion, but at least it is methodical. The home computing books, on the other hand, have a severe lack of discipline. This is probably in large part due to the use of BASIC as the grounding point, although Truck Smith tries hard to branch out a little.

This collection of books shows that, in the past, a lot more attention was paid to things that we now mostly ignore. And those things are found in the periods of time that emerge from having to wait for anything to happen. There is a clear advantage to having a wealth of computing power available when it comes to debugging. Being able to capture gigabytes of output from any number of tools and analyze it in short order, all on the same machine on which the program is running, is significantly better than worrying about whether a print statement will send a hundred pages of output to the line printer. The introspection tools that our current execution environments offer is a realization of what was yearned for in the mainframe books and we get to reap the rewards. What is revelatory is the seeming lack of formality in home computing. It’s as if everyone was so enamoured by the fact there was a computer in their home that they forgot about the computers at work. The complaints in Beizer’s book may very well be attributed to the lessons found in Smith, Bruce and Huffman’s work. There isn’t enough in the books to explain this discrepancy in thoroughness, but there’s certainly something to explore.

One thing is clear: the more things change the more they stay the same, they just get faster.

Capsule reviews

For those interested in the specific books I’ve included short reviews. If you’re really keen on a book and want to read it be forewarned: unless you want to study the historical context, you won’t learn much from the actual content, except in one case. Appreciating the content requires some research into, and possibly experience with, the actual computing environments referenced in the books.

The Frozen Keyboard
Five stars. Beizer is a pleasure to read. His clear prose, pointed metaphors, and backhanded compliments elegantly excoriate the tech industry for its hyped promises and general lack of empathy for the user. A lot of what he says—perhaps aside from the buying guide and the extensive backup procedure using a printer—still applies. He’s amusingly colourful: when talking about programming and why bugs exist, his description starts with “Programming is a bitch.” If I were teaching software testing or usability, I’d put this book on the supplemental reading list. Recommended for discussion over drinks. (Shout out to John Regehr for pointing me to Beizer’s work.)
Secrets of Software Debugging
This book is mentioned in another book I reviewed (Debugging by Thinking, by Metzger) where it is described as targeting “an obsolete language/environment and is of historical interest only.” The hell with that. This book is actually fun and somewhat insightful, despite its dearth of rigour. Truck Smith is clearly a programmer writing for aspiring programmers. The content is thin and there’s a lot of grasping at formality, but the discussion about the mindset and attitude you need to tackle tough debugging problems still resonates. There’s also three long chapters that are his journals from developing three different programs (one of which is his first Pascal program) that helps you appreciate what went into developing on home computers. As an added bonus, the edition I have comes with a cover whose genesis demands an explanation.
How to Debug Your Personal Computer
This is a condensed version of Software Debugging for Microcomputers containing select chapters from it, with an extra chapter devoted to describing hardware of the time. The hardware chapter is dull and probably wouldn’t help if you had to debug hardware. The rest is the same as the original book, save for the fact it doesn’t tell you what personal computer the code is for. (It’s a Sol 20 microcomputer.)
Software Debugging for Microcomputers
Review in less than ten words: 300+ pages of PRINT statement examples. The programs are all BASIC, the designs are questionable, and the discussion drags more than experimental avant-garde cinema. Each concept could be explained in a few pages. Instead, they are weaved into the development of medium sized programs that come with grandiose descriptions and pedestrian implementations. Furthermore, the names of known, studied concepts at the time are given strange names: “block debugging” (subroutines), “forcing” (testing edge cases), “snapshots” (subroutines, again), just to name a few. I pity anyone who learned from this book. (The same goes for its condensed cousin mentioned above.)
Program Style, Design, Efficiency, Debugging, and Testing
Van Tassel’s book was probably a good one to have on your shelf if you were a professional programmer or a student of programming in the late 1970s and into the 1980s. Topics germane to programming at the time are presented seriously and with care. By now all the advice has been superseded or obsoleted, but you won’t be guided in the wrong direction by reading it. There are lots of programming exercises provided that would still be enjoyable to tackle. Some descriptions are strange and dated, feeling out of place in a modern text. And through an incredible coincidence, I found a passage plagiarized from Program Debugging, a book he does at least reference.
Program Debugging
A strange little book that describes a formal approach to debugging that the authors call “The Method”. This method is based on a problem solving technique used by managers and is this: study the specification, look for the differences from it, hypothesize why they exist, look at the program to determine if you’re correct, and verify that you are correct. It’s like reading a business book that claims scientific management is why science is so successful. What is interesting is the case study of using The Method in a real programming environment. It’s well documented and fascinating to read. There is also this sexist passage on page 37 that Van Tassel rips off: “…and assume that his program has to survive all the sabotage attempts of cross-eyed punch girls, cretinous data control clerks, and idiotic operators.” Van Tassel (page 240) excises the sexism and shifts the insult: “A robust program should be able to survive cross-eyed keypunchers, cretinous data control clerks, and retarded operators.”
Debugging Techniques in Large Systems
This is a collection of papers and talk summaries from a symposium held in 1970. It’s also the earliest book about debugging that I could obtain. It mostly provides historical context, making it clear that interactive debugging was a good idea but difficult to pull off given the lack of resources and diversity of hardware at the time. There is a fair bit of discussion of using the compiler to prevent bugs, things like typographical errors and basic type checking. A lot of what is talked about here is taken for granted today. It did provide a view into the world of programming in very limited environments. I also learned about the unfortunately named debugging system AIDS.

If you know of some home computing books from the microcomputer era that talk about program development or debugging, or you know of popular references for programming on home computers in the early to mid-1980s, I’d appreciate an email pointing me to them.

January 4, 2019

January 04, 2019

Derek Jones (derek-jones)

Changes in the shape of code during the twenties? January 04, 2019 11:42 PM

At the end of 2009 I made two predictions for the next decade; Chinese and Indian developers having a major impact on the shape of code (ok, still waiting for this to happen), and scripting languages playing a significant role (got that one right, but then they were already playing a large role).

Since this blog has just entered its second decade, I will bring the next decade’s predictions forward a year.

I don’t see any new major customer ecosystems appearing. Ecosystems are the drivers of software development, and no new ecosystems has several consequences, including:

  • No major new languages: Creating a language is a vanity endeavor. Vanity project can take off if they are in the right place at the right time. New ecosystems provide opportunities for new languages to become widely used by being in at the start and growing with the ecosystem. There is another opportunity locus; it is fashionable for companies that see themselves as thought-leaders to have their own language, e.g., Google, Apple, and Mozilla. Invent your language at the right time, while working for a thought-leader company and your language could become well-known enough to take-off.

    I don’t see any major new ecosystems appearing and all the likely companies already have their own language.

    Any new language also faces the problem of not having a large collection packages.

  • Software will be more thoroughly tested: When an ecosystem is new, the incentives drive early and frequent releases (to build a customer base); software just has to be good enough. Once a product is established, companies can invest in addressing issues that customers find annoying, like faulty behavior; the incentive change results in more testing.

    There are other forces at work around testing. Companies are experiencing some very expensive faults (testing may be expensive, but not testing may be more expensive) and automatic test generation is becoming commercially usable (i.e., the cost of some kinds of testing is decreasing).

The evolution of widely used languages.

  • I think Fortran and C will have new features added, with relatively little fuss, and will quietly continue to be widely used (to the dismay of the fashionista).
  • There is a strong expectation that C++ and Java should continue to evolve:
  • I expect the ISO C++ work to implode, because there are too many people pulling in too many directions. It makes sense for the gcc and llvm teams to cooperate in taking C++ in a direction that satisfies developers’ needs, rather than the needs of bored consultants. What are Microsoft’s views? They only have their own compiler for strategic reasons (they make little if any profit selling compilers, compilers are an unnecessary drain on management time; who cares what happens to the language).
  • It is going to be interesting watching the impact of Oracle’s move to charging for runtimes. I have no idea what might happen to Java.

In terms of code volume, the future surely has to be scripting languages, and in particular Python, Javascript and PHP. Ten years from now, will there be a widely used, single language? People have been predicting, for many years, that web languages will take over the world; perhaps there will be a sudden switch and I will see that the choice is obvious.

Moore’s law is now dead, which means researchers are going to have to look for completely new techniques for building logic gates. If photonic computers happen, then ternary notation may reappear again (it was used in at least one early Russian computer); I’m not holding my breath for this to occur.

Maxwell Bernstein (tekknolagi)

Bytecode compilers and interpreters January 04, 2019 02:15 AM

Two fun things happened recently and the proximity of the two made something go click in my head and now I think I understand how bytecode interpreters work.

  1. I went to a class at work called “Interpreters 101” or something of the sort. In it, the presenter walked through creating a dead-simple tree-walking Lisp interpreter. Then he ended by suggesting we go out and re-implement it as a bytecode interpreter or even a JIT.
  2. I joined a new team at work that is working on something Python related. Because of my new job responsibilities, I have had to become rather closely acquainted with CPython 3.6 bytecode.

Since learning by writing seems to be something I do frequently, here is a blog post about writing a small bytecode compiler and interpreter in small pieces. We’ll go through it in the same manner as my lisp interpreter: start with the simplest pices and build up the rest of the stack as we need other components.

Some definitions

Before we dive right in, I’m going to make some imprecise definitions that should help differentiate types of interpreters:

  • Tree-walking interpreters process the program AST node by node, recursively evaluating based on some evaluation rules. For a program like Add(Lit 1, Lit 2), it might see the add rule, recursively evaluate the arguments, add them together, and then package that result in the appropriate value type.
  • Bytecode interpreters don’t work on the AST directly. They work on preprocessed ASTs. This simplifies the execution model and can produce impressive speed wins. A program like Add(Lit 1, Lit 2) might be bytecode compiled into the following bytecode:

    PUSH 1
    PUSH 2
    ADD
    

    And then the interpreter would go instruction by instruction, sort of like a hardware CPU.

  • JIT interpreters are like bytecode interpreters except instead of compiling to language implementation-specific bytecode, they try to compile to native machine instructions. Most production-level JIT interpreters “warm up” by measuring what functions get called the most and then compiling them to native code in the background. This way, the so-called “hot code” gets optimized and has a smaller performance penalty.

My Writing a Lisp series presents a tree-walking interpreter. This much smaller post will present a bytecode interpreter. A future post may present a JIT compiler when I figure out how they work.

Without further ado, let us learn.

In the beginning, there was a tree-walking interpreter

Lisp interpreters have pretty simple semantics. Below is a sample REPL (read-eval-print-loop) where all the commands are parsed into ASTs and then evaled straightaway. See Peter Norvig’s lis.py for a similar tree-walking interpreter.

>>> 1
1
>>> '1'
1
>>> "hello"
hello
>>> (val x 3)
None
>>> x
3
>>> (set x 7)
None
>>> x
7
>>> (if True 3 4)
3
>>> (lambda (x) (+ x 1))
<Function instance...>
>>> ((lambda (x) (+ x 1)) 5)
6
>>> (define f (x) (+ x 1))
None
>>> f
<Function instance...>
>>> +
<function + at...>
>>> (f 10)
11
>>>

For our interpreter, we’re going to write a function called compile that takes in an expression represented by a Python list (something like ['+', 5, ['+', 3, 5]]) and returns a list of bytecode instructions. Then we’ll write eval that takes in those instructions and returns a Python value (in this case, the int 13). It should behave identically to the tree-walking interpreter, except faster.

The ISA

For our interpreter we’ll need a surprisingly small set of instructions, mostly lifted from the CPython runtime’s own instruction set architecture. CPython is a stack-based architecture, so ours will be too.

  • LOAD_CONST
    • Pushes constants onto the stack.
  • STORE_NAME
    • Stores values into the environment.
  • LOAD_NAME
    • Reads values from the environment.
  • CALL_FUNCTION
    • Calls a function (built-in or user-defined).
  • RELATIVE_JUMP_IF_TRUE
    • Jumps if the value on top of the stack is true.
  • RELATIVE_JUMP
    • Jumps.
  • MAKE_FUNCTION
    • Creates a function object from a code object on the stack and pushes it on the stack.

With these instructions we can define an entire language. Most people choose to define math operations in their instruction sets for speed, but we’ll define them as built-in functions because it’s quick and easy.

The Opcode and Instruction classes

I’ve written an Opcode enum:

import enum


class AutoNumber(enum.Enum):
    def _generate_next_value_(name, start, count, last_values):
        return count


@enum.unique
class Opcode(AutoNumber):
    # Add opcodes like:
    # OP_NAME = enum.auto()
    pass

Its sole purpose is to enumerate all of the possible opcodes — we’ll add them later. Python’s enum API is pretty horrific so you can gloss over this if you like and pretend that the opcodes are just integers.

I’ve also written an Instruction class that stores opcode and optional argument pairs:

class Instruction:
    def __init__(self, opcode, arg=None):
        self.opcode = opcode
        self.arg = arg

    def __repr__(self):
        return "<Instruction.{}({})>".format(self.opcode.name, self.arg)

    def __call__(self, arg):
        return Instruction(self.opcode, arg)

    def __eq__(self, other):
        assert(isinstance(other, Instruction))
        return self.opcode == other.opcode and self.arg == other.arg

The plan is to declare one Instruction instance per opcode, naming its argument something sensible. Then when bytecode compiling, we can make instances of each instruction by calling them with a given argument. This is pretty hacky but it works alright.

# Creating top-level opcodes
STORE_NAME = Instruction(Opcode.STORE_NAME, "name")
# Creating instances
[STORE_NAME("x"), STORE_NAME("y")]

Now that we’ve got some infrastructure for compilation, let’s start compiling.

Integers

This is what our compile function looks like right now:

def compile(exp):
    raise NotImplementedError(exp)

It is not a useful compiler, but at least it will let us know what operations it does not yet support. Let’s make it more useful.

The simplest thing I can think of to compile is an integer. If we see one, we should put it on the stack. That’s all! So let’s first add some instructions that can do that, and then implement it.

class Opcode(AutoNumber):
    LOAD_CONST = enum.auto()

I called it LOAD_CONST because that’s what Python calls its opcode that does something similar. “Load” is a sort of confusing word for what its doing, because that doesn’t specify where it’s being loaded. If you want, you can call this PUSH_CONST, which in my opinion better implies that it is to be pushed onto the VM stack. I also specify CONST because this instruction should only be used for literal values: 5, "hello", etc. Something where the value is completely defined by the expression itself.

Here’s the parallel for the Instruction class (defined at the module scope):

LOAD_CONST = Instruction(Opcode.LOAD_CONST, "const")

The argument name "const" is there only for bytecode documentation for the reader. It will be replaced by an actual value when this instruction is created and executed. Next, let’s add in a check to catch this case.

def compile(exp):
    if isinstance(exp, int):
        return [LOAD_CONST(exp)]
    raise NotImplementedError(exp)

Oh, yeah. compile is going to walk the expression tree and merge together lists of instructions from compiled sub-expressions — so every branch has to return a list. This will look better when we have nested cases. Let’s test it out:

assert compile(5) == [LOAD_CONST(5)]
assert compile(7) == [LOAD_CONST(7)]

Now that we’ve got an instruction, we should also be able to run it. So let’s set up a basic eval loop:

def eval(self, code):
    pc = 0
    while pc < len(code):
        ins = code[pc]
        op = ins.opcode
        pc += 1
        raise NotImplementedError(op)

Here pc is short for “program counter”, which is equivalent to the term “instruction pointer”, if you’re more familiar with that. The idea is that we iterate through the instructions one by one, executing them in order.

This loop will throw when it cannot handle a particular instruction, so it is reasonable scaffolding, but not much more. Let’s add a case to handle LOAD_CONST in eval.

def eval(code):
    pc = 0
    stack = []
    while pc < len(code):
        ins = code[pc]
        op = ins.opcode
        pc += 1
        if op == Opcode.LOAD_CONST:
            stack.append(ins.arg)
        else:
            raise NotImplementedError(op)
    if stack:
        return stack[-1]

Note, since it will come in handy later, that eval returns the value on the top of the stack. This is the beginning of our calling convention, which we’ll flesh out more as the post continues. Let’s see if this whole thing works, end to end.

assert eval(compile(5)) == 5
assert eval(compile(7)) == 7

And now let’s run it:

willow% python3 ~/tmp/bytecode0.py
willow%

Swell.

Naming things

My name, and yours, and the true name of the sun, or a spring of
water, or an unborn child, all are syllables of the great word that
is very slowly spoken by the shining of the stars. There is no other
power. No other name.


-- Ursula K Le Guin, A Wizard of Earthsea

Now, numbers aren’t much fun if you can’t do anything with them. Right now the only valid programs are programs that push one number onto the stack. Let’s add some opcodes that put those values into variables.

We’ll be adding STORE_NAME, which takes one value off the stack and stores it in the current environment, and LOAD_NAME, which reads a value from the current environment and pushes it onto the stack.

@enum.unique
class Opcode(AutoNumber):
    LOAD_CONST = enum.auto()
    STORE_NAME = enum.auto()
    LOAD_NAME = enum.auto()

# ...

STORE_NAME = Instruction(Opcode.STORE_NAME, "name")
LOAD_NAME = Instruction(Opcode.LOAD_NAME, "name")

Let’s talk about our representation of environments. Our Env class looks like this (based on Dmitry Soshnikov’s “Spy” interpreter):

class Env(object):
    # table holds the variable assignments within the env. It is a dict that
    # maps names to values.
    def __init__(self, table, parent=None):
        self.table = table
        self.parent = parent

    # define() maps a name to a value in the current env.
    def define(self, name, value):
        self.table[name] = value

    # assign() maps a name to a value in whatever env that name is bound,
    # raising a ReferenceError if it is not yet bound.
    def assign(self, name, value):
        self.resolve(name).define(name, value)

    # lookup() returns the value associated with a name in whatever env it is
    # bound, raising a ReferenceError if it is not bound.
    def lookup(self, name):
        return self.resolve(name).table[name]

    # resolve() finds the env in which a name is bound and returns the whole
    # associated env object, raising a ReferenceError if it is not bound.
    def resolve(self, name):
        if name in self.table:
            return self

        if self.parent is None:
            raise ReferenceError(name)

        return self.parent.resolve(name)

    # is_defined() checks if a name is bound.
    def is_defined(self, name):
        try:
            self.resolve(name)
            return True
        except ReferenceError:
            return False

Our execution model will make one new Env per function call frame and one new env per closure frame, but we’re not quite there yet. So if that doesn’t yet make sense, ignore it for now.

What we care about right now is the global environment. We’re going to make one top-level environment for storing values. We’ll then thread that through the eval function so that we can use it. But let’s not get ahead of ourselves. Let’s start by compiling.

def compile(exp):
    if isinstance(exp, int):
        return [LOAD_CONST(exp)]
    elif isinstance(exp, list):
        assert len(exp) > 0
        if exp[0] == 'val':  # (val n v)
            assert len(exp) == 3
            _, name, subexp = exp
            return compile(subexp) + [STORE_NAME(name)]
    raise NotImplementedError(exp)

I’ve added this second branch to check if the expression is a list, since we’ll mostly be dealing with lists now. Since we also have just about zero error-handling right now, I’ve also added some asserts to help with code simplicity.

In the val case, we want to extract the name and the subexpression — remember, we won’t just be compiling simple values like 5; the values might be (+ 1 2). Then, we want to compile the subexpression and add a STORE_NAME instruction. We can’t test that just yet — we don’t have more complicated expressions — but we’ll get there soon enough. Let’s test what we can, though:

assert compile(['val', 'x', 5]) == [LOAD_CONST(5), STORE_NAME('x')]

Now let’s move back to eval.

def eval(code, env):
    pc = 0
    stack = []
    while pc < len(code):
        ins = code[pc]
        op = ins.opcode
        pc += 1
        if op == Opcode.LOAD_CONST:
            stack.append(ins.arg)
        elif op == Opcode.STORE_NAME:
            val = stack.pop(-1)
            env.define(ins.arg, val)
        else:
            raise NotImplementedError(ins)
    if stack:
        return stack[-1]

You’ll notice that I’ve

  • Added an env parameter, so that we can evaluate expressions in different contexts and get different results
  • Added a case for STORE_NAME

We’ll have to modify our other tests to pass an env parameter — you can just pass None if you are feeling lazy.

Let’s make our first environment and test out STORE_NAME. For this, I’m going to make an environment and test that storing a name in it side-effects that environment.

env = Env({}, parent=None)
eval([LOAD_CONST(5), STORE_NAME('x')], env)
assert env.table['x'] == 5

Now we should probably go about adding compiler and evaluator functionality for reading those stored values. The compiler will just have to check for variable accesses, represented just as strings.

def compile(exp):
    if isinstance(exp, int):
        return [LOAD_CONST(exp)]
    elif isinstance(exp, str):
        return [LOAD_NAME(exp)]
    # ...

And add a test for it:

assert compile('x') == [LOAD_NAME('x')]

Now that we can generate LOAD_NAME, let’s add eval support for it. If we did anything right, its implementation should pretty closely mirror that of its sister instruction.

def eval(code, env):
    # ...
    while pc < len(code):
        # ...
        elif op == Opcode.STORE_NAME:
            val = stack.pop(-1)
            env.define(ins.arg, val)
        elif op == Opcode.LOAD_NAME:
            val = env.lookup(ins.arg)
            stack.append(val)
        # ...
    # ...

To test it, we’ll first manually store a name into the environment, then see if LOAD_NAME can read it back out.

env = Env({'x': 5}, parent=None)
assert eval([LOAD_NAME('x')], env) == 5

Neat.

Built-in functions

We can add as many opcodes as features we need, or we can add one opcode that allows us to call native (Python) code and extend our interpreter that way. Which approach you take is mostly a matter of taste.

In our case, we’ll add the CALL_FUNCTION opcode, which will be used both for built-in functions and for user-defined functions. We’ll get to user-defined functions later.

CALL_FUNCTION will be generated when an expression is of the form (x0 x1 x2 ...) and x0 is not one of the pre-set recognized names like val. The compiler should generate code to load first the function, then the arguments onto the stack. Then it should issue the CALL_FUNCTION instruction. This is very similar to CPython’s implementation.

I’m not going to reproduce the Opcode declaration, because all of those look the same, but here is my Instruction declaration:

CALL_FUNCTION = Instruction(Opcode.CALL_FUNCTION, "nargs")

The CALL_FUNCTION instruction takes with it the number of arguments passed to the function so that we can call it correctly. Note that we do this instead of storing the correct number of arguments in the function because functions could take variable numbers of arguments.

Let’s compile some call expressions.

def compile(exp):
    # ...
    elif isinstance(exp, list):
        assert len(exp) > 0
        if exp[0] == 'val':
            assert len(exp) == 3
            _, name, subexp = exp
            return compile(subexp) + [STORE_NAME(name)]
        else:
            args = exp[1:]
            nargs = len(args)
            arg_code = sum([compile(arg) for arg in args], [])
            return compile(exp[0]) + arg_code + [CALL_FUNCTION(nargs)]
    # ...

I’ve added a default case for list expressions. See that it compiles the name, then the arguments, then issues a CALL_FUNCTION. Let’s test it out with 0, 1, and more arguments.

assert compile(['hello']) == [LOAD_NAME('hello'), CALL_FUNCTION(0)]
assert compile(['hello', 1]) == [LOAD_NAME('hello'), LOAD_CONST(1),
                                    CALL_FUNCTION(1)]
assert compile(['hello', 1, 2]) == [LOAD_NAME('hello'), LOAD_CONST(1),
                                    LOAD_CONST(2), CALL_FUNCTION(2)]

Now let’s implement eval.

def eval(code, env):
    # ...
    while pc < len(code):
        # ...
        elif op == Opcode.CALL_FUNCTION:
            nargs = ins.arg
            args = [stack.pop(-1) for i in range(nargs)][::-1]
            fn = stack.pop(-1)
            assert callable(fn)
            stack.append(fn(args))
        else:
            raise NotImplementedError(ins)
    # ...

Notice that we’re reading the arguments off the stack — in reverse order — and then reading the function off the stack. Everything is read the opposite way it is pushed onto the stack, since, you know, it’s a stack.

We also check that the fn is callable. This is a Python-ism. Since we’re allowing raw Python objects on the stack, we have to make sure that we’re actually about to call a Python function. Then we’ll call that function with a list of arguments and push its result on the stack.

Here’s what this looks like in real life, in a test:

env = Env({'+': lambda args: args[0] + args[1]}, None)
assert eval(compile(['+', 1, 2]), env) == 3

This is pretty neat. If we stuff lambda expressions into environments, we get these super easy built-in functions. But that’s not quite the most optimal + function. Let’s make a variadic one for fun.

env = Env({'+': sum}, None)
assert eval(compile(['+', 1, 2, 3, 4, 5]), env) == 15
assert eval(compile(['+']), env) == 0

Since we pass the arguments a list, we can do all sorts of whack stuff like this!

Since I only alluded to it earlier, let’s add a test for compiling nested expressions in STORE_NAME.

env = Env({'+': sum}, None)
eval(compile(['val', 'x', ['+', 1, 2, 3]]), env)
assert env.table['x'] == 6

Go ahead and add all the builtin functions that your heart desires… like print!

env = Env({'print': print}, None)
eval(compile(['print', 1, 2, 3]), env)

You should see the arguments passed in the correct order. Note that if you are using Python 2, you should wrap the print in a lambda, since print is not a function.

Conditionals

Without conditionals, we can’t really do much with our language. While we could choose to implement conditionals eagerly as built-in functions, we’re going to do “normal” conditionals. Conditionals that lazily evaluate their branches. This can’t be done with our current execution model because all arguments are evaluated before being passed to built-in functions.

We’re going to do conditionals the traditional way:

(if a b c)

will compile to

[a BYTECODE]
RELATIVE_JUMP_IF_TRUE b
[c BYTECODE]
RELATIVE_JUMP end
b:
[b BYTECODE]
end:

We’re also going to take the CPython approach in generating relative jumps instead of absolute jumps. This way we don’t need a separate target resolution step.

To accomplish this, we’ll add two the opcodes and instructions listed above:

RELATIVE_JUMP_IF_TRUE = Instruction(Opcode.RELATIVE_JUMP_IF_TRUE, "off")
RELATIVE_JUMP = Instruction(Opcode.RELATIVE_JUMP, "off")

Each of them takes an offset in instructions of how far to jump. This could be positive or negative — for loops, perhaps — but right now we will only generate positive offsets.

We’ll add one new case in the compiler:

def compile(exp):
    # ...
    elif isinstance(exp, list):
        # ...
        elif exp[0] == 'if':
            assert len(exp) == 4
            _, cond, iftrue, iffalse = exp
            iftrue_code = compile(iftrue)
            iffalse_code = compile(iffalse) + [RELATIVE_JUMP(len(iftrue_code))]
            return (compile(cond)
                    + [RELATIVE_JUMP_IF_TRUE(len(iffalse_code))]
                    + iffalse_code
                    + iftrue_code)
        # ...

First compile the condition. Then, compile the branch that will execute if the condition passes. The if-false branch is a little bit tricky because I am also including the jump-to-end in there. This is so that the offset calculation for the jump to the if-true branch is correct (I need not add +1).

Let’s add some tests to check our work:

assert compile(['if', 1, 2, 3]) == [LOAD_CONST(1), RELATIVE_JUMP_IF_TRUE(2),
                                    LOAD_CONST(3), RELATIVE_JUMP(1),
                                    LOAD_CONST(2)]
assert compile(['if', 1, ['+', 1, 2], ['+', 3, 4]]) == \
        [LOAD_CONST(1),
         RELATIVE_JUMP_IF_TRUE(5),
         LOAD_NAME('+'), LOAD_CONST(3), LOAD_CONST(4), CALL_FUNCTION(2),
         RELATIVE_JUMP(4),
         LOAD_NAME('+'), LOAD_CONST(1), LOAD_CONST(2), CALL_FUNCTION(2)]

I added the second test to double-check that nested expressions work correctly. Looks like they do. On to eval!

This part should be pretty simple — adjust the pc, sometimes conditionally.

def eval(code, env):
    # ...
    while pc < len(code):
        # ...
        elif op == Opcode.RELATIVE_JUMP_IF_TRUE:
            cond = stack.pop(-1)
            if cond:
                pc += ins.arg  # pc has already been incremented
        elif op == Opcode.RELATIVE_JUMP:
            pc += ins.arg  # pc has already been incremented
        # ...
    # ...

If it takes a second to convince yourself that this is not off-by-one, that makes sense. Took me a little bit too. And hey, if convincing yourself isn’t good enough, here are some tests.

assert eval(compile(['if', 'true', 2, 3]), Env({'true': True})) == 2
assert eval(compile(['if', 'false', 2, 3]), Env({'false': False})) == 3
assert eval(compile(['if', 1, ['+', 1, 2], ['+', 3, 4]]), Env({'+': sum})) == 3

Defining your own functions

User-defined functions are absolutely key to having a usable programming language. Let’s let our users do that. Again, we’re using Dmitry’s Function representation, which is wonderfully simple.

class Function(object):
    def __init__(self, params, body, env):
        self.params = params
        self.body = body
        self.env = env # closure!

The params will be a tuple of names, the body a tuple of instructions, and the env an Env.

In our language, all functions will be closures. They can reference variables defined in the scope where the function is defined (and above). We’ll use the following forms:

((lambda (x) (+ x 1)) 5)
; or
(define inc (x) (+ x 1))
(inc 5)

In fact, we’re going to use a syntax transformation to re-write defines in terms of val and lambda. This isn’t required, but it’s kind of neat.

For this whole thing to work, we’re going to need a new opcode: MAKE_FUNCTION. This will convert some objects stored on the stack into a Function object.

MAKE_FUNCTION = Instruction(Opcode.MAKE_FUNCTION, "nargs")

This takes an integer, the number of arguments that the function expects. Right now we only allow positional, non-optional arguments. If we wanted to have additional calling conventions, we’d have to add them later.

Let’s take a look at compile.

def compile(exp):
    # ...
    elif isinstance(exp, list):
        assert len(exp) > 0
        # ...
        elif exp[0] == 'lambda':
            assert len(exp) == 3
            (_, params, body) = exp
            return [LOAD_CONST(tuple(params)),
                    LOAD_CONST(tuple(compile(body))),
                    MAKE_FUNCTION(len(params))]
        elif exp[0] == 'define':
            assert len(exp) == 4
            (_, name, params, body) = exp
            return compile(['lambda', params, body]) + [STORE_NAME(name)]
    # ...

For lambda, it’s pretty straightforward. Push the params, push the body code, make a function.

define is a little sneaker. It acts as a macro and rewrites the AST before compiling it. If we wanted to be more professional, we could make a macro system so that the standard library could define define and if… but that’s too much for right now. But still. It’s pretty neat.

Before we move on to eval, let’s quickly check our work.

assert compile(['lambda', ['x'], ['+', 'x', 1]]) == \
    [LOAD_CONST(('x',)),
     LOAD_CONST((
         LOAD_NAME('+'),
         LOAD_NAME('x'),
         LOAD_CONST(1),
         CALL_FUNCTION(2))),
     MAKE_FUNCTION(1)]
assert compile(['define', 'f', ['x'], ['+', 'x', 1]]) == \
    [LOAD_CONST(('x',)),
     LOAD_CONST((
         LOAD_NAME('+'),
         LOAD_NAME('x'),
         LOAD_CONST(1),
         CALL_FUNCTION(2))),
     MAKE_FUNCTION(1),
     STORE_NAME('f')]

Alright alright alright. Let’s get these functions created. We need to handle the MAKE_FUNCTION opcode in eval.

def eval(code, env):
    # ...
    while pc < len(code):
        # ...
        elif op == Opcode.MAKE_FUNCTION:
            nargs = ins.arg
            body_code = stack.pop(-1)
            params = stack.pop(-1)
            assert len(params) == nargs
            stack.append(Function(params, body_code, env))
        # ...
    # ...

As with calling functions, we read everything in the reverse of the order that we pushed it. First the body, then the params — checking that they’re the right length — then push the new Function object.

But Functions aren’t particularly useful without being callable. There are two strategies for calling functions, one slightly slicker than the other. You can choose which you like.

The first strategy, the simple one, is to add another case in CALL_FUNCTION that handles Function objects. This is what most people do in most programming languages. It looks like this:

def eval(code, env):
    # ...
    while pc < len(code):
        # ...
        elif op == Opcode.CALL_FUNCTION:
            nargs = ins.arg
            args = [stack.pop(-1) for i in range(nargs)][::-1]
            fn = stack.pop(-1)
            if callable(fn):
                stack.append(fn(args))
            elif isinstance(fn, Function):
                actuals_record = dict(zip(fn.params, args))
                body_env = Env(actuals_record, fn.env)
                stack.append(eval(fn.body, body_env))
            else:
                raise RuntimeError("Cannot call {}".format(fn))
        # ...
    # ...

Notice that the function environment consists solely of the given arguments and its parent is the stored environment — not the current one.

The other approach is more Pythonic, I think. It turns Function into a callable object, putting the custom setup code into Function itself. If you opt to do this, leave CALL_FUNCTION alone and modify Function this way:

class Function(object):
    def __init__(self, params, body, env):
        self.params = params
        self.body = body
        self.env = env # closure!

    def __call__(self, actuals):
        actuals_record = dict(zip(self.params, actuals))
        body_env = Env(actuals_record, self.env)
        return eval(self.body, body_env)

Then eval should call the Function as if it were a normal Python function. Cool… or gross, depending on what programming languages you are used to working with.

They should both work as follows:

env = Env({'+': sum}, None)
assert eval(compile([['lambda', 'x', ['+', 'x', 1]], 5]), env) == 6

eval(compile(['define', 'f', ['x'], ['+', 'x', 1]]), env)
assert isinstance(env.table['f'], Function)

Hell, even recursion works! Let’s write ourselves a little factorial function.

import operator
env = Env({
    '*': lambda args: args[0] * args[1],
    '-': lambda args: args[0] - args[1],
    'eq': lambda args: operator.eq(*args),
}, None)
eval(compile(['define', 'factorial', ['x'],
                ['if',
                ['eq', 'x', 0],
                1,
                ['*', 'x', ['factorial', ['-', 'x', 1]]]]]), env)
assert eval(compile(['factorial', 5]), env) == 120

Which works, but it feels like we’re lacking the ability to sequence operations… because we are! So let’s add that.

One teensy little thing is missing

It should suffice to write a function compile_program that can take a list of expressions, compile them, and join them. This alone is not enough, though. We should expose that to the user so that they can sequence operations when they need to. So let’s also add a begin keyword.

def compile_program(prog):
    return [instr for exp in prog for instr in compile(exp)]  # flatten

And then a case in compile:

def compile(exp):
    # ...
    elif isinstance(exp, list):
        # ...
        elif exp[0] == 'begin':
            return compile_program(exp[1:])
        else:
            args = exp[1:]
            nargs = len(args)
            arg_code = sum([compile(arg) for arg in args], [])
            return compile(exp[0]) + arg_code + [CALL_FUNCTION(nargs)]
    raise NotImplementedError(exp)

And of course a test:

import operator
env = Env({
    '*': lambda args: args[0] * args[1],
    '-': lambda args: args[0] - args[1],
    'eq': lambda args: operator.eq(*args),
}, None)
assert eval(compile(['begin',
    ['define', 'factorial', ['x'],
        ['if',
        ['eq', 'x', 0],
        1,
        ['*', 'x', ['factorial', ['-', 'x', 1]]]]],
    ['factorial', 5]
]), env) == 120

And you’re done!

You’re off to the races. You’ve just written a bytecode interpreter in Python or whatever language you are using to follow along. There are many ways to extend and improve it. Maybe those will be the subject of a future post. Here are a few I can think of:

  • Make a JIT compiler — generate native code instead of bytecode
  • Re-write this in a language whose runtime is faster than CPython
  • Choose a file format and write the compiler and interpreter so that they eject to and read from the disk
  • Add a foreign-function interface so that the user can call functions from the host language
  • Expose compile and eval in the environment to add metaprogramming to your language
  • Add macros
  • Learn from CPython and store local variables in the frame itself instead of in a dict — this is much faster (see LOAD_FAST and STORE_FAST)
  • Target an existing VM’s bytecode so you can compile to, say, actual CPython or Lua or something
  • Similar to above: target OPy so you can run compiled programs with Andy Chu’s VM
  • Implement this is a statically-typed language like OCaml, SML, Haskell, Rust, Swift, etc

Noon van der Silk (silky)

Van der Silk Fashion at Linux Conf 2019 - New Zealand January 04, 2019 12:00 AM

Posted on January 4, 2019 by Noon van der Silk

Excitingly, this year I’m going to Linux Conference, and giving a talk at the Art+Tech MiniConf!

To celebrate, I redesigned the Van der Silk website:

Also, very coincidentally, PAOM re-designed their website for the new year, so it’s much easier to find all my designs: noonvandersilk on PAOM.

Hope to see you there!

Pepijn de Vos (pepijndevos)

Bucket Brigade Delay January 04, 2019 12:00 AM

So far in my quest to build a synthesizer, I’ve controlled a Game Boy synth with a linear potentionmeter and made a pressure sensing circuit for the SoftPot. Today I want to make an analog delay using a bucket brigade device.

To do this, I used a BL3207 bucket brigade device, and a BL3102 clock generator, which are clones of the MN3207 and MN3102, respectively. The datasheets for the former are a single page and not of much use, but since they are clones, the original datasheets provide more useful information. Most importantly, this reference circuit.

MN3207 reference schematic

I did not build that exact circuit, but kind of used it as a hookup guide for the two chips. The MN3102 basically uses the RC filter to generate two opposing clock signals. It also generates a special voltage that’s 15/16 of Vdd for some reason. The MN3207 then takes an analog input, and with some pull-down resistors, produces a delayed output signal. In the above circuit there is a lot of filtering going on, while my circuit uses a simple first order RC filter for now. Both the delayed output and the input signal are fed into an opamp adder to make a nice feedback loop. In the above video I’m simply turning the feedback gain.

ball of wires

To be honest, the current circuit sounds quite bad and is not very flexible. It just adds a fixed delay and loops it back on itself, going from inaudible to metallic to oscillating in the blink of an eye. The best sound I got out of it is the click track when it’s almost oscillating and becomes this funny tone by itself. Maybe all of the filtering in the reference circuit make it a lot better, but I have some other ideas.

I have like 5 of these delay chips, so it’d be fun to chain them together for a longer delay. The other thing is the clock generator: You can disconnect the RC and drive the oscillator externally. I’m thinking I could create an external clock, and modulate that with an LFO to create a vibrato effect.

Update: Vibrato works like a charm. I simply drove the clock with an Arduino as follows. Then I played a sine wave and some acoustic guitar into it for demonstration.

void setup() {
}

void loop() {
  int i;
  for(i=0; i<2000; i+=30) {
    tone(2, 40000+i);
    delay(1);
  }
  for(i=2000; i>0; i-=30) {
    tone(2, 40000+i);
    delay(1);
  }
}

January 03, 2019

Jan van den Berg (j11g)

To the Lighthouse – Virginia Woolf January 03, 2019 08:47 PM

To the Lighthouse – Virginia Woolf (1927) – 254 pages

This was an exhausting book to read. Because it reads like a prolonged poem with a stream of thoughts about everything and anything and with very little dialogue. Two things are very clear though: Woolf went very deep and therefore this book can not be anything else than autobiographical. Feeling already exhausted as a reader, I couldn’t even imagine what the process must have been for a writer to put this on paper.

It’s not my favorite kind of fiction but I am still glad I read it. It’s remarkable how Woolf is able to capture moments of human thought and emotion in an identifiable way: tempestuous and full of unfulfilled intent. And where love and hate are often not as indistinguishable as we think.

The post To the Lighthouse – Virginia Woolf appeared first on Jan van den Berg.

Noon van der Silk (silky)

2018 Books January 03, 2019 12:00 AM

Posted on January 3, 2019 by Noon van der Silk
div.book { display: flex; flex-direction: row; padding: 10px; } div.book img { margin-right: 10px; } div.alt { background: #eaeaea; }

I’ve seen a few people post the books that they’ve read over the last year; so I thought I’d also get in on the action. Here are the books that I read (or partially read), in no particular order, last year:

Non-Fiction

The Oregon Experiment by Christopher W. Alexander, Murray Silverstein, Shlomo Angel, Sara Ishikawa, Denny Abrams

Anyone that knows me, knows that Christopher Alexander is basically my personal Jesus. This book is amazing, to me in it’s demonstration of Alexander’s (and collaborators) standard approach to architecture; i.e building via small local activities in an endless cycle according to a specific and developing pattern language. It also contains very important insights, in my mind, in terms of project management and budgeting.

Educated by Tara Westover

I learned about this one from Bill Gates. It’s amazing. I found it really fascinating to read about Tara’s life, and how she understood her upbringing. It makes you think quite a lot about the things you tell yourself to make sense of the situations you’re in.

Dark Emu by Bruce Pascoe

Soon to be (if not already) an Australian classic; it’s very interesting to learn an entirely different history about our country and the Aboriginal people. I found it to be reasonably hard reading because there are some solid descriptions of various items and processes that it would’ve been amazing to have pictures of. In any case, necessary reading.

Radical Candor by Kim Malone Scott

I basically hated this book. I think, on the one hand, it does have useful information in it; but for me it came across too much like standard Silicon Valley propoganda. I do think there is good stuff in here, for leaders that struggle with feedback.

Cities for People by Jan Gehl

Recommended to me by Ryan, who was working as a Barista at from Streat, this book is, apparently, a classic of urban planning. It’s amazing. It has a incredibly readable style; with lots of pictures and summaries of each paragraph. Jan has had a lot of impact in designing Melbourne, so it’s really interesting to read this book and see and understand the impact he and his team have had.

Factfulness by Hans Rosling, Ola Rosling, Anna Rosling Rönnlund

Quite good. From this book I learned about the website Dollar Street, which is incredible and well worth a visit. Hans has many questions in the book; almost all of which I answered wrong, much like most of his audience that he’s asked the questions to. I learned a lot about how to think about world-wide progress, and that there are interesting organisations out there doing interesting work. I also learned, much to my surprise, that the World Bank isn’t a bank in the sense that I would’ve assumed; it’s main goal is to end poverty!

Enlightenment Now by Steven Pinker

Not bad. This is the first book of Steven Pinker I’ve read. I’m unlikely to read more, after learning about some sexist views he’s had from Delusions of Gender (not a book I’ve read yet; but heard thoughts from my partner).

I found this book to be reasonably good, and make some solid arguments for continuing to fight against people that don’t think progress is real. Unfortunately, to paraphrase a joke from the Simposns, Steven seems like the kind of person that can use graphs to prove anything that is even remotely true. It contains too many anecdotes and cherry-picked examples.

I’d recommend Factfulness over this one.

The Hidden Life of Trees by Peter Wohlleben

Really great. I’ve always been a bit of a hippie, and am mostly vegetarian, so when my partner found out I’d be reading this one, she started to get a bit concerned that I’d soon not even eat greens! Now, I haven’t quite gone that far, but I do really love the ideas and science in this book; and I think it’s really interesting to think of trees and their root-systems as being some giant entity. The one downside to this book is that it occasionally lacks pictures of the concepts that are being described.

Surfaces and Essences by Douglas R. Hofstadter, Emmanuel Sander

Amazing. Probably not for people feeling a bit impatient. The book goes into many examples of the ideas; and makes a strong argument for the idea that all thinking is based on analogies. I’ve always loved analogies, so I’m naturally a big fan.

As someone presently working in AI, I found this book full of many incredibly interesting and relevant ideas; one of which we discussed previously.

If you’re at all interested in when computers might be able to get to think like humans, definitely have a read!

Why Buddhism is True by Robert Wright

Amazing. Recommended to me by my friend Julia. Probably one of the best books I’ve ever read, really; right up there with Christopher Alexander’s “Nature of Order” and “The Timeless Way of Building”. This book is particularly neat, of course, because it takes a scientific approach to making sense of buddhist ideas. Overall, I find lots of agreement between this book, and the ideas of Christopher Alexander, and Surfaces and Essences.

Inferior by Angela Saini

Pretty interesting. In this book Angela debunks many myths about the supposed biological differences between men and women. This is the first book of this kind that I’ve read; it won’t be the last.

I read this because my partners dad had it in the house. It’s pretty good; it’s really interesting to see the thoughts from someone from so far back in history. For me, this book resonated with the ideas I read in “Why Buddhism is True”. Worth reading if you happen upon it.

Siddartha by Hermann Hesse

I was not a fan of this. Recommended to me by my partner; I found it to be way too “on the nose” about enlightenment. I felt like I was being dragged around to understand certain things, in an obliquely mystical way. Maybe it’s because I read it after I read Why Buddhism is True; which is much more direct and scientific.

The Last Lecture by Randy Pausch

Quite good; but quite “American”. If you don’t like too much american-style dreams of success and culture, you might not like this. There’s also the video of the talk that inspired it that is well worth watching.

I watched the video way back in the day; and thought I’d finally read the book. I wasn’t disappointed, but I’m not sure I got much new out of the book that I didn’t get out of the talk. Still, worth grabbing if you got something out of the talk.

The Big Picture by Sean Carroll

Not great. Before I’d read this, I really had thought I liked Sean Carroll. But, this book comes across to me as very arrogant. As someone who’s read way too many popular physics books, it contained a fair amount that wasn’t new to me (but that’s okay), but also a fair amount of just plain arrogance about what is known, where we can expect progress and what areas are accessible to science.

Lost Connections by Johann Hari

I really enjoyed this. I’ve also suffered (mild, I think) depression, and can really relate to the ideas in this book; namely making connections with people. Probably not the solution for everyone, but interesting reading.

Made by Humans by Ellen Broad

Not bad. Fairly introductory thoughts on the topic, but from someone working in policy on this area. Nice to see an Australian view on these issues. Worth reading it you want a broad overview.

Reality is Not What it Seems by Carlo Rovelli

This is okay. I get very frustrated these days by books that are implicitly sexist; using language like “when man discovered this”, etc, and this book does that a fair bit. I think I was hoping for this to answer more questions than it did; or to be a bit more detailed. I think I probably just misjudged the intended audience.

Your Brain Is a Time Machine by Dean Buonomano

Pretty good. I haven’t read much about neuroscience, and I am really mostly confused about the physics of time, and problems like the so-called “arrow of time”, and such. So I found this to be pretty interesting, and have some good ideas in it that I hadn’t thought about, such as different “clocks” in the body for different purposes.

I Feel You by Cris Beam

Interesting, but odd. I found this book to be worthwhile reading; in particular because it lead me to some interesting ideas around the ethics of AI. Cris goes into weird details of empathy in use in businesses, and life. Overall, worth taking a look at.

How to Change Your Mind by Michael Pollan

Great. Never having taken psychadelics myself, and basically having the view that they are a bit dangerous, this book totally changed my mind. I found it totally fascinating, alongside Why Buddhism is True, in terms of a potential path towards enlightenment. It’s certainly interesting to learn about the history of psychadelic drugs, and think about the connection to mindful meditation.

Incredible. As we learned at the start, Christopher Alexander is my personal Jesus and this set of books is essentially his bible. As a result, this is amazing reading for me. He describes what it means for something to “have life”; he goes into detail by way of examples, a mathematical model, and a practical, hands-on theory. He breaks it down into what he refers to as “centers”; things that build on each other.

The Value of Everything by Mariana Mazzucato

Amazing. I’m only part of the way through this book, but I love it. We saw Mariana talk at some event a few weeks ago; and she’s an amazing speaker with a incredibly ability to talk with enthusiasm about complicated ideas.

This book is all about “value” vs “price”; and what industries should be considered “productive” in how they contributed to the economoy. In particular it’s really opened my eyes about the finance industry. Amazing reading.

How To Sit, How To Walk by Thich Nhat Hanh

Amazing, and useful. I’ve found these two (small) books really good to read when I’m feeling stressed or overwhelmed. They contain many little thoughts that help ground and relax you.

Fiction

The Power by Naomi Alderman

Pretty good. Best read without too much knowledge of what it’s about. Worth reading.

The Martian by Andy Weir

Awesome. Instant classic. Great fun.

All the Birds in the Sky by Charlie Jane Anders

Pretty fun. A nice and simple read. I enjoyed it.

The Broken Earth Series: The Fifth Season, The Obselisk Gate, The Stone Sky, by N. K. Jemisin

Incredible. The first series of books I read by N. K. Jemisin. I couldn’t put these down, and as soon as I’d finished the first one I bought the remaining two and couldn’t wait to read them.

The Inheritance Trilogy by N. K. Jemisin

Really great. N. K. Jemisin is easily my favourite Sci-Fi author at the moment and I’ll probably read anything she writes. So interesting, unique, and innovative.

The Dreamblood Duology by N. K. Jemisin

Really great. Quite different, really, from The Inheritance Trilogy, but still amazing.

Less by Andrew Sean Greer

Not bad. I picked it up off the table in the bookshop because it had won the Pulitzer Prize, and I figured it might be good. I found it to be pretty good; but utlimately not quite my style.

The Stormlight Archive The Way of Kings, Words of Radiance, Oathbringer, by Brandon Sanderson

Awesome. I discovered Brandon after looking for something new after finishing The Wheel of Time. I’m really enjoying this series, and can’t wait for the next one!

Hope you enjoyed this list! If you have any suggestions of books, I’d be really interested!

Pepijn de Vos (pepijndevos)

Sensing pressure with a SoftPot January 03, 2019 12:00 AM

In my previous post I connected a SoftPot to an Arduino and used a Game Boy as the synth. In this post I’m focusing more on the SoftPot, and specifically on reading pressure.

The SoftPot consists of a resistive bottom layer and a conductive top layer. When pressed together, the bottom resistor forms a voltage divider with the top conductor. When the conductive top side is pressed down, it shorts part of the bottom track, creating a lower total resistance. The harder you press down, the larger the area of the bottom track that is shorted, the lower the resistance.

So to measure both pressure and position, we’ll need to simultaneously read the voltage on the top track, and the resistance of the bottom track. To get an accurate position, we’d like to have the full voltage swing available, without any changes in voltage due to pressure. Meanwhile, we’d like to sense small DC variations in resistance of a large total resistance. This is my best attempt so far.

circuit

It uses a current mirror to maintain an almost fixed voltage on the pot (R1) and only lose 0.6V of the full swing. The current mirror then feeds into a biased transimpedance amplifier. The rightmost voltage divider is used to set the DC output voltage. (I later added a capacitor in parallel to the TIA feedback resistor to reject noise)

In reality the voltage divider on the positive TIA input is a trim pot. The changes in resistance of the SoftPot are smaller than the process variations. For example, this particular pot changes from 20.7k to 20.3k when pressed, so the bias voltage has to be adjusted for this.

sensing pressuer and pitch

In the above figure, the blue line is me sliding my finger up and down, while the yellow line is the pressure of my finger, with the green/red lines being the max/min. This seems to work just fine, except after letting it run for a while the pressure value starts to drift. Remember, we need the constant, absolute value, not some high-pass filtered version.

I was really confused, and tried various things to reproduce the problem and improve the situation. Then I blew on the circuit.

temperature sensitivity

OMG, look at that! The current mirror goes completely crazy when it gets warm. One solution is to add more degeneration, which adds voltage drop. Another is to use a Wilson current mirror, which also adds voltage drop. And finally, you could get two integrated transistors on the same die, which makes them the same temperature and improves matching. I guess that’s what I’ll have to do. To be continued.

const int pitchPin = A0;
const int pressurePin = A1;
const int pwmPin = 12;

const int pressureThreshold = 20;

int pitchValue = 0;        // value from SoftPot
int pressureValue = 0;        // value from TIA
int minPressureValue = INT16_MAX;
int maxPressureValue = 0;
int pressure = 0;
int frequency = 0;
int polarity = 1;

void setup() {
  Serial.begin(9600);
}

void loop() {
  delay(1);
  pitchValue = analogRead(pitchPin);
  delay(1);
  pressureValue = analogRead(pressurePin);
  delay(1);
  minPressureValue = min(pressureValue, minPressureValue);
  maxPressureValue = max(pressureValue, maxPressureValue);
  pressure = map(pressureValue, minPressureValue, maxPressureValue, 0, 10);
  frequency = map(pitchValue, 0, 1023, 440, 1320);
  if (pressureValue < minPressureValue+pressureThreshold) {
    pitchValue = 0;
    noTone(pwmPin);
  } else if (pressure > 6) {
    tone(pwmPin, frequency+pressure*polarity);
    polarity *= -1;
  } else {
    tone(pwmPin, frequency);
  }
  Serial.print(pitchValue);
  Serial.print(", ");
  Serial.print(minPressureValue);
  Serial.print(", ");
  Serial.print(maxPressureValue);
  Serial.print(", ");
  Serial.println(pressureValue);
  delay(50);
}

January 02, 2019

Indrek Lasn (indreklasn)

Here are some of the best resources to improve your coding skills January 02, 2019 12:49 PM

Photo by Ash Edmonds on Unsplash

There are lots of high-quality coding tutorials, books, and overall resources on the web. I’ve gathered the very best for you and put them in a convenient list. Without further ado, let’s dive in!

Devhints

Devhints has the biggest collection of cheat-sheets on the Internet. Big time saver and a great way to familiarize yourself with all the concepts.

Udemy

Udemy is by far the most popular place to learn to code in my opinion. Dropping $10 to learn new skills which yield huge returns is a no-brainer.

Egghead

Egghead has many courses by industry leaders. Definitely a top-notch bump that’ll take your coding skills to a new level. Courses are both free and paid.

Exercism

Exercism has tons of exercises in most programming languages. Understanding how to approach and solve critical problems is a very valuable trait to have in programming.

freeCodeCamp

freeCodeCamp.org is a tiny 501(c)(3) non-profit that’s helping millions of people learn to code for free. Their curriculum will prepare you to become a great coder. freeCodeCamp teaches both the necessary fundamentals and also makes you code real projects.

Mozilla Developer Network (MDN) Web Docs

If you are a complete beginner, web development can be challenging. The Mozilla Developer Network offers a hold-your-hand approach and provides enough detail for you to feel comfortable and learn the topics properly.

You should feel at home whether you are a student learning web development (on your own or as part of a class), a teacher looking for class materials, a hobbyist, or someone who just wants to understand more about how web technologies work.

Reactdom

Reactdom provides a platform allowing you to learn from the best programming tutorials & books.

Khan Academy

Khan Academy does what it states in its mission. To provide a free, world-class education to anyone, anywhere. Khan Academy is a 501(c)(3) nonprofit organization and offers various web development tutorials.

Eloquent JavaScript

This is a book about JavaScript, programming, and the wonders of the digital. You can read it online for free here.

This should be a great start to your coding journey. Now go out there and start coding!

Here are some project ideas you could build to improve your coding skills;

If you have any questions, I’m actively answering questions on Twitter.

Indrek Lasn (@lasnindrek) | Twitter

Here are some of my previous articles.


Here are some of the best resources to improve your coding skills was originally published in freeCodeCamp.org on Medium, where people are continuing the conversation by highlighting and responding to this story.

January 01, 2019

Nikita Voloboev (nikivi)

Activating things instantly by pressing two keys will be faster than activating Alfred and typing… January 01, 2019 11:48 PM

Activating things instantly by pressing two keys will be faster than activating Alfred and typing something.

December 31, 2018

Átila on Code (atilaneves)

Comparing Pythagorean triples in C++, D, and Rust December 31, 2018 01:16 PM

EDIT: As pointed out in the Rust reddit thread, the Rust version can be modified to run faster due to a suble difference between ..=z and ..(z+1). I’ve updated the measurements with rustc0 being the former and rustc1 being the latter. I’ve also had to change some of the conclusions. You may have recently encountered […]

December 30, 2018

Nikita Voloboev (nikivi)

Karabiner God Mode December 30, 2018 03:29 PM

Karabiner is a low level macOS kernel extension that gives you a virtual keyboard you can heavily customize to your own liking.

I found out about the power of Karabiner by reading an old article on how you could remap caps lock key into a key that’s actually useful. That is you could remap caps lock to act as escape key when pressed alone and some four/five key modifier when held together. With that article, the notion of a hyper (dual purpose) key was born but most people seemed to have stopped there. This article is about how to take this idea of a hyper key and apply it to every key on your keyboard using Karabiner and a special DSL named Goku that makes editing your Karabiner configuration a breeze.

Karabiner Elements uses JSON to define rules which lets you program keys in various ways. The most powerful of which is the ability to create custom hyper keys. Like the useful caps lock I mentioned earlier but applied to every other key on the keyboard. It is not enough to talk about this and praise how game changing this concept is in using a computer until you try it though. So let’s do that.

The problem with using JSON to define Karabiner rules is that it gets unwieldy complex and nearly impossible to edit by hand. Here is a gist of what my Karabiner configuration looks like in JSON format. It’s a 22,692 lines long file. Inside it there are definitions for how to make all of my QWERTY keyboard (minus some letters and delete/return keys because those are too far from the hjkl hand position) into custom hyper keys. There is no way a sane person would open this file and try to modify it in any meaningful way. It contains too much ‘boilerplate’ that Karabiner actually needs to make its magic. When as all you really need is to just point to a key that you want to make a hyper key, then point to a key that when inside that hyper key would do some custom action (run applescript/shell-script/open-app/..).

Fortunately this problem was solved by creating a special DSL that has the minimum amount of syntax needed to define these kind of hyper key definitions and more. Goku tool actually supports all of Karabiner Elements features but abstracts away the JSON mess so you don’t have to deal with it. Here is my 600 line Karabiner configuration written with Goku. 600 line file that generates 20,000+ lines of JSON that not only is a joy to edit but has the nice feature of auto reloading config. Meaning that as soon as I press save on that Goku config file, I have my keyboard modifications applied instantly.

Defining our first hyper key

If this sparked your interest and you are reading this article on a mac. Let’s try this magic in action and you can see for yourself if that’s something you may fancy doing to your computer.

Let’s install all the stuff we need first.

  1. Install Karabiner Elements (Press the big blue button to install latest version)
  2. Install Goku (Karabiner DSL) (brew install yqrashawn/goku/goku)
  3. Run goku service to watch over Goku file and instantly apply changes (brew services start goku)
  4. Open Karabiner Elements (with Alfred/Spotlight) and create a Goku profile

More instructions on how Goku works are in the README alongside its short tutorial that goes over all its features. There is also a Telegram chat where you can ask for help in any Karabiner modification you like to make.

Now that we have everything we need installed and running on our system, we can get to work.

Goku’s config file by default lives inside ~/.config/karabiner.edn. Since I like to keep all my useful config files in my dotfiles dir, I only keep a symlink there and original Goku config is placed inside ~/.dotfiles/karabiner/karabiner.edn.

Make caps lock great again

I have my caps lock mapped to escape key when pressed alone (VIM is life) and ⌥ ⌃ when held. This one is actually quite easy, especially since macOS already provides you with ability to map caps lock to escape key in System Preferences.

Of course we could of done that in Goku configuration itself too but since macOS gives us ability to do it in settings, let’s use that.

Assuming you’ve done that and everything worked, you can now press caps lock key and instead of going into caps lock mode, you will trigger escape instead. Pretty useful given that new macbooks don’t even ship with a physical escape key (although it was too far away anyway and TouchBar is genuinely useful). But holding onto the caps lock key now does nothing useful. Let’s change that.

Open ~/.config/karabiner.edn file in your favorite editor (create it if doesn’t exist). I edit my config with neovim. Then Paste this inside and save the file:

{:simlayers {:caps_lock-mode {:key :caps_lock}},
:main [{:des "capskey",
:rules [:caps_lock-mode
[:open_bracket :!TOopen_bracket]
[:close_bracket :!TOclose_bracket]
[:semicolon :!TOsemicolon]
[:quote :!TOquote]
[:comma :!TOcomma]
[:period :!TOperiod]
[:slash :!TOslash]
[:a :!TOa]
[:b :!TOb]
[:c :!TOc]
[:d :!TOd]
[:e :!TOe]
[:f :!TOf]
[:g :!TOg]
[:h :!TOh]
[:i :!TOi]
[:j :!TOj]
[:k :!TOk]
[:l :!TOl]
[:m :!TOm]
[:n :!TOn]
[:o :!TOo]
[:p :!TOp]
[:q :!TOq]
[:r :!TOr]
[:s :!TOs]
[:t :!TOt]
[:u :!TOu]
[:v :!TOv]
[:w :!TOw]
[:x :!TOx]
[:y :!TOy]
[:z :!TOz]
[:1 :!TO1]
[:2 :!TO2]
[:3 :!TO3]
[:4 :!TO4]
[:5 :!TO5]
[:6 :!TO6]
[:7 :!TO7]
[:8 :!TO8]
[:9 :!TO9]
[:0 :!TO0]]}]}

Goku is still a tool that is in beta so unfortunately the only way to check if your Goku configuration worked without any errors is to type in goku in the terminal. When I do that, I get:

~
❯ goku
Done!

Which means success. If there is a mistake anywhere in the configuration, you will get a nasty stack trace which usually means you forgot to close out some bracket.

If you did get Done! like me printed, congratulations! You now have yourself your first hyper key. Now if you hold caps lock and press another key, you will trigger ⌥ ⌃ + another-key. ⌃ ⌥ + keys are by default empty keys so you have to map them to something. Let’s map one of these keys to one of my favorite mappings on macOS.

Go to System Preferences -> Keyboard -> Shortcuts -> Services -> Find Search With Google action -> Add shortcut -> Press and hold caps lock briefly and with caps lock still held press g key. It should show the same hotkey as I have in screenshot below.

Now select some text in any app (not VS Code because electron) and press caps lock (hold), then press g key. If everything worked, it should make a Google query with the selected text. Pretty damn useful to have this action at your finger tips.

Hope you now got the idea of what these hyper keys are. You have to play a bit with it to get comfortable because if you hold onto your hyper key for long, it will trigger repeating of escape key and not leave you in the hyper key mode. You have to press caps lock key, hold it and instantly (whilst caps lock key is held) press on the activation key (any other key on the keyboard). In the case of our caps lock key, the activation keys will simply trigger our ⌥ ⌃ hotkeys. But macOS and Karabiner is a lot more awesome and powerful than that. Let’s take this power to its limits.

Actual God Mode

To fully appreciate what Karabiner can do for you, you need to buy and use these two very powerful macOS apps. Keyboard Maestro and Alfred.

Alfred is a programmable launcher for macOS. Like Spotlight but better because you can search anything with it and use a wide array of powerful workflows people have built and freely shared for anyone to use. I also wrote an article on how to start writing these workflows in Go.

Keyboard Maestro is an automation tool that lets you create macros and execute them. A macro is a series of actions to do things. It’s like shell scripting for macOS but better because it just makes sense and is intuitive. Here is a simple macro that activates Safari app on macOS.

I ran this macro this many times:

And that’s because I have this macro mapped nicely to w + k keys. So w key is my hyper key and k my activation key. Only instead of some boring hotkey, I now run a KM macro. I have 1,000’s of KM macros I’ve made and use and if you look into my Goku config you will see on 1st line, this code:

{:templates {:alfred "osascript -e 'tell application \"Alfred 3\" to run trigger \"%s\" in workflow \"%s\" with argument \"\"'"
:km "osascript -e 'tell application \"Keyboard Maestro Engine\" to do script \"%s\"'"}

Specifically this line:

:km "osascript -e 'tell application \"Keyboard Maestro Engine\" to do script \"%s\"'"

Creates an alias :km that will call a KM macro by name I pass into it. What it’s doing is actually calling an AppleScript code via osascript shell command (can man osascript if you like). The same thing happens here:

:alfred "osascript -e 'tell application \"Alfred 3\" to run trigger \"%s\" in workflow \"%s\" with argument \"\"'"

Only instead of KM macro, it calls an Alfred Workflow. To call Alfred workflows with AppleScript, you need 2 things. The external trigger name:

And the bundle.id of the workflow in which this trigger lives which can be found in the description of the workflow here:

And here is an example of calling this specific Alfred workflow in Goku:

[:n [:alfred "search lists" "nikivi.learn.anything"]]

Although you can’t just paste this in into Goku config. This just defines what happens when n key is pressed in some hyper key. What hyper key? Well that is defined as a block of code.

For example in my keyboard configuration I activate this Alfred Learn Anything workflow by pressing o key and then pressing n key inside my o key layer. In code that looks like this:

{:des "okey (alfred)"
:rules [:o-mode
[:n [:alfred "search lists" "nikivi.learn.anything"]]]}

Goku syntax is odd, I agree, it’s optimized for typing less (thus iterating on config faster), not necessarily clarity. But that short snippet actually defines a hyper key only now this hyper key lives on the actual keys you type on. The letter o. So I can still type things away (like this article I am writing) but if I ever want to search curated lists with a workflow, I press o key briefly and whilst not releasing it, press n key. And I get an Alfred prompt show up just as I programmed my keyboard to do this:

Of course my actual o key definition doesn’t just have a binding for :n key. In reality it looks like this:

{:des "okey (alfred)"
:rules [:o-mode
[:1 :!C1]
[:2 :!C2]
[:3 :!C3]
[:4 :!C4]
[:5 :!C5]
[:6 :!C6]
[:w :!CObackslash]
[:v :!CT4]
[:spacebar [:alfred "search PDF files" "nikivi.search.files"]]
[:g [:alfred "search snippets" "com.renfei.SnippetsLab.AlfredWorkflow"]]
[:grave_accent_and_tilde [:alfred "search processes" "com.vitorgalvao.alfred.processcontrol"]]
[:e [:km "Make 2Do task"]]
[:r [:km "Make 2Do task with Safari title as name"]]
[:q [:alfred "search downloads" "recentdownloads.ddjfreedom"]]
[:t [:km "2Do with current url as note (read)"]]
[:b [:km "2Do task (ask)"]]
[:tab [:km "2Do with current url as note (check)"]]
[:h [:km "2Do with current url as note (watch)"]]
[:a [:alfred "search files" "nikivi.manage.wiki"]]
[:y [:alfred "search trash" "com.vitorgalvao.alfred.directories"]]
[:c [:alfred "search emoji" "com.github.jsumners.alfred-emoji"]]
[:i [:alfred "search tty sessions" "net.isometry.alfred.tty"]]
[:f [:alfred "search repos" "net.deanishe.alfred-git-repos"]]
[:n [:alfred "search lists" "nikivi.learn.anything"]]
[:m [:alfred "search lists" "nikivi.awesome.lists"]]
[:s [:alfred "search clones" "com.vitorgalvao.alfred.directories"]]
[:d [:alfred "search desktop" "com.vitorgalvao.alfred.directories"]]
[:j [:alfred "search km macros" "iansinnott.keyboardmaestro"]]
[:caps_lock [:alfred "search folders" "nikivi.search.folders"]]
[:z [:alfred "search workflows" "org.jeef.workflowdirectory"]]
[:k [:alfred "search menu bar" "com.tedwise.menubarsearch"]]]}

Now the description okey (alfred) makes more sense. My o key layer is responsible for calling various Alfred workflows. Making layer mappings by theme makes things easier to remember. For example my w key is used to activate my most used applications (VSCode/Safari/iTerm/2Do/..). Here is how it looks in code:

{:des "wkey (apps)"
:rules [:w-mode
[:k [:km "open: Safari"]]
[:h [:km "open: Activity Monitor"]]
[:i [:km "open: Chrome Canary"]]
[:g [:km "open: Bee"]]
[:t [:km "open: Trello"]]
[:spacebar :!CSO7]
[:comma [:km "open: Spotify"]]
[:caps_lock [:km "open: Finder"]]
[:m [:km "open: Tower"]]
[:b [:km "open: BetterTouchTool"]]
[:r [:km "open: Fantastical"]]
[:e :!OSC1] ; Open Dash
[:semicolon [:km "open: Xcode"]]
[:period [:km "open: SnippetsLab"]]
[:f [:km "open: 2do"]]
[:j [:km "open: iTerm"]]
[:n [:km "open: Sublime Text"]]
[:l [:km "open: VS Code"]]
[:o [:km "open: Keyboard Maestro"]]]}

In this layer, instead of Alfred workflows, I am calling KM macros. Like the one I showed above:

I share all these KM macros I made and use here alongside all the Alfred workflows.

Go wild

I’ve been using Karabiner with custom modifier keys for over 3 years now. I am still in awe that we have this ability to modify our computers to this degree. It’s beyond powerful what you can do with it and I am sad that this isn’t known to the public. Aside from mine and creator’s of Goku configuration, there seems to be no one doing this! And that’s a shame. 😞

Hopefully this article changes it. Because I want to live in a world where everyone pushes the boundaries of what computers can do for us.

If you enjoyed reading this, give it a 👏. And look into other articles I wrote. You can support me on Patreon and get updates on next articles I write on Twitter.💜

December 28, 2018

Jan van den Berg (j11g)

Ten notable books I read in 2018 December 28, 2018 08:19 PM

In 2018 I read 24 books (25th in progress). Here is a list of 10 books that stood out in one way or another. Some are recommendations.

Besides music biographies, computer history and management/business books, I tend to read bestseller novels from a few years ago. By then the crowd has decided what’s good and thrift stores will carry a lot of these books — which can be an indication of something worth reading.

Yes, I read on paper. Screen reading is a different experience for me.

The first couple of books on the list are fiction, the rest are non-fiction. Otherwise the list is in no particular order. Spoilers ahead.

The Underground Railroad – Colson Whitehead

This Pulitzer price winning book will get under your skin. Everything you read about it is true. It is disturbing and unsettling in every way. It is a book about slavery, and the structural dehumanization and killing of millions of Africans over hundreds of years. And it will shed light on the completely hopeless situation of entire generations of humans. But it is also about redemption and the human spirit (and sadly not just the good parts).

The after-the-fact writing style Whitehead uses is very effective. For instance he will write a build up to a rape scene, so you know it’s coming, and there’s nothing you can do. But when it does, the first sentence is of what happened after (“the women of the camp stitched her up”). So without actually describing the event he will make your heart sink. He uses this technique quite a few times, to great effect. I could write a lot more about this book, but just know that this book will not leave you and it will change how you look at things.

The book blends realism with allegory. So even though it is a fictional story (there was of course no literal underground railroad), large parts are based on extensive research of real events. Therefore this book greatly upset my understanding of the history of the USA and slavery.

The Kite Runner – Khaled Hosseini

The Kite Runner is another book that will get under your skin. It is a universal story of the friendship of two boys. About their relationship and the unselfishness and sacrifice. What makes this story extra remarkable is how the writer interweaves the setting (the nature, the culture, the city) with the story. He makes them work really well together. In doing so he not only underscores the idea that people, and therefore their choices, are a product of their environment. But also, this writing style will make you feel as if you are running your kite with the two boys through the crowded and narrow streets, leaving a dust trail while the hot sun is beating on you. A small part of the book near the end I thought was a bit far-fetched, but that might just be me. All in all it’s one of the best books I read this year.

I’ll Steal You Away – Niccolò Ammaniti

This book is in a lot of Dutch thrift stores. And for good reason, because the writing talent of Ammaniti is evident. Of all books, this book made me laugh out loud the most (I mean, the donkey on a catapult). Ammaniti tells imaginative and captivating but still recognizable stories, that are great fun to read. Even though he adds some unnecessary coarse details in my opinion. For good measure, I also read ‘I Am Not Afraid’ by Ammaniti to see if it was a one-off. But it is not. You can probably pick up any Ammaniti book and enjoy yourself. Overall I would say his writing style feels like a snack instead of a wholesome meal. But sometimes that’s what you want.

The Dinner – Herman Koch

And when you crave a snack, I can also recommend the Dinner. Many thrift stores’ bookshelves bend by the weight of the amount of copies of this book. It was one of the most sold books ever in the Netherlands. And I enjoyed reading it. It is written in such a way that you can tell the writer really carefully crafted sentences and chapters to keep the flow going. But after finishing I do remember some of the characters, but I struggle to remember the main plot reveal. But that is because the plot is secondary, this book is all about dialogue and human interaction which are both masterfully written down.

Man in Black: His Own Story in His Own Words  – Johnny Cash

This bio is from 1975, when Johnny Cash already had quite the life and a biography was well in order. We now know he would also have another life, artistically, in the nineties. So reading this book from a 1975 perspective was great. Cash, of course, seems like a man of few words. But the words are heartfelt and honest, and they shine through the paper. You can sometimes just hear the sentences in his voice. Reading this book left me with a deeper understanding of the man and his thoughts and his faith.

But as with most (music) autobiographies I’ve read, this one also falls into the trap of spending a disproportional amount of pages on more recent, less relevant events. Neil Young’s bio for instance talks way too much about his digital music player. And Cash dedicates a lot of time to the charities he runs. That’s probably fine when you read it in 1975 but 43 years later, that’s not what people remember him for and readers will have little connection to something that was topical then.

The Soul of a New Machine – Tracy Kidder

I’ve written a blog specifically on this book. I can think of no other book that blends two of my favorite subjects, management and computer history, in such an exciting manner that you would think it is fiction, when in fact it is not.

The Phoenix project – Gene Kim, George Spafford, and Kevin Behr

And that can also be said about this book. In a different way, it is fiction, but it is also not. Read my blog post about it here.

How Google Works – Eric Schmidt

I would say this book was written when Google was at its peak (2014). So I couldn’t help but compare what I read to what we know now. And that is not always a positive story. Therefore this book sometimes struck me as being a bit too much self-congratulatory and trying to be funny when clearly they are not telling the whole story. So it deliberately withholds information, which of course makes sense for a billion dollar company, but you can feel tricked as a reader. They only tell you what they want to tell you. And if you’re fine with that there are still quite a few really good lessons in here so I would certainly recommend it. Also, I am still rooting for Google to not be evil.

Zero to One – Peter Thiel

I reread this book this year, because I think it holds good reminders of how some things really work. Thiel himself has become kind of a controversial figure, but this book has ideas and asks questions that help you make see things clearer. So rereading it once in a while is recommended.

The Innovators – Walter Isaacson

I reread 4 books in 2018, including Zero to One, the Rider, the Old Man and the Sea. These three I’ve read multiple times already, but this was my second read through of the Innovators. It is the near complete story of the age of computers. It’s my favorite kind of subject by my favorite biographer. I love it.

This year I had a chance to meet, and have my copy of the Innovators signed by, Walter Isaacson himself, which was a lot of fun. I also picked up of his latest book on Leonardo da Vinci, which I plan on reading of course.

About reading

When you consider that tens of thousands of new books are published each year, reading around 25 books per year is not a lot. The books combine for a total of 6,123 pages, so averaging around 250 pages per book. But more importantly this comes down to reading around 16 pages per day. Now that does not sound like a lot does it? So maybe there is still room for improvement.

These are all the 2018 books I read (expect two, which I borrowed) in chronological order of reading. The top book, from Donald Knuth, I read on a loop (chapter per week), so I am actually always reading this book.

Art Garfunkel has a log of every book he ever read. Over 50 years of reading and still he ‘only’ read around 1300 books. Which is impressive, but of course still very little compared to the millions of books available. So when you can only read so many books, you have to be picky. But I am always looking for tips!

I am looking forward to reading ‘How the Internet Happened’ and see how it compares to or complements the Innovators. And a lot of people seem to recommend Bad Blood because it would reveal a lot about Silicon Valley culture. Both books came out recently.

So, any other books I need to read?

The post Ten notable books I read in 2018 appeared first on Jan van den Berg.

Derek Jones (derek-jones)

Foundations for Evidence-Based Policymaking Act of 2017 December 28, 2018 04:27 PM

The Foundations for Evidence-Based Policymaking Act of 2017 was enacted by the US Congress on 21st December.

A variety of US Federal agencies are responsible for ensuring the safety of US citizens, in some cases this safety is dependent on the behavior of software. The FDA is responsible for medical device safety and the FAA publishes various software safety handbooks relating to aviation (the Department of transportation has a wider remit).

Where do people go to learn about the evidence for software related issues?

The book: Evidence-based software engineering: based on the publicly available evidence sounds like a good place to start.

Quickly skimming this (currently draft) book shows that no public evidence is available on lots of issues. Oops.

Another issue is the evidence pointing to some suggested practices being at best useless and sometimes fraudulent, e.g., McCabe’s cyclomatic complexity metric.

The initial impact of evidence-based policymaking will be companies pushing back against pointless government requirements, in particular requirements that cost money to implement. In some cases this is a good, e.g., no more charades about software being more testable because its code has a low McCable complexity.

In the slightly longer term, people are going to have to get serious about collecting and analyzing software related evidence.

The Open, Public, Electronic, and Necessary Government Data Act or the OPEN Government Data Act (which is about to become law) will be a big help in obtaining evidence. I think there is a lot of software related data sitting on disks and tapes, waiting to be analysed (NASA appears to have loads to data that they have down almost nothing with, including not making it publicly available).

Interesting times ahead.

Gustaf Erikson (gerikson)

Advent of Code 2018 December 28, 2018 08:30 AM

This blog post is a work in progress

Project website: Advent of Code 2018.

Previous years: 2015, 2016, 2017.

I use Perl for all the solutions.

Most assume the input data is in a file called input.txt in the same directory as the file.

A note on scoring

I score my problems to mark where I’ve finished a solution myself or given up and looked for hints. A score of 2 means I solved both the daily problems myself, a score of 1 means I looked up a hint for one of the problems, and a zero score means I didn’t solve any of the problems myself.

My goals for this year (in descending order of priority):

  • match or beat last year’s score of 49/50.
  • solve all problems within 24 hours of release

Link to Github repo.

TODO

This year’s todo list:

  • Complete day 24
  • Complete day 23 part 2
  • Complete day 22 part 2
  • Publish day 21
  • Complete day 20
  • Publish day 19
  • Complete day 17
  • Day 16 - solve part 2 without manual intervention.
  • Complete day 15
  • Day 14 - faster part 2.
  • Day 12 - rewrite part 2 to find the answer programmatically.
  • General - implement blogging and listing publication using templates
  • Publish day 18
  • Day 8 - clean up and publish solution
  • Day 11 - faster solution

Day 1 - Day 2 - Day 3 - Day 4 - Day 5 - Day 6 - Day 7 - Day 8 - Day 9 - Day 10 - Day 11 - Day 12 - Day 13 - Day 14 - Day 16 - Day 18 - Day 19 - Day 25

Day 1 - Chronal Calibration

Day 1 - complete solution

Capsule summary:

Investigate the properties of summing an infinite list of integers.

A nice start to this year’s puzzles.

I had planned on getting up at 6AM to start with this but I spent the night before playing Fallout 4 so that didn’t happen.

Score: 2.

Day 2 - Inventory Management System

Day 2 - complete solution

Capsule summary:

Investigate the properties of the strings given as input.

Another O(n^2) solution, but for the small sizes involved it barely matters.

Perl’s string munging is helpful as always.

Score: 2.

Day 3 - No Matter How You Slice It

Day 3 - complete solution

Capsule summary:

Investigate the properties of overlapping areas given as input.

Nice and clean. I was worried I’d have to do some complicated lookups to single out the correct swatch in part 2, but it turns out that there was only one possible solution in my input.

Score: 2.

Day 4 - Repose Record

Day 4 - complete solution

Capsule summary:

Given a list of activities, determine properties about the times these activities occur.

Getting the correct answer without resorting to an external run through sort -n was the hardest part of this problem!

There are some nice solutions using a Schwartzian transform out there but to be honest I’d rather have something straightforward and dumb that can understood later.

TODO: better variable naming.

Score: 2.

Day 5 - Alchemical Reduction

Day 5 - complete solution

Capsule summary:

Remove elements from a list, based on rules on which neighbors an element has.

A quick and simple problem.

Score: 2.

Day 6 - Chronal Coordinates

Day 6 - complete solution

Capsule summary:

Investigate areas on a grid, expressed in Manhattan coordinates.

Keywords: Voronoi diagram

This took some careful perusal of the text and example to figure out. I had a lot of problems with my solution when I tried to merge part 1 and part 2 into one solution. Unfortunately I had overwritten the code that actually gave me the first star, and I broke it.

I felt justified to check out other solutions and finally re-wrote using those (essentially using a representation of the board to keep track of the data). The day after I took the time to re-write it again into a version closer to my original idea.

Runtime is a dismal 16s.

Score: 2.

Day 7 - The Sum of Its Parts

Day 7 - part 1 Day 7 - part 2

Capsule summary:

Find a path through a graph, given that some steps must be done to proceed.

Keywords: DAG, directed acyclic graph, topological sort

I’m really happy with solving this. I’m graph-aphasic, as in I’ve never ever grokked them in code, and was dreading finding some random Perl module that I didn’t really understand. In the end I just found the “endpoints” and processed it using a queue.

This made part 2 reasonably easy to code.

Score: 2.

Day 8 - Memory Maneuver

Day 8 - complete solution

Capsule summary:

Investigate the properties of a tree structure given in the form of a string of integers.

Keywords: tree structure, recursive

I had a lot of trouble with this, which wasn’t helped by stuff like people coming over to visit, dinner to cook and eat, and wine to drink. After these distractions were done with I could revisit the problem with fresh eyes.

Score: 2.

Day 9 - Marble Mania

Day 9 - complete solution

Capsule summary:

A game is played by placing marbles in a circle. Investigate the end state given the puzzle input.

Keywords: double-ended circular list

This was a fun problem that was marred by some irritating off-by-one errors in the results. We were blessed with a wealth of example inputs, and I could make most of them work by taking the largest or the next largest value. This gave me my second star, but it was hardly repeatable.

Double-checking my logic against other solution revealed I was thinking correctly, but had a stupid overrun where I managed to take more marbles than were actually available…

Runtime for part 2 is 30s, but you need quite a lot of memory to run it. My wimpy VPS cannot run it natively.

Score: 2.

Day 10 - The Stars Align

Day 10 - complete solution

Capsule summary:

Given a number of points, their positions and velocities, investigate whether they form a coherent message when printed.

Keywords: visualization

A nice palate-cleanser after the horrors of the week-end.

Runtime: 4.6s.

Score: 2.

Day 11 - Chronal Charge

Day 11 - complete solution

Capsule summary:

Investigate the values of subsquares on a 300x300 grid.

Keywords: summed-area table

Trying to solve part 2 using brute force took 173 minutes. Implementing a summed-area table reduced it to 3s.

Score: 2.

Day 12 - Subterranean Sustainability

Day 12 - part 1 Day 12 - part 2

Capsule summary:

A number of plants grow in an infinite line of pots. The existance of a plant for the next generation depends on its neighbors. Investigate what plants remain after a number of generations.

Keywords: cellular automata

This was a fiddly one. I got bit by the shallow copying nature of Perl’s data structures, but implementing the Clone module took care of that.

Then is was just a question of dealing with the inevitable off-by-one errors.

Part 2 was found by reasoning that the pattern would stabilize, and after that is was just a matter of finding the pattern and identifying the offsets.

My Part 1 code can either give the answer to the question with no input, or output some useful metadata if given another argument.

Part 2 answers the question by default, and gives the value for an arbitrary input if that’s given as an argument.

Score: 2.

Day 13 - Mine Cart Madness

Day 13 - complete solution

Capsule summary:

Carts move along a track with turns and junctions. Determine whether they collide.

I’m both proud and ashamed of this one. Proud because my basic code determining how the carts move was correct from the get-go. Ashamed because it took me all day to figure out the collision detection.

Runtime: 0.4s

Score: 2.

Day 14 - Chocolate Charts

Day 14 - part 1 Day 14 - part 2

Capsule summary:

A list is generated programmatically. Investigate its properties as it grows.

Keywords: pattern matching, brute force

Not my favorite problem, not least because my brute-force solution is killed by my VPS.

Runtime for part 2: 2m14s.

Score: 2.

Day 16 - Chronal Classification

Day 16 - complete solution

Capsule summary:

An obfuscated register manipulation routine is presented. Solve the program.

Keywords: registers

This years Register Rodeo! It was fun apart from the horrid input parsing.

Score: 2.

Day 18 - Settlers of The North Pole

Day 18 - complete solution

Capsule summary:

Investigate the properties of a Game of Life setup.

Keywords: game of life

A scruffy solution. I had no problems with part 1, but part 2 was plagued by off-by-one errors.

Day 19 - Go With The Flow

Day 19 - complete solution

Capsule summary:

Decompile a deliberately inefficient algorithm.

Keywords: registers, decompilation

Decompilation problems are not my favorite problems.

Day 25 - Four-Dimensional Adventure

Day 25 - complete solution

Capsule summary:

Investigate points in 4D space.

Keywords: union

Last problem this year. Figured this out through brute force.

December 27, 2018

Unrelenting Technology (myfreeweb)

Wow! I've been thinking about how a WebAssembly runtime in CloudABI would be an... December 27, 2018 04:29 PM

Wow! I've been thinking about how a WebAssembly runtime in CloudABI would be an awesome "write once, run everywhere" thing… turns out the Cranelift developers are already looking into CloudABI! Awesome!

December 26, 2018

Monkey Snatch Banana (SeanTAllen)

Things I'd like in my code management tool December 26, 2018 06:31 PM

This morning, I was reading a blog post from Joe Armstrong about how he got a Chandler like Triage system working in Tiddly-Wiki. A Triage system is one of the things I’d like to see GitHub add. That got me thinking, what are some of the other things I’d like to see in a GitHub. What follows is far from a complete listing of some things that readily jumped to mind.

Brian Hicks (brianhicks)

State of Elm 2018 Results December 26, 2018 09:05 AM

The State of Elm survey takes the pulse of the Elm community. What experiences do newcomers have, and are they learning at decent pace? What companies have Elm in production?

This year, the survey ran from the end of Janary to the beginning of March, and collected 1,176 responses (about the same as last year.) After the survey ended, I scrubbed each field for 17 of the questions to make sure we had good data, reduced five of the questions to tags, and performed the analysis below. I know it’s been a long wait; thank you for your patience while I did this, and to all the people who checked in and asked if they could help!

Let’s get going!


Continue Reading

December 25, 2018

Gustaf Erikson (gerikson)

Advent of Code 2018 wrapup December 25, 2018 02:27 PM

This year was a tough one. I believe the abundance of week-ends this December added to the number of quite tough problems. Day 15 seems to have been a really big stumbling block for many people.

I’m still not finished at time of writing, I’m at 38 of 50 stars. The TODO list is at the main entry for this year’s contest.

Despite miserably failing in completing each challenge in the day it was released I’m still happy I have some problems remaining going forward.

Pepijn de Vos (pepijndevos)

Making a new music instrument December 25, 2018 12:00 AM

A while a go I sent a message to Martin from Wintergatan that I want to help with the second version of his Modulin that he briefly talked about during one of his Marble Machine X videos.

He never replied, so I just decided to play around with my own ideas. The immediate goal is not so much to design a music instrument as it is to play around with touch input, digital signal processing, and analog filters.

For my first instrument I decided to build a chiptune violin by using a Game Boy as the synthesizer.

Step one was to figure out touch input. At first I wanted to do capacitive touch, but sensing position is not as easy as it’s made out to be. Then I found these SoftPot lineair potentiometers, which seem pretty good. At first I thought I would need a separate sensor strip to sense pressure, but by pressing down, a small track section of the SoftPot is shorted, resulting in a smaller total resistance that can be measured.

After playing with the Game Boy sound registers in the simulator for a while, I made a new copy of GALP for all the boilerplate code, then I added the following snippet to read an address and a byte from the Game Link port and simply write it to hiram. This code basically allows changing any register at all, not just the sound ones. The full code lives here.

; enable sound
  ld a,%10000000 ; enable
  ldh [rAUDENA],a
  ld a,$ff ; all channels
  ldh [rAUDTERM],a
  ld a,$77 ; max volume
  ldh [rAUDVOL],a

.loop

  call SerialTransfer
  ldh a,[rSB] ; new address
  ld c, a
  call SerialTransfer
  ldh a,[rSB] ; new data
  ld [$FF00+c],a ; write register
  jr .loop
  
SerialTransfer:
  ld a, $80 ; start external transfer
  ldh [rSC], a
.transferWait
  ldh a,[rSC]
  bit 7, a ; is transfer done?
  jr NZ, .transferWait
  ret

Then I connected the SoftPot to an Analog input of an Arduino with a 100k pull-down, and connected the Arduino SPI pins to the Game Link port. The code then simply reads the analog pin and sets the correct sound register on the Game Boy to play the corresponding note. The full code lives here.

int oldValue = 0;
void loop() {
      int sensorValue = analogRead(A9);
      bool lowValue = sensorValue < 1;
      bool oldLowValue = oldValue < 1;
      int outputValue = map(sensorValue, 0, 1023, 440, 1760);
      int oldOutputValue = map(oldValue, 0, 1023, 440, 1760);
      if(!lowValue && oldLowValue) { // rising edge
        ch2Length(63, 2);
        ch2Envelope(15, 0, 0);
        ch2Play(outputValue, 0, 1); 
      } else if (lowValue && !oldLowValue) { // falling edge
        ch2Envelope(15, 3, 0);
        ch2Play(oldOutputValue, 0, 1);
      } else if(!lowValue) { // high
        ch2Play(outputValue, 0, 0); 
      }
      oldValue = sensorValue; 
      delay(10);
}

The current code does not sense pressure and just uses a square wave channel on the Game Boy with an envelope on release. Next items on the list of things I want to try are sensing pressure and playing with some bucket brigade chips that have been sitting in my drawer.

Visualising your WhatsApp conversations December 25, 2018 12:00 AM

I was curious how much I chat with my friends, so I decided to find out. It’s pretty easy and fun, I can promise.

My first thought was to go after the message database, but it turns out this is encrypted with a key that can only be obtained by rooting your device or performing some downgrade attack. Not sure what the security model is here, as the key is obviously on my device and on their servers. (Else you could not restore a backup on another phone)

Instead I went for the chat export feature that is hidden in Menu > Settings > Chats > Chat history > Export chat. This is per chat, so a bit tedious. But I have a rough idea who I talk to a lot, so it’s not too bad.

From there you can do whatever you want. Look at file sizes, grep for random things, or write a simple Python script. I stole a regex from SO, fixed it for Python, put everything in a Pandas DataFrame, and from there you can just play with the df and then call plot on it.

Most of what I did so far is count the messages/characters per month. Very interesting to see how things change over time and in response to real life events. Picture omitted as a reminder how much you can tell about someone by meta-data alone.

import re
import sys
import pandas as pd
import matplotlib.pyplot as plt

regex = """(?P<datetime>\d{1,2}\/\d{1,2}\/\d{1,4}, \d{1,2}:\d{1,2}( (?i)[ap]m)*) - (?P<name>.*(?::\s*\w+)*|[\w\s]+?)(?:\s+(?P<action>joined|left|was removed|changed the (?:subject to "\w+"|group's icon))|:\s(?P<message>(?:.+|\n(?!\d{1,2}\/\d{1,2}\/\d{1,4}, \d{1,2}:\d{1,2}( (?i)[ap]m)*))+))"""

files = sys.argv[1:]

for fname in files:
    with open(fname) as f:
        text = f.read()
    matches = re.findall(regex, text)
    messages = []
    for match in matches:
        messages.append(match[::2])

    df = pd.DataFrame(messages, columns=['datetime', 'name', 'message'])
    df.datetime = pd.to_datetime(df.datetime, dayfirst=True)
    df.set_index('datetime', inplace=True)
    lengths = df.message.str.len()
    monthly = lengths.groupby(pd.Grouper(freq='M')).sum()
    #monthly = lengths.groupby(pd.Grouper(freq='D')).count().rolling('30d', min_periods=1).sum()
    #plt.figure()
    #ax = monthly.plot(title=fname[19:-4])
    ax = monthly.plot()

ax.legend([fname[19:-4] for fname in files])
plt.ylabel("bytes/month")
#plt.ylabel("messages/month")
plt.show()

Oliver Charles (ocharles)

Solving Planning Problems with Fast Downward and Haskell December 25, 2018 12:00 AM

In this post I’ll demonstrate my new fast-downward library and show how it can be used to solve planning problems. The name comes from the use of the backend solver - Fast Downward. But what’s a planning problem?

Roughly speaking, planning problems are a subclass of AI problems where we need to work out a plan that moves us from an initial state to some goal state. Typically, we have:

  • A known starting state - information about the world we know to be true right now.
  • A set of possible effects - deterministic ways we can change the world.
  • A goal state that we wish to reach.

With this, we need to find a plan:

  • A solution to a planning problem is a plan - a totally ordered sequence of steps that converge the starting state into the goal state.

Planning problems are essentially state space search problems, and crop up in all sorts of places. The common examples are that of moving a robot around, planning logistics problems, and so on, but they can be used for plenty more! For example, the Beam library uses state space search to work out how to converge a database from one state to another (automatic migrations) by adding/removing columns.

State space search is an intuitive approach - simply build a graph where nodes are states and edges are state transitions (effects), and a find a path (possibly shortest) that gets you from the starting state to a state that satisfies some predicates. However, naive enumeration of all states rapidly grinds to a halt. Forming optimal plans (least cost, least steps, etc) is an extremely difficult problem, and there is a lot of literature on the topic (see ICAPS - the International Conference on Automated Planning and Scheduling and recent International Planning Competitions for an idea of the state of the art). The fast-downward library uses the state of the art Fast Downward solver and provides a small DSL to interface to it with Haskell.

In this post, we’ll look at using fast-downward in the context of solving a small planning problem - moving balls between rooms via a robot. This post is literate Haskell, here’s the context we’ll be working in:

If you’d rather see the Haskell in it’s entirety without comments, simply head to the end of this post.

Modelling The Problem

Defining the Domain

As mentioned, in this example, we’ll consider the problem of transporting balls between rooms via a robot. The robot has two grippers and can move between rooms. Each gripper can hold zero or one balls. Our initial state is that everything is in room A, and our goal is to move all balls to room B.

First, we’ll introduce some domain specific types and functions to help model the problem. The fast-downward DSL can work with any type that is an instance of Ord.

A ball in our model is modelled by its current location. As this changes over time, it is a Var - a state variable.

A gripper in our model is modelled by its state - whether or not it’s holding a ball.

Finally, we’ll introduce a type of all possible actions that can be taken:

With this, we can now begin modelling the specific instance of the problem. We do this by working in the Problem monad, which lets us introduce variables (Vars) and specify their initial state.

Setting the Initial State

First, we introduce a state variable for each of the 4 balls. As in the problem description, all balls are initially in room A.

Next, introduce a variable for the room the robot is in - which also begins in room A.

We also introduce variables to track the state of each gripper.

This is sufficient to model our problem. Next, we’ll define some effects to change the state of the world.

Defining Effects

Effects are computations in the Effect monad - a monad that allows us to read and write to variables, and also fail (via MonadPlus). We could define these effects as top-level definitions (which might be better if we were writing a library), but here I’ll just define them inline so they can easily access the above state variables.

Effects may be used at any time by the solver. Indeed, that’s what solving planning problems is all about! The hard part is choosing effects intelligently, rather than blindly trying everything. Fortunately, you don’t need to worry about that - Fast Downward will take care of that for you!

Picking Up Balls

The first effect takes a ball and a gripper, and attempts to pick up that ball with that gripper.

  1. First we check that the gripper is empty. This can be done concisely by using an incomplete pattern match. do notation desugars incomplete pattern matches to a call to fail, which in the Effect monad simply means “this effect can’t currently be used”.

  2. Next, we check where the ball and robot are, and make sure they are both in the same room.

  3. Here we couldn’t choose a particular pattern match to use, because picking up a ball should be possible in either room. Instead, we simply observe the location of both the ball and the robot, and use an equality test with gurad to make sure they match.

  4. If we got this far then we can pick up the ball. The act of picking up the ball is to say that the ball is now in a gripper, and that the gripper is now holding a ball.

  5. Finally, we return some domain specific information to use if the solver chooses this effect. This has no impact on the final plan, but it’s information we can use to execute the plan in the real world (e.g., sending actual commands to the robot).

Moving Between Rooms

This effect moves the robot to the room adjacent to its current location.

This is an “unconditional” effect as we don’t have any explicit guards or pattern matches. We simply flip the current location by an adjacency function.

Again, we finish by returning some information to use when this effect is chosen.

Dropping Balls

Finally, we have an effect to drop a ball from a gripper.

  1. First we check that the given gripper is holding a ball, and the given ball is in a gripper.

  2. If we got here then those assumptions hold. We’ll update the location of the ball to be the location of the robot, so first read out the robot’s location.

  3. Empty the gripper

  4. Move the ball.

  5. And we’re done! We’ll just return a tag to indicate that this effect was chosen.

Solving Problems

With our problem modelled, we can now attempt to solve it. We invoke solve with a particular search engine (in this case A* with landmark counting heuristics). We give the solver two bits of information:

  1. A list of all effects - all possible actions the solver can use. These are precisely the effects we defined above, but instantiated for all balls and grippers.
  2. A goal state. Here we’re using a list comprehension which enumerates all balls, adding the condition that the ball location must be InRoom RoomB.

So far we’ve been working in the Problem monad. We can escape this monad by using runProblem :: Problem a -> IO a. In our case, a is SolveResult Action, so running the problem might give us a plan (courtesy of solve). If it did, we’ll print the plan.

fast-downward allows you to extract a totally ordered plan from a solution, but can also provide a partiallyOrderedPlan. This type of plan is a graph (partial order) rather than a list (total order), and attempts to recover some concurrency. For example, if two effects do not interact with each other, they will be scheduled in parallel.

Well, Did it Work?!

All that’s left is to run the problem!

> main
Found a plan!
1: PickUpBall
2: PickUpBall
3: SwitchRooms
4: DropBall
5: DropBall
6: SwitchRooms
7: PickUpBall
8: PickUpBall
9: SwitchRooms
10: DropBall
11: DropBall

Woohoo! Not bad for 0.02 secs, too :)

Behind The Scenes

It might be interesting to some readers to understand what’s going on behind the scenes. Fast Downward is a C++ program, yet somehow it seems to be running Haskell code with nothing but an Ord instance - there are no marshalling types involved!

First, let’s understand the input to Fast Downward. Fast Downward requires an encoding in its own SAS format. This format has a list of variables, where each variable contains a list of values. The contents of the values aren’t actually used by the solver, rather it just works with indices into the list of values for a variable. This observations means we can just invent values on the Haskell side and careful manage mapping indices back and forward.

Next, Fast Downward needs a list of operators which are ground instantiations of our effects above. Ground instantiations of operators mention exact values of variables. Recounting our gripper example, pickUpBallWithGrippper b gripper actually produces 2 operators - one for each room. However, we didn’t have to be this specific in the Haskell code, so how are we going to recover this information?

fast-downward actually performs expansion on the given effects to find out all possible ways they could be called, by non-deterministically evaluating them to find a fixed point.

A small example can be seen in the moveRobotToAdjacentRoom Effect. This will actually produce two operators - one to move from room A to room B, and one to move from room A to room B. The body of this Effect is (once we inline the definition of modifyVar)

Initially, we only know that robotLocation can take the value RoomA, as that is what the variable was initialised with. So we pass this in, and see what the rest of the computation produces. This means we evaluate adjacent RoomA to yield RoomB, and write RoomB into robotLocation. We’re done for the first pass through this effect, but we gained new information - namely that robotLocation might at some point contain RoomB. Knowing this, we then rerun the effect, but the first readVar gives us two paths:

This shows us that robotLocation might also be set to RoomA. However, we already knew this, so at this point we’ve reached a fixed point.

In practice, this process is ran over all Effects at the same time because they may interact - a change in one Effect might cause new paths to be found in another Effect. However, because fast-downward only works with finite domain representations, this algorithm always terminates. Unfortunately, I have no way of enforcing this that I can see, which means a user could infinitely loop this normalisation process by writing modifyVar v succ, which would produce an infinite number of variable assignments.

Conclusion

CircuitHub are using this in production (and I mean real, physical production!) to coordinate activities in its factories. By using AI, we have a declarative interface to the production process - rather than saying what steps are to be performed, we can instead say what state we want to end up in and we can trust the planner to find a suitable way to make it so.

Haskell really shines here, giving a very powerful way to present problems to the solver. The industry standard is PDDL, a Lisp-like language that I’ve found in practice is less than ideal to actually encode problems. By using Haskell, we:

  • Can easily feed the results of the planner into a scheduler to execute the plan, with no messy marshalling.
  • Use well known means of abstraction to organise the problem. For example, in the above we use Haskell as a type of macro language - using do notation to help us succinctly formulate the problem.
  • Abstract out the details of planning problems so the rest of the team can focus on the domain specific details - i.e., what options are available to the solver, and the domain specific constraints they are subject to.

fast-downward is available on Hackage now, and I’d like to express a huge thank you to CircuitHub for giving me the time to explore this large space and to refine my work into the best solution I could think of. This work is the result of numerous iterations, but I think it was worth the wait!

Appendix: Code Without Comments

Here is the complete example, as a single Haskell block:

{-# language DisambiguateRecordFields #-}

module FastDownward.Examples.Gripper where

import Control.Monad
import qualified FastDownward.Exec as Exec
import FastDownward.Problem


data Room = RoomA | RoomB
  deriving (Eq, Ord, Show)


adjacent :: Room -> Room
adjacent RoomA = RoomB
adjacent RoomB = RoomA


data BallLocation = InRoom Room | InGripper
  deriving (Eq, Ord, Show)


data GripperState = Empty | HoldingBall
  deriving (Eq, Ord, Show)


type Ball = Var BallLocation


type Gripper = Var GripperState

  
data Action = PickUpBall | SwitchRooms | DropBall
  deriving (Show)


problem :: Problem (Maybe [Action])
problem = do
  balls <- replicateM 4 (newVar (InRoom RoomA))
  robotLocation <- newVar RoomA
  grippers <- replicateM 2 (newVar Empty)

  let
    pickUpBallWithGrippper :: Ball -> Gripper -> Effect Action
    pickUpBallWithGrippper b gripper = do
      Empty <- readVar gripper
  
      robotRoom <- readVar robotLocation
      ballLocation <- readVar b
      guard (ballLocation == InRoom robotRoom)
  
      writeVar b InGripper
      writeVar gripper HoldingBall
  
      return PickUpBall


    moveRobotToAdjacentRoom :: Effect Action
    moveRobotToAdjacentRoom = do
      modifyVar robotLocation adjacent
      return SwitchRooms


    dropBall :: Ball -> Gripper -> Effect Action
    dropBall b gripper = do
      HoldingBall <- readVar gripper
      InGripper <- readVar b
  
      robotRoom <- readVar robotLocation
      writeVar b (InRoom robotRoom)
  
      writeVar gripper Empty
  
      return DropBall

  
  solve
    cfg
    ( [ pickUpBallWithGrippper b g | b <- balls, g <- grippers ]
        ++ [ dropBall b g | b <- balls, g <- grippers ]
        ++ [ moveRobotToAdjacentRoom ]
    )
    [ b ?= InRoom RoomB | b <- balls ]

  
main :: IO ()
main = do
  plan <- runProblem problem
  case plan of
    Nothing ->
      putStrLn "Couldn't find a plan!"

    Just steps -> do
      putStrLn "Found a plan!"
      zipWithM_ (\i step -> putStrLn $ show i ++ ": " ++ show step) [1::Int ..] steps


cfg :: Exec.SearchEngine
cfg =
  Exec.AStar Exec.AStarConfiguration
    { evaluator =
        Exec.LMCount Exec.LMCountConfiguration
          { lmFactory =
              Exec.LMExhaust Exec.LMExhaustConfiguration
                { reasonableOrders = False
                , onlyCausalLandmarks = False
                , disjunctiveLandmarks = True
                , conjunctiveLandmarks = True
                , noOrders = False
                }
          , admissible = False
          , optimal = False
          , pref = True
          , alm = True
          , lpSolver = Exec.CPLEX
          , transform = Exec.NoTransform
          , cacheEstimates = True
          }
    , lazyEvaluator = Nothing
    , pruning = Exec.Null
    , costType = Exec.Normal
    , bound = Nothing
    , maxTime = Nothing
    }

December 24, 2018

Bit Cannon (wezm)

Windows December 24, 2018 11:25 PM

At various points in my Finding an Alternative to Mac OS series I’ve made promises about trying Windows and writing about the experience. This is as close as I’m going to get to that post:

Regarding Windows, I should say that I am strongly biased towards *nix style operating systems and find it unlikely that I’d be happy using Windows full time. However, in the interests of keeping an open mind I will give it a try in the next few months. I have backed the Eve V campaign. The Eve V is a 2-in-1 laptop tablet that will come with Windows 10. For now Windows is off the table.

Finding an Alternative to Mac OS X — Part 2 (Jan 2017)

It took a lot longer than expected, but the V finally arrived in February 2018. I backed the campaign in December 2016. I’ve been getting a feel for what a 2-in-1 Windows machine with a pen has to offer. I still can’t see myself making Windows my primary OS but I will keep exploring. I’ve been tweeting my adventures with the #wesonwindows hash tag. At some point I’ll attempt installing Linux on it too.

[…]

I’ll post more about the V and Windows in the future.

A Year Away From Mac OS (Mar 2018)

On and off over the last year or so I have spent some time in Windows 10. Mostly with the Eve V, but also on my XPS 15. I have found the experience pretty terrible on many occasions. The hostility and lack of respect towards the user shown by Microsoft and some Windows software developers is egregious and angers me every time I encounter it.

I’m aware that the vast majority of computer users manage to use Windows but I’ve concluded it’s not for me. I find it unpleasant to use, slow, full of junk, and offensive to its users. I’m not going to spend any more time entertaining the idea that it’s a viable alternative to BSD, Linux, or Mac OS for me.

I will end this post with a series of screen shots, each taken with a degree of incredulity when the situation was captured. These culminate in the Windows install being broken on the Eve V for unknown reasons.


Periodically a notification appears asking, “How likely are you to recommend this PC to a friend or colleague?”. Clicking it launches the, “Feedback Hub”. As the tweet above shows I was astonished the first time I got this notification.

How likely are you to recommend this PC to a friend or colleague? Not likely at all now.

Initially every time I logged in I got a nice notification from OneDrive. A service I did not install, nor need. I had to remove it from the auto-started applications to make it stop.

OneDrive: Finish installing sync icons notification.

Not long after the first boot of my XPS 15 I was shown this lovely train wreck. You could argue that the McAfee thing is Dell’s fault for pre-loading it but we don’t see this crapware on BSD, Mac OS, and most Linux systems.

Get to know your new PC notification atop a horrid McAfee notification window thing.

More needy notifications, this time from Edge. Not captured here is the pleading that takes place when you change the default browser to something other than Edge. Please just let me use my computer how I want!

Microsoft Edge showing a popover: Enjoying this app?

Windows support for HiDPI displays seems less complete than Mac OS and GNOME.

This Open dialog is very confused by the HiDPI scaling. The whitespace around the image was part of the window.

It’s less obvious but this dialog spawned from the settings app was smaller than the rest of the UI on the actual display of the Eve V, seemingly ignoring the scaling setting.

Poorly scaled handwriting recognition dialog.

Lastly the death loop the Eve V is now in. After multiple attempts to install updates (or do something?) with spinners of unspecified reasoning and a complete lack of visible progress for large stints of time, it remains in this broken state.

C:\WINDOWS\system32\config\systemprofile\Desktop is unavailable.

I enjoyed watching this spinner for nearly 10 minutes. No progress information. Not even a hint as to what I'm waiting for.

Then we move on to more waiting:

This might take several minutes. What might take several minutes?

Leave everything to us... What other choice do I have? BSD and Linux!

Goodbye Windows.

Jeff Carpenter (jeffcarp)

2018 Year in Review December 24, 2018 10:39 PM

For the past few years I’ve been summing up my yearly highlights in a blog post. It doesn’t matter if anyone reads these posts—it’s nice to have these to be able to look back on the big things that happened in my life each year. So without further ado, here’s an overview of the big things that happened in my life in 2018, from Running, to Reading, Learning Work, and Health, finally looking at how I did on my 2018 goals and setting 2019 goals.

Jan van den Berg (j11g)

Can we replace paper? December 24, 2018 04:34 PM

Paper always beats rock and scissors. Because one of the few inventions greater than writing itself, is writing on paper. Paper writings are absolute, self-contained and transferable units of knowledge, which after publishing become and stay available and accessible for hundreds of years or more.

Don’t take my word for it, there is this great quote by J.C.R. Licklider found in Libraries of the Future and brought to my attention by Walter Isaacson in The Innovators.

Message and medium

Take da Vinci’s work. We are able to witness and experience and read the exact paper he put his thoughts on some 500 years ago. Our language may have changed but the medium and therefore message survived. You can pick it up, look at it, and see exactly what he saw (if you can afford it).

And in the same vein, I can easily pick up a book, written and printed 100 years ago, and read it. Or nearer by, I can open any textbook I used in college from my bookshelf and read it. And my class notes just sit in a box, unchanged, ready to be read. All I need are my eyeballs. But my 3.5 inch floppies from that era, I can no longer access those (with ease). And the CD-ROMs, I wonder if they would even work. And when the medium becomes inaccessible the message is lost.

Part of my bookshelf

The internet

So as I am typing this on an electronic digital device, that translates key presses into binary numbers which are stored on a solid state disk on another computer somewhere else, which is connected with my device through countless other specialised electronic devices and protocols, I can’t help but wonder about what will be left in 100 years — or more — from what is written everyday on the internet.

The internet is right up there with the written word as one of our greatest inventions, but it is much more fragile and dependant on many layers (i.e. electricity, storage, network, specialised devices, formats) that all interact with one another.

We have accumulated large parts of human knowledge in millions of paper books over the past millennium, but most written text nowadays is digital. And digital formats and transfer methods change. Fast and often. So I wonder how we can best preserve our written thoughts for the next millennium: self-contained and transferable. But I can’t come up with anything better than paper?

The post Can we replace paper? appeared first on Jan van den Berg.

December 23, 2018

Ponylang (SeanTAllen)

Last Week in Pony - December 23, 2018 December 23, 2018 04:22 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, our users’ mailing list or join us on IRC.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

December 21, 2018

Siddhant Goel (siddhantgoel)

Developer to Manager December 21, 2018 11:00 PM

TL;DR I've been working on Developer to Manager for the last few weeks. It's a collection of interviews with software developers who moved to management, and explores what they learned, the best advice they received on this topic, and so on. Check it out!


In the software industry, the progression from being a "software developer" to being an "engineering manager" is considered fairly normal. Well performing developers are often "promoted" to a role where they are not writing code anymore, but are instead responsible for managing a team of developers.

As a software developer, you write/review (at least a little bit of) code every single day. You're fixing bugs, implementing product features, pushing commits, reviewing/merging pull requests, you know the drill. Your life basically revolves around code.

As an engineering manager though, most of this changes. Depending on your company, you might still write code every now and then, but it won't be your main job. Your main job now is to support a team of developers and figure out how to juggle everything to be able to add maximum value to the business you're working for. Sounds vague? Yes, it is. You are now responsible for shielding your developers from external factors, aligning their career goals with the daily work they do at your company, regular one-on-ones with them to make sure things are going alright, some other responsibilities you don't know about yet, and a few more responsibilities you don't even know that you don't know.

In a nutshell though, your life is now revolving around people.

If the two roles are that different, is a "developer -> manager" transition really a promotion?

The Back Story

About two years ago, I co-founded a startup with a few friends, where initially I was the only person responsible for software development. We did bring a few other people on board later who now handle a major chunk of work, but the team is still quite small.

Anyway, when you're the only developer co-founding a startup, you're often considered the "face of software" for the people doing business development. This additional responsibility requires you to do a lot more than just write code. Additionally, since you're personally vested in the success of the company, you're not in the "employee mindset" anymore, which means you can't just sit back and wait for someone else to give you things to work on. You need to interface with business development, make sure that software is being written smoothly, there are no bottlenecks, the developers working along with you are happy, their career goals are being met, you're meeting deadlines, and so much more.

It's a very delicate role, and depending on what makes you tick, it can either be the best thing to happen to your career or the worst. I personally don't think there's anything in the middle.

The Motivation

When I found myself in a similar role (which was way outside my comfort zone btw), I realized I was both excited and apprehensive at the same time.

Excited, because this is an opportunity that allows you to impact your organization like you could have never imagined. Such a position requires you to step out of the "coder in the corner" mindset and try to figure out how to help the business succeed, and then go ahead and do all of it. The leverage you have here is immense.

Apprehensive, because with great power comes great responsibility. And how do you assume responsibility without knowing much about what you're getting into?

With this chain of thought, I started looking for resources to prepare myself to handle things better. I like learning from books, so my first instinct was to search for books on this topic. It's difficult to sift through the typical "how to become a manager in 24 hours" noise, but I managed to find two really good books - The Manager's Path, and Managing Humans. Both of them are written from the point of view of a software developer, and are highly recommended reading if you're interested in this domain.

To my surprise though, I couldn't find that many more resources. Maybe my DuckDuckGo skills are getting rusty, but I found it frustrating that there aren't that many good resources written from a developer point of view, especially given how crucial such a role can be to a software company.

What I would have really liked to have when starting out, was to just know how others handled it. Nothing fancy, but just having some other engineering managers answer a common set of questions about their transition so I could get a sense of how best to prepare myself for something like that.

I wasn't looking for an exact set of steps I should take to become a good manager (not to imply that something like that exists), but just knowing that someone else has been in your shoes and managed to come out successful goes a long way, and helps in a way no article/book/teacher probably can.

The Process

Since I couldn't find something like that, I decided to try building it on my own.

I started by contacting a few of my manager friends and talked to them on this topic. After almost all of them agreed with the premise, I realized that the idea cannot be completely stupid and that there could be something worth exploring here.

I then brain-stormed a short list of questions that I would have liked to have other engineering managers answer, and then emailed it to a few of my friends and a few more people I found on the web. The idea was to have different people answer a common set of questions so that the readers can get different perspectives on the same topics.

The response I received was extremely encouraging. People were more than willing to invest the time into talking about their transitions to help younger developers make this transition successfully. And there was a ton of stuff to explore here - best practices, things to keep in mind, things to avoid doing, and much more.

The Product

Developer to Manager is what has come out of all this.

The site currently hosts a collection of interviews with software developers who moved to management. I ask them a few basic questions about their background, what their transition felt like, what their takeaways were, and so on. As I mentioned before, the idea is to get different perspectives on similar situations.

I try to publish one interview every week, but of course this depends on the availability of interviewees.

In the longer term I would like to see "Developer to Manager" evolve into the website you open when you want to learn more about how to become a good engineering manager.

How do I plan to do that? I don't know yet. There are a ton of things floating around in my head right now which don't necessarily fit together, but I have the feeling that given enough time it should be possible to incorporate all of them into the product.

Derek Jones (derek-jones)

Distorting the input profile, to stress test a program December 21, 2018 09:31 PM

A fault is experienced in software when there is a mistake in the code, and a program is fed the input values needed for this mistake to generate faulty behavior.

There is suggestive evidence that the distribution of coding mistakes and inputs generating fault experiences both have an influence of fault discovery.

How might these coding mistakes be found?

Testing is one technique, it involves feeding inputs into a program and checking the resulting behavior. What are ‘good’ input values, i.e., values most likely to discover problems? There is no shortage of advice for manually writing tests, suggesting how to select input values, but automatic generation of inputs is often somewhat random (relying on quantity over quality).

Probabilistic grammar driven test generators are trivial to implement. The hard part is tuning the rules and the probability of them being applied.

In most situations an important design aim, when creating a grammar, is to have one rule for each construct, e.g., all arithmetic, logical and boolean expressions are handled by a single expression rule. When generating tests, it does not always make sense to follow this rule; for instance, logical and boolean expressions are much more common in conditional expressions (e.g., controlling an if-statement), than other contexts (e.g., assignment). If the intent is to mimic typical user input values, then the probability of generating a particular kind of binary operator needs to be context dependent; this might be done by having context dependent rules or by switching the selection probabilities by context.

Given a grammar for a program’s input (e.g., the language grammar used by a compiler), decisions have to be made about the probability of each rule triggering. One way of obtaining realistic values is to parse existing input, counting the number of times each rule triggers. Manually instrumenting a grammar to do this is a tedious process, but tool support is now available.

Once a grammar has been instrumented with probabilities, it can be used to generate tests.

Probabilities based on existing input will have the characteristics of that input. A recent paper on this topic (which prompted this post) suggests inverting rule probabilities, so that common becomes rare and vice versa; the idea is that this will maximise the likelihood of a fault being experienced (the assumption is that rarely occurring input will exercise rarely executed code, and such code is more likely to contain mistakes than frequently executed code).

I would go along with the assumption about rarely executed code having a greater probability of containing a mistake, but I don’t think this is the best test generation strategy.

Companies are only interested in fixing the coding mistakes that are likely to result of a fault being experienced by a customer. It is a waste of resources to fix a mistake that will never result in a fault experienced by a customer.

What input is likely to interact with coding mistakes to be the root cause of faults experienced by a customer? I have no good answer to this question. But, given there are customer input contains patterns (at least in the world of source code, and I’m told in other application domains), I would generate test cases that are very similar to existing input, but with one sub-characteristic changed.

In the academic world the incentive is to publish papers reporting loads-of-faults-found, the more the merrier. Papers reporting only a few faults are obviously using inferior techniques. I understand this incentive, but fixing problems costs money and companies want a customer oriented rationale before they will invest in fixing problems before they are reported.

The availability of tools that automate the profiling of a program’s existing input, followed by the generation of input having slightly, or very, different characteristics make it easier to answer some very tough questions about program behavior.

December 20, 2018

Pete Corey (petecorey)

Advent of Code: Subterranean Sustainability December 20, 2018 12:00 AM

Day twelve of this year’s Advent of Code essentially asks us to implement a basic one dimensional cellular automata that can look two spaces to the left and right of itself. We’re given an infinite row of “pots”, and an initial configuration of pots that contain living plants. We’re asked, after twenty generations, the sum of the pot numbers that contain living plants.

Let’s take a stab at this using the J programming language.

Our sample starting state already looks a lot like a bit mask, so let’s do a little massaging and get it into a form we can work with:

    replacements =. 'initial state: ';'';'=> ';'';'.';'0 ';'#';'1 '
    path =.  '/Users/pcorey/advent_of_code_2018/day_12/input'
    replaced =. cutopen replacements rplc~ (1!:1) < path

    NB. Parse out initial state as boolean array
    initial =. ". > {. replaced
1 0 0 1 0 1 0 0 1 1 0 0 0 0 0 0 1 1 1 0 0 0 1 1 1

Great. Now let’s work with the patterns, or cellular automata rules, we were given and work them into a structure we can deal with:

    NB. Build matrix of replacement patterns
    patterns =. }:"1 ". > }. replaced
1 1 1 1 0
1 1 1 0 1
1 1 1 0 0
...

Similarly, we’ll build up our list of resulting pot values, or replacements, if we find any of those matching patterns:

    NB. Build array of replacement values
    replacements =. {:"1 ". > }. replaced
0 1 1 1 1 1 0 0 1 0 1 0 0 0 0 0 1 0 0 1 1 1 0 1 0 0 0 1 1 0 0 0

Great. Now let’s write a monadic verb that takes a 5-length array of bits, or booleans, and returns the corresponding replacement value:

    NB. Replace a length-5 y with a replacement value
    replace =. replacements"_ {~ patterns&i.

    replace 0 1 0 1 0
1

And now we can tie everything together with an iterate verb that takes our initial configuration, breaks it into overlapping chunks of 5-length arrays, and repeatedly applies replace to each chunk (with some extra padding thrown in to catch edge cases):

    iterate =. 3 : 0
      (1 ,: 5) replace;._3 (0,0,0,0,y,0,0,0,0)
    )

We can iterate twenty times over our initial starting configuration:

    iterated =. iterate^:20 initial

At this point we could iterate <20 times to build up an array of each iteration, and visualize the growth of our plants using viewmat:

Our plants' growth over time.

Finally we can convert our bit mask of pots with living plants into a list of coordinates, and print the sum:

    echo +/ iterated # (iterations * 2) -~ i. # iterated
325

Part Two

Part two asks us for the sum of the pot numbers with living plants after fifty billion (50,000,000,000) generations. Obviously, we can’t run our simulation for that long! We need to find some kind of pattern in our iterations.

To help, I decided to refactor my part one solution to simply keep track of an array of pot numbers with living plants, instead of a bit mask of all possible pots:

    NB. Build matrix of replacement patterns
    patterns_and_output =. ". > }. replaced
    patterns_to_keep =. {:"1 patterns_and_output
    patterns =. patterns_to_keep # ([: < 2 -~ I.)"1 }:"1 patterns_and_output
    NB. patterns =. patterns #~ 1 {:: patterns

    check_pattern =. 4 : 0
      pot_to_check =. 0 {:: x
      pots_with_plants =. 1 {:: x
      pattern_to_check =. > y

      pots_above_cutoff =. pots_with_plants >: pot_to_check - 2
      pots_below_cutoff =. pots_with_plants <: pot_to_check + 2
      pots_to_check =. pots_above_cutoff *. pots_below_cutoff
      pots_to_check =. pots_to_check # pots_with_plants

      (pot_to_check + pattern_to_check) -: pots_to_check
    )

    check_pot =. 4 : 0
      pot_to_check =. y
      pots_with_plants =. x
      +./ (pot_to_check;pots_with_plants)&check_pattern"0 patterns
    )

    iterate =. 3 : 0
      pots_with_plants =. > y
      pots_to_check =. (2 -~ <./ pots_with_plants) + i. 4 + (>./ - <./) pots_with_plants
      next_pots_with_plants =. < pots_to_check #~ pots_with_plants&check_pot"0 pots_to_check
      next_pots_with_plants
    )

As you can see, I tried to be extra clear and verbose with my naming to keep myself from getting confused. I think the result is some fairly readable code.

As an example, I can grab the twentieth generation of pots with living plants like so:

    iterate^:20 <initial
┌─────────────────────────────────────────────────────┐
│_2 3 4 9 10 11 12 13 17 18 19 20 21 22 23 28 30 33 34│
└─────────────────────────────────────────────────────┘

My plan of attack for finding the repeating pattern was to look for a cycle. If a generation, offset to zero, or normalized, ever matches a normalized generation we’ve seen previously, we’ve found a cycle.

I wrote two verbs to help me find this cycle and return some information about it:

    normalize =. <@:(<./ -~ ])@:>

    find_cycle =. 3 : 0
      previous =. 0 {:: y
      previous_normalized =. 1 {:: y
      next =. iterate {: previous
      next_normalized =. normalize next
      if. (next_normalized e. previous_normalized) do.
        next_min =. <./ > next
        previous_min =. <./ > {: previous
        (# previous);(previous_normalized i. next_normalized);(next_min - previous_min);({: previous)
      else.
        if. 1000 < # previous do.
          return.
        end.
        find_cycle (previous,next);<(previous_normalized,next_normalized)
      end.
    )

The find_cycle is a recursive verb that maintains a list of previous generations, and their normalized form. Every call it calculates the next generation and its normalized form. If it finds a cycle, it returns the number of previous generations we’ve seen, the index the repeated generation loops back to, the number of pots we’ve moved, and the last non-cyclical generation we processed, all boxed together.

    find_cycle (<initial);(<normalize<initial)                                
┌──┬──┬─┬───────────────────────────────────────────────────────────┐
│87│86│1│12 14 20 22 28 30 40 42 48 50 61 63 69 71 76 78 87 89 96 98│
└──┴──┴─┴───────────────────────────────────────────────────────────┘

So it looks like our cycle starts at pot eighty six, and has a cycle length of one. This actually simplifies things quite a bit. This basically means that after reaching generation eighty six, our plants just move to the right each generation.

Let’s change our find_cycle function to return just the generation the cycle starts, and the set of pots with living plants at that generation. We can use that to find out how many position we need to add to that set of pots before we sum them for our final answer:

    result =. find_cycle (<initial);(<normalize<initial)
    to_add =. 50000000000 - (0 {:: result)
    final_pots_with_plants =. to_add + (1 {:: result)
    +/ final_pots_with_plants

Notes

  • Pass a boxed number into ^: to return an array of applications of the verb, not just the last result.
  • Use the cut verb (;.) to chunk an array into overlapping sub-arrays.

eta (eta)

A guide to STM32F103 microcontrollers - part 1 December 20, 2018 12:00 AM

So, there are these cheap microcontrollers I’ve recently started looking into based off the STM32F103 architecture, which itself belongs to the Arm Cortex-M3 processor family. I originally looked into them because they’re actually full 32-bit processors instead of the 8-bit kind usually found on an Arduino Uno, which uses the 8-bit ATmega328P processor. This means that they’re a lot easier to work with (you can actually have native 32-bit integers…!), way faster than their AVR competitors - and, as an added bonus, you can do cool things like run Rust on them, with the stm32f103xx-hal crate.

However, documentation on them is rather scarce and fragmented, given that these processors don’t have the same kind of community around them that Arduino users can benefit from. Therefore, I’m going to do a blog series about them, summarizing what I’ve learnt while using them, in the hope that it might be useful to someone else!

This first post mainly focuses on what hardware is available, and how to configure it to upload simple code. It’ll get more interesting in later posts, though!

Origins

The STM32F103 originally made its fame through the Maple Mini and Maple Arduino clones made by a company called LeafLabs, which no longer manufactures them. However, LeafLabs built a whole bunch of tooling to let you write code for these boards as if they were Arduinos (using digitalWrite(), pinMode(), and other familiar Wiring-style functions). This lives on as the open-source STM32duino project, and a lot of the info in this blog post comes from their wiki.

Buying

There are two main boards (that I’ve heard of and tried personally): the Maple Mini, and the Blue Pill. I got mine off Amazon UK; I believe it’s also possible to get these boards via AliExpress or eBay, if you prefer.

(You’ll have to excuse my terrible soldering in the following photos…)

Maple Mini

Picture of Maple Mini

link to STM32duino wiki · buy for £9 on Amazon (affiliate link)

As discussed above, this is the most supported STM32 board, and it’s pretty much plug & play. Mine seemed to come with the bootloader preloaded, making things even easier; if that isn’t the case, it’s not too hard to reflash it (I had to do so myself after botching an upload).

The USB port is a miniUSB, not a microUSB, which can be a bit annoying given how many things use microUSB nowadays.

A pinout is available on the wiki link above; notably, this board uses its own naming scheme for the pins, and you have to use this table to convert between those and the actual GPIO names if you want to program it in Rust or use something apart from the STM32duino environment.

Blue Pill

Picture of Blue Pill link to STM32duino wiki · buy for £6 on Amazon (affiliate link)

This board is £3 cheaper than the Maple Mini (in fact, if you buy it from not-Amazon, it costs about $2), but has a number of drawbacks. Firstly, the on-board voltage regulator (used to convert from 5V to 3.3V) is lousy, meaning that you can’t draw much current (more than 100mA) from any of the pins on the board, or you risk frying the whole thing (if the regulator fails, it passes the 5V through to the STM32 directly, breaking it).

You also can’t power the thing with USB power and external 5V power, otherwise it breaks, and some PCs may refuse to work with its USB interface due to an incorrect pull-up resistor on one of the data lines. Despite that, though, the Blue Pill is still pretty useful.

Unlike the Maple Mini, its pins are actually labeled in a sane manner; like the Maple Mini, there’s also a pinout diagram on the STM32duino wiki link above.

Tools

While it is possible to upload code to the Maple Mini via USB only, the Blue Pill requires one of the below tools to upload anything (you can flash a bootloader that lets you upload via USB, and then do that, but you still need the tool for the first flash).

Picture of ST-Link

buy for £9 on Amazon (affiliate link)

The ST-Link is ST Microelectronics’ tool for flashing STM32-based boards. It’s pretty handy to have - not only does it let you program the board, but it also provides on-chip debugging support, allowing you to pause the program mid-execution, inspect variables, etc.

The pinout varies depending on which kind you end up with - see the below diagram (the one pictured above is the one on the right below).

ST-Link pinout

USB to TTL serial adapter

Picture of USB to TTL serial adapter

buy for £6 on Amazon (affiliate link)

This is just a handy tool to have anyway - for example, if you want to send data using one of the boards’ USART interfaces and receive it using this thing. Incidentally, this also works with the serial console connection on a Raspberry Pi.

For STM32 boards, this can be used to upload code if you don’t have a ST-Link (q.v.).

STM32duino

This section details how to use the STM32duino IDE and firmware. This is probably the easiest way to write code for these boards at present, and there are 3 ways (that I’ve tested) to upload code to the boards: via the bootloader (very easy), with a USB to TTL serial adapter, and with an ST-Link.

Install the IDE and software

Follow the STM32duino Installation guide. You can ignore the “Burning the bootloader” bit for now, as we’ll cover that in more detail later - if you want to use the bootloader method, jump ahead to the relevant section below. I’ve tested it with Arduino IDE 1.8.7, so you can probably ignore the bit about 1.8.5 being the latest supported version.

Upload code - using the bootloader

If you’re using a Maple Mini, or have installed the USB bootloader for the Blue Pill, uploading code is pretty easy. However, note that this method is incompatible with any of the other methods - uploading code with another method will mean that you’ll need to flash the bootloader again (see below).

  1. Open the Arduino IDE, and go to File → Examples → A_STM32_Examples → Digital → Blink to load the example blinking LED code.
    • If you’re using a Blue Pill, change PB1 to PC13 where it appears in the code.
  2. Go to Tools → Board, and choose the appropriate board type.
    • Maple Mini” is, unsurprisingly, the option for a Maple Mini.
    • Choose “Generic STM32F103C series” for a Blue Pill.
  3. If you’re using a Blue Pill, change Tools → Upload method to “STM32duino bootloader”.
  4. Change Tools → Port to the serial port for your microcontroller (it should be fairly obvious; there’s usually only one entry).
  5. Hit Upload (the right-facing arrow on the main toolbar).
    • If it doesn’t work, you might need to press the Reset button on the MCU before trying again; occasionally, you need to time it just right.
    • On the Maple Mini, you can hold the second button (but=32) after pressing Reset to keep it in the bootloader, making uploading somewhat easier.
    • You also need to ensure that you have permission to write to the serial port (usually /dev/ttyACM0 or /dev/ttyUSB0). This might entail installing these udev rules, adding yourself to the plugdev group, or something else; consult your Linux distro’s documentation.
  6. Observe blinking LED.

Upload code - using a serial adapter

STM32F103 chips have an internal mode built into the chip that lets you flash programs via a USB to TTL serial adapter, such as the one featured above. This has different setup instructions depending on the board you’re using.

(This is also a relatively easy way to upload code to a Blue Pill without futzing around with the ST-Link.)

Blue Pill

Picture of Blue Pill wired to TTL serial adapter

For the Blue Pill, wire up 5V and GND from your serial adapter to the identically-named pins on the board. Then, connect RX on your serial adapter to A9 on the board, and TX to A10.

You’ll also need to ensure that the jumpers (e.g. the yellow thingies above the RESET button) are set in the correct position.

  • During normal operation (i.e. when running stored code), the jumpers should both be in the 0 position.
  • To flash code using this method, the top jumper (closest to B10) should be in the 1 position, with the other left in 0.

Do remember to restore the jumper to the correct position after uploading!

Maple Mini

Picture of Blue Pill wired to TTL serial adapter

For the Maple Mini, wire up 5V on the adapter to Vin on the board, GND to GND, RX to 26 (tx1), and TX to 25 (rx1). Then, connect but (aka Boot 0) on the board to Vcc, and 2 (aka Boot 1) to GND.

You’ll need to remove the connections from but and 2 after you’ve finished flashing if you want the board to run on its own (rather like the jumpers on the Blue Pill).

Method

  1. Open the Arduino IDE as before, and go to File → Examples → A_STM32_Examples → Digital → Blink to load the example blinking LED code.
    • If you’re using a Blue Pill, change PB1 to PC13 where it appears in the code.
  2. Go to Tools → Board, and choose “Generic STM32F103C series” (even if you have a Maple Mini).
  3. Change Tools → Upload method to “Serial”.
  4. Change Tools → Port to the serial port for your TTL serial adapter (it should be fairly obvious; there’s usually only one entry).
  5. Hit Upload (the right-facing arrow on the main toolbar).
    • If you get errors, check you have permissions to write to the serial port, and that you’ve connected TX and RX correctly.
  6. Observe blinking LED.

This works like the serial method, but with far less wiring (you don’t have to change any jumpers). Once again, setup instructions differ depending on board.

Blue Pill

Picture of Blue Pill programming header

Use the ST-Link pinout diagram from above. Connect the 4 pins on the 4-pin programming header on the right of the board, from top to bottom, to GND, SWCLK, SWDIO, and 3V3 (or 3.3V) on the ST-Link. (i.e. GND should be at the top, next to the pin labeled 3VB, and 3V3 next to the pin labeled 3.3).

Maple Mini

Use the ST-Link pinout diagram from above. Connect 3V3 on the ST-Link to Vcc on the Maple Mini, GND to GND, SWCLK to 21, and SWDIO to 22.

Method

  1. Open the Arduino IDE as before, and go to File → Examples → A_STM32_Examples → Digital → Blink to load the example blinking LED code.
    • If you’re using a Blue Pill, change PB1 to PC13 where it appears in the code.
  2. Go to Tools → Board, and choose “Generic STM32F103C series” (even if you have a Maple Mini).
  3. Change Tools → Upload method to “STLink”.
  4. Hit Upload (the right-facing arrow on the main toolbar).
  5. Observe blinking LED.

Command-line flashing

In this section, connect up the boards as described in the STM32duino section earlier. These instructions only work for Linux at the moment.

You can compile a STM32duino sketch into a flashable binary using Sketch → Export compiled binary in the IDE; the binary will be placed in your sketch folder. We’ll cover how to create programs without the STM32duino IDE in a later blog post.

All of the tools required in this section can either be installed from your Linux distribution’s package manager, or you can install the STM32duino IDE, which contains a copy in hardware/Arduino_STM32-master/tools/linux/ - stm32flash is found in stm32flash/, dfu-util in dfu-util/ and st-{info, probe} in stlink/. Depending on your installation, Arduino_STM32-master may be called something else (it may not have the -master suffix, for example).

Via serial adapter

On Arch Linux, install the stm32flash package from the AUR. Similar packages may be available for other distributions.

Flash a binary

$ stm32flash -g 0x8000000 -b 115200 -w BINARY /dev/ttyUSB0

…writes BINARY to the flash memory of the board connected to the /dev/ttyUSB0 serial adapter. You might need to change /dev/ttyUSB0 to, for example, /dev/ttyACM0; look in dmesg to figure out what your serial adapter is called.

On Arch Linux, install the stlink package (# pacman -S stlink). Similar packages may be available for other distributions.

As discussed earlier, the ST-Link also allows you to perform live debugging.

Check connections

$ st-info --chipid

Should return a non-zero value (mine says 0x0410). If it returns nothing or 0x0000, check that everything is connected correctly.

If you have a newer version of the stlink tools, st-info --probe may also tell you some interesting information.

Flash a binary

$ st-flash write BINARY 0x8000000

…writes BINARY to the flash memory of the connected board.

GUI

If you installed the stlink package, you might also be able to run

$ stlink-gui

to get a neat GUI interface for doing things. Nifty!

Via USB

On Arch Linux, install the dfu-util package (# pacman -S dfu-util). Similar packages may be available for other distributions.

There are two ways to upload something via USB: you either cave in and use the maple_upload script from the STM32duino tools, which saves you a lot of pain, or you decide to only use dfu-util.

Using maple_upload

The maple_upload script lives in hardware/Arduino_STM32-master/tools/linux, and is invoked like so:

$ ./maple_upload ttyACM0 2 1eaf:0003 BINARY

You might need to replace ttyACM0 with ttyUSB0 if your board or Linux kernel is different to mine. This uploads BINARY to the board, resetting it properly before and after to make everything work with minimal stress.

I still don’t know why 2 is specified (for an option called altid), but it works.

Using dfu-util

To do this, you need to have one hand on the RESET button and one hand on the Enter key, with this command pre-typed. You might have to try it a number of times before it works, and you might also need to try unplugging & reinserting the USB cable.

$ dfu-util -d 1eaf:0003 -a 2 -D BINARY -R

This uploads BINARY to the board, and doesn’t do much else. You’ll want to unplug and reinsert the board after uploading, otherwise future uploads might fail for some mysterious reason.

Burning the bootloader

This is required if you want to upload via USB with the STM32duino IDE. You’ll need to use one of the command-line methods above (apart from USB, of course).

Blue Pill

Download generic_boot20_pc13.bin, and flash it to the board using one of the command line flashing methods. To recap, that means connecting up a serial adapter or ST-Link as described in the STM32duino section, then running the relevant command from the Command line flashing section, replacing BINARY with the path to generic_boot20_pc13.bin.

For example:

$ wget https://github.com/rogerclarkmelbourne/STM32duino-bootloader/raw/master/binaries/generic_boot20_pc13.bin # download the bootloader

For ST-Link:
$ st-flash write generic_boot20_pc13.bin 0x8000000
For a serial adapter connected to /dev/ttyUSB0:
$ stm32flash -g 0x8000000 -b 115200 -w generic_boot20_pc13.bin /dev/ttyUSB0

Maple Mini

As above, except you’ll need maple_mini_boot20.bin instead.

Conclusion

That’s pretty much all I have to say about the basics of working with STM32 boards, hardware-wise. In future blog posts, we’ll explore how to write programs with toolchains more interesting than just the STM32duino IDE - for instance, how to use Rust and its growing embedded ecosystem, or perhaps how to use the more traditional C HAL libraries to get a bit closer to the metal. Thanks for reading!

(If you’d like to be notified of new posts, there’s a RSS link below in the footer!)

December 18, 2018

David Wilson (dw)

Mitogen comes to Opsmop December 18, 2018 04:00 PM

Last week Mitogen gained an exciting user in the form of Opsmop, a brand new automation tool undergoing heavy development from Michael DeHaan, the original creator of Ansible.

Opsmop should be conceptually familiar to infrastructure folk, differentiating itself with a clean and more expressive default syntax (plain Python!), a model-driven internal design offering stronger possibilities for pre-run validation, decoupling of resources from their implementation, and now a push mode that exploits unique zero deploy, connection proxying and message routing capabilities available with Mitogen.

Approach

Opsmop uses a Mitogen remote function call to trigger deployments via SSH without having first been installed on remote hosts, transferring Python modules alongside the user's role definitions on demand, without writing temporary files to disk, and gaining a message bus that provides a sound basis for any future functionality or communication pattern. File transfer reuses this bus alongside RPCs, with one network round-trip per file, offering a big performance improvement for small files compared to scp or sftp.

Unlike in Mitogen for Ansible, most role evaluation in Opsmop already runs remotely, including logic like deciding which task to run next, or whether to fetch a file from the controller, eliminating a major cause of latency and bottlenecking. On the controller, Mitogen's scatter/gather functionality is used to schedule evaluations from a single process, streamily processing completions and progress updates sent by tasks as they occur, with file transfer running on demand in the background, and without the need to fork per-task workers or threads.

Opsmop offloads much more processing than the controller-heavy Ansible while maintaining its agentless simplicity. It has rich online communication with a privileged coordinator as in agentful designs like Salt, Chef, Puppet, or mgmt, yet avoids the versioning and operational hassle of preconfigured agents and servers running 24/7, or external dependencies like etcd and ZeroMQ. It avoids any need for target-local configuration as in purely pull-based designs like ansible-pull or chef-solo, while avoiding pull's pitfalls of lacking a coordinator, such as the inability to sequence tasks across machines, share expensive intermediate steps, or selectively grant access to secrets.

From Mitogen's perspective, Opsmop is exciting because it makes heavier use of network import than previous consumers – most of the program really runs on the remote, striking truer to the library's design promise than the Ansible extension. I worried this might reveal lurking nasties, but except for some noisy logs and unintuitive behaviour, so far no showstoppers were hit.

Lessons Learned

This integration provided invaluable perspective, revealing problems and misconceptions fresh eyes encounter with the library, and helpful sanity checks that are currently missing. The experience supplied all the fuel for a good Getting Started guide, a task delayed for over a year mostly due to having no real clue how to approach it.

Opsmop benefits from planned features like asynchronous connect, along with plenty of scope for improvement unique to it, and so there is renewed motivation to get Mitogen's user-reported tickets under control in preparation for another round of development.

Finally it is painfully clear that Mitogen's APIs are fairly nasty on initial approach. As a result, I've begun to think about a better Session API to wrap common tasks in a more intuitive interface.

Conclusion

Despite adopting a complex young library, progress was quick and involved minimal support. From a fresh user with no background to a functionally complete integration only took around a week, an experience that was extraordinarily refreshing to observe, and a candid insight into the origins of the most popular automation tool around.

Opsmop is shaping up to be an excellent option in the automation space, and by delegating connection handling to a library expressly designed for it, enjoys easy access to already-tested functionality like automagic proxying, Docker, Kubernetes and FreeBSD Jail connections, forming a virtuous cycle of free fixes and exciting improvements long into the future.

Until next time!

Just tuning in?

December 17, 2018

Pete Corey (petecorey)

Optional Notes and Exact Pitches in Chord December 17, 2018 12:00 AM

Currently my Elixir-powered Chord project does a lot of really cool things. It can generate a huge number of guitar chords given a set of notes you want included in those chords. It also computes the voice leading distance and fingering distance between those chords, which let’s us map out “ideal” chord progressions.

While this functionality is awesome in and of itself, it’s missing a few key features that would bring it to the next level.

Traditionally, musicians quickly learn song with the help of lead sheets. A lead sheet consists of a melody laid out over a set of chords. It’s up to the musician to interpret and play those chords with the given melody in a way that makes sense for both the player and the listener.

I want Chord to be able to generate possible interpretations of a lead sheet by giving us chord progressions that include optional notes and specific melody notes.

Supporting Optional Notes

It may be surprising to hear, but often times many of the notes that make up a chord are entirely optional!

For example, if I’m playing a Cmaj7 chord, which is made up of the root of the chord, the third, the fifth, and the major seventh, it’s usually acceptable to omit the fifth of the chord. The fifth usually just serves to add harmonic stability to the root note, and isn’t necessary to convey the color of the chord to the listener.

The ability to mark a note as optional drastically expands the possible set of chords we can generate for a given set of notes. For each optional note, we need to generate all of the possible chords that include that note, all of the possible chords that do not include it, and merge the results together.

Let’s update our Chord.Voicing module to do that now.

Within our Chord.Voicing module is a function, all_note_sets/1 that takes a set of notes, and returns a list of all possible “note sets” that can be spread across the strings of the guitar to build chords.

A note set is really just a collection of notes we want to play. For example, if we’re trying to play a Cmaj7 with an optional fifth, some of our note sets might look like this:


[[0, 4, 11],            # root, third, seventh
 [0, 4, 11, 0],         # root, third, seventh, root
 [0, 4, 11, 4],         # root, third, seventh, third
 [0, 4, 11, 11],        # root, third, seventh, seventh
 [0, 4, 11, 7],         # root, third, seventh, fifth
 [0, 4, 11, 7, 0],      # root, third, seventh, fifth, root
 ...
 [0, 4, 7, 11, 11, 11]] # root third, seventh, seventh, seventh, seventh

Notice that the smallest note set is the set of all three required notes. Also note that after those first three required notes is every possible permutation of every possible note in the chord, required and optional notes included.

We can implement this fairly easily in our all_note_sets/1 function. Let’s start by filtering the provided set of notes down to just the required notes:


required_notes =
  Enum.filter(notes, fn
    {:optional, note} -> false
    _ -> true
  end)

We’ll assume that optional notes are keyword tuples with :optional as the first element and the actual note as the second value. Require notes are simply bare note values.

Next, let’s filter notes down to the list of just optional notes:


optional_notes =
  Enum.filter(notes, fn
    {:optional, note} -> true
    _ -> false
  end)

Finally, let’s get a list together of all possible notes, optional and required included:


all_notes =
  Enum.map(notes, fn
    {_, note} -> note
    note -> note
  end)

Now that we’ve put our ducks in a row, generating all of our possible note sets if fairly straight forward.

We know that every note set will start with our set of required notes. That means that the length of each note set will range in length from the length of the required notes to 6, the number of strings on a guitar:


length(required_notes)..6

We also know that after the set of required notes the remaining space in the note set will be filled by every permutation of all possible notes (allowing repetition):


Permutation.generate(all_notes, length - length(required_notes), true)

We can loop over each of these sets of values and combine the results in a list comprehension to come up with our final list of note sets:


for length <- length(required_notes)..6,
    tail <- Permutation.generate(all_notes, length - length(required_notes), true) do
  required_notes ++ tail
end

Supporting Exact Pitches

Once we’ve built our note sets, we need to translate them into actual chords. Our Chord.Voicing module does this with the help of the all_notes/3 function, which takes a single note from our note set and finds all possible locations on the fretboard where that note can be played.

As we talked about in a previous article, it does this by building a complete fretboard and then filtering out, or sieving, any notes on the fretboard that aren’t the note we’re trying to play.

The original code that decided if the provided target_note matched the note at the given fret (index) and string looked something like this:


if rem(note, 12) == target_note do
  {string, index}
else
  nil
end

If the pitch class of the note (rem(note, 12)) matches our target_note, add the current string and fret to the list of tuples to be returned by our all_notes/3 function.

This solution assumes that all of the notes in our note sets are pitch classes, or values between 0 and 11. If we’re looking for a C and our target_note is 0, it will match on any octave of C it finds across the fretboard.

We can modify this solution to support exact pitches with minimal effort. If we assume that exact pitches will be passed in through the target_note parameter just like pitch classes (as plain numbers), we can add a fallback check to our condition that checks for exact equality:


cond do
  rem(note, 12) == target_note -> {string, index}
  note == target_note -> {string, index}
  true -> nil
end

If the pitch class of the current note doesn’t match our target_note, the untouched value of note still might. For example, if we’re looking specifically for a middle C (60), this condition would match on only those exact pitches, and not any higher or lower octaves of C.

Final Thoughts

Our Chord.Voicing module now supports building chords out of note sets that include both optional notes and exact pitches. We’re one step closer to modeling lead sheets!

As an interesting aside, when I started this refactor, I noticed that the original implementation of all_note_sets/1 was completely wrong. I’m not sure what was going through my mind when I wrote that first version, but it was only returning a small subset of all possible note sets. Equipped with the new implementation, Chord is generating many times the number of possible chords for us to play with.

Be sure to check out the entire Chord project on Github, and stay tuned for more updates and experiments.

Bogdan Popa (bogdan)

Try Firefox December 17, 2018 12:00 AM

Since Microsoft officially announced that they will switch Edge’s rendering engine to Chromium, many people have written about how this poses a danger to the future of the web. I’m not going to repeat those same arguments, as I feel others have done a good job of it. What I want to do is urge you to try Firefox for a couple of days this week. That’s it. Give it a try.

December 16, 2018

Ponylang (SeanTAllen)

Last Week in Pony - December 16, 2018 December 16, 2018 03:58 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, our users’ mailing list or join us on IRC.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

jfb (jfb)

Movie Review - The Meg (2018) December 16, 2018 12:00 AM

Untitled code{white-space: pre-wrap;} span.smallcaps{font-variant: small-caps;} span.underline{text-decoration: underline;} div.column{display: inline-block; vertical-align: top; width: 50%;}

There is a lot that I should like about this movie:

  1. Jason Statham (including a hilariously unnecessary shot of his eight? 10? 12? pack);
  2. the strong international cast (although I can’t see Cliff Curtis and not get the residual Once Were Warriors creeps);
  3. it’s a Chinese/US coproduction, with all of the bi-cultural rah-rah silliness that that entails;
  4. the titular creature is a GIANT RELICT SHARK, I mean, precisely in my wheelhouse.

But as it happens, this movie stinks. It’s stupid, of course, but that’s expected. The real sin is that it’s boring. Everything is telegraphed, and the movie moves on the rails, but a stupid movie often does. It’s just so totally lifeless, so that where a movie about a 100’ long Carcharodon should feel like a Toho Studios joint, featuring humanity helpless in the face of some unstoppable rubber suit monster; it’s really a lot more like an episode of Law and Order, a lame and stilted procedural.

I give this movie two stars – one for the shark, and one for the Chinese co-production.

Rating: ★ ★

December 15, 2018

Indrek Lasn (indreklasn)

How to use destructuring and arrow functions to improve your JavaScript code December 15, 2018 05:55 PM

Photo by Fabian Grohs on Unsplash

Without further ado, here are some everyday “tricks” to make your Javascript code much more readable.

Destructuring

Take this (intentionally) deeply nested object. Reading the age of a mammal would lead us to a very verbose and repetitive code.

Instead of doing that, we can use shorten our code by a huge margin with object destructing.

Object destructuring

Destructing bear and deer

Much better and less verbose. Note that we’re using constants since we don’t want to change the age. But think about it: age is not a constant… once per year in fact the age does increment.

If we declare it as a constant and try to change it later — we get an error. That’s the point of a constant… it’s immutable.

To play around this (intentional) restriction, we can use the let keyword (mutable) instead:

Cool it works. Let’s try deconstructing using both let and const keywords. Give the bear and deer a name. Now — usually, the name of an individual does not change too often — if ever — thus we can treat it as immutable.

We can also apply deconstruction on arrays, like so:

Array deconstruction

Cool huh — but there’s a lot more to it!

I recommend reading more about the ups and downs about deconstucting — Below you can find a finely detailed chapter about destructuring .

http://exploringjs.com/es6/ch_destructuring.html

Arrow functions

Arrow functions are a cool way to shorten your code — they’re not quite the same as traditional functions. Arrow functions are lexically scoped — we’ll get into that in a moment.

Take our previous array of animals — how would you loop over every animal and console.log the each animal?

Nothing too special, traditional forEach loop. For each animal we log out the animal. Let’s refactor our forEach loop to use arrow functions.

Much cleaner and less verbose. The less code we have to write, less we have to maintain in the future. A good rule of thumb is to write less code, but not too clever code.

Here’s a more complex example:

This pattern is called currying — we return a function inside a function. Most notable example which utilizes currying would be the connect()() function from Redux.

note: ** is the exponentiation operator and is the equivalent of Math.pow

The multiplyAndAdd function explained:

  • multiplyAndAddtakes a number, returns the base to the exponent power (multiplication)
  • Adds a number to the exponent

multiplyAndAdd(2)(4) is the same as 2x2 + 4

Let’s write the same functionality using arrow functions.

Less verbose for sure — but we can do even more.

Woah! We’re down from 6 lines to just 1 line of code.

Caveat #1: Notice there are no return statements. Normally the return statements are required to return values from functions. Arrow functions call return implicitly, as long as there are no braces. If we use arrow functions with braces, we have to call the return explicitly.

If this is new to you, check out this awesome article about return rules.

Caveat #2: The this keyword works differently with arrow functions.

Take this example: we have a person object — inside the object we have two properties, the name of the person and a function which returns the name of the person

If we call the sayName method, the first this points to the person object but the second this is undefined?

Note: The second this actually points to the window object but since we’re using Quokka, there is no window object.

Second this points to the window

Why does this happen? Just to point out, this is a very common interview question as well — answering this shows good understanding of the language mechanics. Check out this StackOverflow answer for an in-depth answer.

Here’s a common workaround for this with regular functions.

Arrow functions are lexically scoped, and its value within the arrow function is determined by the surrounding scope — thus they don’t need this work-around.

If you found this article to be useful, please give us some claps so more people can see it.

You can reach out to me with questions on Twitter.

Indrek Lasn (@lasnindrek) | Twitter

Here are some articles you might enjoy:


How to use destructuring and arrow functions to improve your JavaScript code was originally published in freeCodeCamp.org on Medium, where people are continuing the conversation by highlighting and responding to this story.

December 14, 2018

Derek Jones (derek-jones)

Growth of conditional complexity with file size December 14, 2018 05:35 PM

Conditional statements are a fundamental constituent of programs. Conditions are driven by the requirements of the problem being solved, e.g., if the water level is below the minimum, then add more water. As the problem being solved gets more complicated, dependencies between subproblems grow, requiring an increasing number of situations to be checked.

A condition contains one or more clauses, e.g., a single clause in: if (a==1), and two clauses in: if ((x==y) && (z==3)); a condition also appears as the termination test in a for-loop.

How many conditions containing one clause will a 10,000 line program contain? What will be the distribution of the number of clauses in conditions?

A while back I read a paper studying this problem (“What to expect of predicates: An empirical analysis of predicates in real world programs”; Google currently not finding a copy online, grrr, you will have to hassle the first author: durelli@icmc.usp.br, or perhaps it will get added to a list of favorite publications {be nice, they did publish some very interesting data}) it contained a table of numbers and yesterday my analysis of the data revealed a surprising pattern.

The data consists of SLOC, number of files and number of conditions containing a given number of clauses, for 63 Java programs. The following plot shows percentage of conditionals containing a given number of clauses (code+data):

Percentage of conditions containing a given number of clauses in 63 large Java programs.

The fitted equation, for the number of conditionals containing a given number of clauses, is:

conditions = 3*slen^pred e^{10-10pred-1.8 10^{-5}avlen^2}

where: slen={SLOC}/{sqrt{Number of Files}} (the coefficient for the fitted regression model is 0.56, but square-root is easier to remember), avlen={SLOC}/{Number of Files}, and pred is the number of clauses.

The fitted regression model is not as good when slen or avlen is always used.

This equation is an emergent property of the code; simply merging files to increase the average length will not change the distribution of clauses in conditionals.

When slen = e^{10} = 22,026, all conditionals contain the same number of clauses, off to infinity. For the 63 Java programs, the mean slen was 2,625, maximum 11,710, and minimum 172.

I was expecting SLOC to have an impact, but was not expecting number of files to be involved.

What grows with SLOC? Number of global variables and number of dependencies. There are more things available to be checked in larger programs, and an increase in dependencies creates the need to perform more checks. Also, larger programs are likely to contain more special cases, which are likely to involve checking both general and specific values (i.e., more clauses in conditionals); ok, this second sentence is a bit more arm-wavy than the first. The prediction here is that the percentage of global variables appearing in conditions increases with SLOC.

Chopping stuff up into separate files has a moderating effect. Since I did not expect this, I don’t have much else to say.

This model explains 74% of the variance in the data (impressive, if I say so myself). What other factors might be involved? Depth of nesting would be my top candidate.

Removing non-if-statement related conditionals from the count would help clarify things (I don’t expect loop-controlling conditions to be related to amount of code).

Two interesting data-sets in one week, with 10-days still to go until Christmas :-)

Pete Corey (petecorey)

Advent of Code: Chronal Charge December 14, 2018 12:00 AM

Advent of Code’s day eleven puzzle asks us to compute a three hundred square grid of values and to find the three by three sub-grid that contains the highest sum of values. In my heart of hearts I knew that this would be a problem well suited to the J programming language.

I started by working on a verb to break a grid into a number of sub-grids of a given size. This reminded me quite a bit of Elixir’s Enum.chunk_every/4 function, so I decided to name it accordingly:

    grid =. 5 5 $ i. 25
    grid
 0  1  2  3  4
 5  6  7  8  9
10 11 12 13 14
15 16 17 18 19
20 21 22 23 24
    chunk_every =. [ <\"2 [: |:"2 [: > <\ 
    3 chunk_every grid
┌────────┬────────┬────────┐
│0 5 10  │1 6 11  │2 7 12  │
│1 6 11  │2 7 12  │3 8 13  │
│2 7 12  │3 8 13  │4 9 14  │
├────────┼────────┼────────┤
│5 10 15 │6 11 16 │7 12 17 │
│6 11 16 │7 12 17 │8 13 18 │
│7 12 17 │8 13 18 │9 14 19 │
├────────┼────────┼────────┤
│10 15 20│11 16 21│12 17 22│
│11 16 21│12 17 22│13 18 23│
│12 17 22│13 18 23│14 19 24│
└────────┴────────┴────────┘

To be totally transparent, I originally came up with this verb by tinkering in the REPL, and converted it into a tacit verb after the fact.

Now that we have chunk_every, we can define a few more needed values, like our initial grid, our grid size, and our grid serial number:

    size =. 300
    grid =. (size,size) $ i. size * size
    grid_serial_number =. 9424

The puzzle tells us how to convert our initial grid’s x/y coordinates into “fuel cell values”. Let’s write an init verb that takes our initial verb and calculates and populates those values:

    init =. 3 : 0
      'cy cx' =. (0,size)#:y
      rack_id =. cx + 10
      power =. rack_id*cy
      power =. power + grid_serial_number
      power =. power * rack_id
      power =. 10 | power <.@% 100
      power =. power - 5
      power
    )

Now we’re ready to start. We’ll begin by generating our grid of fuel cells:

    grid =. init"0 grid

Next, we’ll break our grid into three by three chunks:

    chunks =. 3 chunk_by grid

Once we have our sub-grids, we’ll calculate the sum of each and flatten that into a one dimensional array of sums:

    flat_chunks =. , +/"1 ,"2 > chunks

And find the maximum sub-grid sum:

    max =. >./ flat_chunks

And the corresponding index of that maximum sum:

    highest =. flat_chunks i. >./ flat_chunks

Finally, we can turn that index into a pair of x/y coordinates:

    |. (0,(size-2))#:highest

This is the answer to our puzzle. Victory!

Our fuel cells, visualized.

Just for fun, we can check out what our newly initialized fuel cell matrix looks like with the help of viewmat. We can see some cool Moiré patterns in the resulting visualization as a side effect of our calculations.

Part Two

Part two wants us to vary the size of our sub-grids, and find the sub-grid size, and x/y coordinate pair that has the most amount of available fuel, or the highest sum.

My first instinct was to run chunk_by multiple times against chunk sizes ranging from 1 to 300:

    chunks =. (grid&(<@:chunk_by~))"0 (1 + i. size)

I wrote a verb to count the amount of available fuel within each new set of sub-grids, and ran that against all of the sub-grid sets I was generating:

    count =. 3 : 0
      chunks =. > y
      chunk_size =. # 0 0 {:: chunks
      flat_chunks =. , +/"1 ,"2 > chunks
      max =. >./ flat_chunks
      highest =. flat_chunks i. >./ flat_chunks
      coord =. |. (0,(size-(chunk_size-1)))#:highest
      max;coord,chunk_size
    )

    counts =. count"_1 chunks

From there, I could grab the best score of all of the sub-grid sizes, find the max, and return a tuple of that winning sub-grid’s size, and its x/y coordinate:

    scores =. > 0 {"1 counts
    idx =. scores i. >./ scores
    1 {:: idx { counts

Unfortunately, this turned out to be too slow of a solution.

Thankfully, there were some patterns to be found that let us speed things up considerably! If we plot our counts with a max grid size of 10, 50, and 100, we can see that our score always peaks almost immediately. We don’t need to test larger sub-grid sizes because we know our answer won’t be there.

A plot of our sub-grid size vs our maximum fuel availability.

Let’s change our solution to only check sub-grid sizes from one to twenty:

    chunks =. (grid&(<@:chunk_by~))"0 (1 + i. size)

And just like that, we get an answer almost immediately.

Thanks J!

Advent of Code: Marble Mania December 14, 2018 12:00 AM

Advent of Code day nine asks us to play a game. The game is played by placing marbles, or numbers, around a circle, or a circular list. If you play a marble that’s a multiple of 23, you keep that marble, and the marble seven marbles behind your current position. We’re to figure out the winning player’s score after thousands of rounds of this game.

As usual, we start things off by pulling in our input and massaging it into a workable form. Let’s kick things off by defining some replacements:

    replacements =. 'players; last marble is worth';''
    replacements =. replacements;'points';''

Next let’s load our puzzle input, execute our replacements, and parse the resulting numbers:

    path =.  '/Users/pcorey/advent_of_code_2018/day_09/input'
    NB. Load the file, remove everything that's not a number, and assign.
    'players highest_marble' =. ". replacements rplc~ (1!:1) < path

Some destructing helps us pull out the number of players and the number of turns we should play out, or the highest marble to be played.

Now let’s write a turn verb that takes the current state of the board, the current player, the current marble being played, and all players’ scores. We’ll place our marble in the correct position in the circular board, potentially update the current player’s score, and return our modified state:

    turn =. 3 : 0
      NB. Destructure my arguments.
      circle =. 0 {:: y
      player =. players | 1 {:: y
      marble =. 2 {:: y
      scores =. 3 {:: y

      if. 0 = 23 | marble do.
        NB. Grab the current player's current score.
        score =. player { scores
        NB. Add the marble they would have placed to their score.
        score =. score + marble
        NB. Rotate circle 7 counter-clockwise
        circle =. _7 |. circle
        NB. Add the marble we landed on to the player's score.
        score =. score + {. circle
        NB. Update the scores list with the player's new score.
        scores =. score (player}) scores
        NB. Remove the marble we landed on.
        circle =. }. circle
      else.
        NB. 
        circle =. marble , 2 |. circle
      end.

      NB. Return our updates arguments.
      circle;(player + 1);(1 + marble);scores
    )

It turns out modeling a circular repeating list is easy using J’s “rotate” (|.) verb.

Because we’re returning the same data from turn as we’re passing in, we can use the “power” verb (^:) to repeatedly apply turn to an initial set of inputs:

    scores =. 3 {:: (turn^:highest_marble) 0;0;1;(players $ 0)

Applying turn highest_marble times to our initial state (0;0;1;(players $ 0)) gives us a final list of player scores.

We find out final answer by returning the maximum score:

    >./scores

Part Two

Part two asks for the highest player’s score if we continue playing for highest_marble*100 turns. This turned out to be an incredibly difficult problem for me to solve using J.

The problem here is performance. Playing two orders of magnitude more turns increases the runtime of our first solution from a few seconds to significantly longer than I’m willing or capable of waiting. We need a better solution. The obvious solution is to use a data structure that’s better suited to rotations, insertions, and removals than a plain array. A doubly linked, circular linked list would be perfect here.

I started researching how to implement a doubly linked list in J. It turns out that this type of low-level data manipulation goes against the grain of J’s intended use. Apparently J code is intended to be descriptive, while the interpreter does the heavy lifting of optimization. Unfortunately, it wasn’t doing a great job with my part one solution.

I was hell bent on building a doubly linked list. My first implementation was modeled after this (seemingly hacky) exploitation of J “locatives”, or namespaces:

    init =. 3 : 0
      l =. <": y
      v__l =. y
      n__l =. 0
      p__l =. 0
    )

    insert =. 4 : 0
      head =. x
      l =. <": y
      nl =. <": head
      pl =. <": p__nl
      v__l =. y
      n__l =. n__pl
      p__l =. p__nl
      n__pl =. y
      p__nl =. y
      v__l
    )

    remove =. 3 : 0
      l =. <": y
      nl =. <": n__l
      pl =. <": p__l
      n__pl =. n__l
      p__nl =. p__l
      v__nl
    )

    rotate_cw =. 3 : 0
      l =. <": y
      n__l
    )

    rotate_ccw =. 3 : 0
      l =. <": y
      p__l
    )

Unfortunately, while this was faster than my original solution, it was still too slow to give me my answer in a reasonable amount of time.

My next attempt led me directly allocating, reading, and writing my own data structures directly into memory using J’s mema, memr, and memw utility verbs. At this point I was basically justing writing C code with weird syntax:

    init =. 3 : 0
      NB. Allocate space for a new node.
      addr =. mema 8*type
      NB. Write the value, prev ptr, and next ptr.
      res =. (0,addr,addr) memw (addr,0,3,type)
      addr
    )

    insert =. 4 : 0
      'v pp pn'    =. memr x, 0,3,type
      'pv ppp ppn' =. memr pp,0,3,type
      'nv npp npn' =. memr pn,0,3,type

      NB. Allocate and write new node.
      addr =. mema 8*type

      if. *./ x = pp , pn do.
        NB. Only 1 element in the list.
        (y,x,x) memw addr,0,3,type
        (v,addr,addr) memw x,0,3,type
      else.
        if. pp = pn do.
          NB. Only 2 elements in the list.
          (y,pn,x) memw addr,0,3,type
          (v,addr,pn) memw x,0,3,type
          (nv,x,addr) memw pn,0,3,type
        else.
          NB. Normal insertion case.
          (y,pp,x) memw addr,0,3,type
          (v,addr,pn) memw x,0,3,type
          (nv,x,npn) memw pn,0,3,type
          (pv,ppp,addr) memw pp,0,3,type
        end.
      end.

      addr
    )

    remove =. 3 : 0
      'v pp pn' =. memr y,0,3,type
      'pv ppp ppn' =. memr pp,0,3,type
      'nv npp npn' =. memr pn,0,3,type

      NB. Free the current node.
      memf y

      NB. Update neighbor nodes
      (pv,ppp,pn) memw pp,0,3,type
      (nv,pp,npn) memw pn,0,3,type

      pn
    )

    rotate_cw =. 3 : 0
      'v pp pn' =. memr y,0,3,type
      pn
    )

    rotate_ccw =. 3 : 0
      'v pp pn' =. memr y,0,3,type
      pp
    )

Mercifully, this solution was much faster than my previous two. I was able to find my answer in roughly two minutes.

Check out the full source for all three solutions on Github, if you’re curious.

Notes

  • Rotate (|.) is awesome.
  • J is meant to be descriptive?
  • I needed to allocate more than 12 bytes to comfortably fit three integers. But only sometimes. Why?
  • You can time things using the 6!:2 foreign.

December 13, 2018

Unrelenting Technology (myfreeweb)

Cool ARM laptop news that I missed somehow: the cause of custom EFI binaries... December 13, 2018 03:15 PM

Cool ARM laptop news that I missed somehow:

Vincent (vfoley)

purp, an Emacs theme of few colors December 13, 2018 01:02 AM

A few years ago, I asked on Reddit whether anyone had recommendations for a theme that used only a few colors. I found that too many colors—what many refer to as a Christmas tree—was not helping me understand code better, but in fact was distracting me. I also thought that turning off syntax highlighting entirely was too radical a solution. I wanted a theme that would just highlight a few, well-chosen elements. Unfortunately, most of the minimal themes that I found were monochrome, which is not what I was looking for.

So I began working on a theme of few colors that I’ve been using and tweaking for a while now. I’ve finally decided to make it available on Github and publish it to MELPA. You can install purp by invoking the command M-x package-install RET purp-theme. A light version called purp-light is also included in the package.

I have not gone and modified the color of every face in every mode, just the ones that I encountered daily in my work. In programming modes, you’ll see the following colors:

  • Purple for function definitions
  • Green for comments
  • Orange for string literals
  • Yellow on red for dangerous stuff (e.g., the unsafe keyword in Rust)
  • White (or black) for everything else

The colors for function definitions and comments makes them easy to find and the color for strings helps to spot an un-escaped double quote.

I hope you like purp, there’s a lot of work left to be done, including how to not have it activate as soon as it’s installed, which is a really bad first impression.

Pete Corey (petecorey)

Advent of Code: The Stars Align December 13, 2018 12:00 AM

The day ten Advent of Code challenge gives us a list of points and velocity pairs. Each point is updated by its corresponding velocity every second. At some point in time, these points will converge and spell out a message. Our task is to find that message using the J programming language!

As usual, we’ll start by loading our input and massaging it into a form we can work with:

    replacements =. 'position=<';'';',';'';'> velocity=<';' ';'>';'';'-';'_'
    path =.  '/Users/pcorey/advent_of_code_2018/day_10/input'
    input =. ". > cutopen replacements rplc~ (1!:1) < path

I’m using a trick I learned from zxw on Twitter to easily replace and reformat the data before passing it into the “numbers” verb (".).

Next let’s write a tick verb that updates each point with its corresponding velocity. We’ll also keep track of the maximum spread between Y coordinate values and return that as the second value in a boxed tuple along with our next set of points and velocities:

    tick =. 3 : 0
      input =. 0 {:: y
      prev =. 1 {:: y
      next =. +/"1 |:"2 (2 2 $ ])"1 input
      max =. >./ 1 {"1 next
      min =. <./ 1 {"1 next
      diff =. | max - min
      if. diff < prev do.
        (next (0 1})"1 input);diff
      else.
        y
      end.
    )

Notice that if applying the next tick results in a lower spread, we return the new values. Otherwise, we return the old values. This means we can “converge” (^:_) on a result for this verb. The result we converge on will be the set of points with the lowest spread in the vertical dimension.

It turns out that this is our answer!

We can use J’s viewmat library to quickly visualize our answer (after some more rotating and massaging):

    load 'viewmat'

    to_mat =. 3 : 0
      min_x =. <./ 0 {"1 y
      min_y =. <./ 1 {"1 y
      max_x =. >./ 0 {"1 y
      max_y =. >./ 1 {"1 y
      coords =. 0 1 {"1 y
      coords =. (min_x,min_y) -~"1 coords
      mat =. ((1 + | max_y - min_y),(1 + | max_x - min_x)) $ 0
      updates =. (<@:;/@:|.)"1 coords
      1 updates} mat
    )

    viewmat to_mat 0 {:: tick^:_ input;_

The stars align.

Part Two

Part two turned out to be a simple modification of our part one solution. The challenge asked us to return how many ticks we went through before we found our message.

All we need to do to figure this out is to add a third element to the tuple we pass in and out of tick that holds an incrementing count:

    tick =. 3 : 0
      input =. 0 {:: y
      prev =. 1 {:: y
      count =. 2 {:: y
      next =. +/"1 |:"2 (2 2 $ ])"1 input
      max =. >./ 1 {"1 next
      min =. <./ 1 {"1 next
      diff =. | max - min
      if. diff < prev do.
        (next (0 1})"1 input);diff;(count + 1)
      else.
        y
      end.
    )

    2 {:: tick^:_ input;_;0

The answer to our challenge is the value of this final count.

Noon van der Silk (silky)

The Quantum Computing Slack Channel December 13, 2018 12:00 AM

Posted on December 13, 2018 by Noon van der Silk

Just a quick note to introduce the quantum computing slack channel: Quantum Computing on Slack.

You can join by requesting an invite at this link: Quantum Computing Slack Invite.

Hope to see you there!

We’ll follow the Code of Conduct from SciRate.

December 12, 2018

Unrelenting Technology (myfreeweb)

The best Cloud-to-Butt replacement I've seen yet, thanks to this page: Butthole was released... December 12, 2018 12:52 PM

The best Cloud-to-Butt replacement I've seen yet, thanks to this page:

Butthole was released in 2016 for Firefox to make Buttflare captchas less painful

Pages From The Fire (kghose)

Tests and code coverage in Python December 12, 2018 03:30 AM

Not only is Python a nice language but it has always had a lot of tooling around it. I’ve always taken advantage of Python’s tooling around testing (okay, not always …) and recently I began to pay attention to code coverage again. Python makes it all so simple and delicious. I have used nose in …

Gokberk Yaltirakli (gkbrk)

Free Hotel Wifi with Python and Selenium December 12, 2018 02:33 AM

Recently I took my annual leave and decided to visit my friend during the holidays. I stayed at a hotel for a few days but to my surprise, the hotel charged money to use their wifi. In $DEITY‘s year 2000 + 18, can you imagine?

But they are not so cruel. You see, these generous people let you use the wifi for 20 minutes. 20 whole minutes. That’s almost half a Minecraft video.

If they let each device use the internet for a limited amount of time, they must have a way of identifying each device. And a router tells devices apart is by their MAC addresses. Fortunately for us, we can change our MAC address easily.

Enter macchanger

There is a really useful command-line tool called macchanger. It lets you manually change, randomize and restore the MAC address of your devices. The idea here is randomizing our MAC regularly (every 20 minutes) in order to use the free wifi over and over indefinitely.

There are 3 small commands you need to run. This is needed because macchanger can’t work while your network interface is connected to the router.

# Bring network interface down
ifconfig wlp3s0 down

# Get random MAC address
macchanger -r wlp3s0

# Bring the interface back up
ifconfig wlp3s0 up

In the commands above, wlp3s0 is the name of my network interface. You can find yours by running ip a. If you run those commands, you can fire up your browser and you will be greeted with the page asking you to pay or try it for 20 minutes. After your time is up, you can run the commands again and keep doing it.

But this is manual labor, and doing it 3 times an hour is too repetitive. Hmm. What’s a good tool to automate repetitive stuff?

Enter Selenium

First, lets get those commands out of the way. Using the os module, we can run macchanger from our script.

import os

interface = 'wlp3s0'

os.system(f'sudo ifconfig {interface} down')
os.system(f'sudo macchanger -r {interface}')
os.system(f'sudo ifconfig {interface} up')

After these commands our computer should automatically connect to the network as a completely different device. Let’s fire up a browser and try to use the internet.

d = webdriver.Chrome()
d.get('http://example.com')
d.get('https://www.wifiportal.example/cp/sponsored.php')

The sponsored.php URL is where I ended up after pressing the Free Wifi link, so the script should open the registration form for us. Let’s fill the form.

In my case, all it asked for was an email address and a full name. If there are more fields, you can fill them in a similar fashion.

num   = random.randint(0, 99999)
email = f'test{num}@gmail.com'

d.find_element_by_name('email').send_keys(email)
d.find_element_by_name('name').send_keys('John Doe\n')

This should fill the form and press enter to submit it. Afterwards, the portal asked me if I wanted to subscribe to their emails or something like that. Of course, we click Reject without even reading it and close the browser.

d.find_elements_by_class_name('reject')[0].click()
d.close()

After this, you should have an internet connection. You can either run the script whenever you notice your connection is gone, or put it on a cron job / while loop.

December 11, 2018

Andreas Zwinkau (qznc)

Waterfall December 11, 2018 12:00 AM

The "Waterfall" methodology was a historic accident and they knew it.

Read full article!

Alex Wilson (mrwilson)

Notes from the Week #12 December 11, 2018 12:00 AM

I was expecting a big parliamentary slap-fight today, but the Prime Minister had other ideas, so I’m going to write my week-notes instead — it’s a very short one because we’re about to start the new quarter, so most of my time is taken up with discussions that I can’t write about yet!

Two Things That Happened

One

There’s an art to enabling yourself to do things, at work and at home — and as with anything, there is a balance.

On one hand if you only ever do what you want then there’s going to likely be conflict with those around you. On the other hand if you never do what you want and put everyone else’s needs above your own then this is equally destructive but far more insidious.

Eventually your self-censuring will bottle up your own needs inside you and possibly explode in a very unhelpful way, depending on how you handle stress.

It’s been pointed out to me this week that I have a habit of doing the latter, and getting inevitably frustrated when I don’t get to attend to my needs for an entirely arbitrary and self-imposed reason.

I’ll be trying to keep an eye out for this in future and making an effort to be kinder to myself.

Two

This week Shift have been working with other teams to improve our department-wide business continuity plans — making our answers more efficient/better to questions like

“What do we do if the office is consumed in a tragic X-based accident?”

Where X can be:

  • Fire
  • Politics
  • Blancmange
  • Irony

It’s really felt like having multiple pots boiling on the hob which is great for the part of me that loves context switching and dashing from problem to problem but not great for my ability to generally keep calm and collected.

On the flipside, it was a great opportunity for us (Shift) to dive into the nitty-gritty with other teams and show them what we’re able to do — I’ve written a bit about building social capital in previous week notes and this is exactly what we’ve been doing this week.

Originally published at blog.probablyfine.co.uk on December 11, 2018.

December 10, 2018

Jeff Carpenter (jeffcarp)

Book Review: Mindset: The New Psychology of Success December 10, 2018 11:38 PM

This book is about two ways of thinking: the fixed mindset and the growth mindset. In the fixed mindset you’re a finished product. Expending any extra effort is unthinkable because supposedly you’re already perfect. Then there’s the growth mindset, which tells us the only way you learn is from mistakes, talent doesn’t get you very far, and the people who succeed are the ones who work the hardest.

Frederik Braun (freddyb)

logging with MOZ_LOG on the try server December 10, 2018 11:00 PM

Preamble

NB: This is mostly for my own public reference. I had written about this elsewhere in 2016 but when arriving at a similar problem, failed to reproduce this. You may skip the following section, if you're familiar the terminology in the title

what is MOZ_LOG?

MOZ_LOG is an environment variable Firefox developers can use to tell specific code sections to emit verbose (or very verbose) status messages for some if its inner workings. This is also called Gecko Logging.

what is the try server

The try server is a repository that allows you to submit code without actually checking it into the public repository. Pushes to try get run through all of our tests, which helps identifying problems and test failures before they are part of our code.

logging with MOZ_LOG on the try server

There is a test failure on Mac OS X, that I can hardly debug. As a first step, I'll push this to the try-server with more logging output enabled.

My test is a mochitest, so I modified testing/mochitest/runtests.py:

diff --git a/testing/mochitest/runtests.py b/testing/mochitest/runtests.py
index 45545b4..5afdffd 100644
--- a/testing/mochitest/runtests.py
+++ b/testing/mochitest/runtests.py
@@ -91,7 +91,7 @@ here = os.path.abspath(os.path.dirname(__file__))
 # Try run will then put a download link for all log files
 # on tbpl.mozilla.org.

-MOZ_LOG = ""
+MOZ_LOG = "nsDocShellLogger:4,CSPParser:4,CSPUtils:4,CSPContext:4,CSP:4"

And now we play the waiting game.

Derek Jones (derek-jones)

Impact of group size and practice on manual performance December 10, 2018 02:04 PM

How performance varies with group size is an interesting question that is still an unresearched area of software engineering. The impact of learning is also an interesting question and there has been some software engineering research in this area.

I recently read a very interesting study involving both group size and learning, and Jaakko Peltokorpi kindly sent me a copy of the data.

That is the good news; the not so good news is that the experiment was not about software engineering, but the manual assembly of a contraption of the experimenters devising. Still, this experiment is an example of the impact of group size and learning (through repeating the task) on time to complete a task.

Subjects worked in groups of one to four people and repeated the task four times. Time taken to assemble a bespoke, floor standing rack with some odd-looking connections between components was measured (the image in the paper shows something that might function as a floor standing book-case, if shelves were added, apart from some component connections getting in the way).

The following equation is a very good fit to the data (code+data). There is theory explaining why log(repetitions) applies, but the division by group-size was found by suck-it-and-see (in another post I found that time spent planning increased with teams size).

There is a strong repetition/group-size interaction. As the group size increases, repetition has less of an impact on improving performance.

time = 0.16+ 0.53/{group size} - log(repetitions)*[0.1 + {0.22}/{group size}]

The following plot shows one way of looking at the data (larger groups take less time, but the difference declines with practice), lines are from the fitted regression model:

Time taken (hours) for various group sizes, by repetition.

and here is another (a group of two is not twice as fast as a group of one; with practice smaller groups are converging on the performance of larger groups):

Time taken (hours) for various repetitions, by group size.

Would the same kind of equation fit the results from solving a software engineering task? Hopefully somebody will run an experiment to find out :-)

December 09, 2018

Vincent (vfoley)

Rust 2019 December 09, 2018 03:45 PM

(This is my response to the call for Rust 2019 Roadmap blog posts.)

In 2019, there are three areas where I would like to see the Rust community focus its efforts:

  • Improved compile times
  • A community effort to review crates
  • More “80% solutions”

Improved compile times

I already wrote about compile times in Rust (see Rust at Work—Two Paint Points and How to alleviate the pain of Rust compile times), and I don’t have much more to add. I just want to re-iterate that this issue is very important and that it should be given the attention it deserves.

The Rust compiler is much slower than the compilers of some “competing languages” such as Go, D, or Jai. As Jonathan Blow put it when he first announced that he was working on a programming language, long compile times are a constant source of friction for developers: they lengthen the write-test loop and they disrupt the programmer’s flow. The success of Go can partly be attributed to its quick compile times; it compiles fast enough that it feels like an interpreted language. (Of course, other factors contributed to Go’s success—an extensive standard library, a vibrant ecosystem, a friendly community, great tooling, good performance, etc.) I want new Rust programmers to not be turned off by Rust’s long compile times and I want veteran Rust programmers to iterate more quickly and wait less on rustc.

The situation is improving. Incremental recompilation has been merged and tools like sccache can speed up compilation in CI. Nicholas Nethercote detailed his experiments in making rustc faster. A new, upcoming backend, cranelift, appears to be faster than the current LLVM backend.

I think most Rust programmers realize that improved compile times would be beneficial to everyone, and I look forward to seeing improvements on that front in 2019.

A community effort to review crates

Rust would not be used nearly as much if it weren’t for Cargo and crates.io. The rich ecosystem of Rust is what made it go from an interesting programming language project to a tool that many now use in production. In 2019, I would like to see a code review effort from the Rust community. The goal of these reviews is to provide answers to two questions for anyone looking for a crate.

What are the quality crates? What are the best crates for making HTTP requests, for consuming from Kafka, or for manipulating dates? Are those crates reliable enough for business-critical services? Should I have concerns about their performance or security? Is the code easy to understand and modify? In a nutshell, how does one find quality crates?

At the moment, we try and decide if a crate is worth checking out based on word-of-mouth, number of downloads, number of GitHub stars, the name of the contributors, etc. Code reviews would be stronger indicators of quality, because someone actually read the code. On Reddit, Dawid Ciężarkiewicz announced crev, a language-agnostic tool for publishing and signing code reviews and forming webs of trust. A tool like that would be ideal for the crates ecosystem. At my place of work, we’ve reviewed a number of the crates that we use in production, to convince ourselves that they were worth using. We’d love to have a place to share that work with the reset of the community.

Is this crate compromised? In 2018, we heard a few stories (1, 2) about malicious packages in NPM, the Node.js package repository. I don’t think I’ve heard about any backdoors in a crate yet, but I see nothing that would make such attacks impossible. (We did have an issue back in October with squatting that actually led to a degradation of service.)

These attacks will undoubtedly become more common in the coming years. Some aspects of Cargo make these attacks more difficult: the immutability of crates and the ability to pin dependencies at a particular version (ideally, a versio that has been reviewed). Active code reviews would provide an extra line of defense against these attacks. They would also, hopefully, increase the overall quality of the most popular crates.

More “80% solutions”

Improved compile times and code reviews require time and effort, and are not likely to occur overnight. While we wait, I’d like to see more “80% solutions”—projects that perform a task well enough, but without being perfect. We are lucky to have some incredibly high-quality projects in the Rust ecosystem: serde, hyper, reqwest come to mind. They are also what I’d call 100% solutions: they offer all the features that users could possibly want and they have extremely good performance. They are also complex beasts, and that makes them more difficult to review. Some of these projects (e.g., hyper) also have many dependencies, which makes the review process even more difficult, and also increases compile times.

An example of an 80% project is miniserde: unlike serde, JSON is the only serialization format that miniserde supports, and some Rust datatypes like enums cannot be encoded. Miniserde has also 10x less code than serde (tokei reports 2379 lines of Rust code for miniserde vs. 25,205 for serde) so it’s much easier for a single person to do a full review of the code.

I’d love to see more such projects: a simple HTTP server for services that just need a /status endpoint; an HTTP client that can send simple requests and write the responses in a Write object.

Ponylang (SeanTAllen)

Last Week in Pony - December 9, 2018 December 09, 2018 02:41 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, our users’ mailing list or join us on IRC.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

December 08, 2018

Benjamin Pollack (gecko)

The Death of Edge December 08, 2018 01:16 AM

Edge is dead. Yes, its shell will continue, but its rendering engine is dead, which throws Edge into the also-ran pile of WebKit/Blink wrappers. And no, I’m not thrilled. Ignoring anything else, I think EdgeHTML was a solid rendering engine, and I wish it had survived because I do believe diversity is good for the web. But I’m not nearly as upset as lots of other pundits I’m seeing, and I was trying to figure out why.

I think it’s because the other pundits are lamenting the death of some sort of utopia that never existed, whereas I’m looking at the diversity that actually exists in practice.

The people upset about Edge’s death, in general, are upset because they have this idea that the web is (at least in theory) a utopia, where anyone could write a web browser that conformed to the specs and (again, theoretically) dethrone the dominant engine. They know this hasn’t existed de facto for at least some time–the specs that now exist for the web are so complicated that only Mozilla, with literally hundreds of millions of dollars of donations, can meaningfully compete with Google–but it’s at least theoretically possible. The death of Edge means one less browser engine to push back against Chrome, and one more nail in the coffin of that not-ever-quite-here utopia.

Thing is, that’s the wrong dynamic.

The dynamic isn’t Gecko v. EdgeHTML v. Blink v. WebKit. It’s any engine v. native. That’s it. The rendering engine wars are largely over: while I hope that Gecko survives, and I do use Firefox as my daily driver, that’s largely irrelevant; Gecko has lost by at least as much as Mac OS Classic ever lost. What does matter is that most people access the web via mobile apps now. It’s not about whether you like that, or whether I like that, or whether it’s the ideal situation; that’s irrelevant. The simple fact is, most people use the web through apps, period. In that world, Gecko v. Blink v. WebKit is an implementation detail; what matters is the quality of mobile app you ship.

And in that world, the battle’s not over. Google agrees. You know how I know? Because they’re throwing a tremendous amount of effort at Flutter, which is basically a proprietary version of Electron that doesn’t even do desktop apps.1 That only makes sense if you’re looking past the rendering engine wars–and if already you control effectively all rendering engines, then that fight only matters if you think the rendering engine wars are already passé.

So EdgeHTML’s death is sad, but the counterbalance isn’t Gecko; it’s Cocoa Touch. And on that front, there’s still plenty of diversity. Here’s to the fight.


  1. Yeah, I know there’s an effort to make Flutter work on desktops. I also know that effort isn’t driven by Google, though. [return]

Pete Corey (petecorey)

Advent of Code: Memory Maneuver December 08, 2018 12:00 AM

Today’s Advent of Code challenge asks us to parse a sequence of numbers that describe a tree. Each node of the tree consists of metadata, a list of numbers, and zero or more children. We’re asked to find the sum of all metadata entries throughout the tree. Let’s use the J programming language to solve this problem!

My gut reaction when I hear the word “tree” is to reach for recursion. Let’s write a recursive verb in J that processes each node described by our input and builds up our tree as we go:

    process =. 3 : 0
      siblings =. 0 {:: y
      childrens =. 0 { 1 {:: y
      metadatas =. 1 { 1 {:: y
      rest =. 2 }. 1 {:: y
      if. childrens = 0 do.
        children =. 0 1 $ 0
      else.
        next =. process^:childrens (0 1 $ 0);rest
        children =. 0 {:: next
        rest =. 1 {:: next
      end.
      metadata =. (i. metadatas) { rest
      rest =. metadatas }. rest
      (siblings,children,metadata);rest
    )

The recursion here is fairly straight forward. If the current node has children, I’m using the ^: adverb to repeatedly, recursively apply the process verb to each of its sibling nodes.

I return any passed in siblings appended to the children we just processed, along with the set of metadata on each node.

We can find our final answer by raveling together all of the collected metadata and summing them together:

    echo +/,0{::process (0 1 $ 0);input

Part Two

Part two revealed that the metadata in each node actually refers to the (1-based) indexes of that node’s children. Calculating the cost of nodes with children is done by adding up the cost of each node specified in the metadata list. The cost of a leaf node is the sum of its metadata.

I figured that the best way to tackle this was to rework my process verb to return the entire, correctly structured tree:

    process =. 3 : 0
      siblings =. 0 {:: y
      childrens =. 0 { 1 {:: y
      metadatas =. 1 { 1 {:: y
      rest =. 2 }. 1 {:: y
      if. childrens = 0 do.
        children =. 0 1 $ 0
      else.
        next =. process^:childrens (0 1 $ 0);rest
        children =. 0 {:: next
        rest =. 1 {:: next
      end.
      metadata =. (i. metadatas) { rest
      node =. metadata;<children
      rest =. metadatas }. rest
      (siblings,node);rest
    )

The final structure of the sample input looks like this:

┌─────┬─────────────────┐
│1 1 2│┌────────┬──────┐│
│     ││10 11 12│      ││
│     │├────────┼──────┤│
│     ││2       │┌──┬─┐││
│     ││        ││99│ │││
│     ││        │└──┴─┘││
│     │└────────┴──────┘│
└─────┴─────────────────┘

For each node, the metadata is on the left, and the boxed list of children is on the right.

I wrote a count verb that recursively counts the cost of a given node. If the node has no children, I return the sum of its metadata. Otherwise, I return the sum of count applied to its children:

    count =. 3 : 0
      metadata =. 0{::y
      children =. 1{::y
      if. 0 = # children do.
        +/ metadata
      else.
        indexes =. 1 -~ metadata
        indexes =. indexes #~ _1 < indexes
        indexes =. indexes #~ -. (1 -~ # children) < indexes
        +/ count"_1 indexes { children
      end.
    )

I can use these two together to get my final answer:

    tree =. 0{0{::process(0 1 $ 0);input
    echo count tree

Notes

  • This page on working with trees in J was incredibly helpful.
  • I’ve been using #~ quite a bit to build a mask and remove items from an array based on that mask.
  • I made heavy use of the if control structure when solving these problems. No need to be a hero.

Andreas Zwinkau (qznc)

Dependency Abstraction December 08, 2018 12:00 AM

A design pattern which generalizes Dependency Inversion and can also be applied on an architectural level.

Read full article!

December 07, 2018

Pete Corey (petecorey)

Advent of Code: The Sum of Its Parts December 07, 2018 12:00 AM

Day seven of this year’s Advent of Code asks us to find the order in which we must complete a set of steps in a directed graph. Let’s see how well we can do with this task using the J programming language!

My high level plan of attack for this task is to keep each pair of dependencies in their current structure. I’ll build a verb that takes a list of “completed” steps, and the list of pairs relating to uncompleted steps. My verb will find the first (alphabetically) step that doesn’t have an unmet dependency in our list, append that step to our list of completed steps, and remove all pairs that are waiting for that step being completed.

Thankfully, parsing our input is easy today:

    parse =. (5 36)&{"1
    pairs =. |."1 parse input
AC
FC
BA
DA
EB
ED
EF

We can write a helper that takes our list of pairs and returns all of the steps referenced in them in a raveled list:

    steps =. [: /: [: . ,
    steps pairs
ABCDEF

Now we can write our verb that completes each step of our instructions:

    next =. 3 : 0
      done =. 0 {:: y
      pairs =. 1 {:: y
      steps =. steps pairs
      left =. {."1 pairs
      right =. {:"1 pairs
      next_step =. {. steps #~ -. steps e. ~. left
      next_pairs =. pairs #~ -. right e. next_step
      remaining_pairs =. pairs #~ right e. next_step

      append =. (done,next_step)"_
      return =. (done)"_
      next_step =. (append ` return @. (0 = # remaining_pairs)"_) _

      next_step;next_pairs
    )

I’m trying to be more explicit here, and rely less on tacit verbs. Last time I found myself getting lost and hacking together solutions that I didn’t fully understand. I’m trying to pull back and bit and do things more intentionally.

We can converge on the result of repeatedly applying next to our list of pairs and an empty starting set of completed steps:

    0{:: next^:_ '' ; pairs
CABDF

An unfortunate side effect of our algorithm is that our last step in our graph is never appended to our list. We need to find this step and append it ourselves:

    append_last =. 4 : 0
      steps =. steps x
      missing =. steps #~ -. steps e. ~. y
      y,missing
    )
    echo pairs append_last 0{:: next^:_ '' ; pairs
CABDFE

And that’s all there is to it!

Part Two

Part two was much more complicated than part one. Each step takes a specified amount of time to complete, and we’re allowed to work on each step with up to four workers, concurrently.

This was the hardest problem I’ve solved so far throughout this year’s Advent of Code. My general strategy was to modify my next verb (now called tick) to additionally keep track of steps that were actively being worked on by concurrent workers.

Every tick, I check if there are any available steps and any space in the worker queue. If there are, I move the step over. Next, I go through each step being worked on by each worker and subtract 1. If a step being worked on reaches 0 seconds of work remaining, I add it to the done list.

Eventually, this solution converges on my answer.

I’m not at all happy with my code. I found myself getting deeply lost in the shape of my data. After much struggling, I started to make heavy use of $ to inspect the shape of nearly everything, and I peppered my code with echo debug statements. The final solution is a nasty blob of code that I only just barely understand.

Enjoy.

Notes

December 06, 2018

Stjepan Golemac (stjepangolemac)

The first thing that comes to my mind is throttling the logout saga. Did you try that? December 06, 2018 10:33 AM

The first thing that comes to my mind is throttling the logout saga. Did you try that?

https://redux-saga.js.org/docs/api/#throttlems-pattern-saga-args

Pete Corey (petecorey)

Advent of Code: Chronal Coordinates December 06, 2018 12:00 AM

Today’s Advent of Code challenge asked us to plot a Manhattan distance Voronoi diagram of a collection of points, and to find the area of the largest, but finite, cell within our diagram.

I’ll be honest. This was a difficult problem for me to solve with my current level of J-fu.

My high level plan of attack was to build up a “distance matrix” for each of the points in our diagram. The location of a point would have a value of 0, neighbors would have a value of 1, and so on. In theory, I’d be able to write a verb that combines two matrices and returns a new matrix, with tied distances represented as _1. I could insert (/) this verb between each of my matrices, reducing them down to a final matrix representing our Voronoi diagram.

I wrote some quick helper verbs to find the distance between two points:

    d =. [: +/ |@-

Find the width and height of the bounding rectangle of my input:

    wh =. 1 + >./ - <./

Generate the set of coordinates for my matrices (this one took some serious trial and error):

    coords =. 3 : 0
      'w h' =. (1 + >./ - <./) y
      (<./y) +"1  (i. h) ,"0~ ((h,w) $ i. w)
    )

And to fill that matrix with the distances to a given point:

    grid =. 4 : 0
      (x d ])"1 coords > y
    )

The process of adding together two matrices was more complicated. I went through many horribly broken iterations of this process, but I finally landed on this code:

    compare =. 4 : 0
      'vx ix' =. x
      'vy iy' =. y
      vx = vy
    )

    tie =. 4 : 0
      (0 {:: x);_1
    )

    pick =. 4 : 0
      'vx ix' =. x
      'vy iy' =. y
      v =. vx ((y"_) ` (x"_) @. <) vy
    )

    add =. 4 : 0
      x (pick ` tie @. compare) y
    )

With that, I could compute my final grid:

    numbers =. ". input
    grids =. ([ ;"0 i.@#) numbers grid"1 <numbers
    sum =. add"1/ grids

Our sum keeps track of closest input point at each position on our grid, and also the actual distance value to that point. The closest input point is what we’re trying to count, so it’s probably the more interesting of the two values:

    groups =. (1&{::)"1 sum
 0  0  0 0 _1 2 2  2
 0  0  3 3  4 2 2  2
 0  3  3 3  4 2 2  2
_1  3  3 3  4 4 2  2
 1 _1  3 4  4 4 4  2
 1  1 _1 4  4 4 4 _1
 1  1 _1 4  4 4 5  5
 1  1 _1 4  4 5 5  5
 1  1 _1 5  5 5 5  5

We could even render the grid using J’s viewmat utility. Awesome!

Our sample inputs, visualized with viewmat.

Using viewmat to visualize my matricies like this actually helped my find and fix a bug in my solution incredibly quickly. I’m a big fan and plan on using it more in the future.

Because of how Manhattan distance works, cells with infinite volume are the cells that live on the border of our final matrix.

To find those infinite groups that live along the edges of my final matrix, I appended the edge of each edge of my matrix together and returned the nub of those values. I found the idea for this matrix rotation helper from this video on J I watched many months ago. I’m glad I remembered it!

    rot =. [: |. |:

    edges =. 3 : 0
      top =. 0 { y
      right =. 0 { rot^:1 y
      bottom =. 0 { rot^:2 y
      left =. 0 { rot^:3 y
      ~. top , right , bottom , left
    )

To find my final answer, I raveled my matrix, removed the infinite groups, used the “key” (/.) adverb to count the size of each group, and returned the size of the largest group.

    without =. _1 , edges groups
    raveled =. ,groups
    0 0 {:: \:~ ({. ;~ #)/.~ raveled #~ -. raveled e. without

This definitely isn’t the most efficient solution, but it works. At this point, I’m happy with that.

Part Two

Part two turned out to be much easier than part one. We simply needed to iterate over each point in our grid, counting the total distance to each of our input points. The set of points that was less than a fixed number from all input points defined a circular “landing area”. We were asked to find the size of that area.

I gutted most of my part one solution and replaced the values returned by my grid verb with the total distance to each input point:

    distances =. 4 : 0
      +/ ((>x) d"1~ ])"1 y
    )

    grid =. 3 : 0
      (<y) distances"1 coords y
    )

Finding my final answer was as easy as calculating my grid, checking which points were less than 10000, removing all of the 0 values, and counting the result.

    numbers =. ". input
    # #~ , 10000 > grid numbers

Notes

  • Rotating a matrix (|. |:) is a great trick.
  • viewmat is awesome. It very quickly helped me find and fix a bug in my solution.
  • Boxes can be treated like arrays in most cases. I was under the wrong impression that a box was a single unit in terms of rank.

December 05, 2018

Derek Jones (derek-jones)

Coding guidelines should specify what constructs can be used December 05, 2018 05:42 PM

There is a widespread belief that an important component of creating reliable software includes specifying coding constructs that should not be used, i.e., coding guidelines. Given that the number of possible coding constructs is greater than the number of atoms in the universe, this approach is hopelessly impractical.

A more practical approach is to specify the small set of constructs that developers that can only be used. Want a for-loop, then pick one from the top-10 most frequently occurring looping constructs (found by measuring existing usage); the top-10 covers 70% of existing C usage, the top-5 55%.

Specifying the set of coding constructs that can be used, removes the need for developers to learn lots of stuff that hardly ever gets used, allowing them to focus on learning a basic set of techniques. A small set of constructs significantly simplifies the task of automatically checking code for problems; many of the problems currently encountered will not occur; many edge cases disappear.

Developer coding mistakes have two root causes:

  • what was written is not what was intended. A common example is the conditional in the if-statement: if (x = y), where the developer intended to write if (x == y). This kind of coding typo is the kind of construct flagged by static analysis tools as suspicious.

    People make mistakes, and developers will continue to make this kind of typographical mistake in whatever language is used,

  • what was written does not have the behavior that the developer believes it has, i.e., there is a fault in the developers understanding of the language semantics.

    Incorrect beliefs, about a language, can be reduced by reducing the amount of language knowledge developers need to remember.

Developer mistakes are also caused by misunderstandings of the requirements, but this is not language specific.

Why do people invest so much effort on guidelines specifying what constructs not to use (these discussions essentially have the form of literary criticism)? Reasons include:

  • providing a way for developers to be part of the conversation, through telling others about their personal experiences,
  • tool vendors want a regular revenue stream, and product updates flagging uses of even more constructs (that developers could misunderstand or might find confusing; something that could be claimed for any language construct) is a way of extracting more money from existing customers,
  • it avoids discussing the elephant in the room. Many developers see themselves as creative artists, and as such are entitled to write whatever they think necessary. Developers don’t seem to be affronted by the suggestion that their artistic pretensions and entitlements be curtailed, probably because they don’t take the idea seriously.

Stjepan Golemac (stjepangolemac)

Hi Shawn, December 05, 2018 03:38 PM

Hi Shawn,

Yes I would go with isRefreshing flag somewhere in your store too. First refresh sets it to true, and all subsequent ones wait until it is changed.

You can have refreshError value in your store too that will be null by default, and that would change if error fails.

After refresh finishes all sagas that were waiting for it can check was it successful or not by checking the refreshError value, and act accordingly.

You could even race refreshSuccess and logout events, and if the logout is first you cancel refresh.

You can find more info here:

I hope this helps you!

Pete Corey (petecorey)

Advent of Code: Alchemical Reduction December 05, 2018 12:00 AM

Today’s Advent of Code problem is to repeatedly remove corresponding “units” from a “polymer” until we’re left with an unreducable polymer string. Unit pairs that can be removed are neighboring, matching characters, with differing cases, like C and c. The answer to the problem is the length of the resulting string.

I’m really happy with my J-based solution to this problem, which is a relief after yesterday’s disaster. I started by writing a function that compares two units and returns a boolean that says whether they react:

    test =. ([ -.@:= ]) * tolower@:[ = tolower@:]
    'C' test 'c'
1

Next, I wrote a dyadic verb that takes a single unit as its x argument, and a string of units as its y argument. If x and the head of y react, it returns the beheaded (}.) y. Otherwise it returns x appended to y:

    pass =. ([,]) ` (}.@:]) @. ([ test {.@:])
    'C' pass 'cab'
ab

This pass verb can be placed between each element of our polymer string using the insert (/) adverb. This gives us a reduced polymer string.

    pass/ 'dabAcCaCBAcCcaDA'
dabCBAcaDA

Finally, we can repeatedly apply pass until the result is stable, essentially converging on a solution. Once we’ve got our fully reduced polymer string, we count its length and print the result:

    echo # pass/^:_ 'dabAcCaCBAcCcaDA'
10

And that’s it!

Part Two

Part two tells us that one of the unit pairs is causing trouble with our polymer reduction. It wants us to remove each possible unit pair from the input string, count the length of the resulting reduction, and return the lowest final polymer string length.

My solution to part two builds nicely off of part one.

We’ll keep test and pass as they are. We’ll start by writing a remove verb that takes a character to remove as x, and a string to remove it from as y. I use i. to build a map that shows me where x isn’t in y, and then use # to omit those matching characters.

    remove =. ] #~ [ i. [: tolower ]
    'y' remove 'xyz'
xz

Next I wrote a remove_nubs verb that calculates the nub of our polymer string, and uses remove to remove each nub from our original string. I box up the results to avoid J appending spaces to end of my strings to fill the matrix.

    remove_nubs =. [ <@:remove"1 (1 , #@:nub) $ nub =. [: ~. tolower
    remove_nubs 'aabc'
┌──┬───┬───┐
│bc│aac│aab│
└──┴───┴───┘

Finally, I apply remove_nubs from my input, and converge on a solution for each new polymer string, count their resulting lengths, and return the minimum length:

    echo <./ ([: # [: pass/^:_"1 >)"0 remove_nubs 'dabAcCaCBAcCcaDA'
4

Notes

  • The application of / modified verbs is from right to left. I would have expected left to right, for some reason. This makes sense though, considering J’s execution model.
  • Visualizing verb trains makes it so much easier to write them. I actually found myself getting them right the first time, thanks to “tree view” ((9!:3) 4.
  • Boxing can be helpful when I don’t want J to pad the value to fit the dimensions of the array it lives in.

Pepijn de Vos (pepijndevos)

Building Bad Amplifiers December 05, 2018 12:00 AM

My brother scavenged and repaired an old bass guitar, and asked if I could make him an equally bad amplifier to go with it to create the ultimate bad sound.

I was happy to oblige, so I threw everything I know about good amplifiers out of the window and googled around for some inspiration. Thus resulted in a lot of badly designed amplifiers, that subject your speaker to DC currents or don’t deliver any power at all.

So I started making some myself. First thought was to make a class A amplifier with a single power MOSFET and power resistor. I got two 1 Ω resistors in series, rated for 5 W. This gives a maximum voltage of 2.2 per resistor.

The MOSFET is rated for 30 A, so that’s probably fine. Then I used a potentiometer to bias the gate and a capacitor to drive it. Something like this.

class a

Problem is, while it’s a very nice and simple space heater, it does’t sound bad enough. It’s inefficient and non-linear, but the sound is kind of fine.

So the next step was to make an amplifier that sounds worse. What better than a pure class B output stage with its sweet unmitigated crossover distortion?

I pulled a complementary pair of BJTs from a drawer, and drove it with some random small-signal MOSFET and a 1K resistor. A diode was added to reduce the cross-over a bit. The output cap is just a large one. At the bottom of the MOSFET I added a 100 Ω degeneration resistor that I bypassed in AC with a capacitor for more gain. I again added a potentiometer to bias the MOSFET.

My brother liked the bad sound, but it wasn’t loud enough, so I added another MOSFET gain stage. Same story, 1K resistor, small bypassed degeneration resistor, and a potentiometer to bias the MOSFET. Except now I put the potentiometer as the degeneration resistor, for no good reason.

class b

Neither of these amplifiers involved much design, calculation, or simulation. They were directly constructed on overboard with potentiometers where I would have needed math to find the correct value, and I just made these drawings for this post.

Normally what you’d do is calculate the gate voltage created by the voltage divider.

The voltage across the degeneration resistor is then roughly

For a BJT the threshold voltage is roughly 0.6 V while a small-signal MOSFET is more like 2 V. Ohms law then gives you the current through the degeneration resistor, which is the same current as through the 1k resistor at the top, so you know how much voltage drops across that one.

December 04, 2018

Gustaf Erikson (gerikson)

November December 04, 2018 03:15 PM

Pete Corey (petecorey)

Advent of Code: Repose Record December 04, 2018 12:00 AM

Today’s Advent of Code challenge asks us to parse and process a set of time-series data that describes when guards start their shift, when they fall alseep, and when they wake up. Our task is to find the guard that sleeps the most. We need to multiple their ID together with the minute they’re asleep the most.

This was an incredibly difficult problem for me to solve using J. My plan of attack was to build a “sleep matrix” for each guard. Each matrix would have a row for each day the guard was on duty, and each row would be sixty columns wide, with each row/column representing whether the guard was asleep during that minute of the twelfth hour of that day.

I was immediately stumped by how to parse each string and organize all of the data into useful, easily manipulatable structures.

After sorting my lines (/:~ input), I checked if each line had a 'G' character at index nineteen. If it did, I raised a boolean and used +/\ e. and # to build a set of groups for each guard’s shift. Once I’d grouped each shift, I could build my sleep matrix and box it together with the guard’s ID:

    sleep_map =. 3 : 0
      1 y } 60 $ 0
    )

    filter =. (' ' -.@:= 0&{)"1 # ]

    parse_log =. 3 : 0
      head =. {. y
      rest =. filter }. y
      'Y M D h m id' =. numbers head
      sleep =. sleep_map"0 ({:"1 numbers"1 rest #~ (# rest) $ 1 0)
      wake =. _1 * sleep_map"0 ({:"1 numbers"1 rest #~ (# rest) $ 0 1)
      id&;"2 +/\"1 sleep + wake
    )

    parse =. 3 : 0
      groups =. +/\ ('G' = 19&{::)"1 y
      masks =. groups&e."0 ~. groups
      parse_log"_1 masks # y
    )

Next I needed to consolidate each guard’s set of shifts and sleep matrices into a single sleep matrix:

    group_days =. 3 : 0
      id =. 0 {:: {. y
      days =. ,/ 1 {::"_1 y
      id;days
    )

    group =. 3 : 0
      ids =. 0 {::"1 y
      ids group_days/. y
    )

Finally I could box up the needed statistics for each guard and sleep matrix, sort the results, and return the desired calculation:

    stats =. 3 : 0
      id =. 0 {:: y
      days =. 1 {:: y
      overlap =. +/ days
      most =. overlap i. >./ overlap
      slept =. +/ overlap
      slept; most; id; days
    )

    result =. (2&{:: * 1&{::) {. \:~ stats"1 group parse log

Part Two

Part two just wants us to find the guard that is asleep most frequently on the same minute of the night. We’re to return that guard’s ID multiplied by the minute they’re usually asleep.

Thankfully, I was able to recycle all of the hard work I put into part one when it came time to solve part two. All I really needed to do was make a change to the set of statistics I boxed up in my final step:

    stats =. 3 : 0
      id =. 0 {:: y
      days =. 1 {:: y
      overlap =. +/ days
      most =. >./ overlap
      minute =. overlap i. most
      most; minute; id; days
    )

The rest of my code was left unchanged.

Notes

  • The “key” verb (/.) can be incredibly useful for grouping data and performing actions on those subsets.
  • Sorting is interesting in J.
  • Any type of data can be sorted. Sorting arrays of boxes behaves like sorting lists of tuples in Elixir, which is a very handy trick.
  • (9!:3) 4 renders verb trains in “tree view” which I find very helpful.

Alex Wilson (mrwilson)

Notes from the Week #11 December 04, 2018 12:00 AM

I’m trying a slightly different style this week, let’s see how it feels!

Four Things That Happened

One

It’s the end of ProDev’s quarter, and we celebrated with a … science fair. We’ve done one of these before and I absolutely love the creativity that comes out of a such a simple proposition. Each team prepares a “stall” that we take into our clubhouse meeting room and we can go around and see what each team has done during the last quarter.

We had:

  • Super-detailed artwork on movable wipe-boards
  • Large monitors showing off new reporting capabilities
  • Kahoot quizzes about our data platform’s learnings
  • And more …

We themed ours on our Opsgenie integration and had “3 Wishes” that we’ve fulfilled during the last quarter.

These events are a great way to down tools (kind of) and create something to show off what we’ve been working on — a side-effect of Agile/XP that I’ve observed is that working in vanishingly thin slices and focusing on incremental delivery removes a lot of the sense of progress as we’re only taking small steps.

Events like these science fairs give us a way to take a step back and recognise an entire quarter’s worth of work.

Two

Shift have been informal adopters of the Occupy Hand Signals as a way of self-moderating group conversations and making sure everyone has opportunities to speak whilst avoiding the “loudest person wins” degenerate case.

We learned this technique from observing another larger team (their team lead wrote about it here) and found ourselves picking up the very basic signals like “I would like to speak” and “I have a direct response”. Over time we’ve gotten pretty good at the self-enforcement part, such as calling out “X first, then Y” when two people put their hands up to speak.

One of our working agreements in our last retrospective was that we would formally adopt this for meetings of more than two people — a largely ceremonial action but it’s now encoded within our working practices.

There’s also a great GDS blog-post about using these signals.

Finally, I read a brief Twitter thread about conversational interactions, the seed of which was this article — I like to think that over the last few years I’ve become more a member of the Church of Strong Civility than the Church of Interruption.

DOCTRINES OF THE CHURCH OF STRONG CIVILITY
Thou shalt not interrupt.
Thou shalt speak briefly.
Thou shalt use physical cues to indicate your understanding and desire to speak.

Definite food for thought, and a reminder that conversations about how we have conversations are often vital preludes to making conversations themselves productive.

Three

We resumed work on improving our log-aggregation/query platform — our initial assessment of the AWS hosted ElasticSearch was that it was missing key features for our usecase, and that we would be trading off too much configurability for reduced operational overhead.

We’re now looking at Elastic’s own cloud offering and we repeated our investigation workflow that we used for our incident management application trials:

  • Identify use cases and trials to run (with input from our stakeholders)
  • Get our ducks in a row to engage a free trial (i.e. set up infrastructure ready for it)
  • Commence the free trial and evaluate our criteria

My use of the word trial rather than experiment is deliberate, and comes from our CTO’s most recent fortnightly ProDev all-hands session — paraphrasing Linda Rising, there’s no null hypothesis or statistical validation going on, so what we’re performing are trials and not experiments.

This is not a bad thing, but if we don’t have the resources to do proper experimental validation we should at least keep our trials short, effective, and frugal. Our free trial is temporally capped at 14 days, and has no revenue cost, so provided we’re effective with the criteria we evaluate this is shaping up to be a good trial.

Four

We now maintain a number of tools to orchestrate production systems that have historically been un-loved — a Shift developer, Stephen, took it upon himself to spend a bit of time to spike a cleaner version with more user-friendliness.

Shift are lucky: if we scrunched up a post-it and lobbed it a few feet, we’d hit one of our stakeholders.

The original XP book (Extreme Programming Explained) talks about the benefits of having an embedded customer for validating work quickly — we’ll go and user-research stuff that we’re building with the users that will be using them by embedding and watching them use the tool, whether it’s a senior developer, a new hire, or one of our experienced Site-Reliability Engineers.

The tool he spiked, to improve our puppet node management workflow, opens up a lot of opportunities for us to try new technologies. Now that we know there’s desire for the product, we’re umming-and-ahhing about whether we:

  1. Keep it in Bash
  2. TDD it from scratch in Python, a language we’re familiar with
  3. TDD it from scratch in Go/Rust, languages we’re not familiar with.

We’ll be invoking the Improve, No Change, Worsen workflow (outlined in a previous blogpost) to establish pros and cons of each approach — (2) and (3) are almost certainly slower, but (3) gives us the opportunity to broaden our horizons.

Will (3) be worth the overhead of learning a new language/toolchain?

Stay tuned to find out!

Reflections

I’m starting to meditate again, once a day for 15–20 minutes if I can. My mind is full of stuff right now, both at work and in my personal life so I am trying to get better at taking time for myself.

I’m basically being hit in the face with my own oft-repeated quote

If you don’t take time, or make time, how can you ever have time?

I finished Killing Commendatore and have downloaded an audio-book of A Wild Sheep Chase, another Murakami book. There’s something relaxing about his prose that takes me out of myself for a bit.

Originally published at blog.probablyfine.co.uk on December 4, 2018.

December 03, 2018

Jan van den Berg (j11g)

Advent of Code December 03, 2018 06:42 AM

Currently the yearly Advent of Code contest created by Eric Wastl is being held at adventofcode.com. That means the site is sprouting two daily programming challenges, until Christmas, to see who can solve them fastest. But it is not just about being fast of course, Advent of Code is a great way to improve your programming skills through daily clever puzzles. And because everyone gets the same puzzles it is also a great way to share and discuss results, and above all, learn.

Though I knew of Advent of Code, I hadn’t participated before, but it seems two days, and four puzzles later, I am sort of in.

Or at least, after only two days I am already fascinated by what I have seen, so I thought I’d share!

Fascinating findings

  • Python seems to be the most popular language, by far. At least judging by Github repo names, which is of course not an exact measure, but it is more or less an indicator, as a lot of people tend to share their solutions there. Python is popular, and it is not even close:

  • Browsing through the code, it once again becomes apparent that even with the exact same tools (e.g. Python) we all bring different experiences and education to the battle resulting in a colorful variation of solutions for the exact same puzzles. I’ve seen 100+ lines of Python code generate the exact same result as 10 lines. Nowhere is it more clear than a coding challenge that we are all unique individuals, and there is not a specific right way, as long as you get there (at least for the sake of this contest, don’t @ me).
  • If I had to guess I would have picked JavaScript to be the most popular language, but as you can see it comes in second. Ruby, Go and C# are also not surprising entries on this list, but Haskell and Elixir are (to me). These two functional languages seem to have quite a bit of buzz around them as people passionately seem to choose either one, which is interesting as I know very little about either. The creator of Elixir even participates himself in AoC! 
  • Very few people seem to pick PHP. Which I also find surprising, because gigantic parts of the web run PHP. But it seems to have little appeal when it comes to coding challenges?
  • Some people are fast, I mean really fast! Just look at the times on the leader board. Judging from these times, this means some people are able to read around 1000 words explaining a puzzle, and then coding up not one, but two solutions and submitting the correct answer in under four minutes!  I kid you not. This next person live-streamed it, and clocks in around 5 minutes (even without using the command-line shortcut CTRL-R), but it didn’t even put him in the top 20!

  • Of course you can use any language you like or even pen and paper, it is a puzzle after all. And people use some really crazy stuff, I love it. Anything goes (even Excel), and that is of course part of the goal of the contest: try to learn new things. There is one person who deliberately tried a new language for each challenge.

Notable entries

So it’s not all about speed, it’s also about trying new things. Here are some other unexpected examples.

    • Minecraft: This one takes the cake for me. See if you can wrap your head around what is happening here:

Learnings so far

So apart from the fascinating findings, I also got involved myself. I think because I solved the very first challenge with a simple AWK one-liner. But solving the followup challenge seemed trickier in AWK, though people seem to have done so (of course).

Being completely new to Python, and seeing how popular it is, I decided to give it a go, and I must say I think I understand a bit better now why and how Python is so popular. Yes, it is well known that it forces deliberately clean code but it also provides ways for incredibly succinct code (my favorite!). Because so far I have learned about map(), collections.counter, zip(), and cycle() Very handy, built-in functions and datatypes that I was unaware of, but which are incredibly powerful.

Some people tend to disagree (probably very few), as I found this comment on StackOverflow when researching the Counter dict.

I don’t think that’s fair, because in a sense every higher level programming language is an abstraction of differently expressed machine code. So unless you’re typing in machine code directly you are also using general purpose tools, and how narrow or general something is, who’s to say? And as long as it helps humans do things faster, programming languages are tools after all, I’m all for it. And let the computer worry about the zeros and ones.

I was very surprised and pleased with the mentioned Python functions. For example I brute-forced a solution in Bash which took probably more than 10 minutes to run, but ran in 0.07 seconds in only a couple of lines of Python. So of course the knowledge of the right functions and data structures once again proved to be the difference between 100 lines or 10, which reminded me of this quote of Linus Torvalds:

So that’s it and if you want to learn something new, go give it a try!

 

The post Advent of Code appeared first on Jan van den Berg.

Pete Corey (petecorey)

Allow Yourself to do Things Poorly December 03, 2018 12:00 AM

I often find myself stuck at the beginning of things. Haunted by past mistakes, I vow to myself that “I’ll do things right this time.” Unfortunately, insisting on doing everything “right” is crippling. Largely because doing everything “right” is impossible.

Lately I’ve stopped beating myself over the head for doing things that I know aren’t “best practice”, and instead I’ve given myself the freedom to start doing things poorly.

It’s been liberating.

Analysis Paralysis

My Elixir-based Chord project is coming along nicely. While my ASCII-based chord chart renderer is incredibly helpful for visualizing chords in the terminal, it’s still difficult to sift through several thousand chords at a time.

Reluctantly, I realized that my Chord project needed a web-based front-end.

These days, my go-to tool for building a front-end is React. However, I’ve built enough React applications for clients and in my personal work to know that if you’re not vigilant and strict, the complexity of a React project can quickly spiral out of control. I was determined not to allow that to happen this time around.

Not only that, but I wasn’t sure how best to build the user interface. Chord offers users a huge number of potential knobs to tweak, and the resulting sets of data can be massive and hard to sift through. How do we best present everything to the user?

I was paralyzed.

Get Going

After several days of hemming, hawing, and wringing my hands over architectural decisions and uncertainties over how I wanted the user experience to go, I decided that I was wasting my time.

It’s better, I convinced myself, to just get something built. Even if it’s built poorly and even if it isn’t an ideal interface for the tool, something is better than nothing. I essentially decided to start building a rough draft of the application.

The first thing I needed was a way to render chord charts in the browser. After an hour or so of writing some absolutely awful React code, I was there.

Randomly generated chords.

Next, I needed a way to pull chords from the back-end Elixir server and render them using our new chord chart component. After another couple hours of hacking together a (poorly designed and roughly implemented) GraphQL data layer, I was greeted with several hundred Cmaj7 chords:

All Cmaj7 chords.

At this point, I was stuck on how to proceed. How could I easily let users build chord progressions from the possibilities presented? I started iterating on a few ideas, mostly involving nested trees, but nothing seemed to click.

Awash in Inspiration

Several days later, I was browsing Reddit and I stumbled across this screenshot from /r/unixporn. I was completely inspired. This is what I wanted my project to look like!

My inspiration.

I fired up CodePen and started hashing out some mockups of how the application might look.

Codepen mockup.

Happy with the direction I was heading, I quickly translated the hard-coded, mocked-up HTML into my already existing React components. The results were promising.

With data.

Seeing real data on the screen gave me even more ideas on how to interact with the application.

With data.

I was awash in inspiration.

Riding the Wave

This cycle of building and playing with what I’d built kept up for days. Every few hours I’d pester my wife and show her what I’d added, because I was so excited. I didn’t take the time during this process to focus on best practices or code quality. Instead, I focused on getting results.

Eventually, that wave rolled back, and I was left with a mostly functional and mostly ugly codebase. It became more difficult to make changes to the codebase, and with my inspiration fading, I wasn’t motivated to push through the pain.

At this point, I turned my attention towards refactoring my original code. Now was the time to focus on best practices and doing things “right”.

While I’m still not fully happy with the codebase or the user interface, I’m very happy with how far I’ve come in such a short time. I never would have made this progress if I didn’t allow myself to do things poorly, just for the sake of getting things done. If I was still at the beginning, fixated on engineering a “correct” solution, I wouldn’t have had the raw materials required to turn my inspiration into a tangible product.

The current state of things.

December 02, 2018

Ponylang (SeanTAllen)

Last Week in Pony - December 2, 2018 December 02, 2018 11:40 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, our users’ mailing list or join us on IRC.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

Indrek Lasn (indreklasn)

Here are the most interesting developer podcasts — 2019 edition December 02, 2018 05:51 PM

Photo by Matthias Wagner on Unsplash

Who doesn’t love to hear the opinions of peer developers? Frankly put, Some of the most useful resources I’ve picked up is from listening to different opinions and thoughts. I’ve put together a small but comprehensive list of my favorite podcasts. The core topics revolve around coding, well-being philosophy, and how to cope with a different set of challenges.

I find myself listening to podcasts while doing the dishes, while on an airplane, driving, and commuting. Learning from podcasts adds up really quickly — there’s so much to learn yet so little time — why not try to maximize ways to improve?

If yours didn’t make it to the list, please post it in the comments and I’ll make sure to check it out!

Assume that the Person You’re Listening to Might Know Something You Don’t —Jordan Bernt Peterson

Laracast

https://laracasts.simplecast.fm/

The Laracasts snippet, each episode, offers a single thought on some aspect of web development. Nothing more, nothing less. Hosted by Jeffrey Way.

I truly enjoy the way Jeffrey Way unfolds his opinions on web development, life, parenting, hiring and running a small business while keeping the atmosphere friendly. Great podcast with great topics — definitely helps to grow as a person.

Syntax

https://syntax.fm/

A Tasty Treats Podcast for Web Developers.

If you’re looking into how to land freelance gigs, improve your coding or overall get your career to the next level — Syntax is the podcasts for you.

Wes Bos and Scott Tolinski are both great teachers and express themselves very fluently. Keeping up with the latest trends can be a drag. Syntax helps me to keep up to date with my skillsets.

There’s nothing more embarrassing than not knowing the latest trends. Especially crucial in the freelancing landscape since knowing the latest tools can be a huge time saver.

Developer Tea

https://spec.fm/podcasts/developer-tea

A podcast for developers designed to fit inside your tea break

From industry professionals, for industry professionals

In January 2015, two independent podcasts — Design Details and Developer Tea — were started by three individuals who wanted to talk about the work they do every day. After an amazing response from the web community, we’ve teamed up to create the Spec Network to help designers and developers to learn, find great resources and connect with one another.

freeCodeCamp podcast

https://freecodecamp.libsyn.com/

The official podcast of the freeCodeCamp open source community. Learn to code with free online courses, programming projects, and interview preparation for developer jobs.

freeCodeCamp has your back from how to get your first job to how to negotiate salaries, and much more!

Coding Blocks

https://www.codingblocks.net/category/podcast/

We are a few guys who’ve been professional programmers for years. As avid listeners of podcasts and consumers of many things code-related, we were frustrated by the lack of quality programming (pun) available in listenable formats. Given our years of experience and real-world problem solving skills, we thought it might be worth getting into this world of podcasting and “giving back” a shot.

Programming Throwdown

https://www.programmingthrowdown.com/?m=0

From teaching kids to code to concurrency, Patrick Wheeler and Jason Gauci have you covered. Programming Throwdown recommends the best developer tools, books and coding patterns. They’ve been available since 2010 and are a great over-all listening experience.

Away from the keyboard

http://awayfromthekeyboard.com/episodes/

Away From The Keyboard is a podcast that talks to technologists and tells their stories. Stories about how they started, how they grew, how they learned, and how they unwind. It’s hosted by Cecil Phillip and Richie Rump and new episodes are released every Tuesday.

Full Stack Radio

http://www.fullstackradio.com/

A podcast for developers interested in building great software products. Every episode, Adam Wathan is joined by a guest to talk about everything from product design and user experience to unit testing and system administration.

This should give you a nice range of topics to listen to! Stay awesome and thanks for reading!

If you found this article useful, give it some claps!

More about the author;

Indrek Lasn - Medium

Never miss an article — stay up to date by following me on Twitter

Here are some articles you might enjoy as well;


Here are the most interesting developer podcasts — 2019 edition was originally published in freeCodeCamp.org on Medium, where people are continuing the conversation by highlighting and responding to this story.

Pete Corey (petecorey)

Advent of Code: Inventory Management System December 02, 2018 12:00 AM

Today’s Advent of Code challenge asks us to scan through a list of IDs, looking for IDs that two and three repeated letters. Once we have the total number of words with two repeated characters and three repeated characters, we’re asked to multiple them together to get a pseudo-checksum of our list of IDs.

At first I had no idea about how I would tackle this problem in J. After digging through the vocabulary sheet, I stumbled upon the monadic form of e. and I was shown the light.

The e.y verb iterates over every element of y, building a boolean map of where that element exists in y. We can use that result to group together each element from y. Once we’ve grouped each element, we can count the length of each group. Because J pads extra 0/' ' values at the end of arrays to guarantee filled matricies, we’ll measure the length of each group by finding the index of this padding character in every group. Finally, we can see if 2 and 3 exists in each set of character counts, and sum and multiply the results.

    read_lines =. >@:cutopen@:(1!:1) <
    lines =. read_lines '/Users/pcorey/advent_of_code_2018/day_02/input'

    group =. ([: ~. e.) # ]
    count =. [: i."1&' ' group"1
    twos_and_threes =. [: (2 3)&e."1 count

    */ +/ twos_and_threes lines

I’ve been trying to make more use of forks, hooks, and verb trains after reading through Forks, Hooks, and Compound Adverbs and Trains on the J website.

Part Two

Part two asks us the find two IDs that differ by only a single letter. Once we’ve found those two IDs, we’re to return the set of letters they have in common.

My basic idea here was to compare every possible word combination and pack up the two compared words with the number of differences between them:

    compare =. 4 : 0
      differences =. x ([: +/ [: -. =) y
      differences;x;y
    )

    comparisons =. ,/ input compare"1/ input

Using that, we could pick out the set of word pairs that have one difference between them:

    ones =. 1&= 0&{::"1 comparisons

Unbox the word pairs from that set:

    words =. > }."1 ones # comparisons

And then use an inverted nub sieve to find the letters those words share in common, using another nub to filter out the duplicate caused by the inverse comparison:

    ([: . (-.@:: # ])"1/) words

I’ll admit, I found myself getting lost in what felt like a sea of boxes and arrays while working on this solution. I found it difficult to keep track of where I was in my data, and found myself building intermediate solutions in the REPL before moving them over to my project. I need to get better at inspecting my data and getting a feel for how to manipulate these structures in J.

I also found myself heavily relying on the REPL for building my verb chains. I constantly found myself building chains by trail and error.

Does this work? No? Maybe if I add a cap at the end? Yes!

I can handle single hooks and forks, but when things expand beyond that I find myself getting lost. I’m hoping I get better with that over time.

Notes

  • Trying to open a box that holds numbers and string will throw a domain error, because the two types can’t live together in an array.

December 01, 2018

Bogdan Popa (bogdan)

Advent of Racket 2018 December 01, 2018 05:45 AM

I decided to do this year’s Advent of Code in Racket and stream the whole thing. We’ll see how far I make it (getting up this early is rough!), but so far I finished day one. The code is here and the playlist for the recordings is here. If you want to get notified as soon as I jump on to start streaming, you can follow me on Twitch.

Vincent (vfoley)

vi, my favorite config-less editor December 01, 2018 03:14 AM

Before I became an Emacs user, I was a vi user, and though Emacs is my primary editor, I still use vi daily. I use vi for quickly skimming the content of a file and quick edits to configuration files; I use vi when I’m on a remote server or when a command-line utility invokes $EDITOR. The main reason why I still use vi is because I’m able to use it very efficiently even without a configuration file.

I learned vi circa 2001 from a magazine and from vimtutor. It was my primary text editor until 2007 or 2008 when I switched to Emacs. Back then, I had a rather long .vimrc, complete with configurations for Linux and Windows, GTK vim and curses vim, and I used a few plugins such as bufexplorer.vim.

Recently, I deleted my old .vimrc and started a new one from scratch. I no longer use plugins and I don’t use a GUI version of vim. For these reasons, I now consider myself more a vi user than a vim user. (I am however a big fan of text objects, and I want unlimited undo, so using nvi is not something I would consider.) My .vimrc is now only 12 lines and most of these changes are conservative and uncontroversial: backspace works “normally” in insert mode all the time, I enable incremental search, I use 4 spaces for indentations rather than hard tabs, I activate syntax highlighting. Even though vim offers a myriad of settings, a vanilla setup is extremely capable and usable.

I find that being able to use an editor without custom configuration is extremely useful. Here are a couple of examples: I learned how to exploit Linux binaries from the book Art of Exploitation. The book comes with a companion live CD that you can use to follow examples from the book and try to perform the hacks yourself. Emacs is not installed in the live CD, but vim is, and I was perfectly happy and comfortable using it to read the source code of the examples and modify them. Another example: recently at work, I wanted to write a Python script on a server that didn’t have Python installed, but had Docker. I launched a Python container, installed vim-tiny, and proceeded to write, debug, and improve my script without ever feeling that I was editing with a hand tied behind my back.

It is oddly satisfying to know that no matter the kind of machine I find myself on or the kind of restrictions it has (no network to download Emacs and/or my custom configuration, corporate policy against fetching packages from MELPA, etc.), that there will always be an editor that I can happily use without needing to make it “my own”.

Tim O’Reilly and Paul Graham have also cited usability without configuration a reason for preferring vi to Emacs.

Thanks to Richard Kallos for proof-reading an early version of this article.

Pete Corey (petecorey)

Advent of Code: Chronal Calibration December 01, 2018 12:00 AM

I’ve been a huge fan of the Advent of Code challenges since I first stumbled across them a few years ago. Last year, I completed all of the 2017 challenges using Elixir. This year, I decided to challenge myself and use a much more esoteric language that’s held my interest for the past year or so.

It’s my goal to complete all of this year’s Advent of Code challenges using the J programming language.

Before we get into this, I should make it clear that I’m no J expert. In fact, I wouldn’t even say that I’m a J beginner. I’ve used J a handful of times, and repeatedly struggled under the strangeness of the language. That being said, there’s something about it that keeps pulling me back. My hope is that after a month of daily exposure, I’ll surmount the learning curve and get something valuable out of the experience and the language itself.


The first Advent of Code challenge of 2018 asks us to read in a series of “changes” as input, and apply those changes, in order, to a starting value of zero. The solution to the challenge is the value we land on after applying all of our changes. Put simply, we need to parse and add a list of numbers.

The first thing I wanted to do was read my input from a file. It turns out that I do know one or two things about J, and one of the things I know is that you can use foreigns to read files. In particular, the 1!:1 foreign reads and returns the contents of a file.

Or does it?

    1!:1 'input'
file name error

Apparently 1!:1 doesn’t read relative to the current script. I’m guessing it reads relative to the path of the jconsole executable? Either way, using an absolute path fixes the issue.

    input =. 1!:1 < '/Users/pcorey/advent_of_code_2018/day_01/input'

Now input is a string with multiple lines. Each line represents one of our frequency changes. We could use ". to convert each of those lines into a number, but because input is a single string, and not an array of lines, can’t map ". over input:

    0 "./ input
0

After scrambling for ways of splitting a string into an array of lines, I stumbled across cutopen, which takes a string and puts each line into a box. That’s helpful.

    boxed =. cutopen input
┌──┬──┬──┬──┐
│+1│-2│+3│+1│
└──┴──┴──┴──┘

Now if we open boxed, we’ll have our array of lines:

    lines =. > boxed
+1
-2
+3
+1

And now we can map ". over that array to get our array of numbers.

    numbers =. 0 "./ lines
1 _2 3 1

And the answer to our problem is the sum of numbers.

    +/ numbers
3

Here’s my first working solution:

    input =. 1!:1 < '/Users/pcorey/advent_of_code_2018/day_01/input'
    boxed =. cutopen input
    lines =. > boxed
    numbers =. 0 "./ lines
    +/ numbers

Part Two

My first instinct for solving this problem is to do it recursively. I might be able to define a dyadic verb that accepts my current list of frequencies and a list of changes. If the last frequency in my array exists earlier in the array, I’ll return that frequency. Otherwise, I’ll append the last frequency plus the first change to my frequencies array, rotate my changes array, and recurse.

After many struggles, I finally landed on this solution:

    input =. 1!:1 < '/Users/pcorey/advent_of_code_2018/day_01/sample'
    boxed =. cutopen input
    lines =. > boxed
    numbers =. 0 "./ lines

    change_frequency =. 4 : 0
      frequency =. {: x
      change =. {. y
      frequency_repeated =. frequency e. (}: x)
      next_x =. x , (frequency + change)
      nexy_y =. 1 |. y
      next =. change_frequency ` ({:@}:@[) @. frequency_repeated
      next_x next next_y
    )

    0 change_frequency numbers

This works great for example inputs, but blows the top off my stack for larger inputs. It looks like J’s max stack size is relatively small. Recursion might not be the best approach for these problems.

Looking into other techniques for working without loops, I learned that you can use the ^:_ verb to “converge” on a result. It will repeatedly apply the modified verb until the same result is returned.

I refactored my verb to take and return my frequencies array and my changes array as a boxed tuple, and converge on that verb until I get a repeated result. That repeated result holds my repeated frequency:

    input =. 1!:1 < '/Users/pcorey/advent_of_code_2018/day_01/sample'
    boxed =. cutopen input
    lines =. > boxed
    numbers =. 0 "./ lines

    package_next =. 4 : 0
      (x,({:x)+({.y));(1|.y)
    )

    package_result =. 4 : 0
      x;y
    )

    change =. 3 : 0
      frequencies =. >@{. y
      changes =. >@{: y
      frequency =. {: frequencies
      change =. {. changes
      repeated =. frequency e. (}: frequencies)
      next =. package_next ` package_result @. repeated
      frequencies next changes
    )

    result =. change^:_ (0;numbers)
    echo 'Repeated frequency:'
    {:@:{.@:> result

Notes

  • $: doesn’t seem to refer to the outermost named verb. Recursion wasn’t working as I expected with $:. Replacing it with the named verb worked perfectly.
  • J seems to have a short stack. Note to self: avoid deep recursion.
  • J doesn’t support tail call optimization.
  • ^:_ and variants can be used as an iterative alternative to recursion.
  • Use boxes like tuples.
  • Use echo for debug printing.

November 30, 2018

Unrelenting Technology (myfreeweb)

Looks like NetBSD is already working on the EC2 AArch64 instances! My attempt at... November 30, 2018 11:32 PM

Looks like NetBSD is already working on the EC2 AArch64 instances! My attempt at running FreeBSD there failed: for mysterious reasons, the system reboots just after the last loader.efi message..

Trying to do anything system-level on EC2 is incredibly frustrating. There is STILL no read-write access to the serial console, because Bezos doesn't believe in debugging or something >_<

Also, about the ARM instances themselves. I am happy to see a big player enter the ARM space. And with custom (Annapurna) chips, even. (Though they'd have much better performance if they just bought some Ampere eMAGs or Cavium ThunderX2s.)

But what's up with that price? Did anyone at AWS ever look at Scaleway's pricing page?! On-demand pricing for a single core EC2 ARM instance is almost 20 bucks per month! While Scaleway offers four ThunderX cores for three euros per month!! Sure sure Scaleway is not a big player and doesn't have a huge ecosystem and is getting close to being out of stock on these ARM instances.. but still, 1/4 the cores for 5x the price.

(Spot pricing is better of course.)

Frederic Cambus (fcambus)

Running a free public API, a post-mortem November 30, 2018 10:29 PM

It's been a little bit more than three years since Telize public API was permanently shut down on November 15th, 2015. I have previously written about the adventure itself, and about the decommissioning of the API.

Before shutting down the public API of Telize, a paid version was launched on Mashape to ease the transition for those who couldn't host their own instances. The Mashape API Marketplace became a part of RapidAPI last year, the service is still running and will keep doing so for the foreseable future. You can support my work on Telize by subscribing to the service.

While a small fraction of the userbase switched to the paid API, the vast majority didn't, and the number of requests exploded due to retries, as detailed in the article about the API decommission. One thing I wondered at the time was how long it would take for the traffic to become negligible. The Internet is a very strange place and things can go unnoticed for a very long time, years sometimes. Of course, Telize case is no exception.

Every year since the public API was closed down, I've been logging requests for a few days in a row to get a glimpse of how many of them were still being made. While the number of unique IP addresses querying the API kept decreasing, the amount of requests themselves went up again compared to last year.

2016-11-06 - Requests: 51,896,923 - Unique IPs: 2,543,814
2016-11-07 - Requests: 56,427,258 - Unique IPs: 2,756,065
2016-11-08 - Requests: 53,641,121 - Unique IPs: 2,746,005
2016-11-09 - Requests: 53,704,140 - Unique IPs: 2,536,632
2016-11-10 - Requests: 53,194,946 - Unique IPs: 2,525,167
2016-11-11 - Requests: 50,444,003 - Unique IPs: 2,652,730
2016-11-12 - Requests: 49,224,863 - Unique IPs: 2,670,926
2016-11-13 - Requests: 48,526,303 - Unique IPs: 2,492,765
2017-11-10 - Requests: 35,325,037 - Unique IPs: 1,736,815
2017-11-11 - Requests: 33,582,167 - Unique IPs: 1,613,161
2017-11-12 - Requests: 33,334,836 - Unique IPs: 1,587,549
2017-11-13 - Requests: 36,131,909 - Unique IPs: 1,593,255
2017-11-14 - Requests: 34,457,433 - Unique IPs: 1,571,144
2017-11-15 - Requests: 33,225,149 - Unique IPs: 1,563,845
2018-11-12 - Requests: 50,612,559 - Unique IPs:   611,302
2018-11-13 - Requests: 50,858,236 - Unique IPs:   640,836
2018-11-14 - Requests: 51,991,454 - Unique IPs:   661,410
2018-11-15 - Requests: 53,008,712 - Unique IPs:   689,646
2018-11-16 - Requests: 51,651,814 - Unique IPs:   686,646
2018-11-17 - Requests: 49,236,779 - Unique IPs:   662,717
2018-11-18 - Requests: 47,237,596 - Unique IPs:   692,718
2018-11-19 - Requests: 51,679,888 - Unique IPs:   735,396
2018-11-20 - Requests: 50,245,134 - Unique IPs:   755,177
2018-11-21 - Requests: 50,745,725 - Unique IPs:   773,949
2018-11-22 - Requests: 50,609,750 - Unique IPs:   786,963
2018-11-23 - Requests: 49,991,775 - Unique IPs:   687,652
2018-11-24 - Requests: 47,479,703 - Unique IPs:   584,058
2018-11-25 - Requests: 47,346,829 - Unique IPs:   597,153

Bandwidth usage, measured with nload:

Incoming:






                                                                    Curr: 2.44 MBit/s
                                                                    Avg: 2.08 MBit/s
  .  ....        ...  .     ...   ......  ...     ..           ...  Min: 1.76 MBit/s
#########################|######################|#################  Max: 2.70 MBit/s
##################################################################  Ttl: 2217.35 GByte
Outgoing:






                                                                    Curr: 1.22 MBit/s
                                                                    Avg: 1.07 MBit/s
                                                                    Min: 904.10 kBit/s
     ....                   ..                                   .  Max: 1.45 MBit/s
################################################|#################  Ttl: 1111.68 GByte

So more than 3 years after the decommission, I'm still getting around 50 millions daily requests. I'm honestly quite astonished to notice that the numbers went up again significantly this year.

Below is a report of user agents which performed more than 1M daily requests on November 12th 2018, top offenders being Android applications and Wordpress sites… How surprising.

Apache-HttpClient/UNAVAILABLE                  24,729,923
Dalvik/2.1.0                                    1,113,530
WordPress/4.2.21                                3,200,750
WordPress/4.1.24                                2,223,350
WordPress/4.3.17                                1,212,849

On a more positive note, those are recent Wordpress releases, which means it might be possible to identify the plugins performing those requests and contact their authors.

Regarding the open source project itself, I released 2.0.0 back in March, which is now using GeoIP2/GeoLite2 databases, as GeoIP/GeoLite databases have been deprecated since April. I'm currently working on a rewrite in C using Kore, which will bring in a couple of improvements compared to the current version. I will write about the new iteration in a following post.

Wallaroo Labs (chuckblake)

Reasons to Scale Horizontally November 30, 2018 03:00 PM

Here at Wallaroo Labs, we build Wallaroo a distributed stream processor designed to make it easy to scale real-time Python data processing applications. That’s a real mouthful. What does it mean? To me, the critical part is the “scale” from “easy to scale.” What does it mean to easily scale a data processing application? In the case of Wallaroo applications, it means that it’s easy to scale those applications horizontally.

Derek Jones (derek-jones)

The 520’th post November 30, 2018 01:34 AM

This is the 520’th post on this blog, which will be 10-years old tomorrow. Regular readers may have noticed an increase in the rate of posting over the last few months; at the start of this month I needed to write 10 posts to hit my one-post a week target (which has depleted the list of things I keep meaning to write about).

What has happened in the last 10-years?

I probably missed several major events hiding in plain sight, either because I am too close to them or blinkered.

What did not happen in the last 10 years?

  • No major new languages. These require major new hardware ecosystems; in the smartphone market Android used Java and iOS made use of existing languages. There were the usual selection of fashion/vanity driven wannabes, e.g., Julia, Rust, and Go. The R language started to get noticed, but it has been around since 1995, and Python looks set to eventually kill it off,
  • no accident killing 100+ people has been attributed to faults in software. Until this happens, software engineering has a dead bodies problem,
  • the creation of new software did not slow down from its break-neck speed,
  • in the first few years of this blog I used to make yearly predictions, which did not happen (most of the time).

Now I can relax for 9.5 years, before scurrying to complete 1,040 posts, i.e., the rate of posting will now resume its previous, more sedate, pace.

November 29, 2018

Bogdan Popa (bogdan)

Announcing geoip November 29, 2018 10:00 AM

I released geoip today. It’s a Racket library for working with MaxMind’s geolocation databases. It’s got a tiny API surface (3 functions!), but it should be useful to anyone needing to do geolocation in Racket. As always, check it out and let me know what you think! BTW, I streamed the whole process on Twitch so, if that’s your thing, you can check out the recordings here.

November 28, 2018

Derek Jones (derek-jones)

Half-life of software as a service, services November 28, 2018 12:07 AM

How is software used to provide a service (e.g., the software behind gmail) different from software used to create a product (e.g., sold as something that can be installed)?

This post focuses on one aspect of the question, software lifetime.

The Killed by Google website lists Google services and products that are no more. Cody Ogden, the creator of the site, has open sourced the code of the website; there are product start/end dates!

After removing 20 hardware products from the list, we are left with 134 software services. Some of the software behind these services came from companies acquired by Google, so the software may have been used to provide a service pre-acquisition, i.e., some calculated lifetimes are underestimates.

The plot below shows the number of Google software services (red) having a given lifetime (calculated as days between Google starting/withdrawing service), mainframe software from the 1990s (blue; only available at yearly resolution), along with fitted exponential regression lines (code+data):

Number of software systems having a given lifetime, in days

Overall, an exponential is a good fit (squinting to ignore the dozen red points), although product culling is not exponentially ruthless at short lifetimes (newly launched products are given a chance to prove themselves).

The Google service software half-life is 1,500 days, about 4.1 years (assuming the error/uncertainty is additive, if it is multiplicative {i.e., a percentage} the half-life is 1,300 days); the half-life of mainframe software is 2,600 days (with the same assumption about the kind of error/uncertainty).

One explanation of the difference is market maturity. Mainframe software has been evolving since the 1950s and probably turned over at the kind of rate we saw a few years ago with Internet services. By the 1990s things had settled down a bit in the mainframe world. Will software-based services on the Internet settle down faster than mainframe software? Who knows.

Based on this Google data, the cost/benefit ratio when deciding whether to invest in reducing future software maintenance costs, is going to have to be significantly better than the ratio calculated for mainframe software.

Software system lifetime data is extremely hard to find (this is only the second set I have found). Any pointers to other lifetime data very welcome, e.g., a collection of Microsoft product start/end dates :-)

November 27, 2018

Wallaroo Labs (chuckblake)

Using Wallaroo with PostgreSQL November 27, 2018 12:00 PM

Introduction In the last blog post, I gave an overview of our new Connectors APIs, discussing how they work under the hood and going over some of the examples we provide. In this post, we are going to take a deeper dive into how to build a connector to pull data from PostgreSQL. We’ll also talk a bit about some of the different approaches for building both external source and sink connectors.

Indrek Lasn (indreklasn)

Well written article! November 27, 2018 09:41 AM

Well written article!