Planet Crustaceans

This is a Planet instance for lobste.rs community feeds. To add/update an entry or otherwise improve things, fork this repo.

September 17, 2019

Jan van den Berg (j11g)

PHP: how did it become so popular? September 17, 2019 07:42 PM

PHP empowers a gigantic part of the internet. So it is, by definition, a very popular (i.e. prevalent) language. But it also very popular (i.e. well-liked) to dislike PHP as a — serious — language.

This could be explained as one of the side effects of that same popularity. With great exposure, comes greater scrutiny. But that would be too easy.

Picture of probably one of the most pragmatic people on the internet ❤

Because when discussing PHP as a serious computer programming language there are serious arguments of what PHP exactly lacks as a language. And of course a lot of these things are fuel for endless, and useless, debate. Also because even the author of PHP, Rasmus Lerdorf, would probably agree on most!

It seems like a popular past time to debate everything that is bad about PHP. So over the years there have been several (viral) blogposts about this: here, here, here and here (which spawned this great response).

Even with all the changes made to PHP in recent years, this is still a common sentiment regarding PHP. This is also illustrated by a recent remark from highly respected programmer Drew DeVault:

So PHP certainly has it fair share of critique (it is no Lisp!).

So why is it so popular?

I certainly have NO need to add my two cents to this debate. But I am VERY interested to investigate why and how PHP — a widely criticised language — became so incredibly popular.

But you don’t have to read this post, because Rasmus Lerdorf explains it rather well in the first part of this next video. (There is also another video of the same talk at a different location, but the slides are easier to follow in this one.)

I thought the talk was very interesting and he drops some killer quotes, so I highly recommend it! For easy reference, here follow the key slides and quotes from his presentation that, in my opinion, help answer the question how PHP became so popular. Of course this may be a biased view (he is the creator) but I am open to different views.

C API for the web

My grand scheme was to write a C API for the web. So to abstract away all the web specific things that you needed to know to get your business logic up online.

Rasmus Lerdorf

This was the plan.

Straight away Rasmus explains he never intended to design a “full-blown” real programming language, but more a templating system. And he wasn’t happy with CGI.pm (“simply writing HTML in another language”). He wanted a C API for the web, where he abstracted away all boiler plate stuff he always needed to write when making a web application. The idea being that the business logic would be written in a real language (C or C++). And PHP was the templating language to present the data. However “the web moved too fast and there weren’t enough C developers in the world”. And with this, his programming language (by request) grew.

What is most striking about this remark, is his dedication towards the end goal: get your business logic online! This initial, pragmatic approach is something that returns time and time again in the development of PHP. With PHP being a means to an end. Just another tool. Nothing to be religious about.

6 months

Rasmus also explains, more than once, that even though he added features he was never under the impression that this little language would survive. In the first few years he was thoroughly convinced it would only last about six more months before something better would come along. And that something would solve the very real problems his language could already solve.

Therefore he also states that for the first 5 to 7 years PHP was “NOT a language, it was a templating system”. But apparently that something never did and reluctantly his templating system grew into a programming language (with recursion and all that).

LAMP wasn’t an accident

However, for someone who was convinced his templating system would die off within six month, he was exceptionally good in making the right decisions in improving the further adoption of PHP.

The right calls!

mod_php is probably the most important decision from all of these. It made certain that PHP would tie in nicely with Linux, Apache en MySQL and thus create one of the most powerful (free) software stacks ever. Rasmus is very clear on why he thought it was necessary to become an integral part of this ecosystem. Say what you will, but he definitely made the right call here.

mod_perl was too late to the game. And too complex and expensive (you needed a single box in a time when VMs where not a thing) when it did arrive. Python (the other P) had a different focus and things like Django were many years in the future. But there are more choices Rasmus made that worked, and that the early web clearly needed:

More right calls.

Speed and efficiency

PHP is amazingly good at running crap code.

Rasmus Lerdorf

Anybody with a good idea, could put their idea online and this has made the web a much much better place.

Rasmus Lerdorf

PHP is probably the most pragmatic language ever. A kind of pragmatism that lowers the barrier to entry. Which creates one of the main points of criticism. Because it makes for bad programmers. Programmers that dont really know what they’re doing because PHP does all the work for them. You can discuss this all your can, but one thing is clear: Rasmus knows who uses his language and makes certain to lower as many barriers as he can.

Critique

Rasmus is of course a very intelligent guy, he knows all this. He also explains some of the reasons of strange language design decisions he made. Some have a good explanation, others he fully admits he made the wrong decision. Admirably his ego is also subjected to pragmatism.

Some decisions are up for debate.

Pragmatic Hypertext Preprocessor

PHP is a tool. PHP is not important, what you do with it is important.

Rasmus Lerdorf

Arguing about the color of the hammer used to built that thing is just moronic. We can’t lose track of what we’re doing here why we’re programming. We’re programming to solve a problem and hopefully it’s a problem that actually matters.

Rasmus Lerdorf

PHP is not going away. Why? Because some of the things that PHP solves, are not yet solved by any other language (don’t add me). I am comp-sci graduate, and even though I program very little and I have probably written more code in PHP code than any other language. I am not building big important business applications, most of the time I just want something online (from a database) and I want it fast! And PHP is my hammer. Time and time again I try to look at other things. And I get it. There is serious critique. But if you want to change that, you can! Nobody is stopping you. So stop complaining and get to it. Even though I don’t share the otherwise harsh tone of this post I do share that building something compelling is up for grabs!

And when someone finally does that, then maybe Rasmus — after 25 years — will finally get his wish. It just took a little bit longer than six months.

The post PHP: how did it become so popular? appeared first on Jan van den Berg.

September 16, 2019

Pete Corey (petecorey)

Elixir Style Conditions in Javascript September 16, 2019 12:00 AM

Elixir has a useful control flow structure called cond that lets you branch on arbitrary conditions. Unlike the more common switch control structure (case in Elixir), cond doesn’t match against a predetermined value. Instead, it evaluates each condition, in order, looking for the first one that evaluates to a truthy value (not nil or false).


numbers = [1, 2, 3]
result = cond do
  4 in numbers ->
    :four
  6 == Enum.sum(numbers) ->
    :sum
  true ->
    :default
end

This is all probably old hat to you.

As I mentioned, cond can be an incredibly useful control structure, and there are times when I’ve missed it while working in languages like Javascript that only have switch expressions.

A traditional Javascript implementation of the above (with a little help from Lodash) would look something like this:


let numbers = [1, 2, 3];
if (_.includes(numbers, 4)) {
  var result = "four";
} else if (6 === _.sum(numbers)) {
  var result = "sum";
} else {
  var result = "default";
}

However, I recently stumbled upon a trick that lets you implement a switch statement in Javascript that behaves very similarly to a cond expression in Elixir. The key is to switch on the value of true. The case expressions that evaluate to true will match, and their corresponding statements will be evaluated in order.


let numbers = [1, 2, 3];
switch (true) {
  case _.includes(numbers, 4):
    var result = "four";
    break;
  case 6 === _.sum(numbers):
    var result = "sum";
    break;
  default:
    var result = "default";
    break;
}

Whether or not this is any more useful or readable than a series of if/else blocks is debatable. That said, this is definitely an interesting example of perspective shifting and seeing old code in a new light. Hopefully you find it as interesting as I do.

September 15, 2019

Derek Jones (derek-jones)

Team DNA-impersonators create a business plan September 15, 2019 09:12 PM

This weekend I was at the Hack the Police hackthon, sponsored by the Metropolitan Police+other organizations. My plan was to find an interesting problem to help solve, using the data we were told would be available. My previous experience with crime data is that there is not enough of it to allow reliable models to be built, this is a good thing in that nobody wants lots of crime. Talking to a Police intelligence officer, the publicly available data contained crimes (i.e., a court case had found somebody guilty), not reported incidents, and was not large enough to build allow a good model to be built.

Looking for a team to join, I got talking to Joe and Rebecca. Joe had discovered a very interesting possible threat to the existing DNA matching technique, and they were happy for me to join them analyzing this threat model; team DNA-impersonators was go.

Some background (Joe and Rebecca are the team’s genetic experts, I’m a software guy who has read a few books on the subject; all the mistakes in this post are mine). The DNA matching technique used by the Police is based on 17 specific sequences (each around 100 bases, known as loci), within the human genome (which contains around 3 billion bases).

There are companies who synthesize sequences of DNA to order. I knew that machines for doing this existed, but I did not know it was possible to order a bespoke sequence online, and how inexpensive it was.

Some people have had their DNA sequenced, and have allowed it to be published online; Steven Pinker is the most famous person I could find, whose DNA sequence is available online (link not given; it requires work+luck to find). The Personal Genome Projects aims to sequence and make available the complete genomes of 100,000 volunteers (the UK arm of this project is on hold because of lack of funding; master criminals in the UK have a window of opportunity: offer to sponsor the project on condition that their DNA is included in the public data set).

How much would it cost to manufacture bottles of spray-on Steven Pinker DNA? Is there a viable business model selling Pinker No. 5?

The screen shot below shows a quote for 2-nmol of DNA for the sequence of 100 bases that are one of the 17 loci used in DNA matching. This order is for concentrated DNA, and needs to be diluted to the level likely to be found as residue at a crime scene. Joe calculated that 2-nmol can be diluted to produce 60-liters of usable ‘product’.

Quote for synthesis of 100 bases of human DNA.

There was not enough time to obtain sequences for the other 16-loci, and get quotes for them. Information on the 17-loci used for DNA matching is available in research papers; a summer job for a PhD student to sort out the details.

The concentrate from the 17-loci dilutes to 60-liters. Say each spray-on bottle contains 100ml, then an investment of £800 (plus researcher time) generates enough liquid for 600-bottles of Pinker No. 5.

What is the pricing model? Is there a mass market (e.g., Hong Kong protesters wanting to be anonymous), or would it be more profitable to target a few select clients? Perhaps Steven Pinker always wanted to try his hand at safe-cracking in his spare time, but was worried about leaving DNA evidence behind; he might be willing to pay to have the market flooded, so Pinker No. 5 residue becomes a common occurrence at crime scenes (allowing him to plausible claim that any crime scene DNA matches were left behind by other people).

Some of the police officers at the hack volunteered that they knew lots of potential customers; the forensics officer present was horrified.

Before the 1980s, DNA profiling was not available. Will the 2020s be the decade in which DNA profiling ceases being a viable tool for catching competent criminals?

High quality photocopiers manufacturers are required to implement features that make it difficult for people to create good quality copies of paper currency.

What might law enforcement do about this threat to the viability of DNA profiling?

Ideas include:

  • Requiring companies in the bespoke DNA business to report suspicious orders. What is a suspicious order? Are enough companies in business to make it possible to order each of the 17-loci from different company (we think so)?
  • Introducing laws making it illegal to be in possession of diluted forms of other people’s DNA (with provisions for legitimate uses).
  • Attacking the economics of the Pinker No. 5 business model by having more than 17-loci available for use in DNA matching. Perhaps 1,000 loci could be selected as potential match sites, with individual DNA testing kits randomly testing 17 (or more) from this set.

Carlos Fenollosa (carlesfe)

September 14, 2019

Gonçalo Valério (dethos)

Firefox’s DoH, the good, the bad and the ugly September 14, 2019 06:16 PM

First of all, DoH stands for “DNS over HTTPS”.

So last week Mozilla announced that future versions of Firefox will use DoH by default, a decision that at first sight might seem innocuous or even good but thanks to some implementation details it ended up being very controversial.

The reaction that followed the announcement on many technology focused forums and communities was mostly negative (example 1, example 2 and example 3), pointing out many problems, mostly with the way it was implemented and with the default settings used by Mozilla.

One of the first outcomes was OpenBSD, a well known operating system, announcing that the version of Firefox distributed through its “package manager” will have this change disabled by default.

In this post I will try to summarize the core of whole controversy and list the pros and cons of it.

How does a DNS query work?

In a very brief and not 100% accurate way, when you try to visit a website such as www.some-example.com, your computer first asks a DNS server (resolver) for the IP address of that website, this server address is usually defined on your system either manually (you set it up) or automatically (when you join a given WiFi network, for example, the network will tell you a server you can use).

That server address generally is set system wide and will be used by all apps. If the server knows the location of the website it will tell you the answer, otherwise it will try to find the location using one of 2 approaches (I will avoid any details here) and come back to you with the result.

You browser will then use this result to fetch the contents of the website. The bellow image describes this flow:

Diagram of a client making a DNS query to a local server.Source: https://commons.wikimedia.org

This system is kind of distributed across many entities. Different people across the globe will contact different servers according to their settings and network/location.

DNS over HTTPS

The previously described flow already exists for decades and does not change with DoH, what changes is the way you contact the server in order to ask for the website location and the way this data is transmitted.

While the standard implementation uses UDP and the information travels in cleartext throughout the network (everybody can see it) with DoH this is done as an HTTP request that uses TCP with an encrypted connection, protecting you from malicious actors.

In the end this should be a good thing, but as we will see later on the post things will do south.

Current implementation

A great deal of the discussion this week was sparked by a blog post telling people to turn off Firefox’s DoH, the main complains resolve around not the DoH in itself but the way Mozilla decided to introduce it. Being opt-out and not opt-in, the browser ignoring system configuration and using the servers of a single company by default.

With the current implementation we end up with:

The good

The good part is the obvious reason for using DNS over HTTPS, all your websites queries are encrypted and protected while in transit on the network. It is the extra protection that has been needed for “DNS traffic” for a while.

The bad

The first bad part is that the browser will work differently from the rest of the apps which can cause confusion (why this URL work on the browser and not on my app?), the browser no longer will connect to the same server that was defined for the whole systems

Related to the above problem there is also special network configurations that will stop working such as internal DNS names, rules and filters that are often used on private networks and rely on the internal DNS servers. For these scenarios Mozilla described a series of checks and fallbacks (such as “canary domains”) to accommodate this situation, however they look like fragile hacks.

The ugly

The ugly part is that all DNS traffic from the browser will go to a single entity by default, no matter where you are or which network you are using, which raises privacy concerns and increases the centralization of the system. There is the option of manually setting up a different server however 99% of the users will rely on that single provider.

Conclusion

The overall the intention was good and having encrypted DNS resolution is something that has been required for a very long time but hasn’t become mainstream yet.

The core of the problem with Mozilla’s approach is making it “opt-out”, which means all users will now tell a single “Mozilla partner” the websites they visit by default, without being aware of it.

It will also create some problems to solutions that are deployed network wide and rely on setting certain DNS configurations, since Firefox will not respect them. We can also expect an increased centralization on a system that has been previously working the other way around.

Lets hope that in the future DoH and other encrypted alternatives become standardized so we can continue to use DNS as we always did and don’t have to manage it on every application.

Caius Durling (caius)

curl --resolve September 14, 2019 05:10 PM

Sometimes it's useful to be able to craft a request to one server, using a DNS name that's either not defined or currently pointed to a different IP. (Migrating webservers, testing a new webserver config out, etc.)

Historically for HTTP calls this was easy, just set the Host header as you make the http request to the IP directly:

 curl -H "Host: caiustheory.com" http://10.200.0.1/

However, HTTPS throws a bit of a spanner in the works, if we just try to connect using an overridden Host header, we get an error back from the server if it's not configured with a certificate for the IP address:

 $ curl -H "Host: caiustheory.com" https://10.200.0.1/
curl: (51) SSL: no alternative certificate subject name matches target host name '10.200.0.1'

Usually at this point I'd just start editing /etc/hosts to add 10.200.0.1 caius.name to it and carry on testing. This is a pain when you're testing more than one server, or you're on a machine where you don't have root access to edit /etc/hosts.

In later versions of curl there's a solution for this built into the binary, in the form of the --resolve flag. You can tell it to override the DNS lookup for a specific hostname/port combination. This in turn means that the correct host is forwarded to the server for the correct SSL certificate to be chosen to serve the request based on host.

It takes the form --resolve HOST:PORT:IP where HOST is the human-friendly host, PORT is the webserver's port (convention is 80 for HTTP, 443 for HTTPS) and IP being the destination IP you want to hit. (As opposed to the one in DNS currently.)

$ curl --silent --head --resolve caiustheory.com:443:10.200.0.1 https://caiustheory.com | head -n1
HTTP/2 200

And voila, you don't need to fiddle with editing /etc/hosts. Just use --resolve to hit a different IP for a given host.

September 13, 2019

Patrick Louis (venam)

September 2019 Projects September 13, 2019 09:00 PM

In the blink of an eye 6 months have gone by. Since then, I’ve written a single article about time on the internet and thus the blog needs an update on my latest endeavours.

Psychology, Philosophy & Books

Language: brains
Explanation:

The Stand

I’ve let go of my daily diary and happiness tracking application, the ones I hinted in my previous post, and it was apparent that I find genuine happiness in books. In short, I enjoy reading much more than I thought I was.

Here’s the reading list of the past few months.

  • Who moved my cheese - Spencer Johnson
  • Peaks and valeys - Spencer Johnson
  • Mature optimization handbook - Carlos Bueno
  • Leadership presence - HBR Emotional Intelligence Series
  • irobot - Isaac Asimov
  • The pleasure of finding things out - Richard Feynman
  • The stand - Stephen King (My longest read to date)
  • Smoke and Mirror - Neil Gaiman
  • Out of the Dark - Gregg Hurwitz
  • Childhood’s end - Arthur Clarke
  • Clean Architecture - Robert Martin

And the countless articles from the newsletter, books that aren’t physical, or anything that my memory is unkind to.

Furthermore, I’m currently in the process of reading “What If?” by Randall Munroe, “how to” by the same author, “The science of food” by Marty Jopson which I’m enjoying with my SO, and eagerly devouring my copy of “Software Architecture in Practice 3rd edition”.

As for podcasts, I’ve exhausted my list which means I’m keeping up with the dozen subscriptions I have. Let’s mention them.

  • The Joe Rogan Experience
  • Stuff You Should Know
  • Snap Judgement
  • Philosophy Bites
  • Planet Money
  • The Philosopher’s Zone
  • Radiolab
  • Making Sense
  • The Psychology Podcast
  • LKT - Let’s know things
  • You Are not so Smart
  • Hidden Brain
  • Team Human
  • Philosophize This!
  • Intercepted
  • Modern Love
  • The Food Chain
  • Lore
  • InfoQ podcast
  • All in the mind
  • Science Friday
  • Big Questions Podcast
  • Levar Burton Reads
  • The knowledge Project
  • Darknet Diaries
  • CMV - Change my view
  • 50 things that made the modern economy
  • Invisibilia
  • Hi-Phi Nation
  • 30 Animals that made us smarter
  • The history of GNOME
  • The end of the world with Josh Clark
  • A history of the world in a 100 object
  • General philosophy
  • The Rhine

I’ve given The Rhine series a second listen. History gives fantastic insights on where the shape of our world comes from. It draws the lines that aren’t apparent.
Giving such a long list wouldn’t matter if I wasn’t sharing my favorite picks. Obviously, this would include The Rhine, but also Planet Money, The Food Chain, Levar Burton Reads, Science Friday, Hidden Brain, and LKT. Those are the on-going podcasts that keeps me from dying on the road each week.

Learning, Growth, & Community

Language:growth
Explanation:

clean architecture

I’ve matured professionally. It’s been two consecutive years that I’ve taken part in a small team, initially an XP duo, later on a trio, and then virally creating new roles all throughout the company, building a highly secure and accredited eSIM solution from scratch.

The project is based on new specifications and thus uncharted territories. This has allowed me to get even more comfortable reading specifications, protocols, architectures, and RFCs. You got to have patience and reading skills in this field and I don’t think I’m missing any of those. It’s similar to what I used to do in the newsletter but having it as my day-to-day job has been an added benefice.

The XP methodology, though informal, is intellectually stimulating. I’ve learned extensively from my seniors not only through seeing them handle tasks but also via discussions (sometimes heated) about technical decisions that needed to be made. And there were a lot of those decisions, this is the kind of flexibility that let you experiment. Many of my misconceptions have been shattered.
If you know one thing about me, through my blog, newsletter, or podcast, it’s that I’ve got a knack for shinning light on software stacks and how they fall into place, I like to explain systems from above. I have a penchant for software architecture, drawing diagrams, understanding why things are built the way they are.
However, time is a costly parameter in the life’s equation and I had to give up something to get another in return.

Edward

Consequentially, I’ve put the newsletter on hiatus. It pains me to get a bit further away from Unix, already knowing I’ve discontinued the nixers podcast more than a year ago. Unix stays in my heart!
I’m thankful to Vermaden for his participation throughout the newsletter, he’s still continuing his valuable news on his blog.
Nevertheless, next February I’ll be in Belgium for Fosdem, the tickets are booked and I’m ecstatic about finally assisting in person. (See you there)

Now it’s been almost a month that I’ve begun my software architecture studies and I plan to keep track of the path I’m undertaking. I’ll write articles on the topic as soon as I’m experienced enough. Meanwhile, here’s what I have in mind as my future steps.
To help me remember important details as I learn them, I’m using a software called Anki. Apart from the two books I’ve previously listed, namely “Clean Architecture”, which I finished, and “Software Architecture in Practice”, I’m following a Youtube channel called “Software Architecture Mondays”, I’m training to draw diagrams the right way, I’m reading a couple of articles everyday analogous to what I was doing for the newsletter, and I’m planning to register for an official certification later on.
That’s the plan.

Apart from these, I’m reviewing my algorithm knowledge. Doing a couple of exercises a week on leetcode and geeksforgeeks. This helps keep my problem solving skills up to date, or at least prepares me for the modern and radically tough interview style.

And all of this happens on my redesigned workstation! First of all, I’m now on a standing desk. Secondly, I got a new mechanical keyboard with red switches for work. Thirdly, I got o-rings dampeners. Fourthly, I got a 22inch screen for my home-station. Fifthly, I got a wrist rocker for the new keyboard. And finally, I got a trackball.

workplace

Other Learning

Language: gray matter
Explanation:

curiosity stream

My SO influenced me to install a brain teaser application called Elevate. I’ve been playing it ever since, a 142 days streak. I should note too that I’ve bought the pro version.

Another of my girlfriend’s influence, is a subscription to a documentary streaming service named Curiosity Stream, which was initially intended for her but that I happen to use quite often.

To contrast those, I’ve haven’t continued the Spanish refreshment that I was intending. It simply isn’t useful at this point in time.

Ascii Art & Art

Language: ASCII
Explanation:

Abstract

I’ve pushed 4 pieces since the last post, that may not seem like an impressive number but quantity doesn’t mean quality. Of those pieces, one was the last dinosaur in the series, a triceratops, another is the amazing totem pole piece that I’ve added above this section, the next one is a piece dedicated to the now defunct Linux Journal that was featured in one of their issues (command line July 2019), and the last one is a piece christened Alamanni that reached third place at Evoke 2019.
Moreover, the Impure Ascii art group released two packs, 72, and 73.

Aside from ASCII art, I’ve painted a single canvas. Instead, what I’ve focused on the most is singing. I’ve been training since February, using an application called Sing Sharp, another one called Perfect Ear, and wrapping this with a small playlist to sing-along in the car while in traffic. I can affirm objectively, through the app rating, that I’ve progressed.

2bwm and Window management

Language: C
Explanation:

There hasn’t been much change to 2bwm. A user named netcrop has been sending pull requests and I’ve reviewed them.
I’ve tried to implement centered pop-ups but there is still one issue I didn’t catch yet.

Life and other hobbies

Language: life
Explanation:

miami

In June I spent 2 weeks in the USA, one week in South Beach Miami and the other in New York City. Even though the 12-15 hours in the plane were intense it was all worth it. After seeing those two cities I’ve formed a better opinion of urban planning.

During my stay in Miami I reignited my love of swimming. Therefore when I came back to Lebanon I had to give it another try. Still, in spite of all the love I have, my attempts were thwarted. The pollution of the beach in Lebanon is just incomparable to what I’ve experienced in Miami. In sum, I was discouraged.

Summer isn’t the time for mushrooms in Lebanon but I’ve nevertheless relished in the wonders of fresh King Oyster, fresh Shiitake, fresh Portobello, fresh White Button, dried morels, dried Porcini, and more. Adding to those, I’m testing taking Lion’s Mane and Cordyceps supplements, so far so good.
Aside from those, I’ve found ginseng tea to be fantastically savory and revitalizing.

On that topic, I’ve been cooking many new recipes. Trying to use the oven more. My cooking page though hasn’t been updated in a while and it’s not as presentable as I wish it was and thus I haven’t been adding pictures on it.

Finally, summer is all about flowers and plants. So I’ve spent a big amount of time investing in my garden, cutting branching, cleaning, planting pumpkins, growing from cuttings, etc..

Now

Which all leads to what’s in store for tomorrow. More of the same but upgraded.

The next months I’m going to put the emphasis on software architecture. That’s really the single new and most important activity I’m going to partake in.
Plus whatever I’m already doing but more of it!

This is it!

As usual… If you want something done, no one’s gonna do it for you, use your own hands.
And let’s go for a beer together sometime, or just chill.

Gustaf Erikson (gerikson)

July September 13, 2019 07:30 PM

Introducing HN&&LO; September 13, 2019 07:28 PM

HN&&LO is a web page that scrapes the APIs of Hacker News and Lobste.rs and collates those entries that share a URL.

This lets you easily follow the discussion on both sites for these links.

I wrote this tool because I’m much more active on Lobste.rs than on HN, but I would like to keep “tabs” on both sites. I’m especially interested in how the information is disseminated between the sites.

“Slow webapps”

I’ve long liked to grab stuff via a web API, stuff it into a DB, and output a web page based on the data. This project was a first attempt to use a templating engine in Perl, and I’m ashamed to say I’ve taken so long to understand the greatness of this approach.

The data updates hourly, there’s rarely more than a couple of new entries on Lobste.rs per hour.

In fact, this can illustrate the difference in scale between the two sites.

Time period: 5 Jul 2019 11:19 to 8 Aug 2019 23:59 (both times in UTC).

  • Number of submissions with URLS on Hacker News: 26,974
  • Number of submissions on Lobste.rs: 899

Average number of submissions per hour: 33 for HN, 1 for Lobste.rs.

Once updated, a static web page is generated. This keeps overhead low.

Differences between Hacker News and Lobste.rs

Coincidentally, while I was writing this post, an article was published in the New Yorker about the moderators (all two of them) of Hacker News. This sparked a discussion on HN (obviously) but also on Lobster.rs (see both linked here).

To me, what makes Lobste.rs better is that it’s much smaller (one can easily catch up with a day’s worth of comments), the editing tools are more sophisticated, and the tag system allows one to determine whether a post is interesting or not. HN is much harder to grasp.

I think the tendencies noted in the article are common to both sites - male, youngish software developers with a rationalist bent. But I think Lobste.rs is a bit more self-aware of this fact (or rather, critiques are more visible there). In some ways, Lobste.rs, being a reaction to HN, defines itself culturally both for and against the bigger site.

Source and TODO

If you’re interested in pedestrian Perl code, source is here.

TODO list is here

Derek Jones (derek-jones)

Natural elimination, or the survival of the good enough September 13, 2019 01:54 PM

Thanks to Darwin, the world is full of people who think that evolution, in nature, works by: natural selection, or the survival of the fittest. I thought this until I read “Good Enough: The Tolerance for Mediocrity in Nature and Society” by Daniel Milo.

Milo makes a very convincing case that nature actually works by: natural elimination, or the survival of the good enough.

Why might Darwin have gone with natural selection in his book, On the Origin of Species? Milo makes the point that the only real evidence that Darwin had to work with was artificial selection, that is the breeding of farm animals and domestic pets to select for traits that humans found desirable. Darwin’s visit to the Galápagos islands triggered a way of thinking, it did not provide him with the evidence he needed; Darwin’s Finches have become a commonly cited example of natural selection at work, but while Darwin made the observations it was not until 80 years later that somebody else spotted their relevance.

The Origin of Species, or to use its full title: “On the Origin of Species by means of natural selection, or the preservation of favored races in the struggle for life.” is full of examples and terminology relating to artificial selection.

Natural selection, or natural elimination, isn’t the result the same?

Natural selection implies an optimization process, e.g., breeders selecting for a strain of cows that produce the most milk.

Natural elimination is a good enough process, i.e., a creature needs a collection of traits that are good enough for them to create the next generation.

A long-standing problem with natural selection is that it fails to explain the diversity present in a natural population of some breed of animal (there is very little diversity in each breed of farm animal, they have been optimized for consistency). Diversity is not a problem for natural elimination, which does not reduce differences in its search for fitness.

The diversity produced as a consequence of natural elimination creates a population containing many neutral traits (i.e., characteristics that have no positive or negative impact on continuing survival). When a significant change in the environment occurs, one or more of the neutral traits may suddenly have positive or negative survival consequences; the creatures with the positive traits have opportunity time to adapt to the changed environment. A population whose members possess a diverse range of neutral traits has a higher chance of long-term survival than a population where diversity has been squeezed in the quest for the fittest.

I think that natural elimination also applies within software ecosystems. Commercial products survive if enough customers buy them, software developers need good enough know-how to get the job done.

I’m sure customers would prefer software ecosystems to operate on the principle of survival of the fittest (it reduces their costs). Over the long term is society best served by diverse software ecosystems or softwaremonocultures? Diversity is a way of encouraging competition, but over time there is diminishing returns on the improvements.

Pete Corey (petecorey)

Obverse and Under September 13, 2019 12:00 AM

I previously wrote about plotting an “amazing graph” using the J programming language. The solution I landed on looked something like this:


require 'plot'
f =: ] - [: #. [: |. #:
'type dot' plot f"0 p: i. 10000

Our verb, f, is taking a very explicit approach by making judicious use of “capped” ([:) verb trains. We’re essentially saying that f is (=:) the given number (]) minus (-) the base two (#.) of the reverse (|.) of the antibase two (#:) of the given number.

Several members of the J community pointed out to me that this verb could be simplified with the help of the “under” (&.) conjunction. Let’s dig into what “under” is, and how we can use it.

Under What?

The best way to think about “under” (&.), as explained by the NuVoc page on “under”, is to think in terms of domains and transformations in and out of those domains.

Verb v defines a transformation of the argument(s) (x and) y into the v-domain. Next, verb u operates on the transformed argument(s). Lastly the result is transformed back from the v-domain to the original domain.

In our example, the domain of our input is base ten, but the transformation we want to apply (reversal) needs to happen in the base two domain. “Under” (&.) can be used to transform our input into base two (#:), apply our reversal (|.), and transform the result of that reversal back to our original base ten domain with the obverse, or opposite, of our base two verb, anti base (#.):


f =: ] - |. &. #:

Notice that we’re not explicitly stating how to transform the result of our reversal back into our original domain. J knows that the obverse of #: is #., and automatically applies it for us.

Out of the box, J comes with many obverse pairings. “Open” (>), for example, is the obverse of “box” (<), and visa versa. This pairing is especially useful when applying transformations to boxed values:


   1+&.>1;2;3
┌─┬─┬─┐
│2│3│4│
└─┴─┴─┘

Check out a full listing of obverse pairs at the end of this Shades of J article.

Inferred Obverses

Even compound verbs built up of verbs with well-defined obverse pairings can be used with “under” (&.). J will correctly infer and apply the compound obverse without any intervention or instruction.

For example, if we wanted to unbox a list of values and then work with them in the “square root domain” (whatever that means), we could do something like this:


   1+&.([:%:>)1;2;3
┌────────────────┐
│4 5.82843 7.4641│
└────────────────┘

J takes each value, opens it and finds its square root ([:%:>), adds one to the result, and then squares and boxes up ([:*:<) the incremented value.

Explicit Obverses

Even more interestingly, if an obverse pairing isn’t defined or inferable for a given verb, J lets us define our own pairing using the “obverse” (:.) verb.

As an example, imagine that we have a JSON string holding an array of values. We want to parse our string, perform some operation on those values, and then serialize the resulting list back into JSON.

We can use the dec_json and enc_json verbs provided by the convert/json package, and tell J that the obverse of dec_json is enc_json:


   json =: dec_json :. enc_json

Running dec_json on a JSON array like '[1, 2, 3]' will return a list of boxed numbers, so we’ll want to open each of these boxes, perform our operation, and box the results back up. This sounds like another job for “under” (&.):


   transform =: 1&+&.>

All together, we can perform our transform “under” (&.) the json domain:


   transform &. json '[1, 2, 3]'
[2,3,4]

And our result is the JSON string '[2,3,4]'!

“Under” is definitely a very powerful conjunction, and I can see myself using it extensively in the future. Thanks to everyone in the J community who was kind enough to point it out and teach me something new!

September 11, 2019

Pages From The Fire (kghose)

Walk through of a UX decision September 11, 2019 07:44 PM

A software developer has to make decisions all the time about how to implement something. Decisions related to the user interface (UI) of a program strongly affect the user experience (UX) which is critical to the successful adoption of the software. In this post I will talk about the design of a particular aspect of some …

Jan van den Berg (j11g)

Glove Pond – Roger Thorpe September 11, 2019 06:51 PM

Roger Thorpe is just as good a writer as Douglas Coupland is. As a matter of fact, he is also as real as a Douglas Coupland character. Glove Pond is his first novel.

Glove Pond – Roger Thorpe (2007) – 91 pages

I got this book bundled with The Gum Thief, and I was thrown off guard by the high praises on the backcover by Coupland himself. How come I had never heard of this writer? But as soon as I started reading there was the unmistakable Coupland prose and cadence!

This book is the result of the work of the main character from the Gum Thief. Interestingly the main character in Glove Pond also deals with another author who also writes a book. Because, of course he is.

The book has a couple of interesting — and classic Coupland — ideas. E.g. what would happen to the world if everyone’s IQ dropped 50 points, or the other way around, if everyone all of a sudden would become a genius. Entertaining, hilarious and sometimes frightening thought experiments. And exactly what I like about Coupland.

The post Glove Pond – Roger Thorpe appeared first on Jan van den Berg.

Carlos Fenollosa (carlesfe)

La predicción del tiempo en tu calendario September 11, 2019 05:21 PM

(Even though I write my blog in English, this post is in Spanish for obvious reasons. Click here to translate it with Google)

Si te pasas el día mirando el calendario, agendando reuniones y eventos, y echas en falta tener a mano el tiempo que va a hacer, he creado una utilidad que te puere resultar muy interesante.

Se trata de mostrarte la predicción del tiempo en tu municipio en el mismo calendario

Es muy sencillo: escribe el nombre de tu municipio y pulsa el botón. No hay que registrarse, ni dar tus datos, ni pagar nada. Es una herramienta simple y anónima.

Es compatible con todos los teléfonos y ordenadores ya que usa tecnologías estándar. Tu dispositivo se encargará de ir actualizando las predicciones de manera regular.

Creé esta utilidad para mi uso personal al descubrir que no existía nada similar, y tras usarla unas semanas pensé que podía ser útil dar acceso a los demás.

Los datos están sacados de AEMET por lo que sólo funciona en el territorio español.

Tienes el enlace aquí: el tiempo en tu calendario

Tags: software, projects, spanish

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

September 10, 2019

Carlos Fenollosa (carlesfe)

The iPhone 11 & co. September 10, 2019 09:10 PM

This year's phone keynote has delivered, according to Apple, all-new products from the top down

Quite boring hardware unfortunately, as was expected.

  • Better cameras, though for use cases I'm not sure are very useful
  • Better battery life thanks to the A13 chip
  • Marginally better screen on the Pro phone
  • Always-on screen on the Watch, which is nice
  • Simple update on the entry-level iPad

The landing page for each phone is 60% camera features and 40% other features. Not saying that is wrong, on the contrary, the marketing team is doing their job as in my experience most people use their phones as an Instagram device.

Where I think Apple nailed it is with the Watch. They are really, really good at the health and fitness message, and the product itself is fantastic.

However, I will criticise them for two things.

First, the fact that they are not even advertising the full price for the phone, but rather an installment plan first, then a discounted price with the trade of an old phone, and only when you say "no" to these options you get the actual price. Let me reiterate that. When you visit Apple.com the price you see for the phones is not the actual retail price.

They are aware that their hardware is not attractive at those price points, but at the same time they can't lower them because of positioning. Well; to be precise, the iPhone 11 is actually sliiightly cheaper than last year's but, in my opinion, not attractive enough to upgrade. And let's not even mention EU prices. On top of the 21% sales tax —nothing to do there— we are eating up a 1:1 USD:EUR ratio which is bullshit.

Second, Apple is advertising a "Pro" phone that can shoot incredible 4K movies, but stuffing it with only 64 GB of storage. The consumer experience is terrible when you are out of disk space.

My phone memory is full and every time I take a picture it is immediately uploaded to iCloud and deleted from the phone. If I want to show it to somebody later the same day, I have to wait for it to load from the network. My UX is that I have no pictures or videos stored locally, not even for pictures I took 15 minutes ago. That is definitely not a feature you want on a super advanced camera-phone.

The phone market is too mature

Regarding innovation, what can Apple really do? I honestly do not have an answer. The majority of the population is not renewing their phones on a yearly cycle, not even a two year cycle. I have an SE only because my 5S broke. I loved my 5S and there is no feature in current phones that would make me upgrade.

I commute every day and see what "normal people" (excuse me) do on their phones. It is 40% scrolling through Instagram, 30% Whatsapp, 20% watching shows, 5% taking pictures, 5% playing games.

If you want to read the best take on the keynote, read Ben Thompson's The iPhone and Apple’s Services Strategy.

The phone market and phone technology have crossed the chasm long ago and they're on diminishing returns. I stick by my reaction of last year's keynote:

  • Apple should seriously consider 2-year iPhone cycles.
  • People who want smaller phones, regardless of price, may move to Android, myself included, so an updated iPhone SE is strategic for Apple.
  • Hardware improvements are going to be mostly incremental from now on. Therefore...
  • Apple should focus on software, which they are doing very well, and keep coming up with really crazy innovative hardware, which they appeared to be doing but rumors say they scraped at the last minute like the U1 chip.

Apple is a company full of smart people that can reinvent boring products like beige PCs, Nokia phones, and even headphones and watches. I am hopeful for the next wave of hardware, whatever it is. AR glasses? Car stuff? TVs? We will see.

Personally, I am indifferent at this keynote. Since my main need is a laptop, I'm still waiting for the new wave of macbooks to renew my 2013 MBA. I simply refuse to buy any laptop from Apple's current lineup. The rumors are very promising, so let's check what they can come up with!

Tags: apple

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

Leo Tindall (LeoLambda)

Packaging Desktop Apps With Flatpak September 10, 2019 08:00 PM

As mentioned in the previous post, the web platform is popular for creating apps that would make plenty of sense as desktop software. This is partly because it unifies many platforms, but mostly it’s because it solves the distribution problem. That is, having written a program, how do I get it into the hands of users? Consider the process of installing a desktop app for a user with a fresh install of their operating system:

Jan van den Berg (j11g)

Gomorrah – Roberto Saviano September 10, 2019 05:42 PM

When Gomorrah came out in 2007, apart from spawning a popular TV series, it shook the world. The workings of Italy’s lesser known ‘other mafia’ — the Camorra — are laid bare and Saviano subsequently signed his own death certificate. He has been in hiding ever since.

Gomorrah – Roberto Saviano (2007) – 301 pages

It’s a gut-wrenching mosaic of stories, of how this organisation operates from Naples — mind you, a modern West-European city — and their direct impact on anything and everything in the area and indirectly all throughout Europe. No details are spared, murders and tortures included. The title is of course a word play on the the Camorra name and the biblical story of the city that was so foul it got destroyed. It makes complete sense.

Gomarrah is a necessary book, it was just a question of who dared to write it. Roberto Saviano did.

There are however some caveats. In trying to be complete the book features a devastating onslaught of names en characters. Too many to wrap your head around. But maybe that was precisely the point. The other caveat is that there is little explanation for how and why this specific area is the birthplace for the Camorra. But that was not the point, so you would need another book for that.

The post Gomorrah – Roberto Saviano appeared first on Jan van den Berg.

September 09, 2019

Gustaf Erikson (gerikson)

August September 09, 2019 08:00 PM

Andrew Owen (yumaikas)

What 8 years of side projects has taught me September 09, 2019 07:58 PM

I’ve been a professional software developer for almost 8 years now. I’ve been paid to write a lot of software in those years. Far more interesting to me has been the recurring themes that have come up in my side-projects and in the software I’ve been personally compelled to write.

Lesson 0: Programming in a void is worthless

When I wanted to learn programming, I always had to come to the keyboard with a purpose. I couldn’t just sit down and start writing code, I had to have built an idea of where I was going.

For that reason, I’ve always used side-projects as a mean of learning programming languages. When I wanted to learn QBasic, there were a number of games, one based in space, another was a fantasy game. When I wanted to learn Envelop Basic, I attempted making a Yahtzee Clone, a Space Invaders Clone, a Hotel running game, and made a MicroGame based on the ones I’d seen videos of in WarioWare DIY.

When I wanted to learn C#, I went through a book, but I also, at the same time, worked on building a Scientific Calculator. (put a pin in that idea). When I wanted to learn Go, I wrote the CMS for the blog you’re reading right now. To pick up Lua, I used Love2D in several game jams. The only reason I have more than a passing familiarity with Erlang is because I used it for idea.junglecoder.com

Most times, when I’ve tried to learn a programming technology without a concrete goal to get something built, it is hard for me to maintain interest. That hasn’t kept me from trying out a lot of things in the past, but it’s the ones that allowed me to build useful or interesting things that have stuck with me the most. Right now, for side-projects, that list includes Go, Lua, Tcl, and Bash.

Lesson 1: Building a programming language is hard, but rewarding

Ever since I first started cutting my teeth on C#, the ideas of parsing have held a certain fascination to me. Like I said before, I started wanting to write a scientific calculator. But, because I was a new programmer, with no idea of what building a scientific calculator should look like, I did a lot of inventing things from first principles. It felt like a divine revelation when I reasoned out a add/multiply algorithm for parsing numbers. It also took me the better part of two weeks to puzzle it out.

I was so proud of that, in fact, that I copied that code into one of my work projects, a fact which really amuses me now that I know about the existance of Regex and Int.Parse().

Eventually, I worked out a very basic notion of doing recursive descent parsing, and some tree evaluation, so that, on a good day, I had a basic, but working, math expression evaluator.

Working on that calculator, however, set me on a course of wanting to understand how programming languages worked. In the process of wanting to understand them, I’ve looked over papers on compilers more than once, but never quite had the patience to actually write one out. In the process of wanting to make a programming language, I ended up writing two before PISC. One that was a “I want to write something in a night that is Turing Complete” langauge that was basically a bastard version of assembly. At the time, I called it SpearVM. I had intended it to be a compilation target for some higher level language, but it mostly just served as stepping stone for the next two projects.

The second one was a semester-long moonshot project where I wanted to try to make a visual programming language, using either Java Swing or JavaFX, inspired by Google’s Blockly environment. Unfortunately, I could not figure out nesting, so I giving the ideas I’d had in SpearVM a visual representation, and using that for my class assignment.

The combination of all of these experiences, and discovering the Factor language, set me thinking about trying to build a programming language that was stack-based, especially since parsing it seemed a far easier task than what I’d been trying to do until then. A couple late nights later, and I’d built out a prototype in Go.

I’ve had a number of co-workers impressed that I’ve written a scripting language. Thing is, it took me like 7 false starts to find a way to do it that made sense to me (and that was a stack-based language with almost 0 lexing). It’s only now, that I’m on the other side of that learning experience, that I’d feel comfortable approach writing a language with C-like syntax. PISC, as I’ve had time to work on it, has actually started to develop more things like parsing, lexing, and even compiling. In fact, I’ve got a small prototype of a langauge called Tinscript that isn’t nearly so post-fix oriented as PISC, though it’s still stack based.

And, PISC, to boot, is still what I’d consider easy-mode when it comes to developing a programming language. Factor, Poprc, or even a run-of-the mill C-like language all strike me as projects that take more tenacity to pull off.

Lesson 2: Organizing my thoughts is important, but tricky to figure out

If the early years of my programming side-projects often focused on how to build programming languages, and how to better understand computers, the more recent years have a had a much stronger focus on how to harness computers to augment my mind. A big focus here was for me to find ways to reduce the working set I needed in my mind at any given time. This has resulted in no less than 7 different systems for trying to help keep track of things.

  • A TCL application for launching programs I used on a regular basis.
  • A C# application for the same, but with a few more bells and whistles
  • Trying out Trello
  • Trying out various online outliners, ultimately being satisfied with none of them.
  • A months-long foray into trying to learn and apply Org-mode and Magit in Emacs, and ultimately giving up due to slowness on Windows, and the fact that my org-files kept getting too messy.
  • ideas.junglecoder.com, a place meant for me to shunt my stray thoughts to get them off my mind during the work day.
  • Another TCL application called PasteKit, which was designed to help me juggle all of the 4-6 digit numbers I was juggling at one of my jobs.
  • A C# version of PasteKit, that also had a customizeable list of launchers
  • .jumplist.sh
  • Bashmarks, but for CMD.exe

These are all approaches I’ve invested non-trivial amounts of time into over the past three years, trying to figure out a way to organize my thoughts as a software developer, but none of them lasted much longer than a month or so.

All of this came to a head during Thanksgiving weekend of 2018. My work at Greenshades often involved diving deep into tickets and opening a lot of SQL scripts in SSMS, and I had found no good way to organize them all. So, in a move that felt rather desperate at the time, I wrote a C# program that was a simple journal, but one that had a persistent search bar, and stored all of its entries in a SQLite database. And I used a simple tagging scheme for the entries, of marking them with things like @ticket65334, and displaying the most recent 5 notes.

It was finally a system that seemed to actually work for how I liked to think about things. The UI was a fairly simple 3-column layout. In the leftmost column, I had a “scratchpad” where I kept daily notes of what I’d worked on, in the middle I had my draft pad, and on the right I had the feed of notes, based on the search I’d done. I also had a separate screen dedicated to searching through all the notes that had previously been recorded.

There were several benefits to how this system worked:

  • It allowed me to forget things by putting notes under a different tags. That meant they wouldn’t show up on my focused feed, but that I could get back to them later.
  • It allowed me to regain context after getting interrupted much easier. - It gave me a virtual rubber duck. Since I often was trying to figure out issues by writing them to my teammates anyway, the journal gave me a very good first port of call when my stream of consciousness got blocked by an obstacle. This helped dramatically with keeping me off distracting websites like Hackernews or Reddit.
  • It allowed old information to fall out of relevance. One of the biggest problems with all the various tracking systems I’d used before, especially Trello and Org-mode, is that the system filled up, it was hard for old items to fall out of relevance without also being just a bit harder to access. Due to the nature of the feed, this system made it much more natural for information to fall off. And if I wanted it to stick around, I could just copy the relevant bits to a new note, which I often do.

All of this added up to me feeling like I’d found a missing piece of my mind. Almost like I’d created a REPL for my thought process.

Unfortunately, I don’t have that C# version any more. I do have a Go/Lua version, which is webapp based, though I still need to put some time into making the feedback-loop for it tighter, since my first versions weren’t quite as tightly focused on that, as much as they were focused on replicating the UI layout of the C# version. I’d argue that the tight feedback loop that the C# version had would be more important now, and I’ve slowly been working on adding it back.

The nice thing about the Go/Lua journal is that it’s far more flexible than the C# version, due to being able to write pages in Lua. Which means I’ll be able to

Lesson 3: Search is a great tool for debugging and flexible organization

Exhaustive string search of both code and notes has proven to be a surprisingly effective tool for understanding and cataloging large systems for me. To this end, Gills (my journaling software), Everything Search (search over the paths and file names on your laptop) and RipGrep have been extremely handy tools to have on hand. The nice thing about search as a tool is that it can be adapted into other things quite nicely. In fact, I would argue that fast search, both via Google, and via the tools I’d mentioned above, is one of the more influential changes we’ve seen in programming in the last 20 years.

Coda: Sticky ideas

8 years is a long time, and there are lot more ideas that I’d like to get into later. However, these are the ideas and things I’ve worked on that have proven to be surprisingly sticky. Perhaps they might help you, or give you some ideas of where to focus.

Published September 9th, 2019

Carlos Fenollosa (carlesfe)

If Harari's Sapiens was a blog post September 09, 2019 01:08 PM

If Sapiens were a blog post (30 min, via) is, in words of the author:

I spent over 25 hours building a cut-down version of Sapiens. The goal? Future-me should be happy to read this once future-me forgets how we evolved. It's massive for a blog post, just under 30 minutes, but that's the best I could do, condensing 9 hours worth of material.

The book is fantastic, a must-read, despite its flaws and objections. The blogpost has one big problem, despite the meritable effort: it summarizes the whats but not the whys.

I've skimmed through it, trying to find if it talks about what, for me, was the biggest realization of Sapiens: why humans transitioned from hunter-gatherer tribes into agricultural civilizations.

Hunterer-gatherer foraging was enough to feed a small tribe; a group of 30 people can be fed with a deer and some apples. A town of 200 people needs something more, so agriculture was developed. This new technology allowed for bigger human concentrations in a small area.

But why would humans want to live in larger groups, if it brought a lower quality of life? Famines, infections, fights, enslaving work, extreme class differences?

Harari argues that this was due to the appearance of religion.

Religion demanded that people (well, it was self-imposed, but bear with me) overcame bigger and bigger projects, like temples, sacrifices, wars, and other, which require a minimum amount of people to succeed. 30 people cannot build a temple, but 200 people can.

Therefore, it was due to religion that civilization as we know it developed. Religion needs large groups of people to work, and that is why we transitioned from hunting to agricultural societies. Maybe it was the other way around? With agriculture came religion? Again, the direction of this implication is what Harari defends, and I don't know enough to argue otherwise.

The blogpost devotes one chapter to talk about religion but doesn't mention that concept. Since it is one of the main points of the book, I'm not sure to which degree this summary looks over other core conclusions. This is a bit disappointing, but maybe it was not the author's intent to begin with.

For me, the magic of Sapiens is that it's not just a Wikipedia-like compendium of timelines and events, but rather provides some theories as to why things happened.

In any case, if this blogpost encourages you to read Sapiens, it will be time well spent.

Tags: books

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

September 08, 2019

Carlos Fenollosa (carlesfe)

Ponylang (SeanTAllen)

Last Week in Pony - September 8, 2019 September 08, 2019 09:00 AM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

Bit Cannon (wezm)

Visualising Operating System Derivation September 08, 2019 06:21 AM

After I ordered a new laptop yesterday (Huawei MateBook X Pro (2018)) I started pondering what OS I might run on it. I started looking through the DistroWatch.com Top 100 OSes (by Page Hit Ranking). This table (shown in the right side bar on the homepage) ranks OSes by the number of hits to their DistroWatch.com page in the last 6 months. It’s a decent proxy for what’s out there and what people are interested in using at moment.

As I went through the list it struck me how many appeared to based on one of three operating systems: Arch Linux, Debian, and Ubuntu (which is itself based on Debian). I thought it might be interesting to visualise this in order to see the clustering and also get a feel for how many of the top 100 are actually independent.

I whipped up a quick Ruby script to pull the data from DistroWatch.com and then output a dot file for GraphViz to render. The result is below (click to view full size SVG). The script uses Nokogiri to parse the HTML documents. I then use a combination of XPath and CSS selectors to extract the information I need.

Update: Some people have commented to say that some historical derivation is missing, like OpenBSD being originally forked from NetBSD. That’s not what this graph is showing. It’s mostly showing how an OS is constructed today. I.e. when the project makes a new release is it being rebased on another project or not. In this context, OpenBSD is very much independent. Ultimately though it visualises DistroWatch.com’s “Based On” field, whatever that happens to be.

Operating System Derivation Visualisation

The graph confirms my suspicion. There are significant clusters around Arch, Debian, and Ubuntu. This helps with the original goal of picking an OS to try on the laptop. I already run Arch, which in my mind rules out all its derivatives – I’d rather just run the real thing. I have no desire to run Debian or Ubuntu as I much prefer a rolling software model. Of the remaining independent options Alpine and Void are interesting. I run Alpine on my server but I’m not sure it would be what I’m after in a desktop (I did play with a desktop install in a virtual machine though). I’m interested in giving Void more of a go – especially the musl libc variant – so that will be what I try first.

The source of the script is on GitHub if there’s any GraphViz wizards out there that can improve the graph, PRs are most welcome.

Comments

September 07, 2019

Leo Tindall (LeoLambda)

HackRF Tripups with GNURadio September 07, 2019 09:48 PM

The HackRF One is an amazing tool, but the documentation about how to use it with GNURadio is… scattered, to say the least. I got mine working today after a few hours of effort, and I wanted to share what I did so others could benefit, because it’s a little arcane. Specifically, I was using the wrong output sample rate. Symptoms of having the wrong output sample rate (too low) are scratchy audio and/or only transmitting in “chunks” (because the device is waiting to get a full buffer before transmitting).

September 06, 2019

Mark Fischer (flyingfisch)

Sharpie Stainless Steel Pen Refill Hack September 06, 2019 04:00 AM

Backstory

If you have owned a Sharpie Stainless Steel Pen, you will know two things. The first thing you’ll know is that it is the most perfect pen ever invented and no other pen will ever compare. The second thing you’ll know, or find out, is that both the pen and the refills have been discontinued by Sharpie.

WHY SHARPIE?! WHY WOULD YOU DISCONTINUE SUCH AN EXCELLENT PRODUCT???? WHY DID YOU DO THIS TO US?!

Ahem. Anyway… after realizing that refills are now $812.91, I decided to try to figure out if I could find an alternative way to refill them. Below are instructions for creating what I call the Sharpie Stainless Steel Pen Frankenfill Mark I.

Stainless Steel Pen Frankenfill Mark I

Materials

  • Sharpie Stainless Steel Pen
  • Cross Selectip Porous Point Refill (Model number 8443)

Step 1: Remove the original Sharpie Pen refill

Remove the cap from your Sharpie pen. Grasp the stainless barrel with one hand and rubber grip in the other and twist to remove the original Sharpie refill.

Step 2: Disassemble the original Sharpie Pen refill

Grasp the rear cap of the Sharpie refill with a pair of pliers. Pull firmly to remove, it’s press fit and shouldn’t take much effort. Inside there is a plastic covered piece of foam with the nib sticking out of it, remove that. Also remove the metal nib by pressing the point down firmly on a hard surface.

Step 3: Insert the Cross Selectip refill into the original Sharpie

Insert the Cross Selectip refill into the carcass of the Sharpie refill. Press firmly until the nib is exposed. A little bit of the back of the Cross refill will be exposed, this is fine. The rear cap on the Sharpie refill can be discarded at this point.

Step 4: Reassemble the pen

At this point, all that’s left is to reassemble the pen.

Review

I’m satisfied with this as a first try and it does give me hope that my Sharpie Stainless Pens have a future. However, although the Cross refill is very pleasant to write with, it has a couple key differences from the original Sharpie pen. The first difference is that the ink is much darker and less fine. The second difference is that it bleeds more. Below is an image comparing the two from the front and back.

Future Plans

I’m currently searching for a felt tip refill that matches the properties of the original Sharpie pen a little more closely and will hopefully be posting an update in the near future. However, I am happy to say that I have found a way to use standard pen refills in the Sharpie Stainless Steel Grip Pen and I’m hoping to get several more years of use out of my current Sharpie pens. Maybe some day Sharpie will bring the pen back to production, but until then I hope this guide helps others wondering how to continue using their Sharpies.

September 05, 2019

Bogdan Popa (bogdan)

Generators from Scratch September 05, 2019 02:00 PM

Generators in Python One of the nice things about Python is that it comes with in-built support for “generators”, functions that can suspend themselves and be resumed in the middle of processing. Here’s a generator that produces the fibonacci series: def fib(): x = 0 y = 1 while True: y, x = x + y, y yield x A generator is instantiated every time you call the fib function.

September 04, 2019

Gokberk Yaltirakli (gkbrk)

Tampermonkey is not Open Source September 04, 2019 11:00 PM

This post is meant to be a short remark about something I noticed today. It is about Tampermonkey, a browser extension for managing User scripts.

Tampermonkey, like Greasemonkey, lets you create and run User scripts. These are small JavaScript snippets that can execute on any URL you want. You can consider them mini-extensions.

Tampermonkey came out as a Greasemonkey alternative for Chrome. Since writing User scripts is a “developer” thing to do, and since pretty much all software development tools are open source, so was Tampermonkey. This quickly allowed them to gather a user base and get contributions from the community.

At some point (around version 2.9), they switched their project to a proprietary license. They stopped taking contributions of course, and the old version is still on GitHub, so there is no license violation. This post isn’t meant to be a “grab your pitchforks there’s proprietary software” post against Tampermonkey either. I just know that some people don’t want to run non-free software, and their browser might’ve auto-updated to a non-free version.

This was certainly the case for me. I remembered Tampermonkey as an open-source user script manager and started using it, it took me a while to realize the license situation. While this information was available on the Firefox add-ons page, I think it should be more prominent in the install process.

After some time, and with developments like major browsers all implementing the same extension API, Tampermonkey took its place on most Add-On stores. I believe for Firefox at least this was after the license change, so people on FF shouldn’t have had unexpected non-free software.

Tobias Pfeiffer (PragTob)

On Going Freelance September 04, 2019 03:30 PM

At the end of a lengthy job search I decided to become a freelancer helping companies onboard onto Elixir, helping them with their development projects and processes, some performance work, pushing Open Source and maybe even a bit of interim CTOing or other consulting. Who knows what the future will hold? Right now I’m on […]

September 02, 2019

Pete Corey (petecorey)

Animating a Canvas with Phoenix LiveView September 02, 2019 12:00 AM

Phoenix LiveView recently released a new feature called “hooks” that introduces Javascript interoperability into the LiveView lifecycle. Put simply, we can now run arbitrary Javascript every time a DOM node is changed by LiveView! LiveView hooks are a complete game changer, and open the doors to a whole new world of applications that can be built with this amazing technology.

As a proof of concept, let’s use LiveView hooks to animate an HTML5 canvas in real time using data provided by the server!

Getting Set Up

To keep this article short(er), we’ll skip the rigmarole of configuring your application to use LiveView. If you need help with this step, I highly recommend you check out Sophie DeBenedetto’s thorough walkthrough. Be sure to cross reference with the official documentation, as things are moving quickly in the LiveView world.

Moving forward, let’s assume that you have a bare-bones LiveView component attached to a route that looks something like this:


defmodule LiveCanvasWeb.PageLive do
  use Phoenix.LiveView
  
  def render(assigns) do
    ~L"""
    <canvas>
      Canvas is not supported!
    </canvas>
    """
  end
  
  def mount(_session, socket) do
    {:ok, socket}
  end
end

We’ll also assume that your assets/js/app.js file is creating a LiveView connection:


import LiveSocket from "phoenix_live_view";

let liveSocket = new LiveSocket("/live");

liveSocket.connect();

Now that we’re on the same page, let’s get started!

Generating Data to Animate

Before we start animating on the client, we should have some data to animate. We’ll start by storing a numeric value called i in our LiveView process’ assigns:


def mount(_session, socket) do
  {:ok, assign(socket, :i, 0)}
end

Next, we’ll increase i by instructing our LiveView process to send an :update message to itself after a delay of 16 milliseconds:


def mount(_session, socket) do
  Process.send_after(self(), :update, 16)
  {:ok, assign(socket, :i, 0)}
end

When we handle the :udpate message in our process, we’ll schedule another recursive call to :update and increment the value of i in our socket’s assigns:


def handle_info(:update, %{assigns: %{i: i}} = socket) do
  Process.send_after(self(), :update, 16)
  {:noreply, assign(socket, :i, i + 0.05)}
end

Our LiveView process now has an i value that’s slowly increasing by 0.05 approximately sixty times per second.

Now that we have some data to animate, let’s add a canvas to our LiveView’s template to hold our animation:


def render(assigns) do
  ~L"""
  <canvas data-i="<%= @i %>">
    Canvas is not supported!
  </canvas>
  """
end

Notice that we’re associating the value of i with our canvas by assigning it to a data attribute on the DOM element. Every time i changes in our process’ state, LiveView will update our canvas and set the value of data-i to the new value of i.

This is great, but to render an animation in our canvas, we need some way of executing client-side Javascript every time our canvas updates. Thankfully, LiveView’s new hook functionality lets us do exactly that!

Hooking Into LiveView

LiveView hooks lets us execute Javascript at various points in a DOM node’s lifecycle, such as when the node is first mounted, when it’s updated by LiveView, when it’s destroyed and removed from the DOM, and when it becomes disconnected or reconnected to our Phoenix server.

To hook into LiveView’s client-side lifecycle, we need to create a set of hooks and pass them into our LiveSocket constructor. Let’s create a hook that initializes our canvas’ rendering context when the element mounts, and renders a static circle every time the element updates:


let hooks = {
  canvas: {
    mounted() {
      let canvas = this.el;
      let context = canvas.getContext("2d");
      
      Object.assign(this, { canvas, context });
    },
    updated() {
      let { canvas, context } = this;
      
      let halfHeight = canvas.height / 2;
      let halfWidth = canvas.width / 2;
      let smallerHalf = Math.min(halfHeight, halfWidth);
      
      context.clearRect(0, 0, canvas.width, canvas.height);
      context.fillStyle = "rgba(128, 0, 255, 1)";
      context.beginPath();
      context.arc(
        halfWidth,
        halfHeight,
        smallerHalf / 16,
        0,
        2 * Math.PI
      );
      context.fill();
    }
  }
};

let liveSocket = new LiveSocket("/live", { hooks });

liveSocket.connect();

Notice that we’re storing a reference to our canvas and our newly created rendering context on this. When LiveView calls our lifecycle callbacks, this points to an instance of a ViewHook class. A ViewHook instance holds references to our provided lifecycle methods, a reference to the current DOM node in el, and various other pieces of data related to the current set of hooks. As long as we’re careful and we don’t overwrite these fields, we’re safe to store our own data in this.

Next, we need to instruct LiveView to attach this new set of canvas hooks to our canvas DOM element. We can do that with the phx-hook attribute:


<canvas
  data-i="<%= @i %>"
  phx-hook="canvas"
>
  Canvas is not supported!
</canvas>

When our page reloads, we should see our circle rendered gloriously in the center of our canvas.

Resizing the Canvas

On some displays, our glorious circle may appear to be fuzzy or distorted. This can be fixed by scaling our canvas to match the pixel density of our display. While we’re at it, we might want to resize our canvas to fill the entire available window space.

We can accomplish both of these in our mounted callback:


mounted() {
  let canvas = this.el;
  let context = canvas.getContext("2d");
  let ratio = getPixelRatio(context);
  
  resize(canvas, ratio);
  
  Object.assign(this, { canvas, context });
}

Where getPixelRatio is a helper function that determines the ratio of physical pixels in the current device’s screen to “CSS pixels” which are used within the rendering context of our canvas:


const getPixelRatio = context => {
  var backingStore =
    context.backingStorePixelRatio ||
    context.webkitBackingStorePixelRatio ||
    context.mozBackingStorePixelRatio ||
    context.msBackingStorePixelRatio ||
    context.oBackingStorePixelRatio ||
    context.backingStorePixelRatio ||
    1;
  
  return (window.devicePixelRatio || 1) / backingStore;
};

And resize is a helper function that modifies the canvas’ width and height attributes in order to resize our canvas to fit the current window, while fixing any pixel density issues we may be experiencing:


const resize = (canvas, ratio) => {
  canvas.width = window.innerWidth * ratio;
  canvas.height = window.innerHeight * ratio;
  canvas.style.width = `${window.innerWidth}px`;
  canvas.style.height = `${window.innerHeight}px`;
};

Unfortunately, our canvas doesn’t seem to be able to hold onto these changes. Subsequent calls to our updated callback seem to lose our resize changes, and the canvas reverts back to its original, blurry self. This is because when LiveView updates our canvas DOM node, it resets our width and height attributes. Not only does this revert our pixel density fix, it also forcefully clears the canvas’ rendering context.

LiveView has a quick fix for getting around this problem. By setting phx-update to "ignore" on our canvas element, we can instruct LiveView to leave our canvas element alone after its initial mount.


<canvas
  data-i="<%= @i %>"
  phx-hook="canvas" 
  phx-update="ignore"
>
  Canvas is not supported!
</canvas>

Now our circle should be rendered crisply in the center of our screen.

Animating Our Circle

We didn’t go all this way to render a static circle in our canvas. Let’s tie everything together and animate our circle based on the ever-changing values of i provided by the server!

The first thing we’ll need to do is update our updated callback to grab the current value of the data-i attribute:


let i = JSON.parse(canvas.dataset.i);

The value of canvas.dataset.i will reflect the contents of our data-i attribute. All data attributes are stored as strings, so a call to JSON.parse will convert a value of "0.05" to its numeric counterpart.

Next, we can update our rendering code to move our circle based on the value of i:


context.arc(
  halfWidth + (Math.cos(i) * smallerHalf) / 2,
  halfHeight + (Math.sin(i) * smallerHalf) / 2,
  smallerHalf / 16,
  0,
  2 * Math.PI
);

That’s it! With those two changes, our circle will rotate around the center of our canvas based entirely on real-time data provided by our server!

Requesting Animation Frames

Our solution works, but by forcing re-renders on the browser, we’re being bad net citizens. Our client may be forcing re-renders when its tab is out of focus, or it may be re-rendering more than sixty times per second, wasting CPU cycles.

Instead of telling the browser to re-render our canvas on every LiveView update, we should invert our control over rendering and request an animation frame from the browser on every update.

The process for this is straight forward. In our updated callback, we’ll wrap our rendering code in a lambda passed into requestAnimationFrame. We’ll save the resulting request reference to this.animationFrameRequest:


this.animationFrameRequest = requestAnimationFrame(() => {
  context.clearRect(0, 0, canvas.width, canvas.height);
  context.beginPath();
  context.arc(
    halfWidth + (Math.cos(i) * smallerHalf) / 2,
    halfHeight + (Math.sin(i) * smallerHalf) / 2,
    smallerHalf / 16,
    0,
    2 * Math.PI
  );
  context.fill();
});

It’s conceivable that our LiveView component may update multiple times before our browser is ready to re-render our canvas. In those situations, we’ll need to cancel any previously requested animation frames, and re-request a new frame. We can do this by placing a guard just above our call to requestAnimationFrame:


if (this.animationFrameRequest) {
  cancelAnimationFrame(this.animationFrameRequest);
}

With those two changes, our LiveView hooks will now politely request animation frames from the browser, resulting in a smoother experience for everyone involved.

Taking it Further

Using a canvas to animate a numeric value updated in real-time by a LiveView process running on the server demonstrates the huge potential power of LiveView hooks, but it’s not much to look at.

We can take things further by generating and animating a much larger set of data on the server. Check out this example project that simulates over two hundred simple particles, and renders them on the client at approximately sixty frames per second:

Is it a good idea to take this approach if your goal is to animate a bunch of particles on the client? Probably not. Is it amazing that LiveView gives us the tools to do this? Absolutely, yes! Be sure to check out the entire source for this example on Github!

Hooks have opened the doors to a world of new possibilities for LiveView-based applications. I hope this demonstration has given you a taste of those possibilities, and I hope you’re as eager as I am to explore what we can do with LiveView moving forward.

September 01, 2019

Ponylang (SeanTAllen)

Last Week in Pony - September 1, 2019 September 01, 2019 11:52 AM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

August 31, 2019

Carlos Fenollosa (carlesfe)

Jeff Carpenter (jeffcarp)

Things I Learned as a First-Time Intern Host August 31, 2019 06:13 PM

I hosted an intern for the first time this summer. It was my first time being somebody’s manager and it became a huge learning experience for me as well as a really fun time. My intern worked on adding many features to velocity-tracking charts, rewriting both of our ML models in TensorFlow 2.0, and a few other projects. Here are the biggest areas where I struggled as a host and the important lessons I took away from those experiences.

Frederic Cambus (fcambus)

My OpenBSD commits August 31, 2019 05:45 PM

Today marks my three year anniversary as an OpenBSD developer. I got my commit bit on August 31th 2016 during the g2k16 hackathon in Cambridge, UK.

A few months ago, I came across a Perl one-liner script to produce commit time distribution ASCII graphs from a Git repository, and I finally have a good pretext to run it :-)

As of this day, I have done 749 commits to OpenBSD, in the following repositories: src (127), ports(596), www (24), and xenocara (2).

Commits in the src repository:

00 -    0
01 -    0
02 -    0
03 -    0
04 -    0
05 -    0
06 -    1 ***
07 -    4 **************
08 -    8 ****************************
09 -    9 ********************************
10 -   13 **********************************************
11 -    9 ********************************
12 -   10 ***********************************
13 -   11 ***************************************
14 -   13 **********************************************
15 -    4 **************
16 -    5 *****************
17 -    6 *********************
18 -    4 **************
19 -    9 ********************************
20 -   14 **************************************************
21 -    4 **************
22 -    3 **********
23 -    0

Commits in the ports repository:

00 -    1
01 -    0
02 -    0
03 -    0
04 -    0
05 -    2 *
06 -   14 **********
07 -   32 ***********************
08 -   34 *************************
09 -   67 **************************************************
10 -   46 **********************************
11 -   53 ***************************************
12 -   40 *****************************
13 -   38 ****************************
14 -   34 *************************
15 -   34 *************************
16 -   35 **************************
17 -   20 **************
18 -   15 ***********
19 -   24 *****************
20 -   34 *************************
21 -   43 ********************************
22 -   19 **************
23 -   11 ********

Commits in the www repository:

00 -    0
01 -    0
02 -    0
03 -    0
04 -    0
05 -    0
06 -    0
07 -    1 ************
08 -    0
09 -    3 *************************************
10 -    2 *************************
11 -    4 **************************************************
12 -    0
13 -    2 *************************
14 -    1 ************
15 -    1 ************
16 -    1 ************
17 -    1 ************
18 -    1 ************
19 -    3 *************************************
20 -    3 *************************************
21 -    1 ************
22 -    0
23 -    0

Commits in the xenocara repository:

00 -    0
01 -    0
02 -    0
03 -    0
04 -    0
05 -    0
06 -    0
07 -    0
08 -    0
09 -    0
10 -    0
11 -    0
12 -    0
13 -    0
14 -    1 **************************************************
15 -    0
16 -    0
17 -    0
18 -    0
19 -    0
20 -    0
21 -    1 **************************************************
22 -    0
23 -    0

Ponylang (SeanTAllen)

0.31.0 Released August 31, 2019 10:50 AM

Pony version 0.31.0 is now available. This is a “smaller” release than some recent ones. Most of the changes in Pony lately have been happening behind the scenes.

0.31.0 is a breaking release. There’s a small change to Map.sub that might break your code. Details follow. There are no serious bugs fixed in 0.31.0 so you can upgrade at your leisure.

August 30, 2019

Jan van den Berg (j11g)

Sex, Blogs and Rock-‘n-Roll – Ernst Jan Pfauth August 30, 2019 02:06 PM

The rather sensationalist title would normally be a reason to not want to read this book. But since this book came out in 2010, the author E.J. Pfauth has become known for more than just this book. He is the co-founder of the Correspondent (a journalism platform), has written other books and he hosts a rather entertaining podcast. So I thought it would be fun to see where his head was some 10 years ago.

Sex, Blogs and Rock ‘n’ Roll – E.J. Pfauth (2010) – 205 pages

The book is part boyish bold blogging adventures i.e. travelling the world and meeting blog celebrities. All while trying to push the medium of blogging to new meaning. And the other part is a between-the-lines advice of how to exactly do this yourself. It’s not a specific how-to, but more ‘this is how I went about it’ (his second book explores this topic in more detail).

A couple of things stood out. First: the upbeat and ambitious belief that blogging could change the world. And second, while reading this in 2019 the notion that yes, the world has indeed changed. But, not necessarily by blogging. Case and point: the main two blogs from the author himself have vanished from the internet. And some of the mentioned blog celebrities have retreated into obscurity. And of course there’s Facebook and other monoliths that have eaten into the blogosphere and the independent self-publishing spirit it stands for.

Proof is in the pudding

But what stood out most throughout the entire book was the relentless tenacity of the author. His eagerness got him all around the world. Fortune favors the bold indeed! He was young and ambitious and would try everything and anything to get readers.

I have been blogging since 2005 myself, but I do it mainly because I enjoy it. So I have a different view of what blogging is about than the author. But of course, blogging can be many things. None of which is wrong. So in the spirit of this book I deliberately set out to blog something that would attract more readers. And it worked.

The post Sex, Blogs and Rock-‘n-Roll – Ernst Jan Pfauth appeared first on Jan van den Berg.

Carlos Fenollosa (carlesfe)

Terrifying iPhone implant spreads just by visiting a website August 30, 2019 07:03 AM

A very deep dive into iOS Exploit chains found in the wild (via) is a terrifying read of an iPhone implant that installs itself just by visiting a website and exploits five different 0-day vulnerabilities.

The implant phones back home with root access to all activity on your phone: chats, mails, location, pictures, and more.

I think it is fair to criticize Apple because they allowed an unsigned process running as root, using the network and a lot of battery activity, without any kind of monitoring to detect it. That process should not have been running without being discovered.

Given that an iPhone is not a computer, and not even an advanced user could detect and/or clean the implant, Apple's responsibility should be to start being more serious about the possibility of iPhone viruses.

Vice has a non-geek writeup which, at first, seemed like a bit sensationalist, but given the severity of the breach is probably somewhat warranted.

An awesome feat of engineering, though. Kudos to both the criminals and the researchers who detected it.

Tags: security, mobile, Apple

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

Andreas Zwinkau (qznc)

Mindstorms August 30, 2019 12:00 AM

A book about a revolution of education which sadly never happened.

Read full article!

Robin Schroer (sulami)

Why I like Clojure August 30, 2019 12:00 AM

This is somewhat of a response to Uncle Bob’s post of similar nature, which I would say has gotten a mixed to positive reception. I had planned a similar post a week or two before the release of his, but archived the idea upon reading his post. But after having read it over a couple of times I have now decided that I still have something meaningful to write. What follows are the purely subjective reasons for which I enjoy using Clojure. Some have criticised Bob for being very absolute and not giving up any screen estate for more nuanced viewpoints, something I will try to avoid.

Lisp

There is not much to say about this that has not already been said. The homoiconicity, meaning code can be represented as a data structure inside the same language, extensibility through macros which can modify both the evaluation order as well as the very syntax to the point where it looks more like LaTeX than Lisp.Racket is actually a great example of the flexibility of Lisp. A language designed to build other languages in it, call it a meta-language. Even package import can be wrapped up in a “language” trivially, meaning that you can essentially write a tiny DSL for every project. Not saying that is necessarily a good idea, but you can.

This also means that you are not stuck with a paradigm. While OO seems to be out, and FP the new hotness, Lisp can do them all, and historically often did before anyone else. Bob mentions dynamic typing in his signature retorts to (I am guessing) fictional counter-arguments, and he is right to mention clojure.spec, a library for gradual typing (omitting schema, an alternative). Racket has a fully typed variant, there is something that is basically Haskell in a Lisp-bun, and let us not forget that there is actually Typed Clojure, with static type checking and all.There are some issues with it, namely that the coverage is not all that great, but it exists and works, meaning if you really need static types, you can get them.

Being able to generate code without being stuck on a “dumb” level by generating strings and passing them into eval like for example in Python allows for sane hyper-dynamic programming, where the program adapts itself to the conditions. Being able to read and write code in a safe manner enables extremely powerful tooling in the Lisp world. Linters are very smart, because reading code into a data structure is trivial, usually a one-liner, and there are many more tools to automatically rewrite and refactor code than for other languages.

Now I do not want to discount the Lisp Curse, it is a real thing, and one of the reasons that while Lisp has stuck around for over half a century, it has not made it into the mainstream. The other main factor probably being the performance problems due to the gap between software and hardware architecture.There were Lisp machines, which had hardware tailored towards running Lisp, but they never took off either.

But with the advent of the internet, ecosystems like GitHub, and hardware that is fast enough that we consider running basically full web browsers for half of our applicationsLooking at you, Slack.

, I think that these issues have become surmountable.

I do not think I need to mention how incredibly useful the REPL and hot-loading code into a running system are?

Hosted

Clojure is explicitly designed as a hosted language, which I think was a very good move. If you are writing a new language today, it might be better than the established ones, but the cost of leaving an existing ecosystem of libraries and Stack Overflow answers just because the new language is 5% nicer is not a trade off many people will want to make. Clojure being hosted and having excellent interoperability with its host platform means it can benefit from existing ecosystem, let alone platform implementations.Do you really want to implement your runtime for FreeBSD on a smart toaster oven? Raspberry Pis are non x86, BSD is not Linux, and who knows what is up with Windows. This matrix is growing quickly.

While the primary platform is the JVM, superbly uncool but stable and relatively performant, there is a CLR (.NET) version which is “almost even on features” thanks to Davit Miller, as well as a very mature JavaScript version in the shape of ClojureScript. The JVM (and to some extent the CLR) have excellent support by big software vendors, if you buy some kind of software with an API, chances are there is a Java SDK which you can use easily from your Clojure code. The JavaScript ecosystem is the largest in numbersIn part due to left-pad-like five-line-packages, but still.

, and includes Electron and React-Native, both of which can be used with some, but not unreasonable, effort from ClojureScript code. One of the newest additions has been GraalVM, which while not 100% feature-complete yet, already allows compilation to native static binaries of many Clojure programsZprint is one of those CLI tools that takes advantage of the reduced startup time.

, running without the JVM at all, and doing away with the dreaded multi-second startup time.I am planning to write a piece about GraalVM some time later this year.

The platform split could have been one of the big, curse-like problems for Clojure, but there is Clojure Common, used by many popular libraries, which allows you to write platform-independent code by using conditional branching for all platform-specific code.

Community

Despite all the positive points I mentioned, Clojure is still a niche language, and in some way that is good as well. Sure, finding jobs is harderLarge companies like Walmart and CircleCI (my employer) are Clojure shops, so it is far less obscure than one might think.

, but not impossible. Clojure developers, like for example Haskell or Rust ones, tend to be more experienced, as it is not a typical first language, and requires a certain interest in the craft. Many Clojure developers have written widely used tools and libraries, not just in Clojure, but also for example for Emacs, which is understandably quite popular with Clojurists.

Rich Hickey himself, the BDFL of Clojure, is someone with decades of industry experience and a desire to get it right. I think he is doing a pretty good job. While there are some small inconsistencies in places, the bulk of the language, and all the important parts are very well thought out.We can also see right now how clojure.spec is being adapted after community feedback to the first alpha version, which has been available for about 1½ years.

Clojure is a very stable language, which means that smaller problems will stick around for a while, but also means you can trust that your code will not break every time you update your dependencies.

In the end, it comes down to enjoyment. I enjoy working with Clojure. I feel like there is a lot to learn, and the language is inviting me to explore beyond the current possibilities of software development. I feel good about the elegant and concise solutionsconcise ≠ obtuse

I can come up with. It has changed the way I think, in a good way.

August 29, 2019

Derek Jones (derek-jones)

Ecosystems chapter of “evidence-based software engineering” reworked August 29, 2019 10:49 PM

The Ecosystems chapter of my evidence-based software engineering book has been reworked (I have given up on the idea that this second pass is also where the polishing happens; polishing still needs to happen, and there might be more material migration between chapters); download here.

I have been reading books on biological ecosystems, and a few on social ecosystems. These contain lots of interesting ideas, but the problem is, software ecosystems are just very different, e.g., replication costs are effectively zero, source code does not replicate itself (and is not self-evolving; evolution happens because people change it), and resources are exchanged rather than flowing (e.g., people make deals, they don’t get eaten for lunch). Lots of caution is needed when applying ecosystem related theories from biology, the underlying assumptions probably don’t hold.

There is a surprising amount of discussion on the computing world as it was many decades ago. This is because ecosystem evolution is path dependent; understanding where we are today requires knowing something about what things were like in the past. Computer memory capacity used to be a big thing (because it was often measured in kilobytes); memory does not get much publicity because the major cpu vendor (Intel) spends a small fortune on telling people that the processor is the most important component inside a computer.

There are a huge variety of software ecosystems, but you would not know this after reading the ecosystems chapter. This is because the work of most researchers has been focused on what used to be called the desktop market, which over the last few years the focus has been shifting to mobile. There is not much software engineering research focusing on embedded systems (a vast market), or supercomputers (a small market, with lots of money), or mainframes (yes, this market is still going strong). As the author of an evidence-based book, I have to go where the data takes me; no data, then I don’t have anything to say.

Empirical research (as it’s known in academia) needs data, and the ‘easy’ to get data is what most researchers use. For instance, researchers analyzing invention and innovation invariably use data on patents granted, because this data is readily available (plus everybody else uses it). For empirical research on software ecosystems, the readily available data are package repositories and the Google/Apple Apps stores (which is what everybody uses).

The major software ecosystems barely mentioned by researchers are the customer ecosystem (the people who pay for everything), the vendors (the companies in the software business) and the developer ecosystem (the people who do the work).

Next, the Projects chapter.

August 27, 2019

Carlos Fenollosa (carlesfe)

Slow down your laptop August 27, 2019 08:29 AM

800 Mega Hertz is enough (via) argues that for most use cases laptops don't need to run at their top speed to save battery.

Computers are amazing; they are so fast and can do so much. Nevertheless, what if we didn't need it to be so fast or do so much? That was the question I asked myself when I was trying to improve the battery life of my laptop even further. I came up with a simple solution. The processor that came in my computer runs at 2.2GHz, that isn't fast compared to modern laptops today, but it is still much faster than I need it to be. Each clock cycle uses a little bit of power, and if all I'm doing is typing out Latex documents, then I don't need that many clock cycles. I don't need one thousand four hundred of them to be precise.

The only caveat is that modern web browsing is definitely much more taxing than "typing out latex documents." You are running almost an entire graphical OS in each tab. And, if I'm not mistaken, most frequency managers actually slow down CPUs to a crawl when they're idle. Other than that, I'm all in on this.

In my own experience, disabling Turbo Boost is barely noticeable and extends battery life for about an hour, which translates to around 20% extra life. I actually have a script called lpm.sh, for "low power mode" that essentially stops syncthing, an important source of CPU wakeups, and disables turbo boost to grant me those extra minutes. CPU frequency is properly managed in my case and I don't have the need to run cpupower manually.

If you truly want to run the most resource-constrained, "distraction-free typewriter mode", boot into single-user mode and write using vim.

Bonus rabbit hole of the day: CPU power management

Tags: hardware, minimalism

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

August 26, 2019

Derek Jones (derek-jones)

Compiler validation used to be a big thing August 26, 2019 08:38 PM

Compiler validation used to be a big thing; a NIST quarterly validated products list could run to nearly 150 pages, and approaching 1,000 products (not all were compilers).

Why did compiler validation stop being a thing?

Running a compiler validation service (NIST was also involved with POSIX, graphics, and computer security protocols validation) costs money. If there are enough people willing to pay (NIST charged for the validation service), the service pays for itself.

The 1990s was a period of consolidation, lots of computer manufacturers went out of business and Micro Focus grew to dominate the Cobol compiler business. The number of companies willing to pay for validation fell below the number needed to maintain the service; the service was terminated in 1998.

The source code of the Cobol, Fortran and SQL + others tests that vendors had to pass (to appear for 12 months in the validated products list) is still available; the C validation suite costs money. But passing these tests, then paying NIST’s fee for somebody to turn up and watch the compiler pass the tests, no longer gets your product’s name in lights (or on the validated products list).

At the time, those involved lamented the demise of compiler validation. However, compiler validation was only needed because many vendors failed to implement parts of the language standard, or implemented them differently. In many ways, reducing the number of vendors is a more effective means of ensuring consistent compiler behavior. Compiler monoculture may spell doom for those in the compiler business (and language standards), but is desirable from the developers’ perspective.

How do we know whether today’s compilers implement the requirements contained in the corresponding ISO language standard? You could argue that this is a pointless question, i.e., gcc and llvm are the language standard; but let’s pretend this is not the case.

Fuzzing is good for testing code generation. Checking language semantics still requires expert human effort, and lots of it. People have to extract the requirements contained in the language specification, and write code that checks whether the required behavior is implemented. As far as I know, there are only commercial groups doing this, i.e., nothing in the open source world; pointers welcome.

Gonçalo Valério (dethos)

Rust examples and exercises August 26, 2019 05:01 PM

Learning to program in Rust is as easy like other languages out there, because it ends up having different constrains and new concepts that you will have to go through, in the beginning everybody fights the compiler at least a little bit.

I started this journey a while ago, however I’ve been progressing slowly just dedicating some time once in a while when I don’t anything else to do.

I did what many recommendations on the internet tell you to do, start by reading the official book, that is in fact pretty good. But after reading one or two chapters, we need to practice and play with the language to have a feel of it and explore the new concepts you had just learned.

So in this small post I just want to share two open resources that can be used while you read the book to practice what you have just learned.

The first one is a website with examples you can modify and execute live in the browser called Rust by Example.

The second is an official rust project that will put your knowledge up to a test called Rustlings.

You can use it like the above video or with rustlings watch that stop and reload each exercise until you solve it.

This is it, I hope they end being helpful to someone else as well.

Pete Corey (petecorey)

Prime Parallelograms August 26, 2019 12:00 AM

A few weeks ago I wrote about an interesting graph of numbers recently investigated by the Numberphile crew. We used it as an opportunity to journey into the world of agendas and gerunds by implementing the graph using the J programming language.

The second half of that same video outlines another interesting number series which has a similarly interesting implementing in J. Let’s try our hand at plotting it.


The basic idea behind calculating the series of numbers in question is to take any positive prime, represent it in base two, reverse the resulting sequence of bits, and subtract the reversed number from the original number in terms of base ten.

Implemented as a tacit, monadic verb in J, this would look something like:


f =: ] - [: #. [: |. #:

Our verb, f, is (=:) the given number (]) minus (-) the base two (#.) of the reverse (|.) of the antibase two (#:) of the given number.

We can plot the result of applying f to the first ten thousand primes (p: i. 10000) like so:


require 'plot'
'type dot' plot f"0 p: i. 10000

If we’re feeling especially terse, we could write this as an almost-one-liner by substituting f for our implementation of f:


require 'plot'
'type dot' plot (] - [: #. [: |. #:)"0 p: i. 10000

Our implementation of f is a “train of verbs”, which is to say, a collection of verbs that compose together into hooks and forks. We can visualize this composition by looking at the “boxed” representation of our train:


┌─┬─┬──────────────────┐
│]│-│┌──┬──┬──────────┐│
│ │ ││[:│#.│┌──┬──┬──┐││
│ │ ││  │  ││[:│|.│#:│││
│ │ ││  │  │└──┴──┴──┘││
│ │ │└──┴──┴──────────┘│
└─┴─┴──────────────────┘

From right to left, J greedily groups verbs into three verb forks potentially followed by a final two verb hook if the total number of verbs in the train is even.

We can see that the first fork, [: |. #:, is a capped fork, which means it’s roughly equivalent to |. @: #:. In the monadic case, this fork takes its argument, converts it to a base two list of ones and zeroes, and reverses that list. Let’s refer to this newly composed verb as a moving forward.

The next fork in our train, [: #. a, is another capped fork building off of our previous fork. Again, this could be expressed using the @: verb to compose #. and a together: #. @: a. In the monadic case, this fork takes its argument, converts it to a reversed binary representation with a, and then converts that reversed list of ones and zero back to base ten with #.. Let’s call this newly composed verb b.

Our final fork, ] - b, runs our monadic input through b to get the base ten representation of our reversed binary, and subtracts it from the original argument.


If we wanted to make J’s implicit verb training explicit, we could define a, b, and our final f ourselves:


a =: [: |. #:
b =: [: #. a
f =: ] - b

But why go through all that trouble? Going the explicit route feels like a natural tendency to me, coming from a background of more traditional programming languages, but J’s implicit composition opens up a world of interesting readability properties.

I’m really fascinated by this kind of composition, and I feel like it’s what makes J really unique. I’ll never pass up an opportunity to try implementing something as a train of verbs.

August 25, 2019

Carlos Fenollosa (carlesfe)

Ponylang (SeanTAllen)

Last Week in Pony - August 25, 2019 August 25, 2019 10:09 AM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

August 23, 2019

Carlos Fenollosa (carlesfe)

Diccionario libre en español - Free Spanish dictionary August 23, 2019 12:09 PM

(Even though I write my blog in English, this post is in Spanish for obvious reasons. Click here to translate it with Google)

Este diccionario no es una "app" para usuarios. No lo puedes "instalar" en el móvil o el PC. Es una herramienta para investigadores e informáticos.

Descargar el diccionario (10MB .csv.gz; 140MB descomprimido)

Licencia: GNU FDL. Creo que es compatible con todas las licencias de los datos y programas usados para generarlo. Si no fuera así, mándame un correo por favor.

O bien, sigue leyendo el artículo para tener un poco de contexto y entender el fichero.

¿El primer chatbot en español?

Uno de los programas más me fascinaba de niño fue "Herbie", un chatbot para MS-DOS y Windows 3.1, con el que pasé muchas horas muertas en aquella época pre-internet. No exagero cuando digo que ese software fue uno de los motivos que me impulsó a estudiar Inteligencia Artificial en la Universidad.

Casi no hay información de Herbie por la red, pero en su LEEME.TXT indica que el autor es un tal Rafael García de Avilés, e incluye su teléfono, al cual más de una vez he estado tentado de llamar.

herbie

Roberto Bonio

Allá por 2014 empecé a programar un chatbot propio, llamado "Roberto Bonio", que incluí en el grupo de Whatsapp con los amigos. Fue un verdadero hit; Roberto nos enviaba memes, te decía qué pizza escoger del menú, nos organizaba las cenas, y se reía de ti cuando decías tonterías.

Por desgracia, Whatsapp es muy agresivo baneando bots, y al no poder encontrar una manera estable de mantener el bot, lo acabé abandonando. Su última proeza fue hacer de DJ en mi boda, sí, como lo lees. Los invitados podían acceder a una página desde su móvil para pedir canciones, y Roberto las pinchaba en el Spotify conectado a un altavoz.

roberto bonio

Bueno, ¿y a qué viene todo esto?

Actualmente estoy desarrollando una versión pública y mejorada de Bonio. Sólo servirá para echar unas risas, una especie de Herbie moderno con capacidades avanzadas de procesamiento del lenguaje.

Al intentar dotarlo de mejor inteligencia me he encontrado con que hay muy pocos recursos de NLP en español. En inglés hay una barbaridad, incluso APIs avanzadas (Microsoft, Google, IBM) de análisis morfosintáctico, análisis de sentimiento, etc, pero no en nuestro idioma.

Así que me puse manos a la obra, investigando y recopilando más de 500 MB de recursos lingüísticos en español.

Viendo el buen resultado, he decidido publicar el trabajo. Se trata de un diccionario libre en español, anotado, con un corpus extenso de palabras y variantes, aunque por desgracia sin definiciones.

El diccionario

Descargar el diccionario (10MB .csv.gz; 140MB descomprimido)

Licencia: GNU FDL. Creo que es compatible con todas las licencias de los datos y programas usados para generarlo. Si no fuera así, mándame un correo por favor.

el diccionario

El diccionario es un fichero csv con las siguientes columnas:

  • palabra: la palabra en cuestión
  • tipo: si es artículo, nombre, pronombre, adjetivo, etc.
    • Los nombres tienen categorización adicional:
      • nombre/<tipo nombre>: topónimo, nombre común, si es masculino, femenino...
    • Los verbos tienen categorización adicional:
      • si es un infinitivo: verbo/<tipo>/infinitivo, donde <tipo> es transitivo, pronominal, etc.
      • si es una forma conjugada: verbo/<tipo>/conjugado/<tiempo>/<persona>/<modelo de conjugación>
    • En general, tendrás que hacer un listado de este campo para ver todas las opciones. Es un poco un cajón de sastre.
  • género: masculino, femenino, o neutro
  • número: singular o plural
  • raíz: si es una palabra declinada (nombre, verbo, adjetivo...) este campo indica la palabra original
  • afijo: el afijo de RLAE
  • tónica: la sílaba tónica
  • sílabas: separación en sílabas de la palabra
  • locale: si esta palabra corresponde a una región específica del español
  • origen: corpus o fichero de procedencia de la palabra
  • sinónimos: sinónimos para dicha palabra, separados por barras. Sólo aparecen en palabras raíz.

Errores

El fichero ha pasado una inspección ocular, y además todas las líneas del CSV son válidas. Sin embargo, es posible que aparezcan errores puntuales.

Si detectas un error GRAVE DE FORMATO, que impide el procesamiento del fichero, envíame un correo y lo revisaré. No tengo capacidad para revisar los errores lingüísticos, de los que habrá muchos. Este es un corpus generado informáticamente.

Estos son los errores actuales detectados:

  • Es muy habitual que falte información y/o haya campos vacíos
  • Puede haber palabras mal acentuadas. He declinado el género/número de manera simple (-o, -a, -os, -as, etc.)
  • Hay errores de categorización de palabras. Aparecen adverbios o adjetivos como nombres, y problemas similares
  • Falta de consistencia al tipificar los nombres y verbos
  • Las sílabas de palabras compuestas o frases pueden estar mal

Pese a todo, el diccionario es muy usable y de alta calidad, especialmente si tenemos en cuenta que las fuentes de datos provienen de voluntarios.

Conversión a SQL y rendimiento

Mi objetivo era generar un único mega-fichero de fácil distribución y que fuera parseable informáticamente. Como, además, tampoco soy lingüista, no he sido capaz de diseñar una estructura de datos mejor.

En cualquier caso, pienso que es suficiente, y que en el peor de los casos sólo habrá que hacer dos queries al fichero para acceder a toda la información relacionada.

Mi recomandación, al tratarse de un fichero muy grande, es pasarlo a una base de datos. Yo he hecho lo siguiente:

$ sqlite3 palabras.sqlite
sqlite> create table palabras
(
    palabra text,
    tipo text,
    género text,
    número text,
    raíz text,
    afijo text,
    tónica integer,
    sílabas text,
    locale text,
    origen text,
    sinónimos text
);

sqlite> create index palabras_palabra_index on palabras (palabra);
sqlite> create index palabras_raíz_index on palabras (raíz);
sqlite> create index palabras_sinónimos_index on palabras (sinónimos);
sqlite> .mode csv
sqlite> .import palabras.csv palabras
sqlite> .quit

De esta manera es mucho más rápido consultar el diccionario, y se pueden hacer queries complejos, del tipo SELECT palabra FROM palabras WHERE sinónimos LIKE '%/excelente/%' AND tipo LIKE 'nombre%' en milésimas de segundo.

Licencia, origen de datos y referencias

Ha sido realmente difícil encontrar buenas fuentes de palabras, por lo que hay que agradecer la tarea titánica a las personas que han generado estos corpus libres.

Es importante mencionar la falta de datasets de recursos públicos: Universidades y RAE proveen de "buscadores" y "herramientas" online, pero no publican las fuentes de datos para poder trabajar con ellas. ¡¡Muy mal!!

El español en la red es un lenguaje que está a años luz del inglés. Nos faltan herramientas y corpus libres para poder avanzar en IA y procesamiento de lenguaje natural en nuestro idioma. Hasta que no haya recursos disponibles, los investigadores y científicos seguiremos prefiriendo el inglés.

Recursos usados para la generación de palabras.csv:

  • El mejor diccionario tanto en cantidad de palabras como en calidad de las anotaciones es el del Recursos Lingüísticos Abiertos del Español, usado en herramientas libres como LibreOffice.
  • Los verbos los he conjugado con el FreeConj-TIP, de Carreras-Riudavets et al.
  • El silabeador también es del mismo grupo, Silabeador TIP, en este caso de Hernández-Figueroa et al.
  • Diccionario wspanish del GNU Ispell
  • Sinónimos del RLAes
  • Algunas correcciones manuales por mi parte, dentro de mis limitaciones

En resumen, gracias a las comunidades de software libre y a un grupo de investigación que publica sus datasets me ha sido posible generar este recurso.

Trabajo futuro

Por orden de importancia, los siguientes pasos serían:

  1. Asegurarse de usar las formas declinadas correctas (femeninos/plurales) que se han generado con un algoritmo simple, y revisar en general la ortografía del diccionario
  2. Revisar la calidad morfológica. Como apunto más arriba, hay bastantes errores de bulto en la clasificación de las palabras que dificultarán el análisis morfosintáctico de las frases usando este diccionario
  3. Inclusión de definiciones. Lo más sencillo sería parsear el Wikcionario, pero es una herramienta pensada para texto, no APIs, y la información no está estructurada.
  4. Análisis de sentimiento de las palabras. Algo tan básico como añadir una columna sentimiento a cada palabra, con los valores positivo o negativo sería de increíble utilidad para el procesamiento de textos, por ejemplo, en redes sociales

Conclusión

Mi esfuerzo se ha centrado en unificar y centralizar datos ya existentes, así como el uso de herramientas para generar datos derivados. Gracias a unas cuantas horas scripteando ficheros txt y csv he podido generar con poco esfuerzo un corpus bastante completo.

Espero haber generado un recurso útil tanto para aficionados al NLP como para investigadores que necesiten acceso a un diccionario de calidad razonable.

Acabo con un mensaje para las instituciones: ¡Publicad siempre vuestros datasets! Las aplicaciones web de consulta no sirven de nada si no tienen una API potente y permiten volcar los datos.

Por mi parte, he cumplido el objetivo que me propuse: tener un buen diccionaro para mejorar el generador de Markov y que Roberto diga chorradas con más gracia.

conversación con Roberto

Descargar el diccionario (10MB .csv.gz; 140MB descomprimido)

Licencia: GNU FDL. Creo que es compatible con todas las licencias de los datos y programas usados para generarlo. Si no fuera así, mándame un correo por favor.

Tags: projects, nlp, spanish

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

August 22, 2019

Jan van den Berg (j11g)

Popular post postmortem August 22, 2019 03:26 PM

Yesterday I wrote a story about how Git is eating the world. And in less than 24 hours more than 10.000 people visited this article! This is not the normal kind of traffic for this site. So that calls for its own article.

As you can see, there is a slight uptake the last day.

Not only did the above WordPress plugin tell me I hit 10.000, but my server logs said the same.

grep git-is-eating-the-world j11g-access.log|awk '{print $1}'|sort -u|wc -l

I run this WordPress installation on my own VPS. Which, by the way, could handle the load just fine! (PHP 7.3 is great.)

1 core/2GB VPS with Apache + PHP 7.3

How?

I usually write about things that interest me: but those things may not always be of interest to the general public. But in this case I tried to make a conscious effort to write about something topical and a little bit controversial (BitBucket dropping Mercurial support). I also picked a catchy title: a play on Marc Andreessens’ famous quote. And I deliberately tried to push the article. I.e. this is how most content creators do it. But I usually put stuff out there, and see where it lands. I mainly enjoy writing for the sake of writing. My most visited article was a Django tutorial which generated 3000 visits this year (all from generic Google searches). So I thought I could try to give promotion a little bit more effort. This idea came after reading about a Dutch blogger who was a bit obsessed with getting clicks. Which to me is (still) not the best reason to write, but of course I can see why he was obsessed with it. When you write something, it’s more fun when people actually read it!

Submitting it

So after writing I picked lobste.rs and Hacker News to submit my story to. I submit links very frequently to both sites, just not my own content. But this content seemed right for these sites.

On Hacker News it sadly went nowhere. This sometimes happens. Timing is everything. I have often also submitted something there that went nowhere, but that same link would be on the front page the next day via a new submission from someone else. My submission did however generate 234 unique visits. Which normally would be a good day for my site.

Six upvotes. Not enough for the front page.

Lobste.rs however, is a different story. Since submitting it there, it quickly shot up to the number one position. And currently (at time of writing) it was still number one. Also, lobste.rs had the most relevant discussion and comments of all aggregators.

I ❤ the lobste.rs community🦞

Uh?!

After this I also tried submitting my link to Reddit. But much to my surprise someone beat me to it!? He (or she) submitted my post to two subreddits. I don’t know how the submitter found my post. But anyway, in r/programming it received exactly zero upvotes but it did generate 5 comments? But in r/coding it was the number one post for almost the entire day!

This specific subreddit has 168000 readers and around 1000 active users at any given moment. So even if it only received 83 upvotes, this did generate the bulk of my traffic.

Timing is everything

After this, things got shared (organically?) on Twitter, and other tech sites (like Codeproject) also picked it up. People also seemed to share the link by e-mail: 175 unique clicks came from Gmail and 32 from Yahoo! Mail. Which I think is quite a lot!

I also cross post every WordPress article to Medium: these readers don’t come to my site, but they are of course able to read my content. However, the reach there was very low.

Lessons?

Stating the very obvious here, but it helps to pick a topical subject and then trying to push it on news aggregators. Also (obvious) it struck me how difficult it was to get a foot in the door — those first few upvotes. After that it gets easier, because you become more visible.

These are all obvious. But the most fun was actually discovering that the thing that you wrote took on a life of its own. And people you don’t know are sharing, reading and discussing it without your interference. I just love that. I still think that that is part of the magical aspect of the internet. People I have never met who live on the other side of the planet can instantly read and discuss what I’ve written. The internet gives everyone a voice and I love it.

The post Popular post postmortem appeared first on Jan van den Berg.

August 21, 2019

Jan van den Berg (j11g)

Git is eating the world August 21, 2019 06:33 PM

The inception of Git (2005) is more or less the halfway point between the inception of Linux (1991) and today (2019). A lot has happened since. One thing is clear however: software is eating the world and Git is the fork with which it is being eaten. (Yes, pun intended).

Linux and Git

In 2005, as far as Linus Torvalds’ legacy was concerned, he didn’t need to worry. His pet project Linux — “won’t be big and professional” — was well on its way to dominating the server and supercomputer market. And with the arrival of Linux powered Android smartphones this usage would even be eclipsed a few years later. Linux was also already a full-blown day job for many developers and the biggest distributed software project in the world.

And now, with the creation of Git in 2005, Linus Torvalds can stake the claim that he is responsible for not one, but two of the most important software revolutions ever. Both projects grew out of a personal itch, with the latter being needed for the other. The story of both inceptions are of course very well documented in the annals of internet history via mailinglist archives. (Side note: one of Git’s most impressive feats was at the very early beginnings, when Torvalds was able to get Git self-hosted within a matter of days).

Today

Fast forward to today and Git is everywhere. It has become the de facto distributed versioning control system (DVCS). However, it was of course not the first DVCS and may not even be the best i.e. most suitable for some use cases.

The Linux project using Git is of course the biggest confirmation of its powerful qualities. Because very few open source projects are bigger than Linux. So if it’s good enough for Linux it sure should be good enough for all other projects. Right?

Git however is also notorious for being the perfect tool to shoot yourself in the face with. It demands a different way of thinking. And things can quickly go wrong if you’re not completely comfortable with what you’re doing.

Web-based DVCS

Part of these problems were solved by GitHub. Which gave the ideas of Git and distributed software collaboration a web interface and made it social (follow developers, star projects etc.). It was the right timing and in an increasingly interconnected world distributed version control seemed like the only way to go. This left classic client-server version control systems like CVS and SVN in the dust (though some large projects are still developed using these models e.g. OpenBSD uses CVS).

GitHub helped popularize Git itself. And legions of young developers grew up using GitHub and therefore Git. And yet, the world was still hungry for more. This was proven by the arrival of GitLab, initially envisioned as as SaaS Git service, most GitLab revenue now comes from self-hosted installations with premium features.

But of course GitHub wasn’t the only web-based version control system. BitBucket started around the same time and offered not only Git support but also Mercurial support. And even in 2019 new web-based software development platforms (using Git) are born: i.e. sourcehut.

Too late?

However the fast adoption of tools like GitHub had already left other distributed version control systems behind in popularity: systems like Fossil, Bazaar and Mercurial and many others. Even though some of these systems on a certain level might be better suited for most projects. The relative simplicity of Fossil does a lot of things right. And a lot of people seem to agree Mercurial is the more intuitive DVCS.

BitKeeper was also too late to realize that they had lost the war, when they open-sourced their software in 2016. Remember: BitKeeper being proprietary was one of the main reasons Git was born initially.

Yesterday BitBucket announced they would sunset their Mercurial support. Effectively giving almost nothing short of a deathblow to Mercurial, as BitBucket was one of the largest promoters of Mercurial. This set off quite a few discussions around the internet. Partly because of how they plan to sunset their support. But partly also because Mercurial seems to have a lot of sentimental support — the argument being that it is the saner and more intuitive DVCS. Which is surprising because, as stated by BitBucket; over 90% of users use Git. So there is a clear winner. Still the idea of a winner-takes-all does not sit well with some developers. Which is probably a good thing.

Future?

Right now Git is the clear winner, there is no denying that. Git is everywhere, and for many IDEs/workflow/collaboration software it is the default choice for a DVCS. But things are never static, especially in the world of software. So I am curious to see where we’ll be in another 14 years!

The post Git is eating the world appeared first on Jan van den Berg.

August 20, 2019

Jan van den Berg (j11g)

Giacomo Joyce – James Joyce August 20, 2019 10:02 PM

A couple of years ago I stranded about half-way into Ulysses. It did not click. One of the greatest novels ever written, but I had little use for it. So I put it aside for another time. Recently I came across a rather insightful video (highly recommended) about Joyce and his work, and all of a sudden a lot of things clicked!

But not to rush into it too fast, I picked up Giacomo Joyce first. A small and a rather easy read, but classic Joyce.

Giacomo Joyce – James Joyce (1914/1968) – 46 pages

It’s a collection of poem-like loose and short to very short phrases and sentences about Joyce’s encounters with a young female student. My copy has the English text on the left and the Dutch translation on the right, which is very useful when you are reading someone who helped shape the English language. Also the notes by the translator are remarkably enlightening. Joyce could write three sentences, and you would think you understand what he says, that is until you read the explanation about the six references and metaphors Joyce crammed in there.

Since it is a very loose collection of thoughts (that Joyce almost threw out) and that were only collected by someone else, I couldn’t help but wonder: why did you write it down in the first place then (or why do people blog)? Funny, is that he answer that question more or less himself:

Write it!

The post Giacomo Joyce – James Joyce appeared first on Jan van den Berg.

Bogdan Popa (bogdan)

Racket for e-commerce August 20, 2019 06:00 PM

I had originally shared a version of this post with a small, private mailing list, but then I figured there’d be no harm in sharing it with a larger audience so here it is. My girlfriend and I recently launched matchacha.ro, a small e-commerce site selling Japanese green tea here in Romania (the site defaults to Romanian, but you can change the language by scrolling to the bottom – we don’t ship outside of Romania, though, sorry!

Leo Tindall (LeoLambda)

Arduino Geiger Counter Dosimeter August 20, 2019 12:19 AM

It’s a simple and humble apparatus, but it’s quite useful. One of my first electronics projects was the excellent β and γ radiation detector kit, from MightyOhm. It’s all through-hole construction, a beautiful yellow PCB that’s easy to assemble and fun to play with. Recently, though, I found myself interested in more than just a simple beep-beep-beep of incoming radiation - I wanted to actually compare the radioactivity of several samples without just manually counting the ticks.

Pete Corey (petecorey)

TIL About Node.js’ REPL Module August 20, 2019 12:00 AM

Today I learned that Node.js ships with a repl module that can be used to spin up a full-featured REPL on any Node.js process. This can be a fantastic tool for debugging a running server, or manually triggering back-end events.

Let’s assume that we’ve built a Node.js server who’s entry point is a server.js file. Let’s also assume that we have a constant (maybe pulled from our environment, maybe elsewhere) called REPL who’s truthiness determines whether we should start our REPL instance on standard in. Spinning up our REPL is as easy as:


if (REPL) {
    require('repl').start();
}

Once our server starts up, we’ll be greeted by a familiar prompt:


Starting server...
Listening on localhost:8080!
> 

Fantastic! Normal REPL rules apply. Our server will continue to run and its output will continue to stream to standard out. Our REPL prompt will stick to the bottom of the tail, as expected.

More advanced options can be gleaned from the repl documentation. Happy REPLing!

August 19, 2019

Pete Corey (petecorey)

Animating a Canvas with React Hooks August 19, 2019 12:00 AM

A recent React-based client project of mine required an HTML5 canvas animation. Already knee-deep in the new React hooks API, I decided to forgo the “traditional” (can something be “traditional” after just five years?) technique of using componentDidMount and componentWillUnmount in a class-based component, and try my hand at rendering and animating a canvas using React’s new useEffect hook.

Let’s dig into it!


Let’s set the scene by creating a new React component that we want to add our canvas to. We’ll assume that we’re trying to render a circle, so we’ll call our new component, Circle:


import React from 'react';

const Circle = () => {
    return (
        <canvas
            style={{ width: '100px', height: '100px' }}
        />
    );
};

export default Circle;

So far so good.


Our Circle component renders a canvas onto the page, but we have no way of interacting with it. Typically, to interact with an element of the DOM from within a React component, you need a “ref”. The new React hooks API gives us a convenient way to create and use refs:


 import React from 'react';
+import { useRef } from 'react';

 const Circle = () => {
+    let ref = useRef();
     return (
         <canvas
+            ref={ref} 
             style={{ width: '100px', height: '100px' }}
         />
     );
 };

 export default Circle;

Now ref.current holds a reference to our canvas DOM node.


Interacting with our canvas produces “side effects”, which isn’t allowed from within the render phase of a React component. Thankfully, the new hooks API gives us a simple way to introduce side effects into our components via the useEffect hook.


 import React from 'react';
+import { useEffect } from 'react';
 import { useRef } from 'react';
 
 const Circle = () => {
     let ref = useRef();
     
+    useEffect(() => {
+        let canvas = ref.current;
+        let context = canvas.getContext('2d');
+        context.beginPath();
+        context.arc(50, 50, 50, 0, 2 * Math.PI);
+        context.fill();
+    });
     
     return (
         <canvas
             ref={ref} 
             style={{ width: '100px', height: '100px' }}
         />
     );
 };
 
 export default Circle;

Our useEffect callback is free to interact directly with our canvas living in the DOM. In this case, we’re drawing a circle to our canvas.


Unfortunately, our circle may look a little fuzzy or distorted, depending on the pixel density of the screen it’s being viewed on. Let’s fix that by adjusting the scaling of our canvas:


 import React from 'react';
 import { useEffect } from 'react';
 import { useRef } from 'react';
 
+const getPixelRatio = context => {
+    var backingStore =
+    context.backingStorePixelRatio ||
+    context.webkitBackingStorePixelRatio ||
+    context.mozBackingStorePixelRatio ||
+    context.msBackingStorePixelRatio ||
+    context.oBackingStorePixelRatio ||
+    context.backingStorePixelRatio ||
+    1;
+    
+    return (window.devicePixelRatio || 1) / backingStore;
+};
 
 const Circle = () => {
     let ref = useRef();
     
     useEffect(() => {
         let canvas = ref.current;
         let context = canvas.getContext('2d');
         
+        let ratio = getPixelRatio(context);
+        let width = getComputedStyle(canvas)
+            .getPropertyValue('width')
+            .slice(0, -2);
+        let height = getComputedStyle(canvas)
+            .getPropertyValue('height')
+            .slice(0, -2);
         
+        canvas.width = width * ratio;
+        canvas.height = height * ratio;
+        canvas.style.width = `${width}px`;
+        canvas.style.height = `${height}px`;
         
         context.beginPath();
         context.arc(
+            canvas.width / 2,
+            canvas.height / 2,
+            canvas.width / 2,
             0,
             2 * Math.PI
         );
         context.fill();
     });
     
     return (
         <canvas
             ref={ref} 
             style={{ width: '100px', height: '100px' }}
         />
     );
 };
 
 export default Circle;

And with that, our circle should be crystal clear.


Now let’s introduce some animation. The standard way of animating an HTML5 canvas is using the requestAnimationFrame function to repeatedly call a function that renders our scene. Before we do that, we need to refactor our circle drawing code into a render function:


 useEffect(() => {
     ...
+    const render = () => {
         context.beginPath();
         context.arc(
             canvas.width / 2,
             canvas.height / 2,
             canvas.width / 2,
             0,
             2 * Math.PI
         );
         context.fill();
+    };
     
+    render();
 });

Now that we have a render function, we can instruct the browser to recursively call it whenever its appropriate to render another frame:


 const render = () => {
     context.beginPath();
     context.arc(
         canvas.width / 2,
         canvas.height / 2,
         canvas.width / 2,
         0,
         2 * Math.PI
     );
     context.fill();
+    requestAnimationFrame(render);
 };

This works, but there are two problems. First, if our component unmounts after our call to requestAnimationFrame, but before our render function is called, it can lead to problems. We should really cancel any pending animiation frame requests any time our component unmounts. Thankfully, requestAnimationFrame returns a request identifier that can be passed to cancelAnimationFrame to cancel our request.

The useEffect hook optionally expects a function to be returned by our callback. This function will be called to handle any cleanup required by our effect. Let’s refactor our animation loop to properly clean up after itself:


 useEffect(() => {
     ...
+    let requestId;
     const render = () => {
         context.beginPath();
         context.arc(
             canvas.width / 2,
             canvas.height / 2,
             canvas.width / 2,
             0,
             2 * Math.PI
         );
         context.fill();
+        requestId = requestAnimationFrame(render);
     };
     
     render();
     
+    return () => {
+        cancelAnimationFrame(requestId);
+    };
 });

Perfect.


Our second problem is that our render function isn’t doing anything. We have no visual indicator that our animation is actually happening!

Let’s change that and have some fun with our circle:


 let requestId,
+    i = 0;
 const render = () => {
     context.clearRect(0, 0, canvas.width, canvas.height);
     context.beginPath();
     context.arc(
         canvas.width / 2,
         canvas.height / 2,
+        (canvas.width / 2) * Math.abs(Math.cos(i)),
         0,
         2 * Math.PI
     );
     context.fill();
+    i += 0.05;
     requestId = requestAnimationFrame(render);
 };

Beautiful. Now we can clearly see that our animation is running in full swing. This was obviously a fairly contrived example, but hopefully it serves as a helpful recipe for you in the future.

#root { margin: 4em auto; } #root canvas { height: 100%!important; width: 100%!important; }

August 18, 2019

David Wilson (dw)

Mitogen v0.2.8 released August 18, 2019 08:45 PM

Mitogen for Ansible v0.2.8 has been released. This version (finally) supports Ansible 2.8, comes with a supercharged replacement fetch module, and includes roughly 85% of what is needed to implemement fully asynchronous connect.

As usual a huge slew of fixes are included. This is a bumper release, running to over 20k lines of diff. Get it while it's hot, and as always, bug reports are welcome!

Zac Brown (zacbrown)

Using Opusmodus with QuickLisp August 18, 2019 07:00 AM

In my scant free time, I have begun playing with Common Lisp again. It’s been 10-15 years since I last spent any time with it. I recently discovered a program called Opusmodus that uses Common Lisp at its core. It provides a music composition interface using Common Lisp to enable composers to programmatically score their compositions.

Opusmodus wasn’t explicitly designed for programmers though there are a number of programmers that use it from what I can tell. If you have written any Common Lisp in recent history, QuickLisp is a natural tool to use as part of the development cycle. Unfortunately, Opusmodus’s bundled version of Clozure CL (called ccl later) is stripped of the development interfaces so some packages will not compile correctly with (ql:quickload "...").

If you go hunting around for instructions on how to install QuickLisp properly with Opusmodus, there’s a few references on the forums but they’re at least a couple years old. Here’s a quick set of steps to get QuickLisp installed and loaded automatically. It’s broken into updating ccl and then what changes are necessary in Opusmodus.

Updating ccl

Note: Update any instances of <user> with your own user’s name.

  1. Pick a directory where you’d like to have the full ccl installed. A good place would be /Users/<user>/ccl/.
  2. Fetch the sources from git: git clone https://github.com/Clozure/ccl.git /Users/<user>/ccl
  3. Fetch the bootstrapping binaries: curl -L -O https://github.com/Clozure/ccl/releases/download/v1.12-dev.5/darwinx86.tar.gz
    • Note: At the time that I wrote this, Opusmodus was using ccl v1.12-dev.5 which is why the bootstrapping binaries above reference that specific release.
  4. Change directory into the ccl sources: cd /Users/<user>/ccl
  5. Unpack the bootstrapping binaries into the sources directory: tar xf ../darwinx86.tar.gz
  6. Launch ccl: ./Users/<user>/ccl/dx86cl64
  7. Now rebuild ccl: (rebuild-ccl :full t)

If all went well, you should see a lot of output about Loading …” and then at the end:

;Wrote bootstrapping image: #P"/Users/zbrown/Code/lisp/ccl-dev/x86-boot64.image"
;Building lisp-kernel ...
;Kernel built successfully.
;Wrote heap image: #P"/Users/zbrown/Code/lisp/ccl-dev/dx86cl64.image"

Configuring Opusmodus to Load QuickLisp

The default location for Opusmodus to install extensions is in /Users/<user>/Opusmodus/Extensions. Opusmodus v1.3 includes a QuickLisp Start.lisp in the extension directory with a lot of what you need to get started.

There’s a couple of specific things you need to update though to get it working:

  1. First, the entirety of the QuickLisp Start.lisp is block/multiline commented out. If you’re not familiar with the Common Lisp multilane comments, they open with #| and close with |#. In my installation of Opusmodus, the entire file was commented out.
  2. Second, you need to tell Opusmodus to use your own version of ccl instead of the bundled version of ccl. Beneath the line (in-package :Opusmodus), add this line (update the <user>): (setf (logical-pathname-translations "ccl") &apos((#P"ccl:**;*.*" #P"/Users/<user>/ccl/**/*.*")))
    • This updates the path that Opusmodus looks for ccl.
  3. Next we need to uncomment the lines that load QuickLisp and install it. They look like the following:
    • (load "http://beta.quicklisp.org/quicklisp.lisp")
    • (quicklisp-quickstart:install)

Once you’ve finished up the two sections above, you can launch Opusmodus and it should load QuickLisp. To test this, select the Listener window and install a package: (ql:quickload "cl-json"). If all goes well, you should see something like:

Welcome to Opusmodus Version 1.3 (r24952)
Copyright © MMXIX Opusmodus™ Ltd. All rights reserved.
? (ql:quickload "cl-json")
To load "cl-json":
  Load 1 ASDF system:
    cl-json
; Loading "cl-json"

("cl-json")
? 

In my case, I already had cl-json installed so nothing was downloaded.

The end

That’s it. Nothing else to describe.

August 17, 2019

Jan van den Berg (j11g)

The Death of Murat Idrissi – Tommy Wieringa August 17, 2019 03:12 PM

Tommy Wieringa is of course famous for his novel Joe Speedboot. A tremendous novel, where Wieringa demonstrates heaps of writers’ finesse. This book — the Death of Murat Idrissi — is no different. Even though this is a short and easy read, it touches on a lot of subjects and themes and has the Tommy Wieringa flair all over it. He is a master is describing brooding situations, internal struggles and setting a scene.

The death of Murat Idrissi – Tommy Wieringa (2017) – 123 pages

The post The Death of Murat Idrissi – Tommy Wieringa appeared first on Jan van den Berg.

August 16, 2019

Jan van den Berg (j11g)

Gid – Get it done! August 16, 2019 08:34 PM

Last weekend I built a personal ToDo app. Partly as an excuse to mess around a bit with all this ‘new and hip’ Web 2.0 technology (jQuery and Bootstrap) 🙈

But mostly because I needed one, and I couldn’t find a decent one.

Yes, I designed the icon myself. Can you tell?

Decent?

Decent in my opinion would be:

  1. Self hosted
  2. Self contained
  3. Use a plain text file
  4. Mobile friendly
  5. Able to track / see DONE items

And Gid does just that (and nothing more).

  1. Any PHP enabled webserver will do.
  2. No need for third party tools, everything you need is right here (Bootstrap and jQuery are included).
  3. No database setup or connection is necessary. Gid writes to plain text files that can be moved and edited by hand if needed (like todotxt.org).
  4. Works and looks decent on a smartphone.
  5. The DONE items are still visible with a strike through.

I had fun, learned quite a few new things (web development is not my day job) and me and my wife now share our grocery list with this app!

The biggest headache was getting iOS to consistently and correctly handle input form submit events. Being a touch device, this is somehow still a thing in 2019. Thanks Stack Overflow! Anyway, this is what it looks like on my iPhone.

Web development

This was mainly an interesting exercise to try to understand how PHP, Javascript/jQuery and Bootstrap work together on a rather basic level and how with Ajax you are able to manipulate the DOM. I deliberately used an older tech stack, thinking a lot of problems would be solved, however (as explained) some things still seem to be a thing. Also, what I was trying to do is just very, very basic and still I feel somehow this should be way easier! There are a lot of different technologies involved that each have their own specifics and that all have to work together.

Code is on Github: do as you please.

(P.S. Yes, the name. I know.)

The post Gid – Get it done! appeared first on Jan van den Berg.

Derek Jones (derek-jones)

Converting lines in an svg image to csv August 16, 2019 01:40 AM

During a search for data on programming language usage I discovered Stack Overflow Trends, showing an interesting plot of language tags appearing on Stack Overflow questions (see below). Where was the csv file for these numbers? Somebody had asked this question last year, but there were no answers.

Stack Overflow language tag trends over time.

The graphic is in svg format; has anybody written an svg to csv conversion tool? I could only find conversion tools for specialist uses, e.g., geographical data processing. The svg file format is all xml, and using a text editor I could see the numbers I was after. How hard could it be (it had to be easier than a png heatmap)?

Extracting the x/y coordinates of the line segments for each language turned out to be straight forward (after some trial and error). The svg generation process made matching language to line trivial; the language name was included as an xml attribute.

Programmatically extracting the x/y axis information exhausted my patience, and I hard coded the numbers (code+data). The process involves walking an xml structure and R’s list processing, two pet hates of mine (the data is for a book that uses R, so I try to do everything data related in R).

I used R’s xml2 package to read the svg files. Perhaps if my mind had a better fit to xml and R lists, I would have been able to do everything using just the functions in this package. My aim was always to get far enough down to convert the subtree to a data frame.

Extracting data from graphs represented in svg files is so easy (says he). Where is the wonderful conversion tool that my search failed to locate? Pointers welcome.

August 15, 2019

Jan van den Berg (j11g)

Create a Chrome bookmark html file to import list of URLs August 15, 2019 02:20 PM

I recently switched RSS providers and I could only extract my saved posts as a list of URLs. So I thought I’d add these to a bookmark folder in Chrome. However, Chrome bookmark import only accepts a specifically formatted .html file.

So if you have a file with all your urls, name this file ‘url.txt’ and run this script to create a .html file that you can import in Chrome (hat-tip to GeoffreyPlitt).

#!/bin/bash
#
# Run this script on a file named urls.txt with all your URLs and pipe the output to an HTML file.
# Example: ./convert_url_file.sh > bookmarks.html

echo "<!DOCTYPE NETSCAPE-Bookmark-file-1>"
echo '<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">'
echo '<TITLE>Bookmarks</TITLE>'
echo '<H1>Bookmarks</H1>'
echo '<DL><p>'
  cat urls.txt |
  while read L; do
    echo -n '    <DT><A HREF="';
        echo ''"$L"'">'"$L"'</A>';
  done
echo "</DL><p>"

The post Create a Chrome bookmark html file to import list of URLs appeared first on Jan van den Berg.

Pepijn de Vos (pepijndevos)

Open Source Formal Verification in VHDL August 15, 2019 12:00 AM

I believe in the importance of open source synthesis, and think it’s important that open source tools support both Verilog and VHDL. Even though my GSoC proposal to add VHDL support to Yosys was rejected, I’ve still been contributing small bits and pieces to GHDL and its Yosys plugin.

This week we reached what I think is an important milestone: I was able to synthesize my VHDL CPU and then formally verify the ALU of it using completely open source tools. (and then synthesize it to an FPGA which is not supported by open source tools yet) There is a lot to unpack here, so let’s jump in.

Yosys, Nextpnr, SymbiYosys, GHDL, ghdlsynth-beta

Yosys is an open source synthesis tool that is quickly gaining momentum and supporting more and more FPGAs. Yosys currently supports Verilog, and turns that into various low-level netlist representations.

Nextpnr is a place-and-rout tool, which takes a netlist and turns it into a bitstream for any of the supported FPGA types. These bitstream formats are not publicly documented, so this is a huge reverse-engineering effort.

SymbiYosys is a tool based around Yosys and various SAT solvers to let you do formal verification on your code. More on formal verification later. But important to know is that it works on the netlists generated by Yosys.

GHDL is an open source VHDL simulator, and as far as I know, one of its kind. VHDL is notoriously hard to parse, so many other open source attempts at VHDL simulation and synthesis have faltered. Work is underway to add synthesis to GHDL.

And last but not least, ghdlsynth-beta is a plugin for Yosys that converts the synthesis format of GHDL to the intermediate representation of Yosys, allowing it to be synthesized to various netlist formats and used for FPGA, ASIC, formal verification, and many other uses. It is currently a separate repository, but the goal is to eventually upstream it into Yosys.

Formal Verification

I think formal verification sounds harder and more scary than it is. An alternative description is property testing with a SAT solver. Think Quickcheck, not Coq. This is much simpler and less formal than using a proof assistent.

Basically you describe properties about your code, and SymbiYosys compiles your code an properties to a netlist and from a netlist to Boolean logic. A SAT solver is then used to find inputs to your code that (dis)satisfy the properties you described. This does not “prove” that your code is correct, but it proves that it satisfies the properties you defined.

In hardware description languages you describe properties by assertions and assumptions. An assumption constrains what the SAT solver can consider as valid inputs to your program, and assertions are things you believe to be true about your code.

The powerful thing about formal verification is that it considers all valid inputs at every step, and not just the happy case you might test in simulation. It will find so many edge cases it’s not even funny. Once you get the hang of it, it’s actually less work than writing a testbench. Just a few assertions in your code and the bugs come flying at you.

If you want to learn more about formal verification, Dan Gisselquist has a large number of articles and tutorials about it, mainly using Verilog.

Installation

To play along at home, you need to install a fair number of programs, so better get some of your favourite hot beverage.

At this point you should be able to run ghdl --synth foo.vhd -e foo which will output a VHDL representation of the synthesized netlist. You should be able to run yosys -m ghdl and use the Yosys command ghdl foo.vhd -e foo to obtain a Yosys netlist which you can then show, dump, synth, or even write_verilog.

Verifying a bit-serial ALU

To demonstrate how formal verification works and why it is so powerful, I want to walk you through the verification of the ALU of my CPU.

I’m implementing a bit-serial architecture, which means that my ALU operates on one bit at a time, producing one output bit and a carry. The carry out is then the carry in to the next bit. The logic that produces the output and the carry depends on a 3-bit opcode.

  process(opcode, a, b, ci)
  begin
    case opcode is
      when "000" => -- add
        y <= a xor b xor ci; -- output
        co <= (a and b) or (a and ci) or (b and ci); -- carry
        cr <= '0'; -- carry reset value
      -- [...]
    end case;
  end process;

  process(clk, rst_n)
  begin
    if(rising_edge(clk)) then
      if(rst_n = '0') then
        ci <= cr; -- reset the carry
      else
        ci <= co; -- copy carry out to carry in
      end if;
    end if;
  end process;

Important to note is the carry reset value. For addition, the first bit is added without carry, but for subtraction the carry is 1 because -a = (not a) + 1, and similarly for other different opcodes. So when in reset, the ALU sets the carry in to the reset value corresponding to the current opcode.

So now onward to the verification part. Since VHDL only has assert and none of the SystemVerilog goodies, Property Specification Language is used. (that link contains a good tutorial) PSL not only provides restrict, assume, and cover, but also allows you to express preconditions and sequences.

To make my life easier, I want to specify that I want to restrict valid sequences to those where the design starts in reset, processes 8 bits, back to reset, and repeat, so that the reset will look like 011111111011111111...

restrict { {rst_n = '0'; (rst_n = '1')[*8]}[+]};

Then, I want to specify that when the ALU is active, the opcode will stay constant. Else you’ll just get nonsense.

assume always {rst_n = '0'; rst_n = '1'} |=>
  opcode = last_op until rst_n = '0';

Note that I did not define any clock or inputs. Just limiting the reset and opcode is sufficient. With those assumptions in place, we can assert what the output should look like. I shift the inputs and outputs into 8-bit registers, and then when the ALU goes into reset, we can verify the output. For example, if the opcode is “000”, the output should be the sum of the two inputs.

assert always {opcode = "000" and rst_n = '1'; rst_n = '0'} |->
  y_sr = a_sr+b_sr;

After adding the other opcodes, I wrapped the whole thing in a generate block so I can turn it off with a generic parameter for synthesis

formal_gen : if formal generate
  signal last_op : std_logic_vector(2 downto 0);
  signal a_sr : unsigned(7 downto 0);
  signal b_sr : unsigned(7 downto 0);
  signal y_sr : unsigned(7 downto 0);
begin
-- [...]
end generate;

And now all that’s left to do is write the SymbiYosys script and run it. The script just specifies how to compile the files and the settings for the SAT solver. Note that -fpsl is required for reading --psl code in comments, or --std=08 to use VHDL-2008 which supports PSL as part of the core language.

[options]
mode bmc
depth 20

[engines]
smtbmc z3

[script]
ghdl --std=08 alu.vhd -e alu
prep -top alu

[files]
alu.vhd

To load the GHDL plugin, SymbiYosys has to be run as follows:

$ sby --yosys "yosys -m ghdl" -f alu.sby 
SBY 15:02:25 [alu] Removing direcory 'alu'.
SBY 15:02:25 [alu] Copy 'alu.vhd' to 'alu/src/alu.vhd'.
SBY 15:02:25 [alu] engine_0: smtbmc z3
SBY 15:02:25 [alu] base: starting process "cd alu/src; yosys -m ghdl -ql ../model/design.log ../model/design.ys"
SBY 15:02:25 [alu] base: finished (returncode=0)
SBY 15:02:25 [alu] smt2: starting process "cd alu/model; yosys -m ghdl -ql design_smt2.log design_smt2.ys"
SBY 15:02:25 [alu] smt2: finished (returncode=0)
SBY 15:02:25 [alu] engine_0: starting process "cd alu; yosys-smtbmc -s z3 --presat --noprogress -t 20 --append 0 --dump-vcd engine_0/trace.vcd --dump-vlogtb engine_0/trace_tb.v --dump-smtc engine_0/trace.smtc model/design_smt2.smt2"
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Solver: z3
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Checking assumptions in step 0..
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Checking assertions in step 0..
[...]
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Checking assumptions in step 9..
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Checking assertions in step 9..
SBY 15:02:25 [alu] engine_0: ##   0:00:00  BMC failed!
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Assert failed in alu: /179
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Writing trace to VCD file: engine_0/trace.vcd
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Writing trace to Verilog testbench: engine_0/trace_tb.v
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Writing trace to constraints file: engine_0/trace.smtc
SBY 15:02:25 [alu] engine_0: ##   0:00:00  Status: FAILED (!)
SBY 15:02:25 [alu] engine_0: finished (returncode=1)
SBY 15:02:25 [alu] engine_0: Status returned by engine: FAIL
SBY 15:02:25 [alu] summary: Elapsed clock time [H:MM:SS (secs)]: 0:00:00 (0)
SBY 15:02:25 [alu] summary: Elapsed process time [H:MM:SS (secs)]: 0:00:00 (0)
SBY 15:02:25 [alu] summary: engine_0 (smtbmc z3) returned FAIL
SBY 15:02:25 [alu] summary: counterexample trace: alu/engine_0/trace.vcd
SBY 15:02:25 [alu] DONE (FAIL, rc=2)

Oh no! We have a bug! Let’s open the trace to see what went wrong.

gtkwave alu/engine_0/trace.vcd

gtkwave trace

So we’re doing a subtraction, and according to my math 29-150=-121 but the ALU output is -122, so we’re off by one. A little head-scratching later, we can see the problem: On the first cycle of the subtraction the carry in is zero rather than one! Why? Because on the previous clock cycle the instruction was exclusive or, which reset the carry in to zero.

Note that this bug would never show up if you did a test bench that executes a fixed instruction from reset. But the SAT solver managed to find a specific sequence of opcodes that cause the carry to be wrong. Awesome.

So how do we fix it? There are two ways. The first is to change the code to asynchronously determine the carry in. The second is to write you code so the opcode is stable before the ALU comes out of reset, which ended up using less logic. In this case we can change the opcode assumption to

assume always {rst_n = '0'; rst_n = '1'} |->
  opcode = last_op until rst_n = '0';

Note that we used the thin arrow |-> rather than the fat arrow |=> now. The fat arrow triggers after the precondition has been met, while the thin arrow overlaps with the end of the precondition. So now we’re saying that when reset became inactive, the opcode is the same as it was while the device was in reset. Let’s try again.

$ sby --yosys "yosys -m ghdl" -f alu.sby 
SBY 15:31:36 [alu] Removing direcory 'alu'.
SBY 15:31:36 [alu] Copy 'alu.vhd' to 'alu/src/alu.vhd'.
SBY 15:31:36 [alu] engine_0: smtbmc z3
SBY 15:31:36 [alu] base: starting process "cd alu/src; yosys -m ghdl -ql ../model/design.log ../model/design.ys"
SBY 15:31:36 [alu] base: finished (returncode=0)
SBY 15:31:36 [alu] smt2: starting process "cd alu/model; yosys -m ghdl -ql design_smt2.log design_smt2.ys"
SBY 15:31:36 [alu] smt2: finished (returncode=0)
SBY 15:31:36 [alu] engine_0: starting process "cd alu; yosys-smtbmc -s z3 --presat --noprogress -t 20 --append 0 --dump-vcd engine_0/trace.vcd --dump-vlogtb engine_0/trace_tb.v --dump-smtc engine_0/trace.smtc model/design_smt2.smt2"
SBY 15:31:36 [alu] engine_0: ##   0:00:00  Solver: z3
SBY 15:31:36 [alu] engine_0: ##   0:00:00  Checking assumptions in step 0..
SBY 15:31:36 [alu] engine_0: ##   0:00:00  Checking assertions in step 0..
[...]
SBY 15:31:37 [alu] engine_0: ##   0:00:01  Checking assumptions in step 19..
SBY 15:31:37 [alu] engine_0: ##   0:00:01  Checking assertions in step 19..
SBY 15:31:37 [alu] engine_0: ##   0:00:01  Status: PASSED
SBY 15:31:37 [alu] engine_0: finished (returncode=0)
SBY 15:31:37 [alu] engine_0: Status returned by engine: PASS
SBY 15:31:37 [alu] summary: Elapsed clock time [H:MM:SS (secs)]: 0:00:01 (1)
SBY 15:31:37 [alu] summary: Elapsed process time [H:MM:SS (secs)]: 0:00:01 (1)
SBY 15:31:37 [alu] summary: engine_0 (smtbmc z3) returned PASS
SBY 15:31:37 [alu] DONE (PASS, rc=0)

Yay!

Debugging tips

It should be said that all of this is very experimental and you are therefore likely to run into bugs and missing features. I would say that at this point it is feasible to write new code and work around GHDL’s current limitations (or fix them!), but running large existing codebases is unlikely to be successful. (but very much the goal!)

When you run into errors, the first step is to find out if it is a bug in the plugin or GHDL itself.

If you see Unsupported(1): instance X of Y. this means the plugin does not know how to translate a GHDL netlist item to Yosys. These are usually pretty easy to fix. See this pull request for an example. Good to know: Id_Sub is defined in ghdlsynth_gates.h which is generated from netlists-gates.ads. module->addSub is defined in rtlil.h.

If you just see ERROR: vhdl import failed. this likely means GHDL crashed. Run GHDL outside Yosys (ghdl --synth) to see the actual error. Usually it’ll show something like some_package: cannot handle IIR_KIND_SOMETHING (mycode.vhd:26:8) which means that some_pacakge in the src/synth part of GHDL can’t handle some language construct yet. This can be anything from a missing operator to whole language constructs, and the fix can be anything for copy-pasting another operator to a serious project. See this pull request for an example on how to add a missing operator.

If it’s not obvious what is going on, it’s time to break out gdb. It’s important to know that in the GHDL repo there is a .gdbinit that you can source inside gdb. This enables catching exceptions and includes utilities for printing IIR values. If you want to debug inside Yosys, it is helpful to first run the program without breakpoints so all the libraries are loaded and gdb understands there is Ada code involved. Then source .gdbinit, set breakpoints, and run again. (note: GHDL/Yosys command line arguments are passed to run and not gdb)

Happy debugging!

August 12, 2019

Pete Corey (petecorey)

Fly Straight, Dammit! August 12, 2019 12:00 AM

Numberphile recently posted a video about an interesting recursive function called “Fly Straight, Dammit!” which, when plotted, initially seems chaotic, but after six hundred thirty eight iterations, instantly stabilizes.

This sounds like a perfect opportunity to flex our J muscles and plot this function ourselves!

An Imperative Solution

The simplest approach to plotting our “Fly Straight, Dammit!” graph using the J programming language is to approach things imperatively:

a =: monad define
  if. y < 2 do.
    1
  else.
    py =. a y - 1
    gcd =. y +. py
    if. 1 = gcd do.
      1 + y + py
    else.
      py % gcd
    end.
  end.
)

We’ve defined our a monadic verb to return 1 if we pass in a “base case” value of 0 or 1. Otherwise, we recursively execute a on y - 1 to get our py, or “previous y”. Next, we check if the gcd of y and py equals 1. If it does, we return 1 + y + py. Otherwise, we return py divided by gcd.

This kind of solution shouldn’t look too foreign to anyone.

Let’s plot values of a to verify our solution:

require 'plot'
'type dot' plot a"0 i. 1000

This works, but it’s very slow. We know that our recursive calls are doing a lot of duplicated work. If we could memoize the results of our calls to a, we could save quite a bit of time. Thankfully, memoizing a verb in J is as simple as adding M. to the verb’s declaration:

a =: monad define M.
  ...
)

Now our imperative solution is much faster.

Using Forks and Hooks

While our initial solution works and is fast, it’s not taking advantage of what makes J a unique and interesting language. Let’s try to change that.

The meat of our solution is computing values in two cases. In the case when y and py have a greatest common divisor equal to 1, we’re computing 1 plus y plus py. Our imperative, right to left implementation of this computation looks like this:

1 + y + py

We could also write this as a “monadic noun fork” that basically reads as “1 plus the result of x plus y:

a_a =: 1 + +

Similarly, when we encounter the case where the greatest common divisor between y and py is greater than 1, we want to compute py divided by that gcd. This can be written as a “dyadic fork”:

a_b =: [ % +.

We can read this fork as “x divided by the greatest common divisor of x and y.”

Now that we’ve written our two computations as tacit verbs, we can use the “agenda” verb (@.) to decide which one to use based on the current situation:

a_a =: 1 + +
a_b =: [ % +.

a =: monad define M.
  if. y < 2 do.
    1
  else.
    py =. a y - 1
    has_gcd =. 1 = y +. py
    py (a_b ` a_a @. has_gcd) y
  end.
)

If has_gcd is 0, or “false”, we’ll return the result of py a_b y. Otherwise, if has_gcd is 1, we’ll return the result of py a_a y.

More Agenda

We can elaborate on the idea of using agenda to conditionally pick the verb we want to apply to help simplify out base case check.

First, let’s define our base case and recursive case as verbs that we can combine into a gerund. Our base case is simple. We just want to return 1:

base_case =: 1:

Our recursive case is just the (memoized) else block from our previous example:

recursive_case =: monad define M.
  py =. a y - 1
  has_gcd =. 1 = y +. py
  py (a_b ` a_a @. has_gcd) y
)

Our function, a wants to conditionally apply either base_case or recursive_case, depending on whether y is greater or less than one. We can write that using agenda like so:

a =: base_case ` recursive_case @. (1&<)

And because our base_case verb is so simple, we can just inline it to clean things up:

a_a =: 1 + +
a_b =: [ % +.

recursive_case =: monad define M.
  py =. a y - 1
  has_gcd =. 1 = y +. py
  py (a_b ` a_a @. has_gcd) y
)

a =: 1: ` recursive_case @. (1&<)

Using agenda to build conditionals and pseudo-“case statements” can be a powerful tool for incorporating conditionals into J programs.

Going Further

It’s conceivable that you might want to implement a tacit version of our recursive_case. Unfortunately, my J-fu isn’t strong enough to tackle that and come up with a sane solution.

That said, Raul Miller came up with a one-line solution (on his phone) and posted it on Twitter. Raul’s J-fu is strong.

Marc Brooker (mjb)

Kindness, Wickedness and Safety August 12, 2019 12:00 AM

Kindness, Wickedness and Safety

We must build kind systems.

David Epstein's book Range: Why Generalists Triumph in a Specialized World turned me on to the idea of Kind and Wicked learning environments, and I've found the idea to be very useful in framing all kinds of problems.1 The idea comes from The Two Settings of Kind and Wicked Learning Environments. The abstract gets right to the point:

Inference involves two settings: In the first, information is acquired (learning); in the second, it is applied (predictions or choices). Kind learning environments involve close matches between the informational elements in the two settings and are a necessary condition for accurate inferences. Wicked learning environments involve mismatches.

The authors go on to describe the two environments in terms of the information that we can learn from L (for learning), and information that we use when we actually have to make predictions T (for target). They break environments down into kind or wicked depending on how L relates to T. In kind environments, L and T are closely related: if you learn a rule from L it applies at least approximately to T. In wicked environments, L is a subset or superset of T, or the sets intersect only partially, or are completely unrelated.

Simplifying this a bit more, in kind environments we can learn the right lessons from experience, in wicked environment we learn the wrong lessons (or at least incomplete lessons).

From the paper again:

If kind, we have the necessary conditions for accurate inference. Therefore, any errors must be attributed to the person (e.g., inappropriate information aggregation). If wicked, we can identify how error results from task features, although these can also be affected by human actions. In short, our framework facilitates pinpointing the sources of errors (task structure and/or person).

This has interesting implications for thinking about safety, and the role of operators (and builders) in ensuring safety. In kind environments, operator mistakes can be seen as human error, where the human learned the wrong lesson or did the wrong thing. In wicked environments, humans will always make errors, because there are risks that are not captured by their experience.

Going back to Anatoly Dyatlov's question to the IAEA:

How and why should the operators have compensated for design errors they did not know about?

Dyatlov is saying that operating Chernobyl was a wicked environment. Operators applying their best knowledge and experience, even flawlessly, weren't able to make the right inferences about the safety of the system.

Back to the paper:

Since kind environments are a necessary condition for accurate judgments, our framework suggests deliberately creating kind environments.

I found reading this to be something of a revelation. When building safe systems, we need to make those systems kind. We need to deliberately create kind evironments. If we build them so they are wicked, then we set our operators, tooling and automation up for failure.

Some parts of our field are inherently wicked. In large and complex systems the set of circumstances we learn from is always incomplete, because the system has so many states that there's no way to have seen them all before. In security, there's an active attacker who's trying very hard to make the environment wicked.

The role of the designer and builder of systems is to make the environment as kind as possible. Extract as much wickedness as possible, and try not to add any.

Footnotes

  1. The book is worth reading. It contains a lot of interesting ideas, but like all popular science books also contains a lot of extrapolation beyond what the research supports. If you're pressed for time, the EconTalk episode about the book covers a lot of the material.

August 11, 2019

Derek Jones (derek-jones)

My book’s pdf generation workflow August 11, 2019 11:50 PM

The process used to generate the pdf of my evidence-based software engineering book has been on my list of things to blog about, for ever. An email arrived this afternoon, asking how I produced various effects using Asciidoc; this post probably contains rather more than N. Psaris wanted to know.

It’s very easy to get sucked into fiddling around with page layout and different effects. So, unless I am about to make a release of a draft, I only generate a pdf once, at the end of each month.

At the end of the month the text is spell checked using aspell, and then grammar checked using Language tool. I have an awk script that checks the text for mistakes I have made in the past; this rarely matches, i.e., I seem to be forever making different mistakes.

The sequencing of tools is: R (Sweave) -> Asciidoc -> docbook -> LaTeX -> pdf; assorted scripts fiddle with the text between outputs and inputs. The scripts and files mention below are available for download.

R generates pdf files (via calls to the Sweave function, I have never gotten around to investigating Knitr; the pdfs are cropped using scripts/pdfcrop.sh) and the ascii package is used to produce a few tables with Asciidoc markup.

Asciidoc is the markup language used for writing the text. A few years after I started writing the book, Stuart Rackham, the creator of Asciidoc, decided to move on from working and supporting it. Unfortunately nobody stepped forward to take over the project; not a problem, Asciidoc just works (somebody did step forward to reimplement the functionality in Ruby; Asciidoctor has an active community, but there is no incentive for me to change). In my case, the output from Asciidoc is xml (it supports a variety of formats).

Docbook appears in the sequence because Asciidoc uses it to produce LaTeX. Docbook takes xml as input, and generates LaTeX as output. Back in the day, Docbook was hailed as the solution to all our publishing needs, and wonderful tools were going to be created to enable people to produce great looking documents.

LaTeX is the obvious tool for anybody wanting to produce lovely looking books and articles; tex/ESEUR.tex is the top-level LaTeX, which includes the generated text. Yes, LaTeX is a markup language, and I could have written the text using it. As a language I find LaTeX too low level. My requirements are not complicated, and I find it easier to write using a markup language like Asciidoc.

The input to Asciidoc and LuaTeX (used to generate pdf from LaTeX) is preprocessed by scripts (written using sed and awk; see scripts/mkpdf). These scripts implement functionality that Asciidoc does not support (or at least I could see how to do it without modifying the Python source). Scripts are a simple way of providing the extra functionality, that does not require me to remember details about the internals of Asciidoc. If Asciidoc was being actively maintained, I would probably have worked to get some of the functionality integrated into a future release.

There are a few techniques for keeping text processing scripts simple. For instance, the cost of a pass over text is tiny, there is little to be gained by trying to do everything in one pass; handling the possibility that markup spans multiple lines can be complicated, a simple solution is to join consecutive lines together if there is a possibility that markup spans these lines (i.e., the actual matching and conversion no longer has to worry about line breaks).

Many simple features are implemented by a script modifying Asciidoc text to include some ‘magic’ sequence of characters, which is subsequently matched and converted in the generated LaTeX, e.g., special characters, and hyperlinks in the pdf.

A more complicated example handles my desire to specify that a figure appear in the margin; the LaTeX sidenotes package supports figures in margins, but Asciidoc has no way of specifying this behavior. The solution was to add the word “Margin”, to the appropriate figure caption option (in the original Asciidoc text, e.g., [caption="Margin ", label=CSD-95-887]), and have a script modify the LaTeX generated by docbook so that figures containing “Margin” in the caption invoked the appropriate macro from the sidenotes package.

There are still formatting issues waiting to be solved. For instance, some tables are narrow enough to fit in the margin, but I have not found a way of embedding this information in the table information that survives through to the generated LaTeX.

My long time pet hate is the formatting used by R’s plot function for exponentiated values as axis labels. My target audience are likely to be casual users of R, so I am sticking with basic plotting (i.e., no calls to ggplot). I do wish the core R team would integrate the code from the magicaxis package, to bring the printing of axis values into the era of laser printers and bit-mapped displays.

Ideas and suggestions welcome.

Ponylang (SeanTAllen)

Last Week in Pony - August 11, 2019 August 11, 2019 01:31 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

August 10, 2019

Jan van den Berg (j11g)

Generation X – Douglas Coupland August 10, 2019 09:04 AM

I am a Douglas Coupland fan. And I think his debut Generation X still holds up as one of his best novels. I probably read it for the first time over ten years ago. And I have since then read several other Coupland novels. (I also reviewed jPod extensively in 2007 on my Dutch blog). So I am quite familiar with his unique style, which is a large part of the attraction. However rereading this book was quite the revelation.

Generation X – Douglas Coupland (1991) – 253 pages

First, to my own shock, I barely seemed to remember the main story (sure, three friends in a desert town, but that was about it). This might be fuel for an entire blogpost on this subject (“What good is reading when you forget? Does this depend on the story or author? etc.”).

But second, when I was reading I had to double check when this book was written — yes, really 1991! The story also takes place around that time. Sure, there a few outdated references, but mainly it struck me how relevant, fresh and on-point Coupland already was in describing and predicting modern society (our society) and our relationships — with each other or technology — which is a main Coupland theme throughout all of his work. Remember this book was written pre-www (pre-grunge!). So it was definitely a different time. But the characters and their stories hold up well, because it is mainly about human interaction. And the struggle these Generation X characters face in their search for meaning, might be even more relevant today.

So rereading this book left me with even more respect for Coupland as an astute and perceptive writer.

The post Generation X – Douglas Coupland appeared first on Jan van den Berg.

The Trumpet of Conscience – Dr. Martin Luther King Jr. August 10, 2019 08:55 AM

Martin Luther King Jr. was only 39 years (and 2 months and 19 days) old when he was murdered. Thirty-nine. I never realised this — until I am 39 myself now.

When he died he had already received a Nobel prize and over 100 honorary degrees from all over the world, but more importantly, he had changed America forever.

The Trumpet of Conscience – Dr. Martin Luther King Jr. (1968) – 93 pages

Much has been written about MLK and by MLK. And it is especially the latter I am interested in.

This small book is a collection of (Christmas) lectures he wrote and presented on Canadian radio in 1967 — the so-called famous Massey Lectures. This collection itself was published less than a month after he was murdered (which makes reading the foreword by his wife and widow quite chilling to read).

The Dutch translation is a bit rough and outdated. But if you ever heard MLK speak there is very little imagination necessary to hear the man’s voice. It is so powerful that it cuts through time and languages.

The themes he discusses are familiar, but what is most prominent is the very, very long journey that he still saw ahead and that he was not so sure we would ever get there. These lectures help in understanding what MLK saw as the ultimate goal and also pinpoint the root of the problem. A goal which is sadly enough still utterly relevant. And a root cause that is still around.

You don’t have to read the book, here is a YouTube playlist I created with all five speeches where you hear the man himself.

The post The Trumpet of Conscience – Dr. Martin Luther King Jr. appeared first on Jan van den Berg.

Andrew Montalenti (amontalenti)

JavaScript: The Modern Parts August 10, 2019 07:00 AM

In the last few months, I have learned a lot about modern JavaScript and CSS development with a local toolchain powered by Node 8, Webpack 4, and Babel 7. As part of that, I am doing my second “re-introduction to JavaScript”. I first learned JS in 1998. Then relearned it from scratch in 2008, in the era of “The Good Parts”, Firebug, jQuery, IE6-compatibility, and eventually the then-fledgling Node ecosystem. In that era, I wrote one of the most widely deployed pieces of JavaScript on the web, and maintained a system powered by it.

Now I am re-learning it in the era of ECMAScript (ES6 / ES2017), transpilation, formal support for libraries and modularization, and, mobile web performance with things like PWAs, code splitting, and WebWorkers / ServiceWorkers. I am also pleasantly surprised that JS, via the ECMAScript standard and Babel, has evolved into a pretty good programming language, all things considered.

To solidify all this stuff, I am using webpack/babel to build all static assets for a simple Python/Flask web app, which ends up deployed as a multi-hundred-page static site.

One weekend, I ported everything from Flask-Assets to webpack, and to play around with ES2017 features, as well as explore the Sass CSS preprocessor and some D3.js examples. And boy, did that send me down a yak shaving rabbit hole. Let’s start from the beginning!

JavaScript in 1998

I first learned JavaScript in 1998. It’s hard to believe that this was 20 years — two decades! — ago. This post will chart the two decades since — covering JavaScript in 1998, 2008, and 2018. The focus of the article will be on “modern” JavaScript, as of my understanding in 2018/2019, and, in particular, what a non-JavaScript programmer should know about how the language — and its associated tooling and runtime — have dramatically evolved. If you’re the kind of programmer who thinks, “I code in Python/Java/Ruby/C/whatever, and thus I have no use for JavaScript and don’t need to know anything about it”, you’re wrong, and I’ll describe why. Incidentally, you were right in 1998, you could get by without it in 2008, and you are dead wrong in 2018.

Further, if you are the kind of programmer who thinks, “JavaScript is a tire fire I’d rather avoid because it lacks basic infrastructure we take for granted in ‘real’ programming languages”, then you are also wrong. I’ll be able to show you how “not taking JavaScript seriously” is the 2018 equivalent of the skeptical 2008-era programmer not taking Python or Ruby seriously. JavaScript is a language that is not only here to stay, but has already — and will continue to — take over the world in several important areas. To be a serious programmer, you’ll have to know JavaScript’s Modern and Good Parts — as well as some other server-side language, like Python, Ruby, Go, Elixir, Clojure, Java, and so on. But, though you can swap one backend language for the other, you can’t avoid JavaScript: it’s pervasive in every kind of web deployment scenario. And, the developer tooling has fully caught up to your expectations.

JavaScript during The Browser Wars

Browsers were a harsh environment to target for development; not only was Internet adoption low and not only were internet connections slow, but the browser wars — mainly between Netscape and Microsoft — were creating a lot of confusion.

Netscape Navigator 4 was released in 1997, and Internet Explorer 5 was released in 1998. The web was still trying to make sense of HTML and CSS; after all, CSS1 had only been released a year earlier.

In this environment, the definitive web development book of the era was “JavaScript: The Definitive Guide”, which weighed in at over 500 pages. Note that, in 1998, the most widely used programming languages were C, C++, and Java, as well as Microsoft Visual Basic for Windows programmers. So expectations about “what programming was” were framed mostly around these languages.

In this sense, JavaScript was quite, quite different. There was no compiler. There was no debugger (at least, not very good ones). There was no way to “run a JavaScript program”, except to write scripts in your browser, and see if they ran. Development tools for JavaScript were still primitive or inexistent. There was certainly not much of an open source community around JS; to figure out how to do things, you would typically “view source” on other people’s websites. Plus, much of the discussion in the programming community of web developers was how JavaScript represented a compatibility and security nightmare.

Not only differing implementations across browsers, but also many ways for you to compromise the security of your web application by relying upon JavaScript too directly. A common security bug in that era was to validate forms with JavaScript, but still allow invalid (and insecure) values to be passed to the server. Or, to password-protect a system, but in a way that inspection of JavaScript code could itself crack access to that system. Combined with the lack of a proper development environment, the “real web programmers” used JavaScript as nothing more than a last resort — a way to inject a little bit of client-side code and logic into pages where doing it server-side made no sense. I remember one of the most common use cases for JavaScript at the time was nothing more than changing an image upon hover, as a stylistic effect, or implementing a basic on-hover menu on a complex multi-tab form. These days, these tasks can be achieved with vanilla CSS, but, at the time, JavaScript DOM manipulation was your only option.

JavaScript in 2008

Fast forward 10 years. In 2008, Douglas Crockford released the book, “JavaScript: The Good Parts”. By using a language subsetting approach, Crockford pointed out that, not only was JavaScript not a bad language, it was actually a good language, well-designed, with certain key features that made it stand out vs competitors.

Around this time, several JavaScript libraries were becoming popular, notably jQuery, Prototype, YUI, and Dojo. These libraries attempted to provide JavaScript with something it was missing: a cross-browser compatibility layer and programming model for doing dynamic manipulation of pages inside the browser, and especially for a new model of JavaScript programming that was emerging, with the moniker AJAX. This was the beginning of the trend of rich internet applications, “dynamic” web apps, single-page applications, and the like.

JavaScript’s Tooling Leaps

The developer tooling for JavaScript also took some important leaps. In 2006, the Firefox team released Firebug, a JavaScript and DOM debugger for Firefox, which was then one of the world’s most popular web browsers, and open source. Two years later, Google would make the first release of Google Chrome, which bundled some developer tooling. Around the same time that Chrome was released, Google also released V8, the JavaScript engine that was embedded inside of Chrome. That marked the first time that the world had seen a full-fledged, performant open source implementation of the JavaScript language that was not completely tied to a browser. Firefox’s JS engine, SpiderMonkey, was part of its source tree, but was not necessarily marketed to be modularized and used outside the context of the Firefox browser.

I remember that aside from Crockford’s work on identifying the good parts of JavaScript, and aside from the new (and better) developer tooling, a specific essay on Mozilla’s website helped me re-appreciate the language, and throw away my 1998 conception. That article was called “A Reintroduction to JavaScript”. It showed how JavaScript was actually a real programming language, once you got past the tooling bumps. A little under-powered in its standard library, thus you had to rely upon frameworks (like jQuery) to give you some tools, and little micro-libraries beyond that.

A year after reading that essay, I wrote my own about JavaScript, which was called “Real, Functional Programs with JavaScript” (archived PDF here). It described how JavaScript was, quite surprisingly, more of a functional language than Java 8 or Python 2.7. And that with a little focus on understanding the functional core, really good programs could be written. I recently converted this essay into a set of instructional slides with the name, “Lambda JavaScript” (archived notes here), which I now use to teach new designers/developers the language from first principles.

But, let’s return to history. Only a year after the release of Chrome, in 2009, we saw the first release of NodeJS, which took the V8 JavaScript engine and embedded it into a server-side environment, which could be used to experiment with JavaScript on a REPL, to write scripts, and even to write HTTP servers on a performant event loop.

People began to experiment with command-line tools written in JavaScript, and with web frameworks written in JavaScript. It was at this point that the pace of development in the JavaScript community accelerated. In 2010, npm — the Node Package Manager — was released, and it and its package registry quickly grew to represent the full JavaScript open source community. Over the next few years, the browser vendors of Mozilla, Google, Apple, and Microsoft engaged in the “JavaScript Engine Wars”, with each developing SpiderMonkey, V8, Nitro, and Chakra to new heights.

Meanwhile, NodeJS and V8 became the “standard” JS engine running on developer’s machines from the command line. Though developers still had to target old “ECMAScript 3” browsers (such as IE6), and thus had to write restrained JavaScript code, the “evergreen” (auto-updating) browsers from Mozilla, Google, and Apple gained support for ECMAScript 5 and beyond, and mobile web browsing went into ascendancy, thus making Chrome and Safari dominant in market share especially on smartphones.

I remember in 2012, I gave a presentation at a local tech conference entitled, “Writing Real Programs… with JavaScript!?”. The “!?” punctuation was intentional. That was the general zeitgeist I remember in a room full of developers: that is, “is writing real programs with JavaScript… actually possible!?” It’s funny to review those slides as a historical relic. I spent the first half of the talk convincing the audience that JavaScript’s functional core was actually pretty good. And then I spent the second half convincing them that NodeJS might… it just might… create a developer tooling ecosystem and standard library for JavaScript. There are also a few funny “detour” slides in there around things like Comet vs Ajax, a debate that didn’t really amount to much (but it’s good to remind one of fashion trends in tech).

Zooming ahead a few years, in all of this noise of web 2.0, cloud, and mobile, we finally reached “mobilegeddon” in 2015, where mobile traffic surpassed desktop traffic, and we also saw several desktop operating systems move to a mostly-evergreen model, such as Windows 10, Mac OS X, and ChromeOS. As a result, as early as 2015 — but certainly by 2018 — JavaScript became the most widely deployed and performant programming language with “built-in support” on almost every desktop and mobile computer in the world.

In other words, if you wanted your code to be “write once, run everywhere” in 2015 or so (but even as far back as 2009), your best option was JavaScript. Well, that’s even more true today. The solid choice for widespread distribution of your code continues to be JavaScript. As Crockford predicted in 2008: “It is better to be lucky than smart.”

JavaScript in 2018-2019

In 2018-2019, several things have changed about the JavaScript community. Development tools are no longer fledgling, but are, instead, mature. There are built-in development tools in all of Safari, Firefox, and Chrome browsers (and the Firebug project is mostly deprecated). There are also ways to debug mobile web browsers using mobile development tools. NodeJS and npm are mature projects that are shared infrastructure for the whole JavaScript community.

What’s more, JavaScript, as a language, has evolved. It’s no longer just the kernel language we knew in 1998, nor the “good parts” we knew in 2008, but instead the “modern parts” of JavaScript include several new language features that go by the name “ES6” (ECMAScript v6) or “ES2017” (ECMAScript 2017 Edition), and beyond.

Some concepts in HTML have evolved, such as HTML5 video and audio elements. CSS, too, has evolved, with the CSS2 and CSS3 specifications being ratified and widely adopted. JSON has all but entirely replaced XML as an interchange format and is, of course, JavaScript-based.

The V8 engine has also gotten a ton of performance-oriented development. It is now a JIT compiled language with speedy startup times and speedy near-native performance for CPU-bound blocks. Modern web performance techniques are almost entirely based on a speedy JavaScript engine and the ability to script different elements of a web application’s loading approach.

The language itself has become comfortable with something akin to “compiler” and “command line” toolchains you might find in Python, Ruby, C, and Java communities. In lieu of a JavaScript “compiler”, we have node, JavaScript unit testing frameworks like Mocha/Jest, as well as eslint and babel for syntax checking. (More on this later.)

In lieu of a “debugger”, we have the devtools built into our favorite browser, like Chrome or Firefox. This includes rich debuggers, REPLs/consoles, and visual inspection tools. Scriptable remote connections to a node environment or a browser process (via new tools like Puppeteer) further close the development loop.

To use JavaScript in 2018/2019, therefore, is to adopt a system that has achieved 2008-era maturity that you would see in programming ecosystems like Python, Ruby, and Java. But, in many ways, JavaScript has surpassed those communities. For example, where Python 3’s reference implementation, CPython, is certainly fast as far as dynamic languages go, JavaScript’s reference implementation, V8, is optimized by JIT and hotspot optimization techniques that are only found in much more mature programming communities, such as Java’s (which received millions of dollars of commercial support in applied/advanced compiler techniques in the Sun era). That means that unmodified, hotspot JavaScript code can be optimized into native code automatically by the Node runtime and by browsers such as Chrome.

Whereas Java and C users may still have debates about where, exactly, open source projects should publish their releases, that issue is settled in the JavaScript community: it’s npm, which operates similarly to PyPI and pip in the Python community.

Some essential developer tooling issues were only recently settled. For example, because modern JavaScript (such as code written using ES2017 features) needs to target older browsers, a “transpilation” toolchain is necessary, to compile ES2017 code into ES3 or ES5 JavaScript code, suitable for older browsers. Because “old JavaScript” is a Turing complete, functional programming language, we know we can translate almost any new “syntactic sugar” to the old language, and, indeed, the designers of the new language features are being careful to only introduce syntax that can be safely transpiled.

What this means, however, is that to do JavaScript development “The Modern Way”, while adopting its new features, you simply must use a local transpiler toolchain. The community standard for this at the moment is known as babel, and it’s likely to remain the community standard well into the future.

Another issue that plagued 2008-era JavaScript was build tooling and modularization. In the 2008-2012 era, ad-hoc tools like make were used to concatenate JavaScript modules together, and often Java-based tools such as Google’s Closure Compiler or UglifyJS were used to assemble JavaScript projects into modules that could be included onto pages. In 2012, the Grunt tool was released as a JavaScript build tool, written atop NodeJS, runnable from the command-line, and configurable using a JavaScript “Gruntfile”. A whole slew of build tools similar to this were released in the period, creating a whole lot of code churn and confusion.

Thankfully, today, Single Page Application frameworks like React have largely solved this problem, with the ascendancy of webpack and the reliance on npm run-script. Today, the webpack community has come up with a sane approach to JavaScript modularization that relies upon the modern JS support for modules, and then development-time tooling, provided mainly through the webpack CLI tool, allow for local development and production builds. This can all be scripted and wired together with simple npm run-script commands. And since webpack can be itself installed by npm, this keeps the entire development stack self-contained in a way that doesn’t feel too dissimilar from what you might enjoy with lein in Clojure or python/pip in Python.

Yes, it has taken 20 years, but JavaScript is now just as viable a choice for your backend and CLI tooling projects as Python was in the past. And, for web frontends, it’s your only choice. So, if you are a programmer who cares, at all, about distribution of your code to users, it’s time to care about JavaScript!

In a future post, I plan to go even deeper on JavaScript, covering:

  • How to structure your first “modern” JavaScript project
  • Using Modern JS with Python’s Flask web framework for simple static sites
  • Understanding webpack, and why it’s important
  • Modules, modules, modules. Why JS modules matter.
  • Understanding babel, and why it’s important
  • Transpilation, and how to think about evolving JS/ES features and “compilation”
  • Using eslint for bonus points
  • Using sourcemaps for debugging
  • Using console.assert and console for debugging
  • Production minification with uglify
  • Building automated tests with jest
  • Understanding the value of Chrome scripting and puppeteer

Want me to keep going? Let me know via @amontalenti on Twitter.

August 09, 2019

Derek Jones (derek-jones)

Growth and survival of gcc options and processor support August 09, 2019 02:43 PM

Like any actively maintained software, compilers get more complicated over time. Languages are still evolving, and options are added to control the support for features. New code optimizations are added, which don’t always work perfectly, and options are added to enable/disable them. New ways of combining object code and libraries are invented, and new options are added to allow the desired combination to be selected.

The web pages summarizing the options for gcc, for the 96 versions between 2.95.2 and 9.1 have a consistent format; which means they are easily scrapped. The following plot shows the number of options relating to various components of the compiler, for these versions (code+data):

Number of options supported by various components of gcc, over 20 years.

The total number of options grew from 632 to 2,477. The number of new optimizations, or at least the options supporting them, appears to be leveling off, but the number of new warnings continues to increase (ah, the good ol’ days, when -Wall covered everything).

The last phase of a compiler is code generation, and production compilers are generally structured to enable new processors to be supported by plugging in an appropriate code generator; since version 2.95.2, gcc has supported 80 code generators.

What can be added can be removed. The plot below shows the survival curve of gcc support for processors (80 supported cpus, with support for 20 removed up to release 9.1), and non-processor specific options (there have been 1,091 such options, with 214 removed up to release 9.1); the dotted lines are 95% confidence internals.

Survival curve of gcc options and support for specific processors, over 20 years.

Oliver Charles (ocharles)

Who Authorized These Ghosts!? August 09, 2019 12:00 AM

Recently at CircuitHub we’ve been making some changes to how we develop our APIs. We previously used Yesod with a custom router, but we’re currently exploring Servant for API modelling, in part due to it’s potential for code generation for other clients (e.g., our Elm frontend). Along the way, this is requiring us to rethink and reinvent previously established code, and one of those areas is authorization.

To recap, authorization is

the function of specifying access rights/privileges to resources related to information security and computer security in general and to access control in particular.

This is in contrast to authentication, which is the act of showing that someone is who they claim to be.

Authorization is a very important process, especially in a business like CircuitHub where we host many confidential projects. Accidentally exposing this data could be catastrophic to both our business and customers, so we take it very seriously.

Out of the box, Servant has experimental support for authorization, which is a good start. servant-server gives us Servant.Server.Experimental.Auth which makes it a doddle to plug in our existing authorization mechanism (cookies & Redis). But that only shows that we know who is asking for resources, how do we check that they are allowed to access the resources?

As a case study, I want to have a look at a particular end-point, /projects/:id/price. This endpoint calculates the pricing options CircuitHub can offer a project, and there are few important points to how this endpoint works:

  1. The pricing for a project depends on the user viewing it. This is because some users can consign parts so CircuitHub won’t order them. Naturally, this affects the price, so pricing is viewer dependent.
  2. Some projects are owned by organizations, and should be priced by the organization as a whole. If a user is a member of the organization that owns the project pricing has been requested for, return the pricing for the organization. If the user is not in the organization, return their own custom pricing.
  3. Private projects should only expose their pricing to superusers, the owner of the project, and any members of the project’s organization (if it’s owned by an organization).

This specification is messy and complicated, but that’s just reality doing it’s thing.

Our first approach was to try and represent this in Servant’s API type. We start with the “vanilla” route, with no authentication or authorization:

Next, we add authorization:

At this point, we’re on our own - Servant offers no authorization primitives (though there are discussions on this topic).

My first attempt to add authorization to this was:

There are two new routing combinators here: AuthorizeWith and CanView. The idea is AuthorizeWith somehow captures the result of authenticating, and provides that information to CanView. CanView itself does some kind of authorization using a type class based on its argument - here Capture "id" ProjectId. The result is certainly something that worked, but I was unhappy with both the complexity to implement it (which is scope to get it wrong), and the lack of actual evidence of authorization.

The latter point needs some expanding. What I mean by “lacking evidence” is that with the current approach, the authorization is essentially like writing the following code:

If I later add more resource access into doThings, what will hold me accountable to checking authorization on those resources? The answer is… nothing! This is similar to boolean blindless - we performed logical check, only to throw all the resulting evidence away immediately.

At this point I wanted to start exploring some different options. While playing around with ideas, I was reminded of the wonderful paper “Ghosts of Departed Proofs”, and it got me thinking… can we use these techniques for authorization?

Ghosts of Departed Proofs

The basic idea of GDP is to name values using higher-rank quantification, and then - in trusted modules - produce proofs that refer to these names. To name values, we introduce a Named type, and the higher-ranked function name to name things:

Note that the only way to construct a Named value outside of this module is to use name, which introduces a completely distinct name for a limited scope. Within this scope, we can construct proofs that refer to these names. As a basic example, we could use GDP to prove that a number is prime:

Here we have our first proof witness - IsPrime. We can witness whether or not a named Int is prime using checkPrime - like the boolean value isPrime this determines if a number is or isn’t prime, but we get evidence that we’ve checked a specific value for primality.

This is the whirlwind tour of GDP, I highly recommend reading the paper for a more thorough explanation. Also, the library justified-containers explores these ideas in the context of maps, where we have proofs that specific items are in the map (giving us total lookups, rather than partial lookups).

GDP and Authorization

This is all well and good, but how does this help with authorization? The basic idea is that authorization is itself a proof - a proof that we can view or interact with resources in a particular way. First, we have to decide which functions need authorization - these functions will be modified to require proof values the refer to the function arguments. In this example, we’ll assume our Servant handler is going to itself make a call to the price :: ProjectId -> UserId -> m Price function. However, given the specification above, we need to make sure that user and project are compatible. To do this, we’ll name the arguments, and then introduce a proof that the user in question can view the project:

But what is this CanViewProject proof?

A first approximation is to treat it as some kind of primitive or axiom. A blessed function can postulate this proof with no further evidence:

This is a good start! Our price function can only be called with a CanViewProject that matches the named arguments, and the only way to construct such a value is to use canViewProject. Of course we could get the implementation of this wrong, so we should focus our testing efforts to make sure it’s doing the right thing.

However, the Agda programmer in me is a little unhappy about just blindly postulating CanViewProject at the end. We’ve got a bit of vision back from our boolean blindness, but the landscape is still blurry. Fortunately, all we have to do is recruit more of the same machinery so far to subdivide this proof into smaller ones:

Armed with these smaller authorization primitives, we can build up our richer authorization scheme:

Now canViewProject just calls out to the other authorization routines to build it’s proof. Furthermore, there’s something interesting here. CanViewProject doesn’t postulate anything - everything is attached with a proof of the particular authorization case. This means that we can actually open up the whole CanViewProject module to the world - there’s no need to keep anything private. By doing this and allowing people to pattern match on CanViewProject, authorization results become reusable - if something else only cares that a user is a super user, we might be able to pull this directly out of CanViewProject - no need for any redundant database checks!

In fact, this very idea can help us implement the final part of our original specification:

Some projects are owned by organizations, and should be priced by the organization as a whole. If a user is a member of the organization that owns the project pricing has been requested for, return the pricing for the organization. If the user is not in the organization, return their own custom pricing.

If we refine our UserBelongsToProjectOrganization proof, we can actually maintain a bit of extra evidence:

Now whenever we have a proof UserBelongsToProjectOrganization, we can pluck out the actual organization that we’re talking about. We also have evidence that the organization owns the project, so we can easily construct a new CanViewProject proof - proofs generate more proofs!

Relationship to Servant

At the start of this post, I mentioned that the goal was to integrate this with Servant. So far, we’ve looked at adding authorization to a single function, so how does this interact with Servant? Fortunately, it requires very little to change. The Servant API type is authorization free, but does mention authentication.

It’s only when we need to call our price function do we need to have performed some authorization, and this happens in the server-side handler. We do this by naming the respective arguments, witnessing the authorization proof, and then calling price:

Conclusion

That’s where I’ve got so far. It’s early days so far, but the approach is promising. What I really like is there is almost a virtual slider between ease and rigour. It can be easy to get carried away, naming absolutely everything and trying to find the most fundamental proofs possible. I’ve found so far that it’s better to back off a little bit - are you really going to get some set membership checks wrong? Maybe. But a property check is probably gonig to be enough to keep that function in check. We’re not in a formal proof engine setting, pretending we are just makes things harder than they need to be.

August 08, 2019

Jan van den Berg (j11g)

A Supposedly Fun Thing I’ll Never Do Again – David Foster Wallace August 08, 2019 11:36 AM

David Foster Wallace can write. I mean, he can really write. In related news: water is wet. This man’s writing struck me as an epiphany, a beacon of light, a clear and unmistakable differentiator between merely good writing and exceptional writing.

A Supposedly Fun Thing I’ll Never Do Again – David Foster Wallace (1997) – 353 pages

I have known about DFW for some time now, and I have seen his famous commencement speech several times. It strongly resonates with me. As some other interviews do. But his writing? It seemed intimidating.

Infinite Jest, his magnum opus, is this famous thousand page multi-layered beast of a book. So I thought I start with something lighter. ‘A Supposedly Fun Thing…’ is a collection of essays and so it seemed like a good starting place.

It is a collection of 7 stories and essays on tennis, state fairs, TV, irony, David Lynch and a very entertaining cruise among other things. (Each story could validate a blogpost by itself — there is just so much there). Wallace demonstrates with academic skill his philosophical insights on modern life with the essays about other writers, TV and irony. But he is, just as easily, able to make you scream with laughter when he describes a highly anticipated and ultimately disappointing experience with the dessert tasting booth at the state fair. This man could seemingly do anything with a pen.

The words, and sentences (and footnotes!) all just seem to ooze effortlessly out of him. His voice is radically clear and distinct and his vocabulary and attention to detail are unmatched. It is very obvious Wallace operated on a different level, intellectually and talent wise. And I often stopped reading and wondered about how his depression got the best of him in the end, and whether this much talent and severe depression are somehow two sides of the same coin. Because judging by his writing, I don’t think he experienced the world the same way most people do (whatever that is).

The first thing I did after finishing this book, was head to a bookshop where I bought Infinite Jest. It still looks intimidating, but I can now only assume it must be a definitely fun thing to read.

The post A Supposedly Fun Thing I’ll Never Do Again – David Foster Wallace appeared first on Jan van den Berg.

August 07, 2019

Gonçalo Valério (dethos)

Staying on an AirBnB? Look for the cameras August 07, 2019 05:04 PM

When going on a trip it is now common practice to consider staying on an rented apartment or house instead of an hotel or hostel, mostly thanks to AirBnB which made it really easy and convenient for both side of the deal. Most of the time the price is super competitive and I would say a great fit for many situations.

However as it happens with almost anything, it has its own set of problems and challenges. One example of these new challenges are the reports (and confirmations) that some, lets call them malicious hosts, have been putting in place cameras to monitor the guests during their stay.

With a small search on the internet you can find

Someone equipped with the right knowledge and a computer can try to check if a camera is connected to the WiFi network, like this person did:

Toot describing that a camera that was hidden inside a box

If this is your case, the following post provides a few guidelines to look for the cameras:

Finally, try to figure out the public IP address of the network you are on ( https://dshield.org/api/myip ) and either run a port scan from the outside to see if you find any odd open ports, or look it up in Shodan to see if Shodan found cameras on this IP in the past (but you likely will have a dynamic IP address).

InfoSec Handlers Diary Blog

This page even provides a script that you can execute to automatically do most steps explained on the above article.

However, sometimes you don’t bring your computer with you, which means you would have to rely on your smartphone to do this search. I’m still trying to find a good, trustworthy and intuitive app to recommend, since using nmap on Android will not help the less tech-savvy people.

Meanwhile, I hope the above links provide you with some ideas and useful tools to look for hidden cameras while you stay on a rented place.

August 05, 2019

eta (eta)

Presuming mental health issues considered harmful August 05, 2019 11:00 PM

This post isn’t at all technical, unlike the other ones. Here be dragons! Feel free to go and read something more sane if posts about non-technical interpersonal relations aren’t your cup of tea.

It’s generally acknowledged that talking to other people is a hard problem. Generally, though, in order to make it less of one, we have this amazing thing called ‘courtesy’ that gets wheeled out to deal with it, and make everything alright again. The point of being polite and courteous toward people, really, is to acknowledge that not everyone sees things the way you do, and so you need to account for that by giving everyone a bit of leeway. In a way, it shows respect for the other person’s way of thinking – being polite allows you to consider that your opinion may not be better than theirs. And, by and large, it works (except for the examples in that Wait But Why post there, but oh well).

I don’t want to talk about the situations where it works, though, because those are largely uninteresting! Where things get interesting (read: harmful) is when people, for one reason or another, decide that this respect for others is just not something they want to bother with.

Criticism

Of course, you don’t always agree with what other people are saying or doing; in fact, you might believe that, according to the way you see things, they’re being downright horrible! When this happens, there are usually a number of options available to you:

  1. Bring the issue up with the person (i.e. directly give them some constructive criticism)
  2. Tell your friends about the situation, and see what they think
  3. Start bad-mouthing the person when they’re not around

Option (1) might seem like the ‘best’ thing to do, but often it’s not; delivering criticism to people only works if they’re actually willing to receive it. Otherwise, all they hear is someone who thinks they know how to run their life better than them, which they perceive is disrespectful and somewhat rude – exactly the opposite of what you might have intended. Even if they don’t mind criticism, they may disagree with it and refuse to act on it, without getting angry at the fact that you delivered it, which is also fair enough.

Nonviolent Communication

When delivering criticism in this way, it’s also important to emphasise how your opinion is highly subjective; this is kind of what Nonviolent Communication (NVC) is trying to get at. Instead of saying things like:

You were very hurtful when you did X.

(which can come across as absolutism - you were very hurtful, according to some globally defined definition of ‘hurtful’), NVC advocates for statements along the lines of:

When you did X, I felt very hurt by that.

Here, this can’t possibly be disagreed with, as you’re only stating your subjective experience of what happened, instead of making a seemingly objective judgement - you’re not saying the person is bad, or that they necessarily “did anything wrong”; rather, you’re giving them feedback on what effect their actions have had, which they can use to be a better person in future. (Or not, if they don’t think your experience is worth caring about.)

Phoning a friend

Because option (1) is such a minefield sometimes – you have to try and phrase things the right way, avoid getting angry or upset yourself, and even then the person might hate you a bit for the criticism – most people opt for option (2) in everyday life, which is to discuss the problem with some friends.

This is distinct from option (3), in which you actively start spouting abuse about how terrible the person is; in (2), you’re trying to neutrally share your view with only a few friends of yours, with the expectation that they may well turn around and tell you that you’re actually the person at fault here. That is to say, the idea of (2) is not to presume that you’re in the right automatically, and for the people you tell to be blind yes-men who go along with it (!). Rather, the point is to get some crowd-sourced feedback on the problem at hand, in order to give you some perspective. It may even then lead to (1), or maybe one of your friends letting this person know, or whatever – the point is, option (2) is usually a pretty good solution, which is why it’s probably quite popular.

Giving up, or worse

I don’t need to say something like “option (3) [i.e. bad-mouthing other people] is bad”, because you probably knew that already. It’s also not as if every person in the world is a paragon of virtue who would never do anything so shocking, either; people share their negative judgements on other people quite widely all the time, and that’s just part of life. Obviously, it has interesting implications for your relationship with the person you’re bad-mouthing, but that’s your own problem – and is also something you’re unlikely to care about that much, or else you wouldn’t spread your opinion.

No, what I’m more interested in discussing is where this option turns more toxic than it usually is, which is usually helped by the context present in certain institutions nowadays.

Mental health, in 2019

In universities, schools, and other such educational institutions, a rather large focus has recently been placed on the idea of dealing with students’ mental health, for one reason or another. Reports like this 2018 NUS blog post about how Oxbridge may be ‘in decline’, due to how taxing going there can be for your mental health, abound; in many places, safeguarding children is the new buzzword. I’m not wishing to in any way dismiss or trivialize this focus; it’s arguably a good thing, and, at any rate, that’d be for another day and another blog post.

Rather, I want to discuss a rather interesting side-effect of this focus, and how it might interact with the things we’ve just been discussing in the rest of the post.

A not-entirely-fictitious example

Suppose someone has a problem with you. You might not necessarily have much of an idea as to why, or you might do; it doesn’t really matter. Clearly, they think option (1) (telling you about it directly) is right out; you have no way of doing whether or not they did (2), but you decide you don’t care that much. “They have a problem with me, but I don’t think that’s really anything I need to worry about; rather, I think they’re as responsible for this situation as I am.” In a normal situation, this would be alright; everyone’s entitled to their own standpoint, after all, and people generally tend to respect one another.

However, this person really doesn’t like you. As part of that, they’re building their own internal list of things you’ve done that support this standpoint; everything you say, or do, that can be in some way interpreted as validating their view gets added, and their opinion of you gets worse and worse. Which would also be fine, as long as you didn’t actually need to rely on this person for anything important; they can sit over there with their extreme judgements, and it doesn’t really bother you at all: what are they going to do about it?

Well, but, remember the context I was talking about. In a lot of institutions, the management are constantly on the lookout for possible ‘mental health issues’, which isn’t a bad thing at all. However, this can give your fun new friend some extra leverage; all they need to do is find something that would seem to suggest you’re not 100% alright, or that you might be in need of some help. (Either that, or they cherry-pick examples of you supposedly being nasty, and claim that their mental health is being affected. For bonus points, one could even do both!)

Now, they can go and deliver a long spiel about how you’re clearly not right in the head, or something to that effect, and how it’s caused great problems for them. They don’t really have to worry that much about it being traced back to them, of course; usually in these kinds of systems people reporting problems are granted a degree of anonymity, in order to prevent people being put off from reporting anything at all.

And, if you haven’t been very carefully controlling everything you do or say to ensure none of it could possibly be used against you, you might now find yourself a bit stuck; how, after all, do you counter the allegations laid against you? You often don’t even know who raised them, and you’re lucky if you’re even told anything about what they actually are; rather, you now have to contend with the completely fabricated idea that there’s something wrong with you, which tends to be awfully sticky and hard to get rid of once people get wind of it.

Even though that’s not at all true, and never was.

Andrew Montalenti (amontalenti)

Parse.ly’s brand refresh August 05, 2019 04:59 PM

Here’s how Parse.ly’s original 2009 logo looked:

Parse.ly has some fun startup lore from its early days about how we “acquired” this logo. I wrote about this in a post entitled, “Parse.ly: brand hacking”:

Our first Parse.ly logo was designed as a trade for another domain I happened to own. It was the dormant domain for a film project for one of my friends, Josh Bernhard. I had registered it for him while we were both in college. […] It so happened that my friend had picked the name “Max Spector” for his film, and thus registered maxspector.com. The film never came to fruition, so the domain just gathered dust for awhile. But, Max Spector happened to be the name of a prominent San Francisco designer. And Max got in touch with me about buying the domain for his personal website. Acting opportunistically, I offered it in trade for a logo for Parse.ly. To my surprise, he agreed.

I still look back at the logo fondly, though, it being nearly a decade old at this point, it obviously has that dated “web 2.0 startup” feel.

A stroll down Memory Lane…

Every era of tech startups has design trends that influence it.

In 2009, I think our typeface was somehow influenced by two logos, WordPress and Twitter. That kind of makes sense, given how big these were in our community of tech/media at the time. The WordPress logo uses a smallcaps and serifs-heavy typeface to this day:

As for those heavy gradients, shadows, and emboss effects we had with the Parse.ly green colorings, I think that might have come from Twitter; see, for example, this edition of their 2009 logo:

Our simplified revised logo

The first revision to that original logo came in 2012, when we decided to simplify the logo coinciding with the commercial launch of Parse.ly Analytics. That resulted in us simplifying from a complex 7-leaf icon to a much simpler 3-leaf one. It also had us remove “circa 2009 logo flourishes” like typeface coloring with gradient and shadow.

We preserved the custom typeface that Max had designed for us, and coupled it with the simpler icon.

We still had a touch of green gradient in here, but only in the icon itself.

Further refinement

When we hired our head of design, who still works at Parse.ly to this day, we further simplified the logo by removing all gradient effects from the icon, and choosing a friendlier green color which became part of our standard style and palette.

That served us well for several more years. I think this also matched the evolving maturity of tech company brands.

Redesigning Parse.ly’s logotype

This year, Parse.ly is celebrating another year of sustainable growth atop customer revenue. The world of 2019 is very different than the world of 2009, when our logo was originally designed.

In particular, the extreme serifs, small caps style, and washed-out black coloring of our original logotype lacks confidence and affability, despite our brand becoming, over the years, quite confident and affable.

As a result, we’ve decided to give our logo and brand a refresh.

Here it is:

And here is a little style guide snippet to show it in various forms and colors:

What I love about this evolution is that it represents a big change: we have completely thrown away our old logotype. The new type is more open, more friendly, and more confident. The parts I particularly like include the open “P” and “e”, the tight kerning, and the distinctive dot. The color of the logotype is confident and matches the Parse.ly green well.

The icon looks the same, but it, too, evolved slightly.

Click/watch the video below to see the change between the old and new logo:

See if you can spot the difference.

Evolving the Parse.ly icon

For the icon, the changes were subtle and all about continuous refinement.

Specifically: we made the little “leaf slits” a bit more open, so that the icon appears more textured when scaled down to low resolution (e.g. app icons, site favicons, and so on). We also changed the proportion of the icon slightly to better balance with the new type, and tightened the location of the dot.

Those changes are so subtle, you need to blink to notice them. I’ve helped you out by making this video: click/watch this one to see the subtle change to the leaf icon:

I’m really pleased with these changes and I want to thank Sven Kils, the graphic designer in Frankfurt, Germany, who helped us with the brand refresh.

Like all other things at Parse.ly, our brand is a process of perpetual simplification.

This year was special for me because Parse.ly crossed over several important company milestones. This included a 10-year company anniversary (how on earth did we make it this long and this far?), a new office for our staff in NYC (we’re in Chelsea now, instead of Midtown East); and, a new bigger and more ambitious go-to-market strategy, coinciding with the hiring of our excellent Chief Revenue Officer, Nick.

This also coincided with my own personal return to New York City, which is where Parse.ly began before my relocation to Charlottesville, VA for eight years. I’ve come home to where it all began, and just as it’s starting to get really good.

We also have an upcoming website redesign, market re-positioning, and supporting materials (such as explainer videos), which I’m quite excited about.

I’ll give a little preview of our new market positioning here:

We believe the most successful companies in the digital world are the ones with the best content.

At Parse.ly, we built the world’s most innovative content analytics system, which drove the growth of the web’s biggest sites. We took the expertise and data from that success, and we’ve made it available to every company.

Today, every company is a content company. Every marketer is a content marketer. You need to create and measure content like the pros in order to win an audience.

Imagine you had the audience scale of The Wall Street Journal. Or, the visitor loyalty of the NFL. Or, the branding of Bloomberg. What would that do for your business? These companies used Parse.ly to build a content strategy that creates loyalty, engages visitors, and converts them into revenue.

Leading brands like HelloFresh, TheKnot, PolicyGenius, and Harvard University have also recognized the power of content analytics to transform their marketing efforts, by growing their share of traffic and attention.

We have entered the era of data-informed content. And in this era, the web’s best publishers and brands rely on Parse.ly data to grow a loyal and engaged audience for their content. Join them.

I hope we get to start using this logo more widely in our materials as we pursue a web/collateral update through the next couple of months. Here’s to the next 10 years of continuous improvement!

Pete Corey (petecorey)

Embedding React Components in Jekyll Posts August 05, 2019 12:00 AM

Last week I published an article on “clipping convex hulls”. The article included several randomly generated examples. Every time you refresh the article, you’re given a new set examples based on a new set of randomly generated points.

I thought we could dive into how I used React to generate each of those examples, and how I embedded those React components into a Jekyll-generated static page.

Creating Our Examples

To show off the process, let’s create three React components and embed them into this very article (things are about to get meta). We’ll start by creating a new React project using create-react-app:

create-react-app examples
cd examples
yarn start

This examples project will hold all three examples that we’ll eventually embed into our Jekyll-generated article.

The first thing we’ll want to do in our examples project is to edit public/index.html and replace the root div with three new divs, one, two, and three: one to hold each of our examples:


<body>
  <noscript>You need to enable JavaScript to run this app.</noscript>
  <div id="one"></div>
  <div id="two"></div>
  <div id="three"></div>
</body>

Next, we’ll need to instruct React to render something into each of these divs. We can do that by editing our src/index.js file and replacing its contents with this:


import React from 'react';
import ReactDOM from 'react-dom';
import "./index.css";

ReactDOM.render(<One />, document.getElementById('one'));
ReactDOM.render(<Two />, document.getElementById('two'));
ReactDOM.render(<Three />, document.getElementById('three'));

We’ll need to define our One, Two, and Three components. Let’s do that just below our imports. We’ll simply render different colored divs for each of our three examples:


const One = () => (
  <div style=&lbrace{ height: "100%", backgroundColor: "#D7B49E" }} />
);

const Two = () => (
  <div style=&lbrace{ height: "100%", backgroundColor: "#DC602E" }} />
);

const Three = () => (
  <div style=&lbrace{ height: "100%", backgroundColor: "#BC412B" }} />
);

We’re currently telling each of our example components to fill one hundred percent of their parents’ height. Unfortunately, without any additional information, these heights will default to zero. Let’s update our index.css file to set some working heights for each of our example divs:


html, body {
  height: 100%;
  margin: 0;
}

#one, #two, #three {
  height: 33.33%;
}

If we run our examples React application, we’ll see each of our colored example divs fill approximately one third of the vertical height of the browser.

Embedding Our Examples

So now that we’ve generated our example React components, how do we embed them into our Jekyll post?

Before we start embedding our React examples into a Jekyll post, we need a Jekyll post to embed them into. Let’s start by creating a new post in the _posts folder of our Jekyll blog, called 2019-08-05-embedding-react-components-in-jekyll-posts.markdown:


---
layout: post
title:  "Embedding React Components in Jekyll Posts"
---

Last week I published an article on "clipping convex hulls"...

Great. Now that we have a post, we need to decide where our examples will go. Within our post, we need to insert three divs with identifiers of one, two, three, to match the divs in our examples React project.

Let’s place one here:

Another here:

And the third just below the following code block that shows off how these divs would appear in our post:


...

Let's place one here:

<div id="one" style="height: 4em;"></div>

Another here:

<div id="two" style="height: 4em;"></div>

...

<div id="three" style="float: right; height: 4em; margin: 0 0 0 1em; width: 4em;"></div>

...

Notice that we’re explicitly setting the heights of our example components, just like we did in our React project. This time, however, we’re setting their heights to 4em, rather than one third of their parents’ height. Also notice that the third example is floated right with a width of 4em. Because our React components conform to their parents properties, we’re free to size and position them however we want within our Jekyll post.

At this point, the divs in our post are empty placeholders. We need to embed our React components into the post in order for them to be filled.

Let’s go into our examples React project and build the project:


yarn build

This creates a build folder within our examples project. The build folder contains the compiled, minified, standalone version of our React application that we’re free to deploy as a static application anywhere on the web.

You’ll notice that the build folder contains everything needed to deploy our application, including an index.html file, a favicon.ico, our bundled CSS, and more. Our Jekyll project already provides all of this groundwork, so we’ll only be needing a small portion of our new build bundle.

Specifically, we want the Javascript files dumped into build/static/js. We’ll copy the contents of build/static/js from our examples React project into a folder called js/2019-08-05-embedding-react-components-in-jekyll-posts in our Jekyll project.

We should now have three Javascript files and three source map files being served statically by our Jekyll project. The final piece of the embedding puzzle is to include the three scripts at the bottom of our Jekyll post:


<script src="/js/2019-08-05-embedding-react-components-in-jekyll-posts/runtime~main.a8a9905a.js"></script>
<script src="/js/2019-08-05-embedding-react-components-in-jekyll-posts/2.b8c4cbcf.chunk.js"></script>
<script src="/js/2019-08-05-embedding-react-components-in-jekyll-posts/main.2d35bbcc.chunk.js"></script>

Including these scripts executes our React application, which looks for each of our example divs, one, two, and three on the current page and renders our example components into them. Now, when we view our post, we’ll see each of our example components rendered in their full, colorful glory!

August 04, 2019

Ponylang (SeanTAllen)

Last Week in Pony - August 4, 2019 August 04, 2019 08:42 AM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

August 03, 2019

Alex Wilson (mrwilson)

Notes from the Week #28 August 03, 2019 12:00 AM

Notes from the Week #28

Here’s a non-exhaustive summary of what happened this week.

One

The work I mentioned last week to run our smoke-tests from a different data-centre is effectively complete.

We’re running the new system in parallel with the existing system of checks for a bit to ensure that they behave the same before switching off the existing system.

I’m consistently impressed by the thought that’s clearly gone into Concourse CI’s domain concepts and API.

Two

I’ve almost finished the write up for the “Mental Health in Software Development” session that I ran last month.

It turns out that writing up an hour and a half of post-its into a structured blogpost takes a while, but I’m pleased with how it’s coming together.

I’ve had a couple of realisations during the reflection on the session, especially around intersectionality with regards to mental health and other kinds of diversity, but more on that in the post.

Three

Wednesday was a bit of a rough day — for whatever reason, I was off my game and wasn’t as well equipped as I would have liked to contribute during a 2hour mapping session.

I was annoyed at myself about this for a while, but going for a brief walk helped clear my head. In future I need to be better about just stepping out of the room if I need some air, much as I would be okay with it if someone else asked to.

Four

I have to sign some of my commits at work to verify that they’ve come from me. It’s been a while since I’ve needed to do this, and it’s got me thinking about a problem I ran into while pairing/mobbing.

git supports signing commits and tags, but with a single key. How can you verify (a) that a single commit was paired/mobbed on and (b) the people whose names are on the commit were actually involved?

I’m playing with a rough idea of “primary committer signs the commit, secondary committers create and sign unique tags”, with something like a pre-receive-hook on the git server, which has a list of allowed public keys and rejects pushes where there are <2 verified authors.

The disadvantage of this is that tags are deliberately lightweight and easy to create, and so just as easy to delete. I suppose that this could be enforced on the server-side too (forbid deletions of tags).

The tag itself would have to have a unique id, something that uses the short-hash of the commit like a8b7c6-co-author-1. If tags were being used for release management, this might "pollute" your tag history to an unacceptable extent.

However, these might be acceptable trade-offs for being able to verify that everyone who claims to have worked on the commit did actually do so (assuming decent protections of each person’s signing key).

Originally published at https://blog.probablyfine.co.uk on August 3, 2019.

July 30, 2019

Gustaf Erikson (gerikson)

Fotografiska, July 2019 July 30, 2019 03:41 PM

Mandy Barker - Sea of Artifacts

An earnest exploration of plastic in seas and waterways. For some reason this exhibit was in a room even more dimly lit than usual at Fotografiska - and that’s saying something. Sadly nothing that’s not been said or done many times before.

Scarlett Hooft Graafland - Vanishing Traces

This grew on me. Staged images in different environments with surreal components (yet of course it’s all “analog” and there’s no manipulation). Impressive for the visuals and conceptions alone.

Vincent Peters - Light Within

Peters photographs our modern Hollywood idols like they where 1940s Hollywood idols.

James Nachtwey - Memoria

A visual onslaught of images from famines, wars, catastrophes and field hospitals. Almost unbearable in its intensity. I became numb after just a few walls. It depresses me that future epigones will have no shortage of similar subjects, all without having to travel that far.

July 29, 2019

Derek Jones (derek-jones)

First language taught to undergraduates in the 1990s July 29, 2019 05:44 PM

The average new graduate is likely to do more programming during the first month of a software engineering job, than they did during a year as an undergraduate. Programming courses for undergraduates is really about filtering out those who cannot code.

Long, long ago, when I had some connection to undergraduate hiring, around 70-80% of those interviewed for a programming job could not write a simple 10-20 line program; I’m told that this is still true today. Fluency in any language (computer or human) takes practice, and the typical undergraduate gets very little practice (there is no reason why they should, there are lots of activities on offer to students and programming fluency is not needed to get a degree).

There is lots of academic discussion around which language students should learn first, and what languages they should be exposed to. I have always been baffled by the idea that there was much to be gained by spending time teaching students multiple languages, when most of them barely grasp the primary course language. When I was at school the idea behind the trendy new maths curriculum was to teach concepts, rather than rote learning (such as algebra; yes, rote learning of the rules of algebra); the concept of number-base was considered to be a worthwhile concept and us kids were taught this concept by having the class convert values back and forth, such as base-10 numbers to base-5 (base-2 was rarely used in examples). Those of us who were good at maths instantly figured it out, while everybody else was completely confused (including some teachers).

My view is that there is no major teaching/learning impact on the choice of first language; it is all about academic fashion and marketing to students. Those who have the ability to program will just pick it up, and everybody else will flounder and do their best to stay away from it.

Richard Reid was interested in knowing which languages were being used to teach introductory programming to computer science and information systems majors. Starting in 1992, he contacted universities roughly twice a year, asking about the language(s) used to teach introductory programming. The Reid list (as it became known), was regularly updated until Reid retired in 1999 (the average number of universities included in the list was over 400); one of Reid’s ex-students, Frances VanScoy, took over until 2006.

The plot below is from 1992 to 2002, and shows languages in the top with more than 3% usage in any year (code+data):

Normalised returned required for various elapsed years.

Looking at the list again reminded me how widespread Pascal was as a teaching language. Modula-2 was the language that Niklaus Wirth designed as the successor of Pascal, and Ada was intended to be the grown up Pascal.

While there is plenty of discussion about which language to teach first, doing this teaching is a low status activity (there is more fun to be had with the material taught to the final year students). One consequence is lack of any real incentive for spending time changing the course (e.g., using a new language). The Open University continued teaching Pascal for years, because material had been printed and had to be used up.

C++ took a while to take-off because of its association with C (which was very out of fashion in academia), and Java was still too new to risk exposing to impressionable first-years.

A count of the total number of languages listed, between 1992 and 2002, contains a few that might not be familiar to readers.

          Ada    Ada/Pascal          Beta          Blue             C 
         1087             1            10             3           667 
       C/Java      C/Scheme           C++    C++/Pascal        Eiffel 
            1             1           910             1            29 
      Fortran       Haskell     HyperTalk         ISETL       ISETL/C 
          133            12             2            30             1 
         Java  Java/Haskell       Miranda            ML       ML/Java 
          107             1            48            16             1 
     Modula-2      Modula-3        Oberon      Oberon-2     ObjPascal 
          727            24            26             7            22 
       Orwell        Pascal      Pascal/C        Prolog        Scheme 
           12          2269             1            12           752 
    Scheme/ML Scheme/Turing        Simula     Smalltalk           SML 
            1             1            14            33            88 
       Turing  Visual-Basic 
           71             3 

I had never heard of Orwell, a vanity language foisted on Oxford Mathematics and Computation students. It used to be common for someone in computing departments to foist their vanity language on students; it enabled them to claim the language was being used and stoked their ego. Is there some law that enables students to sue for damages?

The 1990s was still in the shadow of the 1980s fashion for functional programming (which came back into fashion a few years ago). Miranda was an attempt to commercialize a functional language compiler, with Haskell being an open source reaction.

I was surprised that Turing was so widely taught. More to do with the stature of where it came from (university of Toronto), than anything else.

Fortran was my first language, and is still widely used where high performance floating-point is required.

ISETL is a very interesting language from the 1960s that never really attracted much attention outside of New York. I suspect that Blue is BlueJ, a Java IDE targeting novices.

Jeremy Morgan (JeremyMorgan)

How to Nail Your Next Coding Interview July 29, 2019 04:45 PM

The room is silent except for the buzzing of the fluorescent lights. The judges across the table are staring at you, expressionless. Some have pen and paper, some don't. They're all staring at you. Your mouth is so dry it feels like you've been eating sawdust all day. You grab the marker and head for the whiteboard. One judge is staring at a laptop. It's time to show them a quicksort.

I've been on both sides of the coding interview. I've interviewed for jobs, sometimes I got the offer, sometimes I didn't. Sometimes I nailed the whiteboard tests and didn't get a call, and vice versa. I've interviewed probably hundreds of people in my career and I did my best to make candidates feel comfortable, but many managers won't. They will try to trip you up, make you choke. We can argue about the effectiveness of a whiteboard interview later, but they will happen.

Let's look at some tips to rock your next coding interview. You can set yourself up for success and nail it. Don't be mistaken, this isn't a set of "hacks", "tricks", or some kind of brain dump. Don't use sneaky tactics or tricks to crack your way into a position you aren't qualified for. They'll just fire you later. Follow this path and you will nail the interview because you'll be a better developer.

What They're Looking for

Here's what most of the interviewers will be looking for, in no particular order:

  • Problem Solving - How well can you solve problems, and more importantly: what is your process. Many tests will look for this. 

  • Coding Skills - You'll need to be ruthlessly good here. I've handed people laptops and told them to write something for me. You can tell how much actual coding people have done by watching them do it. 

  • Technical Knowledge - This is where the trivia questions come in, but your interviewer wants to know just how technical you are. 

  • Experience - This is where you talk about your past projects. Interviewers want to hear your war stories, and what you learned from them. 

  • Culture Fit - I can't help you much with this one. This is where they see how you'll fit into a team. There are ways you can improve in this area..

So what do you need to do to get prepared? Let's dive in. 

Stage 1: Laying the Foundation

So here's what you need to be doing way before any interview. Days, months or weeks before you need to build some core skills. 

Learn Computer Science Basics

You need to get the basics of Computer Science down. You don't have to be Donald Knuth here, but you need to know the theory, language, and idioms. This is the bare minimum for an interview. If an interviewer starts casually mentioning a binary tree in the interview you better know what they're talking about. 

  1. Review The Basics of Computer Science Tutorial - some of this may be pretty basic to you but it gives you a framework of things to pay attention to. Where do your strengths lie? If you find any areas where you are weak at, work to build your knowledge and skills in that area. 

  2. Review Teach Yourself Computer Science - Again we're talking basics foundational stuff that you may be lacking in. You need to know the basics and speak the language. 

  3. You can even Take a course on Computer Science 101 from Stanford, for free.

Learn Different Algorithms

Algorithms run the world and if you're a developer you'll need to know them. So how do you get good at algorithms? It's not black magic or a secret art. 

According to Geeks for Geeks these are the top 10 algorithms in interview questions:

  • Graph
  • Linked List
  • Dynamic Programming
  • Sorting and Searching
  • Tree / Binary Search Tree
  • Number Theory
  • BIT Manipulation
  • String / Array 

It seems pretty accurate to me. It's super helpful to know this. What are you great at in this list? What are you weak at? This link is excellent for getting a high-level view of each and some examples. 

These courses will help you absolutely dominate in this area:

In about 6 hours you'll be able to understand and really talk the talk when it comes to algorithms. 

Action: Study this stuff. Learn it. Know it.

Stage 2: Practice Practice Practice

Here's another thing you need to make a part of your routine: practice. You need to practice this stuff a few times a week or more to really get good. 

The more you practice the better you'll do in any whiteboard situation. Most of the time they ask for pseudocode but if you really put your practice in you can write real compilable code on a whiteboard without blinking. 

  1. Check out the HackerRank Interview Preparation Kit - This is where the rubber meets the road with all that algorithm knowledge you now have. Put it to work with real examples. 

  2. Start participating in HackerRank Challenges - This is how you can really apply your knowledge in different areas. This is the ultimate whiteboard practice area.      
    It's broken down like this:

    Do an exercise a day when you can. Work your way through many of the challenges. If you just do a challenge a day for 30 days you can nail a whiteboard interview and will become a better coder. I promise you. 

  1. Sign up for LeetCode and start participating in challenges - Nothing sharpens your skills like competing with others. There are tons of challenges and fun stuff here. 

  2. Check out Project Euler and start writing code to solve the problems there. Project Euler is a set of math and programming problems that really challenge your problem-solving abilities. Use code to solve your problems here and your skills will grow. 

Action: Write some code. A lot of it. Get ruthlessly good at it.

Stage 3: Build Your Public Profile

You need to have work you've done online and accessible. Every recruiter, manager, or someone interested in you will Google you. Make sure they find your work. 

  1. Put your stuff Online and start a GitHub account if you don't already have one. Put all of your code up there. Your personal projects, stuff you write for tutorials, all of it.

  2. Create accounts on sites like JSFiddle and CodePen - This is mostly for web developers but it creates a good spot where people can find your work. 

  3. Create a Blog - It doesn't matter if it's WordPress, GitHub pages, or a custom hosted server setup like my blog you should create a blog that talks about what you're working on, what you're learning, and what you can teach. 

Note: I have thrown away resumes of people when I couldn't find their GitHub or any public work. This shows that they aren't enthusiastic and managers who look for passionate people will look for what you've posted publicly. Put your work out there no matter how good it is. 

Action: Sign up and start making your code and projects public.

Stage 4: Prepare for Interviews

Preparation ultimately determines your success. Nobody succeeds without preparation. Here's how you can prepare for your interview and kick the ball through the uprights. 

  1. Pick up a copy of Cracking the Code Interview - This book is the bible of coding interviews. Like this article it doesn't tell you how to cheat or shortcut your way into getting hired - it gives you a great framework to upgrade your skills. Practice questions are included and it gives you all the tools you need to really sharpen your skillset. 

  2. Take this Course on preparing for a job interview - This gives you the information you need to really prepare yourself. 

    This course covers:

    • Job Interview Basics
    • Algorithm Based Questions
    • Typical Questions
    • Computer Science Questions
    • Getting Experience  It really covers what you need to know to succeed in 2 1/2 hours. Well worth it. 
  3. Watch this video about how to prepare yourself for Developer Job Interviews - This is created by John Sonmez, who you may already know. There's no question that John is a successful developer and he's more than willing to share what has worked for him. It's well worth checking out. 

  4. Exercise - Ok I know this will sound silly, but here's something that will give you an extra edge. No matter what time of day your interview is, hit the gym or do some cardio an hour or two before the interview. This will ensure:

    • You are refreshed and energetic
    • You have oxygen flowing through your blood
    • Your muscles will be relaxed

    A good hard workout will make sure you are charged and ready to go for your interview. You don't want to seem tired or lethargic in your interview. You want to be at your physical and mental BEST. 

Preparation is everything. The more you prepare the better you'll feel on the interview day.

Action: Start training like a boxer. Get fight ready.

Conclusion

Coding interviews can be brutal. You can take the sting off them by doing the following:

  • Learning 
  • Sharpening your skills
  • Practicing

These things will ensure your success. Trust me, after working on crazy problems in LeetCode and HackerRank the whiteboard tests are so much easier. If you do this alone, you'll be successful. 

If you have comments feel free to share or yell at me on Twitter and we can discuss it. 

Sevan Janiyan (sevan)

Something blogged (on pkgsrcCon 2019) July 29, 2019 02:55 PM

pkgsrcCon 2019 was held in Cambridge this year. The routine as usual was social on the Friday evening, day of talks on the Saturday, hacking on things on the Sunday.Armed with a bag of bits to record the talks, I headed up to Cambridge on Friday evening to meet up with sborrill before heading to …

Pete Corey (petecorey)

Clipping Convex Hulls with Thi.ng! July 29, 2019 12:00 AM

I recently found myself knee-deep in a sea of points and polygons. Specifically, I found myself with two polygons represented by two random sets of two dimensional points. I wanted to convert each set of points into a convex polygon, or a convex hull, and find the overlapping area between the two.

After doing some research, I learned about the existence of a few algorithms that would help me on my quest:

  • Graham’s scan can be used to build a convex hull from a set of randomly arranged points in two dimensional space.
  • The Sutherland–Hodgman algorithm can be used to construct a new “clip polygon” that represents the area of overlap between any two convex polygons.
  • The shoelace formula can be used to compute the area of any simple polygon given its clockwise or counterclockwise sorted points.

My plan of attack is to take my two random sets of two dimensional points, run them through Graham’s scan to generate two convex hulls, computed a new “clip polygon” from those two polygons, and then use the shoelace formula to compute and compare the area of the clip polygon to the areas of the two original polygons.

Before diving in and building a poorly implemented, bug riddled implementation of each of these algorithms, I decided to see if the hard work had already been done for me. As it turns out, it had!

I discovered that the thi.ng project a , which is a set of computational design tools for Clojure and Clojurescript and also includes a smorgasbord of incredibly useful Javascript packages, contains exactly what I was looking for.

  • @thi.ng/geom-hull implements Graham’s scan.
  • @thi.ng/geom-clip implements the Sutherland–Hodgman clipping algorithm.
  • @thi.ng/geom-poly-util implements the shoelace formula.

Perfect!

Let’s use these libraries, along with HTML5 canvas, to build a small proof of concept. We’ll start by writing a function that generates a random set of two dimensional points:


const generatePoints = (points, width, height) =>
  _.chain(points)
    .range()
    .map(() => [
      width / 2 + (Math.random() * width - width / 2),
      height / 2 + (Math.random() * height - height / 2)
    ])
    .value();

Next, let’s use our new generatePoints function to generate two sets of random points and render them to a canvas (we’re glossing over the canvas creation process):


const drawPoints = (points, size, context) =>
  _.map(points, ([x, y]) => {
    context.beginPath();
    context.arc(x, y, size, 0, 2 * Math.PI);
    context.fill();
  });

let points1 = generatePoints(5, width * ratio, height * ratio);
let points2 = generatePoints(5, width * ratio, height * ratio);

context.fillStyle = "rgba(245, 93, 62, 1)";
drawPoints(points1, 10, context);

context.fillStyle = "rgba(118, 190, 208, 1)";
drawPoints(points2, 10, context);

The points on this page are generated randomly, once per page load. If you’d like to continue with a different set of points, refresh the page. Now we’ll use Graham’s scan to convert each set of points into a convex hull and render that onto our canvas:


import { grahamScan2 } from "@thi.ng/geom-hull";

const drawPolygon = (points, context) => {
  context.beginPath();
  context.moveTo(_.first(points)[0], _.first(points)[1]);
  _.map(points, ([x, y]) => {
    context.lineTo(x, y);
  });
  context.fill();
};

let hull1 = grahamScan2(points1);
let hull2 = grahamScan2(points2);

context.fillStyle = "rgba(245, 93, 62, 0.5)";
drawPolygon(hull1, context);

context.fillStyle = "rgba(118, 190, 208, 0.5)";
drawPolygon(hull2, context);

We can see that there’s an area of overlap between our two polygons (if not, refresh the page). Let’s use the Sutherland-Hodgman algorithm to construct a polygon that covers that area and render it’s outline to our canvas:


import { sutherlandHodgeman } from "@thi.ng/geom-clip";

let clip = sutherlandHodgeman(hull1, hull2);

context.strokeStyle = "rgba(102, 102, 102, 1)";
context.beginPath();
context.moveTo(_.first(clip)[0], _.first(clip)[1]);
_.map(clip, ([x, y]) => {
  context.lineTo(x, y);
});
context.lineTo(_.first(clip)[0], _.first(clip)[1]);
context.stroke();

Lastly, let’s calculate the area of our two initial convex hulls and the resulting area of overlap between then. We’ll render the area of each at the “center” of each polygon:


import { polyArea2 } from "@thi.ng/geom-poly-utils";

const midpoint = points =>
  _.chain(points)
    .reduce(([sx, sy], [x, y]) => [sx + x, sy + y])
    .thru(([x, y]) => [x / _.size(points), y / _.size(points)])
    .value();

const drawArea = (points, context) => {
  let [x, y] = midpoint(points);
  let area = Math.round(polyArea2(points));
  context.fillText(area, x, y);
};

context.fillStyle = "rgba(245, 93, 62, 0.5)";
drawArea(hull1, context);
context.fillStyle = "rgba(118, 190, 208, 0.5)";
drawArea(hull2, context);
context.fillStyle = "rgba(102, 102, 102, 1)";
drawArea(clip, context);

As you can see, with the right tools at our disposal, this potentially difficult task is a breeze. I’m incredibly happy that I discovered the thi.ng set of libraries when I did, and I can see myself reaching for them in the future.

Update: The Thi.ng creator put together a demo outlining a much more elegant way of approaching this problem. Be sure to check out their solution, and once again, check out thi.ng!

canvas { width: 100%; height: 100%; }

July 28, 2019

Ponylang (SeanTAllen)

Last Week in Pony - July 28, 2019 July 28, 2019 12:03 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

Andreas Zwinkau (qznc)

Sacrifice of Scar July 28, 2019 12:00 AM

Short Lion King fanfic

Read full article!

July 27, 2019

Ponylang (SeanTAllen)

0.30.0 Released July 27, 2019 10:50 AM

Pony version 0.30.0 is now available. This is a high-priority bug fix release that contains a fix for a bug that could cause Pony programs to segfault. We recommend upgrading as soon as possible. Also note, this release does include a breaking API change (albeit small). Please note that, at this time, the update to Homebrew has yet to be merged, but should be within a day. All other release targets are completed.

July 26, 2019

Awn Umar (awn)

working with forks in git July 26, 2019 12:00 AM

Working with forks entails having a single codebase but being able to push and pull from multiple remote endpoints. This can get fairly annoying, especially for Go projects which have a specified import path which must be reflected in the filesystem path (at least before modules).

Luckily git handles this use-case pretty well.

$ # Working with a hypothetical project hosted on GitHub.
$ cd $GOPATH/src/github.com
$ 
$ # Create a directory for the source owner, to preserve import paths.
$ mkdir source && cd source
$ 
$ # Clone your own fork into this directory.
$ git clone git@github.com:me/repo.git && cd repo
$ 
$ # Add the original repository URL as a new remote.
$ git remote add upstream git@github.com:source/repo.git
$ 
$ git pull upstream master # Pull from original repository
$ git push origin master   # Push to forked repository

July 25, 2019

Derek Jones (derek-jones)

Want to be the coauthor of a prestigious book? Send me your bid July 25, 2019 07:22 PM

The corruption that pervades the academic publishing system has become more public.

There is now a website that makes use of an ingenious technique for helping people increase their paper count (as might be expected, the competitive China thought of it first). Want to be listed as the first author of a paper? Fees start at $500. The beauty of the scheme is that the papers have already been accepted by a journal for publication, so the buyer knows exactly what they are getting. Paying to be included as an author before the paper is accepted incurs the risk that the paper might not be accepted.

Measurement of academic performance is based on number of papers published, weighted by the impact factor of the journal in which they are published. Individuals seeking promotion and research funding need an appropriately high publication score; the ranking of university departments is based on the publications of its members. The phrase publish or perish aptly describes the process. As expected, with individual careers and departmental funding on the line, the system has become corrupt in all kinds of ways.

There are organizations who will publish your paper for a fee, 100% guaranteed, and you can even attend a scam conference (that’s not how the organizers describe them). Problem is, word gets around and the weighting given to publishing in such journals is very low (or it should be, not all of them get caught).

The horror being expressed at this practice is driven by the fact that money is changing hands. Adding a colleague as an author (on the basis that they will return the favor later) is accepted practice; tacking your supervisors name on to the end of the list of authors is standard practice, irrespective of any contribution that might have made (how else would a professor accumulate 100+ published papers).

I regularly receive emails from academics telling me they would like to work on this or that with me. If they look like proper researchers, I am respectful; if they look like an academic paper mill, my reply points out (subtly or otherwise) that their work is not of high enough standard to be relevant to industry. Perhaps I should send them a quote for having their name appear on a paper written by me (I don’t publish in academic journals, so such a paper is unlikely to have much value in the system they operate within); it sounds worth doing just for the shocked response.

I read lots of papers, and usually ignore the list of authors. If it looks like there is some interesting data associated with the work, I email the first author, and will only include the other authors in the email if I am looking to do a bit of marketing for my book or the paper is many years old (so the first author is less likely to have the data).

I continue to be amazed at the number of people who continue to strive to do proper research in this academic environment.

I wonder how much I might get by auctioning off the coauthoship of my software engineering book?

July 24, 2019

Jeremy Morgan (JeremyMorgan)

Creating Trimmed Self Contained Executables in .NET Core July 24, 2019 02:09 PM

I'm going to show you a cool new feature in .NET Core 3.0.

Let's say you want to create a simple, lean executable you can build and drop on to a server.

For an example we'll create a console app that opens a line of text, reads it and displays the output.

First, let's create a new .NET Core app:

dotnet new console

This will scaffold a new console application.

Now run it:

dotnet run

It should look something like this:

Creating Trimmed Self Contained Executables in Net Core

I'm on a Mac here, but it doesn't matter as long as your development box has the .NET Core CLI Installed.

This displays "Hello World" on the console.

Now, lets create a file called file.txt:

this is a file! With some lines whatever

Doesn't matter what you put in here, as long as it has some text in it.

Next we'll create a something that will read those lines and display them. Remove the "Hello World!" code and replace it with this:

``` string[] lines = System.IO.File.ReadAllLines(@"test.txt");

    foreach (string line in lines)
    {
        Console.WriteLine("\t" + line);
    }

Console.WriteLine("Press any key to exit."); System.Console.ReadKey(); ```

This is pretty much your basic cookie cutter code for:

  • opening up a file
  • reading it into a string array
  • loop through the array line by line
  • print each line
  • exit

Pretty simple stuff. When I run it on my machine it looks like this:

Creating Trimmed Self Contained Executables in Net Core

And that's great. But I'm on a Mac here, what if I want it to run on a Windows Machine? Linux? No problem, this is .NET Core right? We'll just publish it to multiple targets.

But what if .NET Core isn't installed on the machine?

What if I just want a simple executable I can run to read this file without a pile of files or .Net core installed?

Publishing in .Net Core

Let's back up a little. .NET Core has had publishing profiles for a long time. The idea behind "target" publishing is one of the biggest selling points of the platform. Build your app, then publish it for a specific target, Windows, OSX, or Linux.

You can publish it a few different ways

  • Framework Dependent Deployment - This means relies on a shared version of .NET Core that's installed on the Computer/Server.
  • Self Contained Deployment - This doesn't rely on .Net Core being installed on the server. All components are included with the package (tons of files usually).
  • Framework Dependent Executables - This is very similar to a framework dependent deployment, but it creates executables that are platform specific, but require the .NET Core libraries.

Ok, so what's this cool new feature I'm going to show?

Well, when you do a self contained deployment it's cool because you don't need the runtime installed, but it ends up looking something like this:

Creating Trimmed Self Contained Executables in Net Core

This is the application we just built published as a Self Contained Deployment for Windows. Yikes.

Let's say you wanted to share this file reader application, and asking someone to copy all these files into a folder to run something to read a text file. It's silly.

New Feature: Self Contained Executables

So to build that self contained executable I ran the following command:

dotnet publish -c release -r win10-x64

This should look pretty familiar to you if you've done it before. But .NET Core 3.0 has a cool new feature:

dotnet publish -r win-x64 -c Release /p:PublishSingleFile=true Using this flag, it will build something a little different:

Creating Trimmed Self Contained Executables in Net Core

That's MUCH better. It's now a single exe and .pdb. If we look at the info, it's not super small:

Creating Trimmed Self Contained Executables in Net Core

But it includes the .NET Core runtime with it as well. And here's another cool feature.

dotnet publish -r win-x64 -c Release /p:PublishSingleFile=true /p:PublishTrimmed=true

So in our example it doesn't change the size, but if you have a large complex application with a lot of libraries, if you just publish it to a single file, it can get HUGE. By adding the PublishTrimmed flag it will only extract the libraries you need to run the application.

So when we copy the files to a Windows 10 machine, we have a nice small package:

Creating Trimmed Self Contained Executables in Net Core

And we run it and it works! Without .NET Core!

Creating Trimmed Self Contained Executables in Net Core

and if I change my target:

dotnet publish -r linux-x64 -c Release /p:PublishSingleFile=true /p:PublishTrimmed=true

I can run it on a Linux server without .NET Core just as easily:

Creating Trimmed Self Contained Executables in Net Core

Just remember on a Linux machine you won't need the .NET Core runtime, but you will need the Prerequisites for .NET Core on Linux installed.

Conclusion

So this is a cool feature of .NET Core 3.0. If you want to build trimmed down self contained executables for any of the platforms, you can do it easily with a couple of flags.

This is great for those stupid simple things: console apps, data readers, microservices, or whatever you want to build and just drop on a machine with no hassle. I thought it was a cool feature to show.

If you want to master .NET Core, check out the ASP.Net Core Path on Pluralsight. The courses go pretty in depth and are a great way to ramp up.

Yell at me on Twitter with questions or comments!



What is your .NET Core IQ?


My ASP.NET Core IQ is 200. Can you beat it? Take the test now to find out your score!!

Benjamin Pollack (gecko)

Falsehoods Programmers Believe About Cats July 24, 2019 12:43 PM

Inspired by Falsehoods Programmers Believe About Dogs, I thought it would be great to offer you falsehoods programmers believe about mankind’s other best friend. But since I don’t know what that is, here’s instead a version about cats.

  1. Cats would never eat your face.
  2. Cats would never eat your face while you were alive.1
  3. Okay, cats would sometimes eat your face while you’re alive, but my cat absolutely would not.
  4. Okay, fine. At least I will never run out of cat food.
  5. You’re kidding me.
  6. There will be a time when your cat knows enough not to vomit on your computer.
  7. There will be a time when your cat cares enough not to vomit on your computer.
  8. At the very least, if your cat begins to vomit on your computer and you try to move it to another location, your cat will allow you to do so.
  9. When your cat refuses to move, it will at least not manage to claw your arm surprisingly severely while actively vomiting.
  10. Okay, but at least they won’t attempt to chew the power cord while vomiting and clawing your hand, resulting in both of you getting an electric shock.
  11. …how the hell are you even alive?2
  12. Cats enjoy belly rubs.
  13. Some cats enjoy belly rubs.
  14. Cats reliably enjoy being petted.
  15. Cats will reliably tell you when they no longer enjoying being petted.
  16. Cats who trust their owners will leave suddenly when they’re done being petted, but at least never cause you massive blood loss.
  17. Given all of the above, you should never adopt cats.
  18. You are insane.

Happy ten years in your forever home, my two scruffy kitties. Here’s to ten more.


  1. Here, ask Dewey, he knows more about it than I do. [return]
  2. Because, while my cat has absolutely eaten through a power cord, this is an exaggeration. The getting scratched while trying to get my cat not to puke on a computer I was actively using happened at a different time from the power cord incident. Although this doesn’t answer the question how she is alive. [return]

Zac Brown (zacbrown)

Book Notes: Deep Work by Cal Newport July 24, 2019 07:00 AM

Note: These notes are lightly organized and reflect my own takeaways from this book. They’re captured here for my own purposes. If you should find them useful, great!

Rule #1 - Work Deeply

Depth Philosophy - styles of achieving depth

  • Monastic - isolationist, doesn’t work well for the vast majority of folks and not likely to work for me
  • Bimodal - a few days on, a few days off
    • utilized by Carl Jung and Adam Grant
    • periods of intense deep work followed by periods of shallower work
    • probably most useful in achieving big leaps” periodically
  • Rhythmic - best for day to day life
    • Cal Newport’s preferred
    • Do it every day. Probably best for growing the muscle initially.
    • For future - probably intersperse this approach with Bimodal.

Ritualize

  • develop habits that reduce willpower consumption
    • consistent routines and organization, e.g.
      • same ritual for starting and ending your work day
      • wearing the same clothing for your work week (Zuckerberg/Jobs style)
    • consistent places to work and for how long
  • pay attention to rituals before you enter Deep Work, look for opportunities to repeat
    • meditation
    • cup of tea
    • quick walk

Make Grand Gestures

  • e.g. holing up in a hotel suite ($1000/night) for a few days to complete something
  • probably less important/practical in most cases
  • practically, this may be as simple as paying for a private office

Don’t Work Alone

  • hard to do this if you’re a remote worker
  • How can I optimize this in business trips? Hole up in a conference room with key folks?
  • What about this will help me create breakthroughs?
  • whiteboard effect is very helpful - like time spent with J and so on

Execute Like a Business

Discipline #1 - Focus on the wildly important.

  • Saying no more than saying yes. Both to yourself and to others.
  • More importantly, say Yes to the truly important and lean into it.
    • Stay out of the attention grabbing fray to avoid saying No so often that it drains willpower.

Discipline #2 - Act on Lead Measures

  • Tighten the OODA loop in other words.
  • Easiest metric is time spent working deeply. If you’re not measuring this then you’re not getting a good sense of the quality of your time.

Discipline #3 - Keep a Compelling Scoreboard

  • People play differently when you’re keeping score.”
  • What’s measured gets managed.”
  • Try tally mark system.
    • Track hours spent working deeply. Each tally is an hour.
    • Circle tally marks where something big is finished.
  • Shoot for at least 50% deep work initially. Move up from there. Ideally >75%.

Discipline #4 - Create a Cadence of Accountability

  • Reviewing periodically helps put progress toward a goal into context.
  • Weekly review of notebook w/ score card.
    • Do it Saturday morning?
    • Review what went well and what went poorly.
    • Revise plan for following weekly to maximize likelihood of success.

Be Lazy

  • Make time for Leisure. Continually keeping the mind busy prevents it from forming those really deep connections.
  • Practically speaking, end the day at 17:30-18:30.
  • Working beyond 8 hours results in working on things which are not that important.
    • ~4-5 hours of deep work is possible per day. Anything else should not dominate.
  • Develop a shutdown ritual.
    • Note everything in flight 15-20 minutes before end of work day.
    • Provide next steps for anything that is in flight.
    • Note what is completed from the todo list of the day.
    • Be consistent. Have a verbal cue. e.g. Shutdown complete.”
    • Basic outline of steps:
      1. Check email/Slack/Basecamp for nay outstanding tasks or requests.
      2. Transfer tasks that are on the mind to task list and add relevant notes for next steps.
        • There may be multiple lists of multiple projects ongoing.
      3. Quickly skim every task list and compare against the next few days of the calendar. Note any deadlines or appointments that impact task prioritization. Rearrange next day as needed.
      4. Say the words. e.g. Shutdown complete.”

Rule #2 - Embrace Boredom

Boredom is directly related to ability to focus for long periods of time. Distractions, e.g. smartphones, reduce our capacity to be focused because they force stimulation. Focus is not strictly stimulating.

Don’t take breaks from distraction. Instead take breaks from focus.

  • Internet Sabbath doesn’t address the underlying problem. Relies on taking a break from distractions.
  • Schedule in advance when Internet use will occur. Record the next time Internet use is allowed.
    1. This should work even with need for Internet as part of your job.
    2. Time outside the block should be absolutely free of Internet use.
    3. Scheduling Internet use at home can further improve concentration.
      • Just always have a notebook for thoughts/queries to look up?
  • This is all about rewiring the brain to be comfortable resisting distracting stimuli.

Work Like Teddy Roosevelt

  • Estimate how long a task should take and publicly commit to a deadline. For example - the person you’re doing the work for.
  • Start small initially. Focusing intensely with no distraction breaks. Over time, quality of intensity and distraction reduction should increase.

Meditate Productively

  • Focus on a single well defined professional problem. Can be done while walking, on transit, or on a car ride.
  • Lets you make use of your brain while doing simple mechanical tasks.
  • This is less about productivity and more about improving deep focus ability. Forces you to resist distraction and return attention repeatedly to a well-defined problem.
  • Suggestions to make this more successful:
    1. Be wary of distractions and looping.
      • Similar to meditation, when you notice the mind wandering, bring attention back to the problem you’re focusing on.
      • Note when you’ve rehashed the same facts or observations repeatedly. Note it and redirect attention to the next problem solving step.
    2. Structure Your Deep Thinking
      • Structure helps decompose a problem. It can help to develop a framework for assessing the given problem and reasoning about it.
      • Useful steps include:
        • review the relevant variables
        • define the next-step question
        • Using the two pieces of info above allows you to plan and iterate on targeting your attention.
        • after solving the next-step question, consolidate your gains by reviewing clearly the answer identified
        • push yourself to the next level of depth by restarting the process

Memorize a Deck of Cards

  • Memory training exercises help improve depth of focus. AKA helps by improving concentration.
  • May be useful to learn some basic memorization training exercises like memorizing the order of a deck of cards.

Rule #3 - Quit Social Media

The perceived value of social media is not nearly as deep or meaningful as they’d have us believe. For example - liking a Facebook post is not a replacement for having dinner and conversation with that same person.

  • Consider: Any benefit approach” - If X tool provides any benefit at all, it is worth using.”
    • This is obviously false, especially with things that consume our attention and willpower.
    • This is where social media pretty much categorically falls.
  • Consider: craftsman approach” - A tool is worth using only if its benefits substantially outweigh the negative impacts.”

Apply the Law of the Vital Few to Your Internet Habits

  • Start by identifying the main high level goals in professional and personal life.
    • Professional Goals
      • Be an effective Engineering Leader for my team, our projects, and the broader company.
      • Be an advocate for the team for work/life balance and reasonable work expectations.
      • Be a resource for my team and peers to help them where possible.
    • Professional Activities that further these goals
      • Engineering Leader
        • regular 1:1’s with my team -realistic costing of our projects
        • keeping SLT expectations realistic
        • deeply consider designs and plans for projects
      • Advocate
        • encourage engineers to point out when things are negatively impacting their lives
        • telling SLT when our workload is too much and manage expectations
  • Any activity should be weighed against aforementioned goals and activities to identify whether a tool significantly or detracts from the goals and activities.
  • While there may in fact be benefits to a given social network, when evaluated holistically, they general don’t outweigh the costs.

The Law of the Vital Few

  • In many settings, 80 percent of a given effect is due to just 20 percent of the possible causes.
  • In context of achieving goals, while there may be 10-15 activities that contribute towards goals, there’s probably 2-3 that really make the biggest difference. You should focus on those top 2-3 activities.
    • This is especially important in view of our limited attention and willpower. E.g. Do a couple things well rather than many things poorly.
    • By limiting focus to the most important activities, this probably magnifies their effectiveness.

Quitting Social Media

  • Use the pack” method. Pack away everything and as you need it, unpack it again back into your life. In practice, this means stop using Facebook, LinkedIn, Twitter, and so on and see who actually reaches out to find out where you went. Chances are, no one will.
  • In the case of Social Media, log out of all of them and login as needed. Block them from your devices in most cases.
  • Stuff, social networks, habits, etc accumulate without us noticing. The approach above helps identify what is truly useful.
  • Practically speaking, ask the following two questions of each social media service:
    1. Would the last thirty days have been notably if I had been able to use this service?
    2. Do people care that I wasn’t using this service?
  • If the answer is no to both, get rid of it.
  • If the answer is yes to both, keep using it.
  • If the answer is ambiguous, use best judgement.
    • Lean toward no” in general - they’re designed to be addictive and easily become a time suck.

Don’t Use the Internet to Entertain Yourself

  • Both from an enrichment and relaxing effect, the Internet of today does a piss poor job of being a leisure activity.
    • The modern Internet is designed to generate FOMO and anxiety.
    • It is best treated with skepticism and avoided.
    • Better activities might include:
      • physical hobby like woodworking
      • reading books, papers, magazines, etc
      • writing blog posts, essays, etc
      • cooking
      • watching a movie or specific show - not surfing
  • Put thought into your leisure time. Schedule if necessary to help yourself make progress toward personal goals.
  • Planning leisure also helps avoid the surfing” trap. You know what you’ll do so no need to surf.
    • This doesn’t mean being inflexible - just have a plan for how you’d like to use your time so that you have a default that isn’t surfing.

Rule #4 - Drain the Shallows

  • You can’t completely eliminate the shallow work. A non-trivial amount of shallow work is required to stay successful at work. Sadly, you cannot exist solely in the Ivory Tower.
    • e.g. You can avoid checking email every ten minutes but probably can’t entirely ignore email like a hermit.
  • There’s a limit to the amount of deep work you can perform in a single day. Shallow work isn’t problematic till it begins to crowd out your deep work time.
  • Fracturing deep work time can be highly detrimental. Ideal to get big chunks of deep work and batch the shallow work together rather than interspersing.
  • Treat shallow work with suspicion in general. If someone asks you to do something and you can find little depth to it, push back.

Schedule Every Minute of Your Day

  • We often spend too much of our days on autopilot - being reactive. Giving too little thought to how we’re using our time.
  • Schedule in a notebook the hours of your work day.
    • block off times for tasks which fall outside meetings
    • if interruptions occur, take a moment to reschedule and shuffle the day
      • be flexible - if you hit an important insight, ignore the schedule and carry on with the insight
    • Periodically evaluate What makes the most sense for me to do with the time that remains?” throughout the day. This affords flexibility and spontaneity to ensure you’re always working on the most important thing.

Quantify the Depth of Every Activity

  • For any task, ask How long would it take in months to train a smart recent college graduate with no special training in my field to complete this task?”
  • If a task would take many months to train someone, then that’s the deep work to focus on.

Ask for a Shallow Work Budget

  • Ask boss or self, how much shallow work should you be doing? What percentage of your time?
  • After some analysis, may find that:
    • you avoid some projects which seem to have a high amount of shallow work
    • begin purging shallow work from current workload
  • Important to be aligned with boss. Don’t want mismatched expectations but be clear on what you believe is shallow work and why. Try to prune as much as possible.
  • Quantifying the percentage can be useful if you ned to show that there’s too much shallow work on your plate and want to force your management’s hand to agree to the max amount of shallow work on your plate.

Finish Work by Five Thirty

  • fixed schedule productivity” - work backwards from the endpoint to identify productivity strategies to ensure a 17:30 end time to work day
  • Identify rules and habits that help enforce these regular work hours. By maintaining the regular work hours, you preserve your ability to sustain Deep Work regularly. It forces draining of the shallows and restoration of your limited willpower/focus.
  • Be extremely cautious with the use of the word Yes.”
    • Be clear in your refusal but vague in your reasons.
      • Giving details allows the requestor of your time wiggle room to get you to say yes in a follow up request.
    • Resist the urge to offer consolation prizes when you turn down a request. Don’t invent shallow work for yourself.
  • The 17:30 cutoff forces good use of time earlier in the day. Can’t afford to waste the morning or to let a big deadline creep up on you.
  • This is all about a scarcity mindset. Drain the shallows mercilessly. Use this Fixed Schedule Productivity ruthlessly and relentlessly.
  • Just because your boss sends email or Slack messages at 20:30 doesn’t mean they’re sitting there waiting on a reply that evening. They’re probably just working through their backlog at their own pace.

Become Hard to Reach

  1. Make People Who Send You Email Do More Work
    • Provide people options for contact with clear expectations.
      • e.g. Shoot an email here if you have something you think I’d find interesting. I guarantee no response unless I am specifically interested in engaging with you about the topic or opportunity.”
    • sender filter” - This helps the sender understand they should only send you something they truly feel you’d be interested in.
      • Most people easily accept the idea that you have a right to control you own incoming communication as they would like to enjoy the same right. More importantly, people appreciate clarity.”
  2. Do More Work When You Send or Reply to Emails
    • Avoid productivity land mines
      • e.g. Great to meet you last week. Do you want to grab coffee?”
      • They prey on your desire to be helpful and you dash off a quick response. In the short term, feels productive, but in the long term, it just adds a lot of extra noise.
    • Best to consider these messages with requests in them as: What is the project represented by this message, and what is the most efficient (in terms of messages generated) process for bringing this project to a successful conclusion?”
    • Better response to request for coffee would be process centric, taking the time to describe the process you identified in the message, points out the current step, and emphasizes the next step.
      • e.g. to meet up, specify a place and a couple of times that you’re available to meet. Provide an out in case the details conflict.
    • Process centric email closes the loop and removes it from mental foreground. You’ve dealt with it, next thing.”
  3. Don’t Respond
  • It’s the sender’s responsibility to convince the receiver that a reply is worthwhile.”
  • Simple rules to Professorial Email Sorting”
    • It’s ambiguous or otherwise makes it hard for you to generate a reasonable response.
    • It’s not a question or proposal that interests you.
    • Nothing really good would happen if you did respond and nothing really bad would happen if you didn’t.
  • This can be an uncomfortable practice due to common conventions around email with expecting replies. There are also exceptions - e.g. your boss emails you.
  • Develop the habit of letting small bad things happen. If you don’t, you’ll never find time for the truly life-changing big things.”
  • People are adaptable. If they realize you only respond to requests relevant to your interests then they’ll adjust.

Ponylang (SeanTAllen)

Pony Stable 0.2.1 Released July 24, 2019 12:00 AM

July 21, 2019

Ponylang (SeanTAllen)

Last Week in Pony - July 21, 2019 July 21, 2019 02:48 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, or our Zulip community.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

July 20, 2019

Leo Tindall (LeoLambda)

Simple Elixir Functions July 20, 2019 07:03 PM

I’ve been playing around with Elixir, which is a pretty cool language - specifically, I’m reading Programming Elixir 1.6 which is free for anyone with a .edu email address from The Pragmatic Bookshelf. reverse/1 I enjoy functional programming, so of course the first two functions I made were list operations. First up is reverse(list) (written as reverse/1, meaning it takes one argument), which reverses a list. # reverse/1 # Reverse the given list.

Sevan Janiyan (sevan)

A week of pkgsrc #13 July 20, 2019 04:56 PM

With the lead up to the release of pkgsrc-2019Q2 I picked up the ball with the testing on OS X Tiger again. It takes about a month for a G4 Mac Mini to attempt a bulk build of the entire pkgsrc tree with compilers usually taking up most days without success. To reduce the turnaround …

Awn Umar (awn)

memory retention attacks July 20, 2019 12:00 AM

In my post on implementing an in-memory encryption scheme to protect sensitive information, I referenced a mitigation strategy called a Boojum. It is described by Bruce Schneier, Niels Ferguson and Tadayoshi Kohno in their book, Cryptography Engineering: Design Principles and Practical Applications.

A number of people asked me about the mechanisms of the attack and the scheme, so I am including the relevant parts here. It is an excellent resource and I recommend that you go and buy it.

page one page two page three

[32] Giovanni Di Crescenzo, Niels Ferguson, Russel Impagliazzo, and Markus Jakobsson. How to Forget a Secret. In Christoph Meinel and Sophie Tison, editors, STACS 99, volume 1563 of Lecture Notes in Computer Science, pages 500-509. Springer-Verlag, 1999.

[57] Peter Gutmann. Secure Deletion of Data from Magnetic and Solid-State Memory. In USENIX Security Symposium Proceedings, 1996. Available from http://www.cs.auckland.ac.nz/~pgut001/pubs/secure_del.html.

[59] J. Alex Halderman, Seth D. Schoen, Nadia Heninger, William Clarkson, William Paul, Joseph A. Calandrino, Ariel J. Feldman, Jacob Appelbaum, and Edward W. Felten. Lest We Remember: Cold Boot Attacks on Encryption Keys. In USENIX Security Symposium Proceedings, pages 45-60, 2008.

July 19, 2019

Jeremy Morgan (JeremyMorgan)

You Can Get the Source Code for Apollo 11 and Take a Course on It July 19, 2019 04:47 AM

In software development you'll hear the term "moon shot". If something is a "moon shot" it's something that's extraordinarily difficult, like landing on the moon. We say this about some app doing something cool, but what about the software that... landed us on the moon? What was the original "moon shot" all about? 

The Software That Put Us on the Moon

Margaret Hamilton

Meet Margaret Hamilton ). She was the director of Software engineering at MIT Instrumentation Laboratory, which was contracted to build the onboard software for the Apollo Space Program. This is her standing with the stack of source code used to launch us to the moon. Today we complain when Visual Studio runs slow. 

If you're a developer of any kind, you owe a lot of thanks to her, since "Software Engineer" wasn't even a term before she came along. 

So imagine having to write software that had to work the first time it ran, and run perfectly, or people would die. Sounds like a lot of pressure right? Margaret and her team did it. They built it, and it worked. It succeeded. 

They made a piece of machinery land on the moon with a computer that had about .08 percent of the processing power of an iPhone. 

You can get a lot more background and details here.

The Source Code

A few years ago, Ron Burkey released the source code for Apollo 11. He put a ton of work into this. This is the source code for the Apollo Guidance Computer, or AGC. 

Not only can you download the source code, but he created a simulator for the AGC, at the Virtual AGC Page. You can dig deep into the systems and how they work, trust me it's a rabbit hole for geeks. Tons of awesome stuff here. There's even a kinder, gentler introduction to the AGC you can check out to get familiar.

The AGC source code is written in assembler, which is foreign to most of us. I've played around enough with x86 assembler to know it's not my calling, but perusing through a lot of this source code, you can piece together how some of this stuff works.  

Comanche and Luminary

If you dig into the code, you'll see it's divided into two parts, Comanche and Luminary. 

Comanche is the software for the Command Module and Luminary is the Lunar Module

The Command Module was the cone that contained the crew and vital equipment, and was the vessel returned back to earth. 

Apollo 11 Source Code

The Lunar Module well, it was the module that landed on the moon. 

Apollo 11 Source Code

It's very interesting to see how these systems interact. When you look through the source code you can see a lot of cool hints how everything works. 

The DSKY 

The DSKY was the user interface for the computer. You could enter commands through a calculator-like interface and it was connected directly to the AGC. The commands contained a verb and a noun for each command. 

Apollo 11 Source Code

If you dig deep enough into the source code you'll see a lot of references to the DSKY and commands related to it. It was a marvel of engineering for its time. Of course, there is a DSKY simulator if you want to play around. 

How Do I Know so Much About This? 

I may sound like a seasoned expert here, but I just took this free course on the code of the Apollo 11, then started digging in the code and researching stuff. 

Apollo 11 Source Code

Click Here to Take This Course for FREE

For the 50th anniversary of the Apollo 11 Moon Landing, Simon Allardice created this awesome course on Pluralsight exploring the AGC. It's a quick course, and it won't teach you how to write AGC specific assembler code, but it will give you a good idea of how things worked back then and how the whole thing fits together. After watching this you'll get a better understanding of the source code, and start looking for easter eggs. 

Conclusion

Ok if you're a nerd like me, you love stuff like this. Obviously this isn't some kind of actively developed code where you can do pull requests and add features. Nobody will hire you because you put AGC knowledge on your resume. But it sure is fun to go through this code and figure out how things worked, and look at the comical comments in the code. It's fascinating and well worth the time to explore if you're genuinely curious about this kind of stuff.

You can always yell at me on Twitter if you'd like to discuss it further. 

Enjoy! 



What is your DevOps IQ?

what's your devops score


My Devops Skill IQ is 232. Can you beat it? Take the test now to find out your Devops IQ score!!

Joe Nelson (begriffs)

History and effective use of Vim July 19, 2019 12:00 AM

This article is based on historical research and on simply reading the Vim user manual cover to cover. Hopefully these notes will help you (re?)discover core functionality of the editor, so you can abandon pre-packaged vimrc files and use plugins more thoughtfully.

physical books

physical books

To go beyond the topics in this blog post, I’d recommend getting a paper copy of the manual and a good pocket reference. I couldn’t find any hard copy of the official Vim manual, and ended up printing this PDF using printme1.com. The PDF is a printer-friendly version of the files $VIMRUNTIME/doc/usr_??.txt distributed with the editor. For a convenient list of commands, I’d recommend the vi and Vim Editors Pocket Reference.

Table of Contents

History

Birth of vi

Vi commands and features go back more than fifty years, starting with the QED editor. Here is the lineage:

  • 1966 : QED (“Quick EDitor”) in Berkeley Timesharing System
  • 1969 Jul: moon landing (just for reference)
  • 1969 Aug: QED -> ed at AT&T
  • 1976 Feb: ed -> em (“Editor for Mortals”) at Queen Mary College
  • 1976 : em -> ex (“EXtended”) at UC Berkeley
  • 1977 Oct: ex gets visual mode, vi

hard copy terminal

You can discover the similarities all the way between QED and ex by reading the QED manual and ex manual. Both editors use a similar grammar to specify and operate on line ranges.

Editors like QED, ed, and em were designed for hard-copy terminals, which are basically electric typewriters with a modem attached. Hard-copy terminals print system output on paper. Output could not be changed once printed, obviously, so the editing process consisted of user commands to update and manually print ranges of text.

video terminal

By 1976 video terminals such as the ADM-3A started to be available. The Ex editor added an “open mode” which allowed intraline editing on video terminals, and a visual mode for screen oriented editing on cursor-addressible terminals. The visual mode (activated with the command “vi”) kept an up-to-date view of part of the file on screen, while preserving an ex command line at the bottom of the screen. (Fun fact: the h,j,k,l keys on the ADM-3A had arrows drawn on them, so that choice of motion keys in vi was simply to match the keyboard.)

Learn more about the journey from ed to ex/vi in this interview with Bill Joy. He talks about how he made ex/vi, and some things that disappointed him about it.

Classic vi is truly just an alter-ego of ex – they are the same binary, which decides to start in ex mode or vi mode based on the name of the executable invoked. The legacy of all this history is that ex/vi is refined by use, requires scant system resources, and can operate under limited bandwidth communication. It is also available on most systems and fully specified in POSIX.

From vi to vim

Being a derivative of ed, the ex/vi editor was intellectual property of AT&T. To use vi on platforms other than Unix, people had to write clones that did not share in the original codebase.

Some of the clones:

  • nvi - 1980 for 4BSD
  • calvin - 1987 for DOS
  • vile - 1990 for DOS
  • stevie - 1987 for Atari ST
  • elvis - 1990 for Minix and 386BSD
  • vim - 1991 for Amiga
  • viper - 1995 for Emacs
  • elwin - 1995 for Windows
  • lemmy - 2002 for Windows

We’ll be focusing on that little one in the middle: vim. Bram Moolenaar wanted to use vi on the Amiga. He began porting Stevie from the Atari and evolving it. He called his port “Vi IMitation.” For a full first-hand account, see Bram’s interview with Free Software Magazine.

By version 1.22 Vim was rechristened “Vi IMproved,” matching and surpassing features of the original. Here is the timeline of the next major versions, with some of their big features:

1991 Nov 2 Vim 1.14: First release (on Fred Fish disk #591).
1992 Vim 1.22: Port to Unix. Vim now competes with Vi.
1994 Aug 12 Vim 3.0: Support for multiple buffers and windows.
1996 May 29 Vim 4.0: Graphical User Interface (largely by Robert Webb).
1998 Feb 19 Vim 5.0: Syntax coloring/highlighting.
2001 Sep 26 Vim 6.0: Folding, plugins, vertical split.
2006 May 8 Vim 7.0: Spell check, omni completion, undo branches, tabs.
2016 Sep 12 Vim 8.0: Jobs, async I/O, native packages.

For more info about each version, see e.g. :help vim8. To see plans for the future, including known bugs, see :help todo.txt.

Version 8 included some async job support due to peer pressure from NeoVim, whose developers wanted to run debuggers and REPLs for their web scripting languages inside the editor.

Vim is super portable. By adapting over time to work on a wide variety of platforms, the editor was forced to keep portable coding habits. It runs on OS/390, Amiga, BeOS and BeBox, Macintosh classic, Atari MiNT, MS-DOS, OS/2, QNX, RISC-OS, BSD, Linux, OS X, VMS, and MS-Windows. You can rely on Vim being there no matter what computer you’re using.

In a final twist in the vi saga, the original ex/vi source code was finally released in 2002 under a BSD free software license. It is available at ex-vi.sourceforge.net.

Let’s get down to business. Before getting to odds, ends, and intermediate tricks, it helps to understand how Vim organizes and reads its configuration files.

Configuration hierarchy

I used to think, incorrectly, that Vim reads all its settings and scripts from the ~/.vimrc file alone. Browsing random “dotfiles” repositories can reinforce this notion. Quite often people publish monstrous single .vimrc files that try to control every aspect of the editor. These big configs are sometimes called “vim distros.”

In reality Vim has a tidy structure, where .vimrc is just one of several inputs. In fact you can ask Vim exactly which scripts it has loaded. Try this: edit a source file from a random programming project on your computer. Once loaded, run

:scriptnames

Take time to read the list. Try to guess what the scripts might do, and note the directories where they live.

Was the list longer than you expected? If you have installed loads of plugins the editor has a lot to do. Check what slows down the editor most at startup by running the following and look at the start.log it creates:

vim --startuptime start.log name-of-your-file

Just for comparison, see how quickly Vim starts without your existing configuration:

vim --clean --startuptime clean.log name-of-your-file

To determine which scripts to run at startup or buffer load time, Vim traverses a “runtime path.” The path is a comma-separated list of directories that each contain a common structure. Vim inspects the structure of each directory to find scripts to run. Directories are processed in the order they appear in the list.

Check the runtimepath on your system by running:

:set runtimepath

My system contains the following directories in the default value for runtimepath. Not all of them even exist in the filesystem, but they would be consulted if they did.

~/.vim
The home directory, for personal preferences.
/usr/local/share/vim/vimfiles
A system-wide Vim directory, for preferences from the system administrator.
/usr/local/share/vim/vim81
Aka $VIMRUNTIME, for files distributed with Vim.
/usr/local/share/vim/vimfiles/after
The “after” directory in the system-wide Vim directory. This is for the system administrator to overrule or add to the distributed defaults.
~/.vim/after
The “after” directory in the home directory. This is for personal preferences to overrule or add to the distributed defaults or system-wide settings.

Because directories are processed by their order in line, the only thing that is special about the “after” directories is that they are at the end of the list. There is nothing magical about the word “after.”

When processing each directory, Vim looks for subfolders with specific names. To learn more about them, see :help runtimepath. Here is a selection of those we will be covering, with brief descriptions.

plugin/
Vim script files that are loaded automatically when editing any kind of file. Called “global plugins.”
autoload/
(Not to be confused with “plugin.”) Scripts in autoload contain functions that are loaded only when requested by other scripts.
ftdetect/
Scripts to detect filetypes. They can base their decision on filename extension, location, or internal file contents.
ftplugin/
Scripts that are executed when editing files with known type.
compiler/
Definitions of how to run various compilers or linters, and of how to parse their output. Can be shared between multiple ftplugins. Also not applied automatically, must be called with :compiler
pack/
Container for Vim 8 native packages, the successor to “Pathogen” style package management. The native packaging system does not require any third-party code.

Finally, ~/.vimrc is the catchall for general editor settings. Use it for setting defaults that can be overridden for particular file types. For a comprehensive overview of settings you can choose in .vimrc, run :options.

Third-party plugins

Plugins are simply Vim scripts that must be put into the correct places in the runtimepath in order to execute. Installing them is conceptually easy: download the file(s) into place. The challenge is that it’s hard to remove or update some plugins because they litter subdirectories in the runtimepath with their scripts, and it can be hard to tell which plugin is responsible for which files.

“Plugin managers” evolved to address this need. Vim.org has had a plugin registry going back at least as far as 2003 (as identified by the Internet Archive). However it wasn’t until about 2008 that the notion of a plugin manager really came into vogue.

These tools add plugins’ separate directories to Vim’s runtimepath, and compile help tags for plugin documentation. Most plugin managers also install and update plugin code from the internet, sometimes in parallel or with colorful progress bars.

In chronological order, here is the parade of plugin managers. I based the date ranges on earliest and latest releases of each, or when no official releases are identified, on the earliest and latest commit dates.

  • Mar 2006 - Jul 2014 : Vimball (A distribution format and associated Vim commands)
  • Oct 2008 - Dec 2015 : Pathogen (Deprecated in favor of native vim packages)
  • Aug 2009 - Dec 2009 : Vimana
  • Dec 2009 - Dec 2014 : VAM
  • Aug 2010 - Nov 2010 : Jolt
  • Oct 2010 - Nov 2012 : tplugin
  • Oct 2010 - Feb 2014 : Vundle (Discontinued after NeoBundle ripped off code)
  • Mar 2012 - Mar 2018 : vim-flavor
  • Apr 2012 - Mar 2016 : NeoBundle (Deprecated in favor of dein)
  • Jan 2013 - Aug 2017 : infect
  • Feb 2013 - Aug 2016 : vimogen
  • Oct 2013 - Jan 2015 : vim-unbundle
  • Dec 2013 - Jul 2015 : Vizardry
  • Feb 2014 - Oct 2018 : vim-plug
  • Jan 2015 - Oct 2015 : enabler
  • Aug 2015 - Apr 2016 : Vizardry 2
  • Jan 2016 - Jun 2018 : dein.vim
  • Sep 2016 - Present : native in Vim 8
  • Feb 2017 - Sep 2018 : minpac
  • Mar 2018 - Mar 2018 : autopac
  • Feb 2017 - Jun 2018 : pack
  • Mar 2017 - Sep 2017 : vim-pck
  • Sep 2017 - Sep 2017 : vim8-pack
  • Sep 2017 - May 2019 : volt
  • Sep 2018 - Feb 2019 : vim-packager
  • Feb 2019 - Feb 2019 : plugpac.vim

The first thing to note is the overwhelming variety of these tools, and the second is that each is typically active for about four years before presumably going out of fashion.

The most stable way to manage plugins is to simply use Vim 8’s built-in functionality, which requires no third-party code. Let’s walk through how to do it.

First create two directories, opt and start, within a pack directory in your runtimepath.

mkdir -p ~/.vim/pack/foobar/{opt,start}

Note the placeholder “foobar.” This name is entirely up to you. It classifies the packages that will go inside. Most people throw all their plugins into a single nondescript category, which is fine. Pick whatever name you like; I’ll continue to use foobar here. You could theoretically create multiple categories too, like ~/.vim/pack/navigation and ~/.vim/pack/linting. Note that Vim does not detect duplication between categories and will double-load duplicates if they exist.

Packages in “start” get loaded automatically, whereas those in “opt” won’t load until specifically requested in Vim with the :packadd command. Opt is good for lesser-used packages, and keeps Vim fast by not running scripts unnecessarily. Note that there isn’t a counterpart to :packadd to unload a package.

For this example we’ll add the “ctrlp” fuzzy find plugin to opt. Download and extract the latest release into place:

curl -L https://github.com/kien/ctrlp.vim/archive/1.79.tar.gz \
	| tar zx -C ~/.vim/pack/foobar/opt

That command creates a ~/.vim/pack/foobar/opt/ctrlp.vim-1.79 folder, and the package is ready to use. Back in vim, create a helptags index for the new package:

:helptags ~/.vim/pack/foobar/opt/ctrlp.vim-1.79/doc

That creates a file called “tags” in the package’s doc folder, which makes the topics available for browsing in Vim’s internal help system. (Alternately you can run :helptags ALL once the package has been loaded, which takes care of all docs in the runtimepath.)

When you want to use the package, load it (and know that tab completion works for plugin names, so you don’t have to type the whole name):

:packadd ctrlp.vim-1.79

Packadd includes the package’s base directory in the runtimepath, and sources its plugin and ftdetect scripts. After loading ctrlp, you can press CTRL-P to pop up a fuzzy find file matcher.

Some people keep their ~/.vim directory under version control and use git submodules for each package. For my part, I simply extract packages from tarballs and track them in my own repository. If you use mature packages you don’t need to upgrade them often, plus the scripts are generally small and don’t clutter git history much.

Backups and undo

Depending on user settings, Vim can protect against four types of loss:

  1. A crash during editing (between saves). Vim can protect against this one by periodically saving unwritten changes to a swap file.
  2. Editing the same file with two instances of Vim, overwriting changes from one or both instances. Swap files protect against this too.
  3. A crash during the save process itself, after the destination file is truncated but before the new contents have been fully written. Vim can protect against this with a “writebackup.” To do this, it writes to a new file and swaps it with the original on success, in a way that depends on the “backupcopy” setting.
  4. Saving new file contents but wanting the original back. Vim can protect against this by persisting the backup copy of the file after writing changes.

Before examining sensible settings, how about some comic relief? Here are just a sampling of comments from vimrc files on GitHub:

  • “Do not create swap file. Manage this in version control”
  • “Backups are for pussies. Use version control”
  • “use version control FFS!”
  • “We live in a world with version control, so get rid of swaps and backups”
  • “don’t write backup files, version control is enough backup”
  • “I’ve never actually used the VIM backup files… Use version control”
  • “Since most stuff is on version control anyway”
  • “Disable backup files, you are using a version control system anyway :)”
  • “version control has arrived, git will save us”
  • “disable swap and backup files (Always use version control! ALWAYS!)”
  • “Turn backup off, since I version control everything”

The comments reflect awareness of only the fourth case above (and the third by accident), whereas the authors generally go on to disable the swap file too, leaving one and two unprotected.

Here is the configuration I recommend to keep your edits safe:

" Protect changes between writes. Default values of
" updatecount (200 keystrokes) and updatetime
" (4 seconds) are fine
set swapfile
set directory^=~/.vim/swap//

" protect against crash-during-write
set writebackup
" but do not persist backup after successful write
set nobackup
" use rename-and-write-new method whenever safe
set backupcopy=auto
" patch required to honor double slash at end
if has("patch-8.1.0251")
	" consolidate the writebackups -- not a big
	" deal either way, since they usually get deleted
	set backupdir^=~/.vim/backup//
end

" persist the undo tree for each file
set undofile
set undodir^=~/.vim/undo//

These settings enable backups for writes-in-progress, but do not persist them after successful write because version control etc etc. Note that you’ll need to mkdir ~/.vim/{swap,undodir,backup} or else Vim will fall back to the next available folder in the preference list. You should also probably chmod the folders to keep the contents private, because the swap files and undo history might contain sensitive information.

One thing to note about the paths in our config is that they end in a double slash. That ending enables a feature to disambiguate swaps and backups for files with the same name that live in different directories. For instance the swap file for /foo/bar will be saved in ~/.vim/swap/%foo%bar.swp (slashes escaped as percent signs). Vim had a bug until a fairly recent patch where the double slash was not honored for backupdir, and we guard against that above.

We also have Vim persist the history of undos for each file, so that you can apply them even after quitting and editing the file again. While it may sound redundant with the swap file, the undo history is complementary because it is written only when the file is written. (If it were written more frequently it might not match the state of the file on disk after a crash, so Vim doesn’t do that.)

Speaking of undo, Vim maintains a full tree of edit history. This means you can make a change, undo it, then redo it differently and all three states are recoverable. You can see the times and magnitude of changes with the :undolist command, but it’s hard to visualize the tree structure from it. You can navigate to specific changes in that list, or move in time with :earlier and :later which take a time argument like 5m, or the count of file saves, like 3f. However navigating the undo tree is an instance when I think a plugin – like undotreeis warranted.

Enabling these disaster recovery settings can bring you peace of mind. I used to save compulsively after most edits or when stepping away from the computer, but now I’ve made an effort to leave documents unsaved for hours at a time. I know how the swap file works now.

Some final notes: keep an eye on all these disaster recovery files, they can pile up in your .vim folder and use space over time. Also setting nowritebackup might be necessary when saving a huge file with low disk space, because Vim must otherwise make an entire copy of the file temporarily. By default the “backupskip” setting disables backups for anything in the system temp directory.

Vim’s “patchmode” is related to backups. You can use it in directories that aren’t under version control. For instance if you want to download a source tarball, make an edit and send a patch over a mailing list without bringing git into the picture. Run :set patchmod=.orig and any file ‘foo’ Vim is about to write will be backed up to ‘foo.orig’. You can then create a patch on the command line between the .orig files and the new ones.

Include and path

Most programming languages allow you to include one module or file from another. Vim knows how to track program identifiers in included files using the configuration settings path, include, suffixesadd, and includeexpr. The identifier search (see :help include-search) is an alternative to maintaining a tags file with ctags for system headers.

The settings for C programs work out of the box. Other languages are supported too, but require tweaking. That’s outside the scope of this article, see :help include.

If everything is configured right, you can press [i on an identifier to display its definition, or [d for a macro constant. Also when you press gf with the cursor on a filename, Vim searches the path to find it and jump there. Because the path also affects the :find command, some people have the tendency to add ‘**/*’ or commonly accessed directories to the path in order to use :find like a poor man’s fuzzy finder. Doing this slows down the identifier search with directories which aren’t relevant to that task.

A way to get the same level of crappy find capability, without polluting the path, is to just make another mapping. You can then press <Leader><space> (which is typically backslash space) then start typing a filename and use tab or CTRL-D completion to find the file.

" fuzzy-find lite
nmap <Leader><space> :e ./**/

Just to reiterate: the path parameter was designed for header files. If you want more proof, there is even a :checkpath command to see whether the path is functioning. Load a C file and run :checkpath. It will display filenames it was unable to find that are included transitively by the current file. Also :checkpath! with a bang dumps the whole hierarchy of files included from the current file.

By default path has the value “.,/usr/include,,” meaning the working directory, /usr/include, and files that are siblings of the active buffer. The directory specifiers and globs are pretty powerful, see :help file-searching for the details.

In my C ftplugin (more on that later), I also have the path search for include files within the current project, like ./src/include or ./include .

setlocal path=.,,*/include/**3,./*/include/**3
setlocal path+=/usr/include

The ** with a number like **3 bounds the depth of the search in subdirectories. It’s wise to add depth bounds where you can to avoid identifier searches that lock up.

Here are other patterns you might consider adding to your path if :checkpath identifies that files can’t be found in your project. It depends on your system of course.

  • More system includes: /usr/include/**4,/usr/local/include/**3
  • Homebrew library headers: /usr/local/Cellar/**2/include/**2
  • Macports library headers: /opt/local/include/**
  • OpenBSD library headers: /usr/local/lib/\*/include,/usr/X11R6/include/\*\*3

See also: :he [, :he gf, :he :find.

Edit ⇄ compile cycle

The :make command runs a program of the user’s choice to build a project, and collects the output in the quickfix buffer. Each item in the quickfix records the filename, line, column, type (warning/error) and message of each output item. A fairly idomatic mapping uses bracket commands to move through quickfix items:

" quickfix shortcuts
nmap ]q :cnext<cr>
nmap ]Q :clast<cr>
nmap [q :cprev<cr>
nmap [Q :cfirst<cr>

If, after updating the program and rebuilding, you are curious what the error messages said last time, use :colder (and :cnewer to return). To see more information about the currently selected error use :cc, and use :copen to see the full quickfix buffer. You can populate the quickfix yourself without running :make with :cfile, :caddfile, or :cexpr.

Vim parses output from the build process according to the errorformat string, which contains scanf-like escape sequences. It’s typical to set this in a “compiler file.” For instance, Vim ships with one for gcc in $VIMRUNTIME/compiler/gcc.vim, but has no compiler file for clang. I created the following definition for ~/.vim/compiler/clang.vim:

" formatting variations documented at
" https://clang.llvm.org/docs/UsersManual.html#formatting-of-diagnostics
"
" It should be possible to make this work for the combination of
" -fno-show-column and -fcaret-diagnostics as well with multiline
" and %p, but I was too lazy to figure it out.
"
" The %D and %X patterns are not clang per se. They capture the
" directory change messages from (GNU) 'make -w'. I needed this
" for building a project which used recursive Makefiles.

CompilerSet errorformat=
	\%f:%l%c:{%*[^}]}{%*[^}]}:\ %trror:\ %m,
	\%f:%l%c:{%*[^}]}{%*[^}]}:\ %tarning:\ %m,
	\%f:%l:%c:\ %trror:\ %m,
	\%f:%l:%c:\ %tarning:\ %m,
	\%f(%l,%c)\ :\ %trror:\ %m,
	\%f(%l,%c)\ :\ %tarning:\ %m,
	\%f\ +%l%c:\ %trror:\ %m,
	\%f\ +%l%c:\ %tarning:\ %m,
	\%f:%l:\ %trror:\ %m,
	\%f:%l:\ %tarning:\ %m,
	\%D%*\\a[%*\\d]:\ Entering\ directory\ %*[`']%f',
	\%D%*\\a:\ Entering\ directory\ %*[`']%f',
	\%X%*\\a[%*\\d]:\ Leaving\ directory\ %*[`']%f',
	\%X%*\\a:\ Leaving\ directory\ %*[`']%f',
	\%DMaking\ %*\\a\ in\ %f

CompilerSet makeprg=make

To activate this compiler profile, run :compiler clang. This is typically done in an ftplugin file.

Another example is running GNU Diction on a text document to identify wordy and commonly misused phrases in sentences. Create a “compiler” called diction.vim:

CompilerSet errorformat=%f:%l:\ %m
CompilerSet makeprg=diction\ -s\ %

After you run :compiler diction you can use the normal :make command to run it and populate the quickfix. The final mild convenience in my .vimrc is a mapping to run make:

" real make
map <silent> <F5> :make<cr><cr><cr>
" GNUism, for building recursively
map <silent> <s-F5> :make -w<cr><cr><cr>

Diffs and patches

Vim’s internal diffing is powerful, but it can be daunting, especially the three-way merge view. In reality it’s not so bad once you take time to study it. The main idea is that every window is either in or out of “diff mode.” All windows put in diffmode (with :difft[his]) get compared with all other windows already in diff mode.

For example, let’s start simple. Create two files:

echo "hello, world" > h1
echo "goodbye, world" > h2

vim h1 h2

In vim, split the arguments into their own windows with :all. In the top window, for h1, run :difft. You’ll see a gutter appear, but no difference detected. Move to the other window with CTWL-W CTRL-W and run :difft again. Now hello and goobye are identified as different in the current chunk. Continuing in the bottom window, you can run :diffg[et] to get “hello” from the top window, or :diffp[ut] to send “goodbye” into the top window. Pressing ]c or [c would move between chunks if there were more than one.

A shortcut would be running vim -d h1 h2 instead (or its alias, vimdiff h1 h2) which applies :difft to all windows. Alternatively, load just h1 with vim h1 and then :diffsplit h2. Remember that fundamentally these commands just load files into windows and set the diff mode.

With these basics in mind, let’s learn to use Vim as a three-way mergetool for git. First configure git:

git config merge.tool vimdiff
git config merge.conflictstyle diff3
git config mergetool.prompt false

Now, when you hit a merge conflict, run git mergetool. It will bring Vim up with four windows. This part looks scary, and is where I used to flail around and often quit in frustration.

+-----------+------------+------------+
|           |            |            |
|           |            |            |
|   LOCAL   |    BASE    |   REMOTE   |
+-----------+------------+------------+
|                                     |
|                                     |
|             (edit me)               |
+-------------------------------------+

Here’s the trick: do all the editing in the bottom window. The top three windows simply provide context about how the file differs on either side of the merge (local / remote), and how it looked prior to either side doing any work (base).

Move within the bottom window with ]c, and for each chunk choose whether to replace it with text from local, base, or remote – or whether to write in your own change which might combine parts from several.

To make it easier to pull changes from the top windows, I set some mappings in my vimrc:

" shortcuts for 3-way merge
map <Leader>1 :diffget LOCAL<CR>
map <Leader>2 :diffget BASE<CR>
map <Leader>3 :diffget REMOTE<CR>

We’ve already seen :diffget, and here our bindings pass an argument of the buffer name that identifies which window to pull from.

Once done with the merge, run :wqa to save all the windows and quit. If you want to abandon the merge instead, run :cq to abort all changes and return an error code to the shell. This will signal to git that it should ignore your changes.

Diffget can also accept a range. If you want to pull in all changes from one of the top windows rather than working chunk by chunk, just run :1,$+1diffget {LOCAL,BASE,REMOTE}. The “+1” is required because there can be deleted lines “below” the last line of a buffer.

The three-way marge is fairly easy after all. There’s no need for plugins like Fugitive, at least for presenting a simplified view for resolving merge conflicts.

Finally, as of patch 8.1.0360, Vim is bundled with the xdiff library and can create diffs internally. This can be more efficient than shelling out to an external program, and allows for a choice of diff algorithms. The “patience” algorithm often produces more human-readable output than the default, “myers.” Set it in your .vimrc like so:

if has("patch-8.1.0360")
	set diffopt+=internal,algorithm:patience
endif

Buffer I/O

See if this sounds familiar: you’re editing a buffer and want to save it as a new file, so you :w newname. After editing some more, you :w, but it writes over the original file. What you want for this scenario is :saveas newname, which does the write but also changes the filename of the buffer for future writes. Alternately, the :file newname command will change the filename without doing a write.

It also pays off to learn more about the read and write commands. Becuase r and w are Ex commands, they work with ranges. Here are some variations you might not know about:

:w >>foo append the whole buffer to a file
:.w >>foo append current line to a file
:$r foo read foo into the end of the buffer
:0r foo read foo into the start, moving existing lines down
:.,$w foo write current line and below to a file
:r !ls read ls output into cursor position
:w !wc send buffer to wc and display output
:.!tr ‘A-Za-z’ ‘N-ZA-Mn-za-m’ apply ROT-13 to current line
:w|so % chain commands: write and then source buffer
:e! throw away unsaved changes, reload buffer
:hide edit foo edit foo, hide current buffer if dirty

Useless fun fact: we piped a line to tr in an example above to apply a ROT-13 cypher, but Vim has that functionality built in with the the g? command. Apply it to a motion, like g?$.

Filetypes

Filetypes are a way to change settings based on the type of file detected in a buffer. They don’t need to be automatically detected though, we can manually enable them to interesting effect. An example is doing hex editing. Any file can be viewed as raw hexadecimal values. GitHub user the9ball created a clever ftplugin script that filters a buffer back and forth through the xxd utility for hex editing.

The xxd utility was bundled as part of Vim 5 for convenience. The Vim todo.txt file mentions they want to make it more seamless to edit binary files, but xxd can take us pretty far.

Here is code you can put in ~/.vim/ftplugin/xxd.vim. Its presence in ftplugin means Vim will execute the script when filetype (aka “ft”) becomes xxd. I added some basic comments to the script.

" without the xxd command this is all pointless
if !executable('xxd')
	finish
endif

" don't insert a newline in the final line if it
" doesn't already exist, and don't insert linebreaks
setlocal binary noendofline
silent %!xxd -g 1
%s/\r$//e

" put the autocmds into a group for easy removal later
augroup ftplugin-xxd
	" erase any existing autocmds on buffer
	autocmd! * <buffer>

	" before writing, translate back to binary
	autocmd BufWritePre <buffer> let b:xxd_cursor = getpos('.')
	autocmd BufWritePre <buffer> silent %!xxd -r

	" after writing, restore hex view and mark unmodified
	autocmd BufWritePost <buffer> silent %!xxd -g 1
	autocmd BufWritePost <buffer> %s/\r$//e
	autocmd BufWritePost <buffer> setlocal nomodified
	autocmd BufWritePost <buffer> call setpos('.', b:xxd_cursor) | unlet b:xxd_cursor

	" update text column after changing hex values
	autocmd TextChanged,InsertLeave <buffer> let b:xxd_cursor = getpos('.')
	autocmd TextChanged,InsertLeave <buffer> silent %!xxd -r
	autocmd TextChanged,InsertLeave <buffer> silent %!xxd -g 1
	autocmd TextChanged,InsertLeave <buffer> call setpos('.', b:xxd_cursor) | unlet b:xxd_cursor
augroup END

" when filetype is set to no longer be "xxd," put the binary
" and endofline settings back to what they were before, remove
" the autocmds, and replace buffer with its binary value
let b:undo_ftplugin = 'setl bin< eol< | execute "au! ftplugin-xxd * <buffer>" | execute "silent %!xxd -r"'

Try opening a file, then running :set ft. Note what type it is. Then:set ft=xxd. Vim will turn into a hex editor. To restore your view, :set ft=foo where foo was the original type. Note that in hex view you even get syntax highlighting because $VIMRUNTIME/syntax/xxd.vim ships with Vim by default.

Notice the nice use of “b:undo_ftplugin” which is an opportunity for filetypes to clean up after themselves when the user or ftdetect mechanism switches away from them to another filetype. (The example above could use a little work because if you :set ft=xxd then set it back, the buffer is marked as modified even if you never changed anything.)

Ftplugins also allow you to refine an existing filetype. For instance, Vim already has some good defaults for C programming in $VIMRUNTIME/ftplugin/c.vim. I put these extra options in ~/.vim/after/ftplugin/c.vim to add my own settings on top:

" the smartest indent engine for C
setlocal cindent
" my preferred "Allman" style indentation
setlocal cino="Ls,:0,l1,t0,(s,U1,W4"

" for quickfix errorformat
compiler clang
" shows long build messages better
setlocal ch=2

" auto-create folds per grammar
setlocal foldmethod=syntax
setlocal foldlevel=10

" local project headers
setlocal path=.,,*/include/**3,./*/include/**3
" basic system headers
setlocal path+=/usr/include

setlocal tags=./tags,tags;~
"                      ^ in working dir, or parents
"                ^ sibling of open file

" the default is menu,preview but the preview window is annoying
setlocal completeopt=menu

iabbrev #i #include
iabbrev #d #define
iabbrev main() int main(int argc, char **argv)

" add #include guard
iabbrev #g _<c-r>=expand("%:t:r")<cr><esc>VgUV:s/[^A-Z]/_/g<cr>A_H<esc>yypki#ifndef <esc>j0i#define <esc>o<cr><cr>#endif<esc>2ki

Notice how the script uses “setlocal” rather than “set.” This applies the changes to just the current buffer rather than the whole Vim instance.

This script also enables some light abbreviations. Like I can type #g and press enter and it adds an include guard with the current filename:

#ifndef _FILENAME_H
#define _FILENAME_H

/* <-- cursor here */

#endif

You can also mix filetypes by using a dot (“.”). Here is one application. Different projects have different coding conventions, so you can combine your default C settings with those for a particular project. The OpenBSD source code follows the style(9) format, so let’s make a special openbsd filetype. Combine the two filetypes with :set ft=c.openbsd on relevant files.

To detect the openbsd filetype we can look at the contents of buffers rather than just their extensions or locations on disk. The telltale sign is that C files in the OpenBSD source contain /* $OpenBSD: in the first line.

To detect them, create ~/.vim/after/ftdetect/openbsd.vim:

augroup filetypedetect
        au BufRead,BufNewFile *.[ch]
                \  if getline(1) =~ 'OpenBSD;'
                \|   setl ft=c.openbsd
                \| endif
augroup END

The Vim port for OpenBSD already includes a special syntax file for this filetype: /usr/local/share/vim/vimfiles/syntax/openbsd.vim. If you recall, the /usr/local/share/vim/vimfiles directory is in the runtimepath and is set aside for files from the system administrator. The provided openbsd.vim script includes a function:

function! OpenBSD_Style()
	setlocal cindent
	setlocal cinoptions=(4200,u4200,+0.5s,*500,:0,t0,U4200
	setlocal indentexpr=IgnoreParenIndent()
	setlocal indentkeys=0{,0},0),:,0#,!^F,o,O,e
	setlocal noexpandtab
	setlocal shiftwidth=8
	setlocal tabstop=8
	setlocal textwidth=80
endfun

We simply need to call the function at the appropriate time. Create ~/.vim/after/ftplugin/openbsd.vim:

call OpenBSD_Style()

Now opening any C or header file with the characteristic comment at the top will be recognized as type c.openbsd and will use indenting options that conform with the style(9) man page.

Don’t forget the mouse

This is a friendly reminder that despite our command-line machismo, the mouse is in fact supported in Vim, and can do some things more easily than the keyboard. Mouse events work even over SSH thanks to xterm turning mouse events into stdin escape codes.

To enable mouse support, set mouse=n. Many people use mouse=a to make it work in all modes, but I prefer to enable it only in normal mode. This avoids creating visual selections when I click links with a keyboard modifier to open them in my browser.

Here are things the mouse can do:

  • Open or close folds (when foldcolumn > 0).
  • Select tabs (beats gt gt gt…)
  • Click to complete a motion, like d<click!>. Similar to the easymotion plugin but without any plugin.
  • Jump to help topics with double click.
  • Drag the status line at the bottom to change cmdheight.
  • Drag edge of window to resize.
  • Scroll wheel.

Misc editing

This section could be enormous, but I’ll stick to a few tricks I learned. The first one that blew me away was :set virtualedit=all. It allows you to move the cursor anywhere in the window. If you enter characters or insert a visual block, Vim will add whatever spaces are required to the left of the inserted characters to keep them in place. Virtual edit mode makes it simple to edit tabular data. Turn it off with :set virtualedit=.

Next are some movement commands. I used to rely a lot on } to jump by paragraphs, and just muscle my way down the page. However the ] character makes more precise motions: by function ]], scope ]}, paren ‘])’, comment ]/, diff block ]c. This series is why the quickfix mapping ]q mentioned earlier fits the pattern so well.

For big jumps I used to try things like 1000j, but in normal mode you can actually just type a percentage and Vim will go there, like 50%. Speaking of scroll percentage, you can see it at any time with CTRL-G. Thus I now do :set noruler and ask to see the info as needed. It’s less cluttered. Kind of the opposite of the trend of colorful patched font powerlines.

After jumping around between tags, files, or within a file, there are some commands to get your bearings. Try :ls, :tags, :jumps, and :marks. Jumping through tags actually creates a stack, and you can press CTRL-T to pop one back. I used to always press CTRL-O to back out of jumps, but it is not as direct as popping the tag stack.

In a project directory that has been indexed with ctags, you can open the editor directly to a tag with -t, like vim -t main. To find tags files more flexibly, set the tags configuration variable. Note the semicolon in the example below that allows Vim to search the current directory upward to the home directory. This way you could have a more general system tags file outside the project folder.

set tags=./tags,**5/tags,tags;~
"                          ^ in working dir, or parents
"                   ^ in any subfolder of working dir
"           ^ sibling of open file

There are some buffer tricks too. Switching to a buffer with :bu can take a fragment of the buffer name, not just a number. Sometimes it’s harder to memorize those numbers than remember the name of a source file. You can navigate buffers with marks too. If you use a capital letter as the name of a mark, you can jump to it across buffers. You could set a mark H in a header, C in a source file, and M in a Makefile to go from one buffer to another.

Do you ever get mad after yanking a word, deleting a word somewhere else, trying paste the first word in, and then discovering your original yank is overwritten? The Vim registers are underappreciated for this. Inspect their contents with :reg. As you yank text, previous yanks are rotated into the registers "0 - "9. So "0p pastes the next-to-last yank/deletion. The special registers "+ and "* can copy/paste from/to the system clipboard. They usually mean the same thing, except in some X11 setups that distinguish primary and secondary selection.

Another handy hidden feature is the command line window. It it’s a buffer that contains your previous commands and searches. Bring it up with q: or q/. Once inside you can move to any line and press enter to run it. However you can also edit any of the lines before pressing enter. Your changes won’t affect the line (the new command will merely be added to the bottom of the list).

This article could go on and on, so I’m going to call it here. For more great topics, see these help sections: views-sessions, viminfo, TOhtml, ins-completion, cmdline-completion, multi-repeat, scroll-cursor, text-objects, grep, netrw-contents.

vim logo

vim logo

July 18, 2019

Awn Umar (awn)

encrypting secrets in memory July 18, 2019 12:00 AM

In a previous post I talked about designing and implementing an in-memory data structure for storing sensitive information in Go. The latest version of memguard adds something new: encryption.

But why? Well, there are limitations to the old solution of using guarded heap allocations for everything.

  1. A minimum of three memory pages have to be allocated for each value: two guard pages sandwiching n≥1 n \geq 1 n1 data pages.
  2. Some systems impose an upper limit on the amount of memory that an individual process is able to prevent from being swapped out to disk.

Memory layout of guarded heap allocation.

Typical layout of a 32 byte guarded heap allocation.

So it is worth looking into the use of encryption to protect information. After all, ciphertext does not have to be treated with much care and authentication guarantees immutability for free. The problem of recovering a secret is shifted to recovering the key that protects it.

But there is the obvious problem. Where and how do you store the encryption key?

We use a scheme described by Bruce Schneier in Cryptography Engineering. The procedure is sometimes referred to as a Boojum. I will formally define it below for convenience.

Define B={i:0≤i≤255} B = \{i : 0 \leq i \leq 255\} B={i:0i255} to be the set of values that a byte may take. Define h:Bn→B32 h : B^n \to B^{32} h:BnB32 for n∈N0 n \in \mathbb{N}_0 nN0 to be a cryptographically-secure hash function. Define the binary XOR operator ⊕:Bn×Bn→Bn \oplus : B^n \times B^n \to B^n :Bn×BnBn.

Suppose k∈B32 k \in B^{32} kB32 is the key we want to protect and Rn∈B32 R_n \in B^{32} RnB32 is some random bytes sourced from a suitable CSPRNG. We initialise the two partitions:

x1=R1y1=h(x1)⊕k \begin{aligned} x_1 &= R_1\\ y_1 &= h(x_1) \oplus k\\ \end{aligned} x1y1=R1=h(x1)k

storing each inside its own guarded heap allocation. Then every m m m milliseconds we overwrite each value with:

xn+1=xn⊕Rn+1=R1⊕R2⊕⋯⊕Rn+1yn+1=yn⊕h(xn)⊕h(xn+1)=h(xn)⊕h(xn)⊕h(xn+1)⊕k=h(xn+1)⊕k \begin{aligned} x_{n+1} &= x_n \oplus R_{n+1}\\ &= R_1 \oplus R_2 \oplus \cdots \oplus R_{n+1}\\ y_{n+1} &= y_n \oplus h(x_n) \oplus h(x_{n+1})\\ &= h(x_n) \oplus h(x_n) \oplus h(x_{n+1}) \oplus k\\ &= h(x_{n+1}) \oplus k\\ \end{aligned} xn+1yn+1=xnRn+1=R1R2Rn+1=ynh(xn)h(xn+1)=h(xn)h(xn)h(xn+1)k=h(xn+1)k

and so on. Then by the properties of XOR,

k=h(xn)⊕yn=h(xn)⊕h(xn)⊕k=0⊕k=k \begin{aligned} k &= h(x_n) \oplus y_n\\ &= h(x_n) \oplus h(x_n) \oplus k\\ &= 0 \oplus k\\ &= k\\ \end{aligned} k=h(xn)yn=h(xn)h(xn)k=0k=k

It is clear from this that our iterative overwriting steps do not affect the value of k k k, and the proof also gives us a way of retrieving k k k. My own implementation of the protocol in fairly idiomatic Go code is available here.

An issue with the Boojum scheme is that it has a relatively high overhead from two guarded allocations using six memory pages in total, and we have to compute and write 64 bytes every m m m milliseconds. However we only store a single global key, and the overhead can be tweaked by scaling m m m as needed. Its value at the time of writing is 8 milliseconds.

The authors of the Boojum claim that it defends against cold boot attacks, and I would speculate that there is also some defence against side-channel attacks due to the fact that k k k is split across two different locations in memory and each partition is constantly changing. Those attacks usually have an error rate and are relatively slow.

OpenBSD added a somewhat related mitigation to their SSH implementation that stores a 16 KiB (static) “pre-key” that is hashed to derive the final key when it is needed. I investigated incorporating it somehow but decided against it. Both schemes have a weak point when the key is in “unlocked” form so mimimising this window of opportunity is ideal.

In memguard the key is initialised when the program starts and then hangs around in the background—constantly flickering—until it is needed. When some data needs to be encrypted or decrypted, the key is unlocked and used for the operation and then it is destroyed.

Diagram showing the high-level structure of the scheme.

High-level overview of the encryption scheme.

The documentation provides a relatively intuitive guide to the package’s functionality. The Enclave stores ciphertext, the LockedBuffer stores plaintext, and core.Coffer implements the Boojum. Examples are available in the examples sub-package.

The most pressing issue at the moment is that the package relies on cryptographic primitives implemented by the Go standard library which does not secure its own memory and may leak values that are passed to it. There has been some discussion about this but for now it seems as though rewriting crucial security APIs to use specially allocated memory is the only feasible solution.

If you have any ideas and wish to contribute, please do get in touch or open a pull request.

Pepijn de Vos (pepijndevos)

VHDL to PCB July 18, 2019 12:00 AM

When learning to program FPGAs using VHDL or Verilog, you also learn that these hardware description languages can be used to design ASICs (application specific integrated circuit). But this is only something big corporations with millions of dollars can afford, right? Even though I later learned it only costs thousands, not millions, to make an ASIC on an older process, it is still far away from hobby budgets.

I had been keeping an eye on Yosys, the open source HDL synthesis tool, which can apparently do ASIC by giving it a liberty file that specifies the logic cells your foundry supports. Meanwhile I also toyed with the idea of making a 7400 series computer, and I wondered if you could write a liberty file for 7400 chips. I had kind of dismissed the idea, but then ZirconiumX came along and did it.

It suffices to say this revived my interest in the idea and a lively discussion and many pull requests followed. First some small changes, then simulations to verify the synthesized result is still correct, and finally a KiCad netlist generator.

You see, generating a Yosys netlist is nice, but eventually these 7400 chips have to end up on a PCB somehow. Normally you draw your schematic in Eeschema, generate a netlist, and import that to Pcbnew. But instead I used skidl to generate the netlist directly. Then all there is to do is add the inputs and outputs and run the autorouter (or do it manually of course).

I decided to do a proof-of-concept “application specific interconnected circuit”, with the goal of making something fun in under 10 chips. (a Risc-V CPU currently sits at about 850) I settled on a fading PWM circuit to drive an LED. I manually added a 555-based clock, and ordered a PCB for a few bucks. A few weeks later, this was the result. It worked on the first try! This feeling is even more amazing than with software, and shows that as long as there are no compiler/library bugs or DRC errors, logic simulations are a good way to prove your PCB design.

To follow along at home you need to install Yosys. A recent release might work, but it’s getting better every day, so building from source is recommended. Then you can just git clone 74xx-liberty and go. There are a number of Verilog programs in benchmarks in case you’d rather make PicoRV32 in 7400 chips.

cd stat
make pwmled.stat # synthesize and run stat
../ic_count.py pwmled.stat # count number of chips used
cd ../sim
make pwmled.vcd # synth to low-level verilog and simulate
gtkwave pwmled.vcd # show test bench results
cd ../kicad
make pwmled.net # generate kicad netlist

But this was all done in Verilog, so where is the VHDL, you might wonder. Well, Yosys does not really support VHDL yet, but Tristan Gingold is hard at work making GHDL synthesize VHDL as a Yosys plugin. I think this is very important work, so I’ve been contributing there as well. After some pull requests I was able to port the breathing LED to VHDL.

Getting VHDL to work in Yosys is a bit of effort. First you need to compile GHDL, which requires installing a recent version of GNAT. Then you need to install ghdlsynth-beta as a plugin, allowing you to run yosys -m ghdl. My fork of 74xx-liberty contains additional make rules for doing the above synthesization for VHDL files, which does something like this before calling the 7400 synthesis script.

cd stat
ghdl -a ../benchmarks/pwmled.vhd # analyse VHDL file
yosys -m ghdl -p "ghdl pwmled; show" # load pwmled entity, show graph

yosys dot graph

A huge thank you to all the people working tirelessly to make open source hardware design a reality. You’re awesome!

July 16, 2019

Jeremy Morgan (JeremyMorgan)

Forget What You've Heard, Now Is the Best Time to Become a Coder July 16, 2019 04:15 PM

Do you want to be a coder? Are you on the fence about trying it? Nervous to get started?

The time is now. Time to pull the trigger. 

There has never been a better time to become a coder.

And I'm going to tell you how to get started.  

My History as a Coder

I started writing code professionally in 2002. Before that, I was building (terrible) websites for myself and friends. I even ran a business for a few years crapping out HTML/Perl sites in the late 90s for good money. I built shoddy software in Visual Basic for businesses.

To learn how to do this, I bought expensive books on HTML, Perl, Visual Basic and Unix. I read them cover to cover and marked them up. Google didn’t exist yet. I consulted Usenet news groups where they insulted you for asking "stupid questions" and figured out most of the stuff the hard way. I learned Unix so I could host my customer's sites and destroyed a lot of stuff before figuring it out. 

I was living in a rural area and had very few friends that were even in to this kind of thing so I had few people to bounce ideas and questions off of. I hadn't yet gone to college and met other technical folks. 

I'm not telling you this so you'll think I'm awesome or some kind of grizzled veteran who is better than you. It was hard, and I wasn't very good for many years, but I still got the job done. I spent countless nights chugging Surge and hacking away trying to figure out things. I got good. I figured things out. But it sucked.

 

Your History

Your story doesn't have to go that way. The world is at your fingertips. You have all the information to get where I am right in front of you. Some grizzled veterans like myself say things like "new developers don't even have to work for it" and pretend like we're so much better because of our struggles. Not true. In reality because you have this information readily available, you'll get better faster. The developer I was at the 3 year mark will be a rookie compared to you.


Today's beginning developers are positioned to be the best generation of developers yet.

That's part of the reason that NOW is the best time to start. You have Google, Stack Overflow, Dev.to, Social Media, Pluralsight, YouTube, you name it. NOW is exponentially better than 1995 as far a starting point goes. 

The Market: We need developers! 

Another great reason now is the time: we are in a talent shortage. Everyone I talk to is looking for developers. Junior, Senior, mid-level, whatever. Can you code? You'll get a job. It doesn't even have to be in the language they're using. 

In the mid 2000s being a developer I had to know everything under the sun to get in the door to interview. They had crowds lined up fighting for every job. In 2019 if you learn the basics, throw some projects on GitHub and start sending out resumes you'll get that call. 

According to code.org, there are 504,199 open computing jobs nationwide. There were 63,744 computer science graduates entering the workforce last year. 

There are more jobs available than coders to fill them.

The numbers are in your favor. 

What do you need? 

  • Do you need a Computer Science Degree? : No
  • Do you need expensive training? No
  • Do you need a MacBook Pro?: No 

You don't need any of these things. If you want to get started, you can do it with a ChromeBook. 

For instance: 

Want to learn JavaScript?

  1. Go to JavaScript.Info and start from the beginning. 
  2. Create an account at JSFiddle and start hacking away. 

Don't worry about details like getting your own website, server, etc. Just do it.   Want to learn some Python? You can install it on your old Dell laptop and start going. CSS? PHP? C#? These are available to download free and will run on nearly any machine built in the last 10 years. 

At some point you will want a faster computer and more in depth training. But to get started this is all you'll need. 

Next Steps

Disclaimer: So, remember how I talked about how hard it was to learn tech? It's what drives me now. This stage of my career is dedicated towards teaching people, and improving their tech skills. In the interest of transparency I work for a company called Pluralsight which is the leading tech skills platform. This is not the only reason I recommend them, I was a customer long before becoming and employee and it's boosted my career tremendously.

So for next steps after taking some tutorials and getting your feet wet, do the following to break into this industry. 

Phase 1

  • Determine what you want to develop (Web, Mobile, Desktop)
  • Find as many tutorials as you can on the subject.  (You can contact me if you need help with this)
  • Create a GitHub account
  • Start uploading code samples you build with tutorials

Phase 2

  • Start a project -No matter how stupid it seems. Make an app to store your favorite jokes, or todo lists.
  • Create a HackerRank account - Start tackling some puzzles. 
  • Pick problems from Project Euler and write code to solve them. 
  • Get to know Stack Overflow - Search it when you have a problem and answer questions if you know it!

Once you get to this point, you'll start feeling comfortable. Keep working and improving your craft. Dig deeper on the tech you've chosen. Build things, even small things. Then think about getting your first job. 

Phase 3

And many more. If you're targeting a specific role, this is a great way to get your skills up. 

  • Build something useful - Build something that solves a problem for you, a friend or your employer. Nothing teaches like you like building something real.
  • Start teaching - Nothing helps you learn a subject like teaching it. When you get comfortable with your knowledge, share it.

Conclusion

Stop making excuses and don't listen to people who tell you you can't or shouldn't learn to code. If you want it bad enough you can do it and get paid for it. Teaching others is a deep passion of mine so if you get stuck on something or need some advice, hit me up I'd be glad to help. 

Now start learning!  



What is your DevOps IQ?

what's your devops score


My Devops Skill IQ is 232. Can you beat it? Take the test now to find out your Devops IQ score!!

Bogdan Popa (bogdan)

Announcing deta July 16, 2019 07:00 AM

I just released the first version of deta, a Racket library for mapping between database tables and structs. A bit like Elixir’s Ecto.

You can install it from the package server with:

raco pkg install deta

Check it out!