Planet Crustaceans

This is a Planet instance for lobste.rs community feeds. To add/update an entry or otherwise improve things, fork this repo.

February 22, 2019

Artemis (Artemix)

2019 and this blog February 22, 2019 06:12 PM

As you can notice, articles are quite slow to come. The reason is mainly due to me being a slow writer, and not having a lot of topics to write on.

Another problem is school, as there's definitely more work than the previous year, giving me less time to maintain this blog, together with some other projects.

# Website changes and this blog

A project I'm currently walking towards is to rework most of my websites.

I have a lot of very small static websites for lots of different things, and it doesn't make a lot of sense in my opinion to spread them so much, so I'll work on joining together a lot of them.

Part of this work has already been done, where several small documentation websites were finally brought together at docs.artemix.org.

I have a few websites left:

In the Portfolios section, there's some heavy duplication, so I intend to remove all of them, and merge them into a single simple page, which will be simpler to find (and link back to me), and much cleaner to maintain.

I also intend to merge my devlog inside this blog, and to, once again (I know), re-work this blog.

This includes a re-design, a change in the categorization system, and a removal of the comment space (since the service I used is pretty much closing his doors).

# A new hope

The first goal is to merge my Devlog into my blog, which should be followed by the integration of my folio (a.k.a. CV) into the blog website, then followed by the integration of a small "About" page.

To organise all written articles, there'll be some categorization.

The design'll probably change to become simpler; I'll probably use a third-party lightweight CSS library, like skeleton or milligram for the base CSS.

I'll probably redevelop the static site generator, not in nodejs but in go, to be simpler and easier to maintain, because it's a real hell as of now, especially around file manipulation.

# Articles access

The direct articles' links won't change, but the RSS feeds will be differently structured: the global one will continue to work, but a new feed by category will be created.

# Overall

Overall, a lot of planned change, which will take some time, especially around the static site generation (with a much more complex layout), but I personally think it's really worth the change and amount of work.

For now, I'll work on that, so I'll see you later!

Noon van der Silk (silky)

2018s Crazy Ideas February 22, 2019 12:00 AM

Posted on February 22, 2019 by Noon van der Silk

I can’t believe I forgot to do this at the start of the year!

Let’s look back over 2018’s ideas:

The Ideas

  1. using ai to reduce medical dosages: i.e. imagine you need such-and-such amount of radiation in order for certain whatever to be seen in some scan can you lower the dosage and then use some ai technique to enhance the quality of the image?
  2. something about shape-based reasoning: i.e. the fact that some grammar-parsing problem teems like a good fit for tensor-network works because they both have the “shape” of a tree another example would be thinking of solving a problem of identifying different types of plants as it fits in the “shape” of a classifier but these things could potentially have other shapes? and therefore fit into other kinds of problems?
  3. a q&a bot that can answer tests on the citizenship test: Q: What is our home girt by? A: ..
  4. buddhist neural network: it features significant self-attention and self-awareness it performs classification, but predicts the same class for every input as it has a nondual mind it’s loss function features a single term for each of the 6 paramitas: - Generosity: to cultivate the attitude of generosity. - Discipline: refraining from harm. - Patience: the ability not to be perturbed by anything. - Diligence: to find joy in what is virtuous, positive or wholesome. - Meditative concentration: not to be distracted. - Wisdom: the perfect discrimination of phenomena, all knowable things.
  5. music-to-image: say you’re listening to greek music you want to generate an image of a singer singing on some clifftop on greece under an olive tree near a vineyard surely a simple matter of deep learning
  6. code story by word frequency: take all the words in a code repo, order them by frequency, then match that up to some standard book, then remap the code according to the frequency
  7. generalisation of deutch-jozsa’s problem: here - https://en.wikipedia.org/wiki/Deutsch%E2%80%93Jozsa_algorithm generalise it so that we have multiple f’s that have different promises; i.e. i’m constant “50% of the time”. what now?
  8. analogy-lang: reading https://maetl.net/membrane-oriented-programming by @maetl i had an idea for a programming language consider two types, “person” and “frog”, let’s say there are at least two problems; one is the one in the article - it’s hard to come up with a complete list of methods/properties that should exactly define one of these things. in surfaces and essences they argue that in reality no categorisation has perfectly defined boundaries; so how to define the types? another problem is what happens if i want to build a thing that is both “person-like” and “frog-like”? you can’t. especially not if the differences are far away (i.e. is frog hopping like “walking”? should it be the implementation of a “walk” method? probably not; but a “move” method? probably? but what about less-clear things? and doesn’t it depend on what your merging with) in this way it seems like it’s impossible to come up with strict types for anything so here’s an idea: analogy-lang: let’s you define types in a much more relaxed way; “this thing is like that thing in these ways, but different in these ways”
  9. multi-layered network: train some network f to produce y_1 from x_1 f(x_1) = y_1 then, wrap that in a new network, g, that produces y_1 and y_2 from x_1 and x_2 g(x_1, x_2) = ( f(x_1,), g’(x_1, x_2) ) and so on. interesting
  10. psychadelics for ai: ideas: - locate some kind of “default-mode network” in your model and inhibit it - after training; allow many more connections - have two modes of network; one this “high-entropy” learning one, which prunes down to a more efficient one that can’t learn but can decide quickly
  11. ultimate notification page: it’s just a page with a bunch of iframes to all your different websites where you get the little notification things and then it just tiles them; showing in-which places you have notifications.
  12. use deep learning to replace a movie character with yourself and your own acting: 1. semantic segmentation to remove an actor 2. film yourself saying their lines 3. plug yourself back into the missing spot 4. dress yourself in the appropriate clothes 5. adjust the lighting 6. ??? 7. profit!
  13. paramterised papers: “specifically, this model first generates an aligned position p_t for each target word at time t” show me this sentence with t = whatever and p = some dimension
  14. on a slide-deck, have a little small game that plays out in a tiny section of each slide
  15. use autocomplete and random phrases to guess things about people: “i love you …” find out who they love “give me …” find out what they want and so on
  16. visualise the difference between streaming and shuffling sampling algorithms: streaming -> for low numbers, misses items shuffling -> for low numbers, bad distribution across indices that are shuffled. @dseuss @martiningram
  17. symbolic tensorflow: so i can do convs and explain them really easily
  18. the “lentictocular”: uses lenticular technology on a sphere with AI so that it watches your gaze and moves itself accordingly so that it always displays the appropriate time
  19. ai email judgement front: intercepts all your emails for every email, it decides the optimal time that someone will respond it sends it at that time so that they respond
  20. “growth maps” for determining affected areas of projects w.r.t. a pattern language
  21. friend tee: lights up when other friends are nearby
  22. lenticular business cards: this is already done by many people
  23. innovative holiday inventor: thinks up cool holidays
  24. buddhist twitter: there’s only one account, and no password
  25. programming ombudsman: @sordina @kirillrdy
  26. the computational complexity of religion: given various religious abilities, what computational problems can you solve? what are the implications on computational complexity by buddhism? and so on.
  27. spacetime calendar: we have calendars for dates but they don’t often contain space constraints so why not a space-time calendar, defined in some kind of light cone?
  28. app to check consistency of items before you leave: it’s an app you configure it to be aware of your keys, laptop, wallet, glasses then, as you leave your house, it can inform you of the status of those items: - “hey, your computer is at home” - “all g, your glasses are at work” etc @gacafe
  29. gradient tug-of-war demonstration: given a function f(x,y) = x + y then if we have competing loss functions then it would be nice to visualise the gradient flow as a tug of war
  30. a website that is entirely defined in the scroll bars/url link bar, whatever: you can move pages by moving your mouse to different parts of the scroll bars and so on in that fashion
  31. quantum calendar: it’s a calendar where on any given day in the future, items can be scheduled at the same time. but up to some limit (say 1 week) the items get collapsed and locked in @silky @dseuss
  32. streaming terminal graph receiving updates over mqtt: then, can use it to plot tensorboard logs to the terminal instead of tensorboard using blessed + blessed-contrib seems to be the easiest way - https://github.com/yaronn/blessed-contrib/blob/master/examples/line-random-colors.js just need to put in the mqtt part and update the data that way.
  33. ai escape room: this is an idea of dave build an escape room controlled entirely by ai the only way out is by interacting with the machine it can control everything: heating, doors, whatever
  34. programmable themepark: here’s a ride; how you interact with it is defined via your own programs you play minigolf, but instead of a club you use programming @sordina
  35. graph+weights to neuronal net rendering: https://github.com/BlueBrain/NeuroMorphoVis
  36. Arbitrary-Task Programming: given that programming is just arranging symbols, and we can use deep learning to interpret the real world into symbols, then it’s possible to do programming by performing arbitrary tasks i.e. any job can be programming, if we can build the deep-learning system that converts actions in that job into symbols in a programming language
  37. Sitcom-Lang: it’s a pre-processor, or something, for an arbtrariy language whenever a symbol is defined, that symbol is embued with a soul and a “nature”. it starts to have wants and needs; and those must be satisfied in order for it to stay in it’s present form (i.e. as a “for loop”), otherwise it might change (i.e. to be a “while” loop, or maybe even a string instead). all the symbols will interact with each other, and in that way a program will be made @sordina
  38. brain2object2teeshirt: this - https://scirate.com/arxiv/1810.02223 - but once it’s decided on the object, it gets rendered on your LED tee-shirt @geekscape @sordina
  39. pix2pix sourcecode2cat: generate pictures of source code convert to pictures of cat instant machine for generating cat pictures from code what cat does your code look like? @sordina
  40. physical xmonad: use the uarm to be a “physical” xmonad you want to write on some piece of paper? no worries, the uarm will re-arrange your physical desk so that everything is conveniently arranged to do that @sordina
  41. collaborative painting in the style of christopher alexander: it has 3 parts done by 3 artists i draw the left part; you draw the middle, and it has to interact coherently with whatever i’ve drawn; then another person draws the right side, again, it must interact that’s it.
  42. lego laptops: laptops that plug together in a lego-like way
  43. business version of 30 kids vs 2 professional soccer players: 30 grads vs 2 ceos 30 ceos vs 2 grads etc.
  44. shops in parks: would make the parks safer/nicer, because people would be in them more could limit the type of shops, and their size, but would be a nice way to build a bit more of a community feeling in them
  45. an icon next to your public email address that indicates how many unread emails you have: then people can gauge what will happen if they email you
  46. ml for configuring linux: “what file should i look at to change default font sizes?” “how can i set up my new gpu? what settings should i set?”
  47. water-t-shirt: the essence of a water-bed, in t-shirt form!
  48. deep antiques roadshow: the idea explains itself
  49. a being that is by-default inherently abstract, instead of inherently practical like us: for them, being practical would be really hard by default they live at the other end of the abstraction spectrum
  50. business card bowl: through all the business cards into a bowl; each day, call them, if they don’t want to do business, throw the card out
  51. “live blog”: whenever someone visits your blog, instead of reading articles, they get to open a chat window with you in your terminal then you tell them what you’ve been up to; and they can ask you questions
  52. software art: take all the source code; stack them up as if the line count is one slice, that’s the structure
  53. t-shirt whiteboard: in essence i.e. you can draw on the t-shirt, and the writing just washes out the next time then you can design whatever you want would this just work?
  54. physics simulation + diagrams: would be great to define things such as “two pieces of rope inter-twined”, and then “drop” them, but then let that resulting expression become a haskell diagrams Diagram, so that you can then do diagram stuff with it
  55. git version flattener: clone a git repo at every revision, into some folder.
  56. see-through jacket that is also warm: optionally also magnetic @sordina
  57. magnetic glass: @sordina
  58. heated keyboard: keeps your fingers warm
  59. run an experiment where monkeys/dogs/whatever are encouraged to learn some kind of programming to solve a task: i.e. a monkey gets 1 food package per day but if learns to program, using the tools provided to it, ( something like a giant physical version of scratch ) then it gets 3 food packages in some sense people have tried this, with them solving problems, but has anyone tried it where the tool they use to solve the problem is general, and can be applied to other areas of their life?
  60. tree to code: physical trees 1. order trees by the number of leaves 2. order code by the number of statements train a deeplearning network to map between these things then, trees can write computer programs @sordina
  61. ethical algorithms testing ground: related to the last two #409 #408 basically, people can sign up to be ethical tester algorithms can join to provide games for people to play how would it work?
  62. ethical testers: beta testers game testers ethics testers
  63. simulation for ethical machine learning problems: consider the situation: “how do i know if this algorithm X is unethical?” well, instead of waiting for the salespeople to tell you, you could just have it run in a simulated environment and see if it’s unethical by the way that it acts.
  64. minecraft file browser: walk around your filesystem in 3d
  65. ocr clipboard copy and paste: select an image region, send it to some text api thing, get the text back in the clipboard
  66. low-powered de-colourisation network: learns to convert colour -> black and white if it doesn’t do a good job, it’ll look awesome
  67. physical quine: a robot that can type on the computer and write code that writes the program that writes itself
  68. deep learning “do” networks: can you include the “do calculus” into neural networks somehow? to make it do some causal things?
  69. plot websites on cesium map, browse the internet that way.: web-world
  70. animated colour schemes for vim: the colour scheme rotates as you code
  71. a tale of three dickens: the movie: it’s an auto-generated movie from locally-coherent slices instead of the book, we make a movie, where all the scenes in the movie are interspersed based on “local coherence” i.e. from two movies select two people having a conversation with someone named bill or, flick between scenes at the beach @sordina
  72. revolutionary walls: the floor is fixed; but the walls are a tube you pull on some part of the wall to rotate it @tall-josh
  73. activation function composer: or more generally, a function composer 1. what does the graph of relu look like? 2. what about the graph of relu . tanh ? and so on, indefinitely and arbitrarily. some features: - what points should be push through? maybe could add certain kinds of initialisations and ranges - add things like drop-out and whatnot.
  74. record videos of people doing interviews but have their voice replaced by obama and their image replaced by obama
  75. hair cut & deep learning deal with the hair-dresser across the road: sit down for a hair cut, get an hour of deep learning consulting as well
  76. live action star wars playing out across many websites in the background of cesium js windows: on my website, a death-star is driving around on it’s way somewhere eventually it reachers your website, and destroy’s it’s logo, or something
  77. deep learning tcp or upd: find something inbetween
  78. meta-search in google: “i want to see all the alternatives to cloud-ranger” it’s impossible to do this search.
  79. umbrella-scarf / fresh-scarf: it’s a scarf but also, it has a hood that you can pull up, maybe even a clear hood, that let’s you see out front of it, but keeps you under cover could also keep smoke out of your face
  80. meme-net: watch video, extract meme i.e. and the rollsafe guy
  81. ultimate computer setup person: someone who just has the worlds best computer set up everything works no data is duplicated whole operating system exists in 1.5 gig; they’ve got 510 gig free no conda/ruby/stack issues
  82. codebase -> readme: looks at an entire codebase; learns to predict the readme
  83. divangulation theorem for websites: @sordina surfaces can be triangulated websites can be divangulated what are the associated theorems?
  84. tasting plates for saas, *aas: instead of just saying “sign up now for 6 months free”; just auto-sign people up for x free things, then let them use it up. easy way to get a billion more dollars for your saas business. @sordina
  85. self-skating skateboard: it drives down to the skate park; skates around on the pipes, does flips, 180s, griding, whatever. @sordina @tall-josh
  86. different password entry forms: 1. any password you type logs you in, but they all take you to a different computer. only your password takes to you yours. “honeypassword” 2. you password consists of the actual letters, but also the timing between the letters @sordina 3. any key you press is irrelevant, all that matters is the spacing; everything is done via morse-code (@geekscape)
  87. congratulations!!!
  88. meeting chaos monkey: every time a meeting is scheduled, a random attendee is replaced by some other random staff member
  89. consulto the consulting teddybear: @sordina “that sounds good in theory” “have you tried kan-ban’ing that” “moving forward that sounds good, but right now i think we should be pragmatic”
  90. small magnets in fabric that can attach to other magnets so-as to create customisable clothing: just put a diff design on by switching out the magnets i just need some small magnets. jaycar sells them
  91. “collaboration card”: some way of listing and engaging with people in various projects you’re interested in
  92. nlp self-defending thesis
  93. rent factor charged in the city based on how innovative your store is: hairdresser: f = 0.85 funky clothing store: f = 0.6 some weird shop that only sells whatever: 0.2 cafe: 1 or some kind of scheme like so
  94. e-fabric: like e-ink, but for fabric
  95. clothes that change colour with respect to the magnetic fields that are around it
  96. grand designs: of computer programs: follow the development of some kind of app, over a few years. hahahaha would be terrible.
  97. giant magnet that aligns all the spins of the atoms of objects (people?!) so that they can pass through each other with different polarisations
  98. dance-curve net
  99. shoes that look like little cars: volvo shoes monster-truck shoes lamboghini shoes fi-car shoes etc.
  100. augmentation reality glasses that convert what people are saying into words that float in front of you that you can read: so you can “hear” what people say to you when you’re wearing headphones
  101. “html/css layout searcher”, like visual image search, but for how to lay things out with flex/css/react/whatever: input: some scribble about how you want your content laid out in boxes: output: the css/html that achieves this. there’s some networks that do this already, where they convert the drawings to code. but maybe that can be augmented by thinking of it like a search across already-existing content?
  102. “relax ai” or “mindful story ai”: it makes up nice stories, like “you are walking on the beach, you see a small turtle; you follow the turtle for a swim in the water …” could also use cool accents of people, and make sure the story is consistent with another NLP after the first generative run
  103. comedy audience that instead of laughing they just say the things people say when they think something is funny: instead of “ahahaha” audience (in unison): “that’s funny” audience (in unison): “good one” audience (in unison): “great joke”
  104. Adversial NLP: a sentence so similar to another sentence as to be humanly-indistinguishable, but makes the AI think it’s something wildly different
  105. Inflatable Whiteboard Room: it’s a large room, inflatable like a balloon or whatever, but you can walk into it and use the internals of it as a whiteboard useful for offices
  106. Collaborative Password Manager: say i want to make a password for a system you will control, but we both need access to maybe i can have my program generate part of it; you’re program generate part, then combine them both on our independent computers, without the entire password leaving either of them could build this on top of the public keys on github somehow; so i just pick the github user i’m going to share a password with could clearly do this immediately by encrypting it with their public key, or something. but maybe something richer can be done
  107. Video Issues: https://vimeo.com/265518095
  108. Faux Project Management Generator Thing: it’s an RNN that generates hundreds of tickets in trello or jira or whatever; with arbitrary due dates makes you feel stressed @sordina could be used for project-management training scenarios
  109. quantum cppn
  110. AIWS: ai for aws you: “hey, i need to computer with whatever to be up at whichever.com and to have some database, blah blah” aiws: “no worries, that’s set up for you!” alt. “talky-form for AWS” @sordina
  111. deep haircut mirror: a mirror infront of hair-dressers that lets you look at potential haircuts on your own head
  112. train a network to learn when to laugh in response to jokes: deep-audience
  113. dance led prompt device: it’s a little led board that sits at the front of a dance thing, like a teleprompter, but for dance it puts out the next dance moves a dance-prompter move-prompter
  114. Easter Egg Evangelist for Enterprises (E^4): A floating employee who embeds on teams to consult on how to best add easter-eggs to the features they build.
  115. self-driving food truck: @martiningram
  116. Stabbucks: Starbucks for knives. * https://i.imgur.com/1ZCIQnh.jpg * Order venti, grande, etc knives
  117. BrainRank: A leaderboard of brain-shaped logos.
  118. DeliveryNet: Reads prose with impeccable timing.
  119. Rant Meetup: Rant about stuff that sucks. * No solutions allowed * Surely james has something to say
  120. submit an AI-entry to every large festival in melbourne in a single year: https://whatson.melbourne.vic.gov.au/Whatson/Festivals/Pages/festival_calendar.aspx
  121. Stochasm: Metal band that plays random notes. * Easy to swap out band members!
  122. Seinfreinds: Have the cast of one sitcom act out an episode of another and see if anyone notices.
  123. hire a comedian to come along to your meetings: they can provide background entertainment me: “hey nice to meet you, this is my associate jerry seinfeld, let’s get started” jerry: “what’s the deal with peanuts?” …
  124. stacked double-coffee-cup holder: it’s just a handle, that holds on to two cups, one above the other useful for carrying multiple cups
  125. Auto-generating face detection camouflage: Aka, auto-generating styles from https://cvdazzle.com/
  126. use the technology of marina (ShapeTex) to make little movable people in jackets: https://www.linkedin.com/in/marina-toeters-a55a685/
  127. “studio gan”: it just makes up every single thing, much like #341 , but in more depth and for everything could use for #343 for example.
  128. the journey of your parcel: imagine you’re waiting for a parcel from auspost you put on your VR headset and you get a real-time view into it’s life; maybe it’s sitting on a boat, on it’s way here, or it’s in an airplane, or it’s driving etc. you’d get a full HD video-style image of the thing moving, that would be completely imagined by a gan or something.
  129. menu democracy: buy a coffee, earn 1 voting right to change the menu in some way buy more coffees, proceed in this fashion other food yields you more votes
  130. dynamic videos built on the fly to answer standard google queries: i.e “use python requests to do post request” a video could be made on the fly using the celeb-generating stuff of stack-gan, then the voice-simulation stuff of lyrebird or whoever, then the lip-moving stuff, the text-to-speech of wavenet or whatever, and some other random backing scene gans and music production networks it would get the content by reading the first answer it finds on google, in some summarised way. @sordina
  131. fully-automated fashion design: 1. Fashion-MNIST CPPN - At random, pick a random item of clothing, figure out what it is, and generate a large version. 2. Pick a random (creative-commons) photo from Flickr, train a style transfer network on it. 3. Apply the style transfer to a bunch of different clothing items? To make a theme? 4. Pick a name from an RNN? 5. Upload to PAOM? Run-time should be several hours for one collection? Not so bad.
  132. remote-controlled magnet: a perfectly spherical magnet that can be rolled around by remote.
  133. use cppn to generate a 3d landscape by determining the height by the colour
  134. lunch formation yaml specification: example: lunch: - sandwhich: - bread - butter - lettuce - cucumber - butter - bread region: cbd elements are ordered by height on the plate. @sordina
  135. a tale of 3 dickens: combine: 1. a christmas carol 2. a tale of two cities 3. great expectations in order page-by-page. @sordina
  136. instead of colouring in the retro-haskell tee with colours, print the source code for the program itself in the previous colour: easy!
  137. reverse twospace - use offices for other purposes out of hours: silverpond -> t-shirt business on the weekends
  138. dance karaoke: like karaoke, but instead of singing you need to dance uses some pose-recognition thing
  139. build a markov chain thing and then run all the words through some “smoothing” operation by way of a word embedding: i.e. somehow pick a few lines within embedding space, and move all of the words closer to those lines maybe something would happen
  140. naive nn visualisation: just reshape all the weights to be in the shape of an image, normalise the values, and output it.
  141. map sentences to “the gist”; just a few words: “an embedding that compresses a piece of text to its core concepts” “like if i can compress an image” “then i should be able to compress a book” “and if i can do that that means that i can also write a compressed book” “and have the neural network write my book” @gacafe
  142. version number which, in ascii, eventually approaches the source code itself
  143. personal world map: it’s one of those scaled world maps, where the scaling is determined by say your gps coordinates over a given year, so that it only enlarges the places you go. @mobeets
  144. water doughnut
  145. Deep-Can-I-Do-Deep-Learning-Here?: it’s a network for which you input a situation and it tells you if you can use deep learning to help.
  146. haskell type journey challenge: get from package x to package y using only the following types once ….
  147. make the 3d wall art that we saw at the house of sonya
  148. novels in binder-form so that you can take out small sections of the pages and read them
  149. multi-agent learning where the agents also watch each other locally and learn from each other
  150. ml for learning the life/centers function from christopher alexander: two pictures which one has more life? alt. something about centers?
  151. bureaucratic-net: instead of a network that is really good at explanationing it’s decisions, this network is really bad at it. nothing it says makes sense, or alternatively it’s really long-winded in it’s responses. or maybe it’s always right, but it never has any idea why.
  152. artistic-arxiv: instead of papers, each day take a random few images from every paper and show that. maybe it’d be cool.
  153. DeepWiki: on normal wikipedia, humans edit pages about concepts in the form of words on deepwiki AIs edit concepts in the form of embeddings by way of adjusting the vectors (or something) they’d need to think about how to manage edits and revisions and so on. but that’s the general idea. @sordina
  154. secret walls: wear the streets (or: graffiti on a wall of clothes; and wear them)
  155. a network that is given the punchline and has to work out the setup: @icp-jesus
  156. reverse website or inverse website: normally, you visit a site and see the website and you can view source to see the source what about if you could visit a site and see the source, then view the source to see the site
  157. endless pasta hat: has a self-pesto’ing tube that pushed out a long piece of spaghetti that you can munch on.
  158. in the gan setting, the discriminator isn’t needed when generating, maybe there’s another setting where the discriminator is still useful at the generative stage?
  159. a jacket that makes amazon’s automated shopping thing think you’re a packet of chips: or something similar
  160. CompromiseApp: two people need to agree on something they both have the app person 1 rates the estimated compromise, on a scale, of person 2 person 2 likewise both people record their own true compromise values then, over time, there’s a bunch of things that can be done, such as comparing predicted compromises, total compromises made, etc.
  161. DerivativeNet: it watches all seinfield episodes and sees if it can generate curb your enthusiasm episodes it reads all smbc comics and sees if it can generate xkcd ones etc.
  162. Cap-Gun mechanical keyboard: You pull back a bunch of hammers then as you type it fires the caps.

February 21, 2019

Patrick Louis (venam)

February 2019 Projects February 21, 2019 10:00 PM

The new year has begun… A while ago!

My last post Was almost 9 months ago, more than half a year has passed. A lot has happened but I still feel like time has passed quickly.

Psychology, Philosophy & Books

Language: brains
Explanation:

Les fleurs du mal

The majority of my reading has been through the newsletter, however I still had the time to go through the following:

  • The man who mistook his wife for a hat - Oliver Sacks
  • Les fleurs du mal - Baudelaire
  • Tribal leadership - Dave Logan
  • Managing business through human psychology - Ashish Bhagoria
  • The new one minute manager - Ken Blanchard
  • Authentic leadership - Harvard Business Review
  • The one thing - Garry Keller

As you might have noticed a lot of them are related to methodologies, approaches of interaction with others, and new ways of thinking. I find it fascinating to gather novel ways of seeing the world, all the mental models, all the manners to make decisions. This was the mindset for most of the past months along with re-energizing, invigorating, and bringing back my artistic sense.
A long long time ago I used to be in love with poetry and thus I’m slowly reincorporating this old habit into my daily life. Too much rationality is a disease of these days and ages which I’m trying not to fall into. I’m also working on my personal “immunity to change” regarding over-planning, so all of this helps.

As far as podcasts go, here’s the new list apart from the usual ones I was already following:

  • The Knowledge Project with Shane Parrish [All]
  • Planet Money [All]
  • The Food Chain [All]
  • Science Friday [All]
  • Hi-Phi Nation [All]
  • The History of GNOME [All]
  • The End Of The World with Josh Clark [All]
  • The Darknet Diaries [Most]
  • LeVar Burton Reads [Most]
  • Team Human [Started]

I’ve gathered more than one thousand hours those past months in AntennaPod, the application I’m using to listen to podcast (Something like 3h a day everyday). So I thought of moving away from the podcast world, at least for a while, for the next 2 or 3 months to learn something else on the road. I’ve chosen to dedicate this time to practicing singing. We’ll see what comes out of it, so far it’s going great but there needs to be a day or two a week for resting.

Learning & Growth

Language:growth
Explanation:

Face to face

I go through phases of learning and then creating. These months it’s been about learning. The emphasis was on work life, management, leadership, and android.

I have in mind some ideas for applications I want to build and I’m slowly gathering info, and in the meantime having fun, learning the various aspects of the android ecosystem.

On the other side, I’m working on my “immunity to change”, something I’ve learned from a Robert Kegan’s book. This relates to how we unknowingly create toxic loops within our lives that stop us to do actual changes we would like to do because those loops inherently defines us. For me it’s about an obsession with time, delegation, and loosening the grip over control of time.
Thus the switch to reading and learning in those leadership, management, and emotional intelligence books instead of digging deeper into projects one after the other like time is running out fast. I would’ve dismissed such content before and the same goes for the reemergence of artistic hobbies in my day to day.

Upgrading the Old

Language: C++
Explanation:

Manga downloader GUI

On that same note, I’ve gone in the past to upgrade an old project of mine, I’ve upgraded my manga downloader to support webtoons, a popular free Korean manwha website.

Get it here.

It has been fun re-reading old code, checking out what kind of mindset I had while writing it. I can’t access that it’s great code, nor that I would write it the same way today, but hey it’s still standing!

Newsletter and community

Language: Unix
Explanation:

nixers sticker

Countless videos, talks, research papers, and articles about Unix were consumed with much pleasure!
The newsletter has had its two years anniversary (issue 104), it has been a blast with a drawing from rocx, a ricing tip section from xero, and the start of a partnership with vermaden.

Every week we’re learning more and topics start to link together in a spider web of ideas. I like to reference previous sections of the newsletter when things are related and long time readers might enjoy this too (Check the archive).

To the joy of the now ~400 readers, vermaden has now partnered to give a nice touch to the newsletter, he’s more into BSD news than I am which keeps the balance.

I’ve dropped the “Others” part of the newsletter because of criticism that it was too offtopic and had a hint of political ideas to it. Let’s cut it short and simply say that I’d rather leave that section out than play the so-trendy game of internet over-interpretative argumentations and red herrings.

As for the podcast, I couldn’t put it on my top list but still felt in my guts that I wanted to write or prepare something similar. So I’ve begun a series named “Keeping time and date” which is similar to having a podcast/research on it. I’m hoping to do more of those in the future. By the time this post is online the series should almost be over.

On the nixers community side of things, the forums don’t get much appreciation but the irc conversation is still going on. I might organize events in the near future but I’m hoping a community push will do some good. My guess is that the forums aren’t as active because it’s seen as too high a pedestal to contribute to, nobody thinks their ideas are worth sharing there and if they find it worth sharing they’d rather do that on their personal platform, which I totally understand.

2bwm and Window management

Language: C
Explanation:

Glue

2bwm now supports a new fancy EWMH for full screen messages. This is a change I wanted to make for a while but didn’t put on my list.

Fortunately, frstrikerman has had the courtesy of starting a pull request with enough energy to make it happen. I’ve guided him a bit and together we’ve implemented it.
I’m looking forward to writing more about changes I want to make, how to make them, but leave the door open for the changes to be done by contributors. This creates a learning opportunity for anyone interested.

Three new articles have seen the light of day on my blog, all of them, unexpectedly, in close tie with the topic of window management. I’ve also added some preloading/caching behavior on the index page of the blog.

All of the articles in themselves were popular although the window automation one got quite a crowd (25k+) the first two days coming from tech news aggregation websites. Clearly, someone had posted it and it attracted readers, all for the best.

Ascii Art & Art

Language: ASCII
Explanation:

Abstract

Many packs have been posted by impure during the past period, impure 69, impure 70, and impure 71.
In 69 all I could manage to pull of was an abstract piece and in 70 and 71 I’ve had a cretaceous dinosaurs series.

In between, there was the Evoke 2018 where I scored fifth position. The idea of that compo was to restrict the size of the piece to 80 columns by 25 lines.

trapped within

I’ve also indirectly participated in Demosplash 2018 and scored the tenth position. I hadn’t really done anything particular for it but simply compiled together all the dinosaurs I had until that point.

ankylosaurus

A similarity you might have noticed is that I’ve toned down on the coloring. I’m trying to extract as much as possible from the art without having to think about anything other than message and form.

To put it bluntly I’ve turned off color mode:

syntax off

Right now I’m letting myself flow through a novel totem piece, a similar but more pronounced style to the one I had for Evoke 2017, that I call “dogmaIamgod” and which I should publish in the next two or three weeks.

Likewise, I’ve done some drawings and paintings too. I bought canvases for the first time and started experimenting, though I haven’t put them on the blog yet. Here’s a peek:

Extrapolating

Life and other hobbies

Language: life
Explanation:

Fungi board game

The quest for wild mushrooms is still ongoing. My SO and I have been gathering and studying mushrooms almost everyday. /r/mycology has become one of my favorite place to browse on lazy mornings.

In autumn we went on many small hikes and some bigger ones too without much luck but with much fun. All we could find were amanita citrina and beautiful though non-edible elfin saddles. There were other non-interesting species too.

elfin saddle

However, we’re not giving up, we’re taking this hobby to the next step. We lately went shopping for some hiking equipments, brand new fancy boots, some sticks, a handmade straw basket, etc.. And we’re planning on going on more hikes in spring. Spring is not a high season for edibles but we’re still hoping to find morels as it’s their prime time.

Overall this hobby has got us closer together and brought excitement to our couple. We’ve even got a card game named fungi for valentine. Moreover, we’ve searched local places that serve wild mushrooms and went to all the ones we could find. They’re usually pricey but worth the culinary experience.

Other than that with my SO we’ve got into retro gaming, which I’ve written about before in the past. These days it has become trendy again and many console manufacturers are relaunching their old brand. I guess nostalgia is a market that is well tapped into.

I’m actively looking for mushroom books to order from local libraries but most of them are not available so I end up reading the PDF versions I can find online. I’ve also been binge watching on Carluccio’s mushroom recipes, such a master.

This got me back into honing my cooking knowledge and art. I got tired of overcooking on Sunday for the rest of the week so I’m trying to juggle 3 days a week of home-cooked meals with restaurants or others for the rest of the week.
Which all drove me into two ideas.

First, the idea for a specific cooking diary app.
Second, the creation of a Zomato account. This goes with the same mindset as when I created the Google map account. I want to contribute to the local community by sharing the places I like the most. My mantra is that there will be no bad reviews and only constructive criticism if any.

Lastly, on the topic of food, fermentation has got my attention and I’ve now got mason jars filled with awesome vegetables. I’m currently on my third batch and exploring different formulas.

fermentation

When appetite is good, life’s good.

And life is!
Every year I normally fix a certain theme to rotate around and focus on. This time I’ve chosen to awaken my artistic side, spend more time in nature, organize more activities with my friends, write and contribute more to communities I’m part of through what I know best.

I did try to refresh my Spanish tongue but it wasn’t really part of anything and I didn’t follow up on it.

And to finish of, I’ve begun a daily diary. A quick summary of what I’ve done during the day, what’s on my mind, what I feel, what I want to do next. It’s a complement to what I was already doing through short, medium, and long term goals, associating it with my global path and intentions in life. In general this is revitalizing to do at the end of the day, it makes me more aware of my actions, appreciate the good parts and reflect on what could be done in the future.

Now

Which all leads to what’s in store for tomorrow.
More of the same but upgraded.

I really want to work on the idea I got for the application. Contribute to community projects more, write more articles about what I know, share ideas. Obviously the newsletter is included with more mini-series.
2bwm needs a big shake to add the wanted EWMH.
Maybe I’ll get back into WeChall, though it’s not part of the priority of the year.
And definitely push other hobbies too!

I’m going to travel in June with a friend to New York and then Miami, that’ll shift my perspective for a while, maybe bring some new insights.

This is it!

As usual… If you want something done, no one’s gonna do it for you, use your own hands.
And let’s go for a beer together sometime, or just chill.

Pages From The Fire (kghose)

Python3: subclassing int February 21, 2019 06:54 PM

Lying awake at night, existential questions cross your mind. What is my purpose? Is there a god? Can you subclass a Python3 int? Only the last one has an answer. You can indeed subclass an int, but there are a few subtleties. In short, in Python3 depending on whether the built in type is immutable …

Joe Nelson (begriffs)

Browsing a remote git repository February 21, 2019 12:00 AM

Git has no built-in way to browse and display files from a remote repo without cloning. Its closest concept is git ls-remote, but this shows only the hashes for references like HEAD or master, and not the files inside.

I wrote a server to expose a git repo in a new way. Watch the short demo:

Watch Video

How to try it yourself

You can get the code at begriffs/gitftp. It’s currently a proof of concept. Once I’ve added some more features I’ll run a public server to host the project code using the project itself.

The server is written in C and requires only libgit2. It’s small and portable.

Why do it this way?

The standard solution is to use a web interface like GitHub, GitLab, cgit, stagit, klaus, GitWeb, etc. However these interfaces are fairly rigid, and don’t connect well with external tools. While some of these sites also provide RESTful APIs, the clients available to consume those APIs are limited. Also desktop clients for these proprietary services are often big Electron apps.

By serving a repo behind an FTP interface, we get these benefits:

  • Web browser supported but not required
  • Minimized network traffic, sending just the file data itself
  • Supported on all platforms, with dozens of clients already written
  • Support for both the command line and GUI

GitFTP reads from a git repo’s internal database and exposes the trees and blobs as a filesystem. It reads from the master branch, so each new connection sees the newest code. Any single connection sees the files in a consistent state, unchanging even if new commits happen during the duration of the connection. The FTP welcome message identifies the SHA being served.

TODOs

This is a proof of concept. I’m putting it out there to gauge general interest. If we want to continue working on it, there are plenty of features to add, like providing a way to browse code at different commits, supporting SFTP so files cannot be changed by a man in the middle, etc etc. See the project issues for more.

February 20, 2019

Indrek Lasn (indreklasn)

Create a blazing fast modern blog with Nuxt and Prismic February 20, 2019 04:25 PM

Let’s build a modern blog with Vue, Nuxt and Prismic.

I chose Vue + Nuxt since they’re fun to work with. It’s easy to start with, offers plenty of essential features out of the box, and provides good performance.

If you’re new to Vue, I encourage to take a look at this article for understanding the basics.

From Zero to Hero with Vue - getting up and running

Nuxt is a Vue framework for server side rendering. It is a tool in the Vue ecosystem that you can use to build server-rendered apps from scratch without being bothered by the underlying complexities of rendering a JavaScript app to a server.

Why Nuxt?

https://nuxtjs.org/

Nuxt.js is an implementation of what we call a Universal Application.

It became famous with React but is currently getting more and more popular for many client-side libraries such as Angular, Vue.js, etc.

A Universal Application is a kind of application that renders your component on the server side.

Nuxt.js offers a simple way to first retrieve your asynchronous data from any data source and then renders it and sends it the the browser as HTML.

In terms of SEO, the Google bot crawler will get the rendered content and index it properly. In addition to that, the fact that your content can be pre-rendered and ready to be served increases the speed of your website and in that way, it also improves your SEO.

The Nuxt ecosystem is a never ending stream of handy tools and packages.

Contents

Fast rendering ensured by virtual DOM and minimal load time

Vue.js is only ~30 KB gzipped with the core module, the router and Vuex.

A minimal footprint offers short load time, meaning higher speed for users and better ranking on the speed criterium for Google crawler.

Virtual DOM!

Vue.js also took inspiration in ReactJS by implementing Virtual DOM under the hood since the version 2.0. Virtual DOM is basically a way to generate a version of the DOM in-memory each time you change a state and compare it to the actual DOM, so you can update only the part that needs to be updated instead of re-rendering everything.

Benchmarking

Vue.js offers some really good overall performance as you can see on the following benchmarks:

Duration in milliseconds ± standard deviation (Slowdown = Duration)

Memory allocation in MB

(Source: third-party benchmarks by Stefan Krause)”

What is Prismic and why should I care?

Prismic is a headless CMS. This means you edit your templates on your own server, but the backend runs on the cloud. This presents a few advantages such as being able to use an API to feed your content into external apps.

Imagine that you built a blog, not for yourself, but for someone else who is not a developer, so they can edit their content. You want to have full control over the layout (built with Vue) but you don’t want to go over the tedious process of deploying every time a new file for a new blog post is created.

This is where including a headless content management system (CMS) into your app is useful — so you don’t have to deal with that.

https://prismic.io/usecases

What’s the difference between a headless CMS and vanilla CMS?

A traditional CMS like Wordpress would provide the editing tools for managing content. But, it also would assume full control of the front-end of your website — the way the content is displayed is largely defined in a CMS.

Headless content management system, or headless CMS, is a back-end only content management system (CMS) built from the ground up as a content repository that makes content accessible via a RESTful API for display on any device.

If you want to know more, Prismic wrote a clear article about headless cms

I chose Prismic as my headless CMS — it’s super simple to set up and has great features out of the box.

Why I chose Prismic

  • Easy to setup — took me only couple hours to set-up the environment and push to production.
  • Live Preview mode — allows editors to preview their content on their website and apps whether it’s in a draft or scheduled to be published later. This allows marketing teams for example to have a full preview of their website for a specific date and time. This can be extremely useful to manage upcoming blog releases, and preview edits.
  • Slices — Slices are reusable components. Enabling Slices in your template will allow writers to choose between adding a text section, an image, or a quote in the piece of content they are creating. This gives writers the freedom to compose a blog post by alternating and ordering as many of these choices/content blocks as they want/need.
  • Simple and comprehensive documentation.
  • Strong community, e.g Google, New Relic, Ebay, etc are using Prismic
  • Friendly free tier

Setting up Prismic is very simple, let’s get started!

Head over to the Prismic website and create a new user.

After creating a new user on Prismic, we should see something like this;

Building our custom type

Custom Types are models of content that we setup for our marketing or writing team. The marketing team will fill them with content (text, images, etc.), and we’ll be able to retrieve this content through Prismic’s API.

There are two kinds of Custom Types, the Single Type and the Repeatable type.

The Single Type is used for pages of which there is only one instance (a home page, a pricing page, an about us page).

Repeatable Custom Types will be templates used in more than one document (ie. having many blog post pages, product pages, landing pages for your website).

We want a blog post. In fact we want many blog posts, so it should be a repeatable type.

choosing the type

Creating a repeatable type blog post.

We should be on the content builder now. Prismic gives us a lot of options to choose from. If you look on the right, you should see a sidebar including lots of options — from images, titles, content relationship to SEO options.

Let’s build a reusable blog post with the prismic builder. Our blog will include a title and the body.

Start with adding the following fields;

  • UID field
  • Title field
  • Rich text field

Each time you add a field you can define formatting options for it. The UID field is a unique identifier that can be used specifically to create SEO and user-friendly website URLs

Creating our blog post title

Don’t forget to save our progress!

Make sure you have the following fields for the blog post;

  • uid
  • blog_post_title
  • blog_content

So far we have the layout for our reusable blog post.

Custom types menu

Time to create a blog post! Head over to the content tab on the left.

Content tab

This will take us to the blog layout we built earlier. Insert the desired text for the uid, post_title, blog_content blocks.

Building our page with Prismic layout builder

Great! We have our blog post set up now. Look at the right-top, we should see a “save” button. Clicking this saves our progress. After saving we can publish our content. Publishing the content makes it available via the API for our front-end to consume.

Starting a new Nuxt project

Open your terminal and run this command. Make sure you have npx installed (shipped by default with npm +5.2.0)

$ npx create-nuxt-app vue-nuxt-prismic-blog

The Nuxt installer conveniently asks us our preferences and creates the project.

We should end up with a project structure like below;

Nuxt project structure

Great! Let’s build our blog now. We need to fetch the blog content from Prismic. Luckily, prismic gives us plenty of handy tools.

https://medium.com/media/0df21286ef3529027e62f310485e7b47/href

The prismic-javascript package includes many utilities, including fetching from our api. The prismic-dom gives us helper functions to render markup.

Prismic NPM package — https://www.npmjs.com/package/prismic-javascript

Let’s quickly create the prismic.config.js file in our root directory. This is where we’ll place our Prismic related configuration.

https://medium.com/media/5f5caa9dfbe727f13858883c0a369265/href

Note: Make sure you use the API endpoint associated with your blog.

Open the pages/index.vue file and import the Prismic library with our config.

https://medium.com/media/2c21de38ae4ae6ea43abe4446422731a/href

Great! Next, we have to call the API somewhere, let’s use the asyncData life-cycle hook for this.

https://medium.com/media/6ee52da5bc7b385518fde18cd7f5ab15/href

First, we initialize our API with the endpoint. Then, we query the API to return our blog post. We can specify the language and document type.

The Prismic API is promise based, which means we can call the API and chain promises. Hurray for promises. We also can use the async/await syntax to resolve promises. Check out this post I wrote about aysnc/await

Prismic response

All we need to do is render the markup now!

https://medium.com/media/7ac5c40614193b76fd9bbd263d596f77/href

There you go. We successfully fetched our blog post from the Prismic API.

Applying the styles — grab a copy and place it in the style section of the Vue component;

https://medium.com/media/beec023f7d55d59fc84f92f750d4f768/href

If we open our app, this is what we should see.

End result

Voila! We have a modern server-side rendered blog built with Nuxt and Prismic.

We barely scratched the surface — we can do a lot more with Nuxt and Prismic. My favorite Prismic features are Slices and Live Preview. I encourage you to check them out!

For example in this project we worked on only one post, but if we had created lots of posts in Prismic, then one really cool thing about Nuxt.js is that automatically creates routes for you.

Behind the scenes, it still uses Vue Router for this, but you don’t need to create a route config manually anymore. Instead, you create your routing using a folder structure — inside the pages folder. But you can read all about that in the official docs on routing in Nuxt.js.

We didn’t get the chance to go deeper, but my favorite Prismic features are Slices and Live Preview. I encourage you to check them out!

Slices will allow you to create “dynamic” pages with richer content, and Live preview will allow you to instantly preview your edits in your webpage.

Slices

Thanks for reading! If you found this useful, please give the article some claps so more people see it! ❤

In case you got lost, here’s the repository for our blog;

wesharehoodies/nuxt-prismic-blog

If you have any questions regarding this article, or anything general — I’m active on Twitter and always happy to read comments and reply to tweets.

Here are some of my previous articles;


Create a blazing fast modern blog with Nuxt and Prismic was originally published in freeCodeCamp.org on Medium, where people are continuing the conversation by highlighting and responding to this story.

February 19, 2019

Dan Luu (dl)

Randomized trial on gender in Overwatch February 19, 2019 12:00 AM

A recurring discussion in Overwatch (as well as other online games) is whether or not women are treated differently from men. If you do a quick search, you can find hundreds of discussions about this, some of which have well over a thousand comments. These discussions tend to go the same way and involve the same debate every time, with the same points being made on both sides. Just for example, these three threads on reddit that spun out of a single post that have a total of 10.4k comments. On one side, you have people saying "sure, women get trash talked, but I'm a dude and I get trash talked, everyone gets trash talked there's no difference", "I've never seen this, it can't be real", etc., and on the other side you have people saying things like "when I play with my boyfriend, I get accused of being carried by him all the time but the reverse never happens", "people regularly tell me I should play mercy[, a character that's a female healer]", and so on and so forth. In less time than has been spent on a single large discussion, we could just run the experiment, so here it is.

This is the result of playing 339 games in the two main game modes, quick play (QP) and competitive (comp), where roughly half the games were played with a masculine name (where the username was a generic term for a man) and half were played with a feminine name (where the username was a woman's name). I recorded all of the comments made in each of the games and then classified the comments by type. Classes of comments were "sexual/gendered comments", "being told how to play", "insults", and "compliments".

In each game that's included, I decided to include the game (or not) in the experiment before the character selection screen loaded. In games that were included, I used the same character selection algorithm, I wouldn't mute anyone for spamming chat or being a jerk, I didn't speak on voice chat (although I had it enabled), I never sent friend requests, and I was playing outside of a group in order to get matched with 5 random players. When playing normally, I might choose a character I don't know how to use well and I'll mute people who pollute chat with bad comments. There are a lot of games that weren't included in the experiment because I wasn't in a mood to listen to someone rage at their team for fifteen minutes and the procedure I used involved pre-committing to not muting people who do that.

Sexual or sexually charged comments

I thought I'd see more sexual comments when using the feminine name as opposed to the masculine name, but that turned out to not be the case. There was some mention of sex, genitals, etc., in both cases and the rate wasn't obviously different and was actually higher in the masculine condition.

Zero games featured comments were directed specifically at me in the masculine condition and two (out of 184) games in the feminine condition featured comments that were directed at me. Most comments were comments either directed at other players or just general comments to team or game chat.

Examples of typical undirected comments that would occur in either condition include ""my girlfriend keeps sexting me how do I get her to stop?", "going in balls deep", "what a surprise. *strokes dick* [during the post-game highlight]", and "support your local boobies".

The two games that featured sexual comments directed at me had the following comments:

  • "please mam can i have some coochie", "yes mam please" [from two different people], ":boicootie:"
  • "my dicc hard" [believed to be directed at me from context]

During games not included in the experiment (I generally didn't pay attention to which username I was on when not in the experiment), I also got comments like "send nudes". Anecdotally, there appears to be a different in the rate of these kinds of comments directed at the player, but the rate observed in the experiment is so low that uncertainty intervals around any estimates of the true rate will be similar in both conditions unless we use a strong prior.

The fact that this difference couldn't be observed in 339 games was surprising to me, although it's not inconsistent with McDaniel's thesis, a survey of women who play video games. 339 games probably sounds like a small number to serious gamers, but the only other randomized experiment I know of on this topic (besides this experiment) is Kasumovic et al., which notes that "[w]e stopped at 163 [games] as this is a substantial time effort".

All of the analysis uses the number of games in which a type of comment occured and not tone to avoid having to code comments as having a certain tone in order to avoid possibly injecting bias into the process. Sentiment analysis models, even state-of-the-art ones often return nonsensical results, so this basically has to be done by hand, at least today. With much more data, some kind of sentiment analysis, done with liberal spot checking and re-training of the model, could work, but the total number of comments is so small in this case that it would amount to coding each comment by hand.

Coding comments manually in an unbiased fashion can also be done with a level of blinding, but doing that would probably require getting more people involved (since I see and hear comments while I'm playing) and relying on unpaid or poorly paid labor.

Being told how to play

The most striking, easy to quantify, difference was the rate at which I played games in which people told me how I should play. Since it's unclear how much confidence we should have in the difference if we just look at the raw rates, we'll use a simple statistical model to get the uncertainty interval around the estimates. Since I'm not sure what my belief about this should be, this uses an uninformative prior, so the estimate is close to the actual rate. Anyway, here are the uncertainty intervals a simple model puts on the percent of games where at least one person told me I was playing wrong, that I should change how I'm playing, or that I switch characters:

table {border-collapse: collapse;}table,th,td {border: 1px solid black;}td {text-align:center;}

Cond Est P25 P75
F comp 19 13 25
M comp 6 2 10
F QP 4 3 6
M QP 1 0 2

The experimental conditions in this table are masculine vs. feminine name (M/F) and competitive mode vs quick play (comp/QP). The numbers are percents. Est is the estimate, P25 is the 25%-ile estimate, and P75 is the 75%-ile estimate. Competitive mode and using a feminine name are both correlated with being told how to play. See this post by Andrew Gelman for why you might want to look at the 50% interval instead of the 95% interval.

For people not familiar with overwatch, in competitive mode, you're explicitly told what your ELO-like rating is and you get a badge that reflects your rating. In quick play, you have a rating that's tracked, but it's never directly surfaced to the user and you don't get a badge.

It's generally believed that people are more on edge during competitive play and are more likely to lash out (and, for example, tell you how you should play). The data is consistent with this common belief.

Per above, I didn't want to code tone of messages to avoid bias, so this table only indicates the rate at which people told me I was playing incorrectly or asked that I switch to a different character. The qualitative difference in experience is understated by this table. For example, the one time someone asked me to switch characters in the masculine condition, the request was a one sentence, polite, request ("hey, we're dying too quickly, could we switch [from the standard one primary healer / one off healer setup] to double primary healer or switch our tank to [a tank that can block more damage]?"). When using the feminine name, a typical case would involve 1-4 people calling me human garbage for most of the game and consoling themselves with the idea that the entire reason our team is losing is that I won't change characters.

The simple model we're using indicates that there's probably a difference between both competitive and QP and playing with a masculine vs. a feminine name. However, most published results are pretty bogus, so let's look at reasons this result might be bogus and then you can decide for yourself.

Threats to validity

The biggest issue is that this wasn't a pre-registered trial. I'm obviously not going to go and officially register a trial like this, but I also didn't informally "register" this by having this comparison in mind when I started the experiment. A problem with non-pre-registered trials is that there are a lot of degrees of freedom, both in terms of what we could look at, and in terms of the methodology we used to look at things, so it's unclear if the result is "real" or an artifact of fishing for something that looks interesting. A standard example of this is that, if you look for 100 possible effects, you're likely to find 1 that appears to be statistically significant with p = 0.01.

There are standard techniques to correct for this problem (e.g., Bonferroni correction), but I don't find these convincing because they usually don't capture all of the degrees of freedom that go into a statistical model. An example is that it's common to take a variable and discretize it into a few buckets. There are many ways to do this and you generally won't see papers talk about the impact of this or correct for this in any way, although changing how these buckets are arranged can drastically change the results of a study. Another common knob people can use to manipulate results is curve fitting to an inappropriate curve (often a 2nd a 3rd degree polynomial when a scatterplot shows that's clearly incorrect). Another way to handle this would be to use a more complex model, but I wanted to keep this as simple as possible.

If I wanted to really be convinced on this, I'd want to, at a minimum, re-run this experiment with this exact comparison in mind. As a result, this experiment would need to be replicated to provide more than a preliminary result that is, at best, weak evidence.

One other large class of problem with randomized controlled trials (RCTs) is that, despite randomization, the two arms of the experiment might be different in some way that wasn't randomized. Since Overwatch doesn't allow you to keep changing your name, this experiment was done with two different accounts and these accounts had different ratings in competitive mode. On average, the masculine account had a higher rating due to starting with a higher rating, which meant that I was playing against stronger players and having worse games on the masculine account. In the long run, this will even out, but since most games in this experiment were in QP, this didn't have time to even out in comp. As a result, I had a higher win rate as well as just generally much better games with the feminine account in comp.

With no other information, we might expect that people who are playing worse get told how to play more frequently and people who are playing better should get told how to play less frequently, which would mean that the table above understates the actual difference.

However Kasumovic et al., in a gender-based randomized trial of Halo 3, found that players who were playing poorly were more negative towards women, especially women who were playing well (there's enough statistical manipulation of the data that a statement this concise can only be roughly correct, see study for details). If that result holds, it's possible that I would've gotten fewer people telling me that I'm human garbage and need to switch characters if I was average instead of dominating most of my games in the feminine condition.

If that result generalizes to OW, that would explain something which I thought was odd, which was that a lot of demands to switch and general vitriol came during my best performances with the feminine account. A typical example of this would be a game where we have a 2-2-2 team composition (2 players playing each of the three roles in the game) where my counterpart in the same role ran into the enemy team and died at the beginning of the fight in almost every engagement. I happened to be having a good day and dominated the other team (37-2 in a ten minute comp game, while focusing on protecting our team's healers) while only dying twice, once on purpose as a sacrifice and second time after a stupid blunder. Immediately after I died, someone asked me to switch roles so they could take over for me, but at no point did someone ask the other player in my role to switch despite their total uselesses all game (for OW players this was a Rein who immediately charged into the middle of the enemy team at every opportunity, from a range where our team could not possibly support them; this was Hanamura 2CP, where it's very easy for Rein to set up situations where their team cannot help them). This kind of performance was typical of games where my team jumped on me for playing incorrectly. This isn't to say I didn't have bad games; I had plenty of bad games, but a disproportionate number of the most toxic experiences came when I was having a great game.

I tracked how well I did in games, but this sample doesn't have enough ranty games to do a meaningful statistical analysis of my performance vs. probability of getting thrown under the bus.

Games at different ratings are probably also generally different environments and get different comments, but it's not clear if there are more negative comments at 2000 than 2500 or vice versa. There are a lot of online debates about this; for any rating level other than the very lowest or the very highest ratings, you can find a lot of people who say that the rating band they're in has the highest volume of toxic comments.

Other differences

Here are some things that happened while playing with the feminine name that didn't happen with the masculine name during this experiment or in any game outside of this experiment:

  • unsolicited "friend" requests from people I had no textual or verbal interaction with (happened 7 times total, didn't track which cases were in the experiment and which weren't)
  • someone on the other team deciding that my team wasn't doing a good enough job of protecting me while I was playing healer, berating my team, and then throwing the game so that we won (happened once during the experiment)
  • someone on my team flirting with me and then flipping out when I don't respond, who then spends the rest of the game calling me autistic or toxic (this happened once during the experiment, and once while playing in a game not included in the experiment)

The rate of all these was low enough that I'd have to play many more games to observe something without a huge uncertainty interval.

I didn't accept any friend requests from people I had no interaction with. Anecdotally, some people report people will send sexual comments or berate them after an unsolicited friend request. It's possible that the effect show in the table would be larger if I accepted these friend requests and it couldn't be smaller.

I didn't attempt to classify comments as flirty or not because, unlike the kinds of commments I did classify, this is often somewhat subtle and you could make a good case that any particular comment is or isn't flirting. Without responding (which I didn't do), many of these kinds of comments are ambiguous

Another difference was in the tone of the compliments. The rate of games where I was complimented wasn't too different, but compliments under the masculine condition tended to be short and factual (e.g., someone from the other team saying "no answer for [name of character I was playing]" after a dominant game) and compliments under the feminine condition tended to be more effusive and multiple people would sometimes chime in about how great I was.

Non differences

The rate of complements and the rate of insults in games that didn't include explanations of how I'm playing wrong or how I need to switch characters were similar in both conditions.

Other factors

Some other factors that would be interesting to look at would be time of day, server, playing solo or in a group, specific character choice, being more or less communicative, etc., but it would take a lot more data to be able to get good estimates when adding it more variables. Blizzard should have the data necessary to do analyses like this in aggregate, but they're notoriously private with their data, so someone at Blizzard would have to do the work and then publish it publicly, and they're not really in the habit of doing that kind of thing. If you work at Blizzard and are interested in letting a third party do some analysis on an anonymized data set, let me know and I'd be happy to dig in.

Experimental minutiae

Under both conditions, I avoided ever using voice chat and would call things out in text chat when time permitted. Also under both conditions, I mostly filled in with whatever character class the team needed most, although I'd sometimes pick DPS (in general, DPS are heavily oversubscribed, so you'll rarely play DPS if you don't pick one even when unnecessary).

For quickplay, backfill games weren't counted (backfill games are games where you join after the game started to fill in for a player who left; comp doesn't allow backfills). 6% of QP games were backfills.

These games are from before the "endorsements" patch; most games were played around May 2018. All games were played in "solo q" (with 5 random teammates). In order to avoid correlations between games depending on how long playing sessions were, I quit between games and waited for enough time (since you're otherwise likely to end up in a game with some or many of the same players as before).

The model used probability of a comment happening in a game to avoid the problem that Kasumovic et al. ran into, where a person who's ranting can skew the total number of comments. Kasumovic et al. addressed this by removing outliers, but I really don't like manually reaching in and removing data to adjust results. This could also be addressed by using a more sophisticated model, but a more sophisticated model means more knobs which means more ways for bias to sneak in. Using the number of players who made comments instead would be one way to mitigate this problem, but I think this still isn't ideal because these aren't independent -- when one player starts being negative, this greatly increases the odds that another player in that game will be negative, but just using the number of players makes four games with one negative person the same as one game with four negative people. This can also be accounted for with a slightly more sophisticated model, but that also involves adding more knobs to the model.

Appendix: comments / advice to overwatch players

A common complaint, perhaps the most common complaint by people below 2000 SR (roughly 30%-ile) or perhaps 1500 SR (roughly 10%-ile) is that they're in "ELO hell" and are kept down because their teammates are too bad. Based on my experience, I find this to be extremely unlikely.

People often split skill up into "mechanics" and "gamesense". My mechanics are pretty much as bad as it's possible to get. The last game I played seriously was a 90s video game that's basically online asteroids and the last game before that I put any time into was the original SNES super mario kart. As you'd expect from someone who hasn't put significant time into a post-90s video game or any kind of FPS game, my aim and dodging are both atrocious. On top of that, I'm an old dude with slow reflexes and I was able to get to 2500 SR (roughly 60%-ile) by avoiding a few basic fallacies and blunders despite have approximately zero mechanical skill. If you're also an old dude with basically no FPS experience, you can do the same thing; if you have good reflexes or enough FPS experience to actually aim or dodge, you basically can't be worse mechnically than I am and you can do much better by avoiding a few basic mistakes.

The most common fallacy I see repeated is that you have to play DPS to move out of bronze or gold. The evidence people give for this is that, when a GM streamer plays flex, tank, or healer, they sometimes lose in bronze. I guess the idea is that, because the only way to ensure a 99.9% win rate in bronze is to be a GM level DPS player and play DPS, the best way to maintain a 55% or a 60% win rate is to play DPS, but this doesn't follow.

Healers and tanks are both very powerful in low ranks. Because low ranks feature both poor coordination and relatively poor aim (players with good coordination or aim tend to move up quickly), time-to-kill is very slow compared to higher ranks. As a result, an off healer can tilt the result of a 1v1 (and sometimes even a 2v1) matchup and a primary healer can often determine the result of a 2v1 matchup. Because coordination is poor, most matchups end up being 2v1 or 1v1. The flip side of the lack of coordination is that you'll almost never get help from teammates. It's common to see an enemy player walk into the middle of my team, attack someone, and then walk out while literally no one else notices. If the person being attacked is you, the other healer typically won't notice and will continue healing someone at full health and none of the classic "peel" characters will help or even notice what's happening. That means it's on you to pay attention to your surroundings and watching flank routes to avoid getting murdered.

If you can avoid getting murdered constantly and actually try to heal (as opposed to many healers at low ranks, who will try to kill people or stick to a single character and continue healing them all the time even if they're at full health), you outheal a primary healer half the time when playing an off healer and, as a primary healer, you'll usually be able to get 10k-12k healing per 10 min compared to 6k to 8k for most people in Silver (sometimes less if they're playing DPS Moira). That's like having an extra half a healer on your team, which basically makes the game 6.5 v 6 instead of 6v6. You can still lose a 6.5v6 game, and you'll lose plenty of games, but if you're consistently healing 50% more than an normal healer at your rank, you'll tend to move up even if you get a lot of major things wrong (heal order, healing when that only feeds the other team, etc.).

A corollary to having to watch out for yourself 95% when playing a healer is that, as a character who can peel, you can actually watch out for your teammates and put your team at a significant advantage in 95% of games. As Zarya or Hog, if you just boringly play towards the front of your team, you can basically always save at least one teammate from death in a team fight, and you can often do this 2 or 3 times. Meanwhile, your counterpart on the other team is walking around looking for 1v1 matchups. If they find a good one, they'll probably kill someone, and if they don't (if they run into someone with a mobility skill or a counter like brig or reaper), they won't. Even in the case where they kill someone and you don't do a lot, you still provide as much value as them and, on average, you'll provide more value. A similar thing is true of many DPS characters, although it depends on the character (e.g., McCree is effective as a peeler, at least at the low ranks that I've played in). If you play a non-sniper DPS that isn't suited for peeling, you can find a DPS on your team who's looking for 1v1 fights and turn those fights into 2v1 fights (at low ranks, there's no shortage of these folks on both teams, so there are plenty of 1v1 fights you can control by making them 2v1).

All of these things I've mentioned amount to actually trying to help your team instead of going for flashy PotG setups or trying to dominate the entire team by yourself. If you say this in the abstract, it seems obvious, but most people think they're better than their rating. A survey of people's perception of their own skill level vs. their actual rating found that 1% of people thought they were overrated, 32% of people thought they were rated accurately, and the other 77% of people thought they were underrated. It doesn't help that OW is designed to make people think they're doing well when they're not and the best way to get "medals" or "play of the game" is to play in a way that severely reduces your odds of actually winning each game.

Outside of obvious gameplay mistakes, the other big thing that loses games is when someone tilts and either starts playing terribly or flips out and says something to enrage someone else on the team, who then starts playing terribly. I don't think you can actually do much about this directly, but you can never do this, so 5/6th of your team will do this at some base rate, whereas 6/6 of the other team will do this. Like all of the above, this won't cause you to win all of your games, but everything you do that increases your win rate makes a difference.

Poker players have the right attitude when they talk about leaks. The goal isn't to win every hand, it's to increase your EV by avoiding bad blunders (at high levels, it's about more than avoiding bad blunders, but we're talking about getting out of below median ranks, not becoming GM here). You're going to have terrible games where you get 5 people instalocking DPS. Your odds of winning a game are low, say 10%. If you get mad and pick DPS and reduce your odds even further (say this is to 2%), all that does is create a leak in your win rate during games when your teammates are being silly.

If you gain/lose 25 rating per game for a win or a loss, your average rating change from a game is 25 (W_rate - L_rate) = 25 (2W_rate - 1). Let's say 1/40 games are these silly games where your team decides to go all DPS. The per-game SR difference of trying to win these vs. soft throwing is maybe something like 1/40 * 25 (2 * 0.08) = 0.1. That doesn't sound like much and these numbers are just guesses, but everyone outside of very high-level games is full of leaks like these, and they add up. And if you look at a 60% win rate, which is pretty good considering that your influence is limited because you're only one person on a 6 person team, that only translates to an average of 5SR per game, so it doesn't actually take that many small leaks to really move your average SR gain or loss.

Appendix: general comments on online gaming, 20 years ago vs. today

Since I'm unlikely to write another blog post on gaming any time soon, here are some other random thoughts that won't fit with any other post. My last serious experience with online games was with a game from the 90s. Even though I'd heard that things were a lot worse, I was still surprised by it. IRL, the only time I encounter the same level and rate of pointless nastiness in a recreational activity is down at the bridge club (casual bridge games tend to be very nice). When I say pointless nastiness, I mean things like getting angry and then making nasty comments to a teammate mid-game. Even if your "criticism" is correct (and, if you review OW games or bridge hands, you'll see that these kinds of angry comments are almost never correct), this has virtually no chance of getting your partner to change their behavior and it has a pretty good chance of tilting them and making them play worse. If you're trying to win, there's no reason to do this and good reason to avoid this.

If you look at the online commentary for this, it's common to see people blaming kids, but this doesn't match my experience at all. For one thing, when I was playing video games in the 90s, a huge fraction of the online gaming population was made up of kids, and online game communities were nicer than they are today. Saying that "kids nowadays" are worse than kids used to be is a pastime that goes back thousands of years, but it's generally not true and there doesn't seem to be any reason to think that it's true here.

Additionally, this simply doesn't match what I saw. If I just look at comments over audio chat, there were a couple of times when some kids were nasty, but almost all of the comments are from people who sound like adults. Moreover, if I look at when I played games that were bad, a disproportionately large number of those games were late (after 2am eastern time, on the central/east server), where the relative population of adults is larger.

And if we look at bridge, the median age of an ACBL member is in the 70s, with an increase in age of a whopping 0.4 years per year.

Sure, maybe people tend to get more mature as they age, but in any particular activity, that effect seems to be dominated by other factors. I don't have enough data at hand to make a good guess as to what happened, but I'm entertained by the idea that this might have something to do with it:

I’ve said this before, but one of the single biggest culture shocks I’ve ever received was when I was talking to someone about five years younger than I was, and she said “Wait, you play video games? I’m surprised. You seem like way too much of a nerd to play video games. Isn’t that like a fratboy jock thing?”

Appendix: FAQ

Here are some responses to the most common online comments.

Plat? You suck at Overwatch

Yep. But I sucked roughly equally on both accounts (actually somewhat more on the masculine account because it was rated higher and I was playing a bit out of my depth). Also, that's not a question.

This is just a blog post, it's not an academic study, the results are crap.

There's nothing magic about academic papers. I have my name on a few publications, including one that won best paper award at the top conference in its field. My median blog post is more rigorous than my median paper or, for that matter, the median paper that I read.

When I write a paper, I have to deal with co-authors who push for putting in false or misleading material that makes the paper look good and my ability to push back against this has been fairly limited. On my blog, I don't have to deal with that and I can write up results that are accurate (to the best of my abillity) even if it makes the result look less interesting or less likely to win an award.

Gamers have always been toxic, that's just nostalgia talking.

If I pull game logs for subspace, this seems to be false. YMMV depending on what games you played, I suppose. FWIW, airmash seems to be the modern version of subspace, and (until the game died), it was much more toxic than subspace even if you just compare on a per-game basis despite having much smaller games (25 people for a good sized game in airmash, vs. 95 for subsace).

This is totally invalid because you didn't talk on voice chat.

At the ranks I played, not talking on voice was the norm. It would be nice to have talking or not talking on voice chat be an indepedent variable, but that would require playing even more games to get data for another set of conditions, and if I wasn't going to do that, choosing the condition that's most common doesn't make the entire experiment invalid, IMO.

Some people report that, post "endorsements" patch, talking on voice chat is much more common. I tested this out by playing 20 (non-comp) games just after the "Paris" patch. Three had comments on voice chat. One was someone playing random music clips, one had someone screaming at someone else for playing incorrectly, and one had useful callouts on voice chat. It's possible I'd see something different with more games or in comp, but I don't think it's obvious that voice chat is common for most people after the "endorsements" patch.

Appendix: code and data

If you want to play with this data and model yourself, experiment with different priors, run a posterior predictive check, etc., here's a snippet of R code that embeds the data:

library(brms)
library(modelr)
library(tidybayes)
library(tidyverse)

d <- tribble(
  ~game_type, ~gender, ~xplain, ~games,
  "comp", "female", 7, 35,
  "comp", "male", 1, 23,
  "qp", "female", 6, 149,
  "qp", "male", 2, 132
)

d <- d %>% mutate(female = ifelse(gender == "female", 1, 0), comp = ifelse(game_type == "comp", 1, 0))


result <-
  brm(data = d, family = binomial,
      xplain | trials(games) ~ female + comp,
      prior = c(set_prior("normal(0,10)", class = "b")),
      iter = 25000, warmup = 500, cores = 4, chains = 4)

The model here is simple enough that I wouldn't expect the version of software used to significantly affect results, but in case you're curious, this was done with brms 2.7.0, rstan 2.18.2, on R 3.5.1.

Thanks to Leah Hanson, Sean Talts and Sean's math/stats reading group, Annie Cherkaev, Robert Schuessler, Wesley Aptekar-Cassels, Julia Evans, Paul Gowder, Jonathan Dahan, Bradley Boccuzzi, Akiva Leffert, and one or more anonymous commenters for comments/corrections/discussion.

February 17, 2019

Simon Zelazny (pzel)

Figuring out a gen_tcp:recv limitation February 17, 2019 11:00 PM

In which a suprisingly pernicious framed payload leads to OTP spelunking.

The setup: sending a string over TCP

Let's say you want to send the ASCII string Fiat lux! to an Erlang process listening on the other side of a TCP connection. Not a big deal, right?

Our sending application is written in Python. Here's what it might look like:

#!/usr/bin/env python3
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(("127.0.0.1", 7777))
data_to_send = b"Fiat Lux!"
sock.sendall(data_to_send)

... and here's the receiving Erlang application:

#!/usr/bin/env escript
main(_) ->
  {ok, L} = gen_tcp:listen(7777, [binary, {active, false}, {reuseaddr, true}]),
  {ok, Sock} = gen_tcp:accept(L),
  {ok, String} = gen_tcp:recv(Sock, 0),
  io:format("Got string: ~ts~n", [String]),
  erlang:halt(0).

If we start the Erlang receiver (in shell 1), then run the Python sender (in shell2), we should see the receiver emit the following:

$ ./receive.escript
Got string: Fiat Lux!
$

As you can see, we optimistically sent all our data over TCP from the Python app, and received all that data, intact, on the other side. What's important here is that our Erlang socket is in passive mode, which means that incoming TCP data needs to be recv'd off of the socket. The second argument in gen_tcp:recv(Sock, 0) means that we want to read however many bytes are available to be read from the OS's network stack. In this case all our data was kindly provided to us in one nice chunk.

Success! Our real production application will be dealing with much bigger pieces of data, so it behooves us to test with a larger payload. Let's try a thousand characters.

More data

We update the sender and receiver as follows:

#!/usr/bin/env python3
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(("127.0.0.1", 7777))
data_to_send = b'a' * 1000
sock.sendall(data_to_send)
#!/usr/bin/env escript
main(_) ->
  {ok, L} = gen_tcp:listen(7777, [binary, {active, false}, {reusaddr, true}]),
  {ok, Sock} = gen_tcp:accept(L),
  {ok, String} = gen_tcp:recv(Sock, 0),
  io:format("Got string of length: ~p~n", [byte_size(String)]),
  erlang:halt(0).

When we run our experiment, we see that our Erlang process does indeed get all 1000 bytes. Let's add one more zero to the payload.

#!/usr/bin/env python3
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(("127.0.0.1", 7777))
data_to_send = b'a' * 10000
sock.sendall(data_to_send)

And we hit our first snag!

Got string of length: 1460

Aha! Our gen_tcp:recv(Sock, 0) call asked the OS to give us whatever bytes it had ready in the TCP buffer, and so that's what we received. TCP is a streaming protocol, and there is no guarantee that a given sequence of bytes received on the socket will correspond to a logical message in our application layer. The low-effort way of handling this issue is by prefixing every logical message on the TCP socket with a known-width integer, representing the length of the message in bytes. "Low-effort" sounds like the kind of thing you put in place when the deadline was yesterday. Onward!

Let's take our initial string as an example. Instead of sending the following sequence of 9 bytes on the wire:

Ascii:     F    i   a    t   ␣    l    u    x   !

Binary:   70  105  97  116  32  108  117  120  33

We'd first prefix it with an 32-bit integer representing its size in bytes, and then append the binary, giving 13 bytes in total.

Ascii:     ␀   ␀  ␀   ␉  F    i   a    t   ␣    l    u    x   !

Binary:    0   0   0   9  70  105  97  116  32  108  117  120  33

Now, the first 4 bytes that reach our receiver can be interpreted as the length of the next logical message. We can use this number to tell gen_tcp:recv how many bytes we want to read from the socket.

To encode an integer into 32 bits, we'll use Python's struct module. struct.pack(">I", 9) will do exactly what we want: encode a 32-bit unsigned Integer (9, in this case) in Big-endian (or network) order.

#!/usr/bin/env python3
import socket
import struct
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(("127.0.0.1", 7777))
data_to_send = b'a' * 10000
header = struct.pack(">I", len(data_to_send))
sock.sendall(header + data_to_send)

On the decoding side, we'll break up the receiving into two parts:

1) Read 4 bytes from the socket, interpret these as Header, a 32-bit unsigned int.

2) Read Header bytes off the socket. The receiving Erlang process will 'block' until that much data is read (or until the other side disconnects). The received bytes constitute a logical message.

#!/usr/bin/env escript
main(_) ->
  {ok, L} = gen_tcp:listen(7777, [binary, {active, false}, {reuseaddr, true}]),
  {ok, Sock} = gen_tcp:accept(L),
  {ok, <<Header:32>>} = gen_tcp:recv(Sock, 4),
  io:format("Got header: ~p~n", [Header]),
  {ok, String} = gen_tcp:recv(Sock, Header),
  io:format("Got string of length: ~p~n", [byte_size(String)]),
  erlang:halt(0).

When we run our scripts, we'll see the Erlang receiver print the following:

Got header: 10000
Got string of length: 10000

Success! But apparently, our application needs to handle messages much bigger than 10 kilobytes. Let's see how far we can take this approach.

Yet more data

Can we do a megabyte? Ten? A hundred? Let's find out, using the following loop for the sender:

#!/usr/bin/env python3
import socket
import struct
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(("127.0.0.1", 7777))
for l in [1000, 1000*1000, 10*1000*1000, 100*1000*1000]:
    data_to_send = b'a' * l
    header = struct.pack(">I", len(data_to_send))
    sock.sendall(header + data_to_send)
sock.close()

...and a recursive receive function for the receiver:

#!/usr/bin/env escript
recv(Sock) ->
  {ok, <<Header:32>>} = gen_tcp:recv(Sock, 4),
  io:format("Got header: ~p~n", [Header]),
  {ok, String} = gen_tcp:recv(Sock, Header),
  io:format("Got string of length: ~p~n", [byte_size(String)]),
  recv(Sock).

main(_) ->
  {ok, L} = gen_tcp:listen(7777, [binary, {active, false}, {reuseaddr, true}]),
  {ok, Sock} = gen_tcp:accept(L),
  recv(Sock).

Running this will lead to our Erlang process crashing with an interesting message:

Got header: 1000
Got string of length: 1000
Got header: 1000000
Got string of length: 1000000
Got header: 10000000
Got string of length: 10000000
Got header: 100000000
escript: exception error: no match of right hand side value {error,enomem}

enomem looks like a strange kind of error indeed. It happens when we get the 100-megabyte header and attempt to read that data off the socket. Let's go spelunking to find out where this error is coming from.

Spelunking for {error, enomem}

First, let's take a look at what gen_tcp:recv does with its arguments. It seems that it checks inet_db to find our socket, and calls recv on that socket.

OK, let's check out inet_db. Looks like it retrieves module information stored via erlang:set_port_data, in the call above.

A grepping for a call to inet_db:register_module reveals that multiple modules register themselves this way. Among these, we find one of particular interest.

lib/kernel/src/inet_tcp.erl
169:        inet_db:register_socket(S, ?MODULE),
177:        inet_db:register_socket(S, ?MODULE),

Let's see how inet_tcp.erl implements recv. Hmm, just a pass-through to prim_inet. Let's look there.

It seems here that our erlang call-chain bottoms out in a call to ctl_cmd, which is itself a wrapper to erlang:port_control, sending control data over into C-land. We'll need to look at out TCP port driver to figure out what comes next.

    case ctl_cmd(S, ?TCP_REQ_RECV, [enc_time(Time), ?int32(Length)])

A slight hitch is finding the source code for this driver. Perhaps the marco ?TCP_REQ_RECV can help us find what we're after?

$  rg 'TCP_REQ_RECV'
lib/kernel/src/inet_int.hrl
100:-define(TCP_REQ_RECV,           42).

erts/preloaded/src/prim_inet.erl
584:    case ctl_cmd(S, ?TCP_REQ_RECV, [enc_time(Time), ?int32(Length)]) of

erts/emulator/drivers/common/inet_drv.c
735:#define TCP_REQ_RECV           42
10081:    case TCP_REQ_RECV: {
10112:  if (enq_async(INETP(desc), tbuf, TCP_REQ_RECV) < 0)

A-ha! inet_drv.c, here we come!

Indeed, this C function here, responsible for the actual call to sock_select, will proactively reject recv calls where the requested payload size n is bigger than TCP_MAX_PACKET_SIZE:

if (n > TCP_MAX_PACKET_SIZE)
    return ctl_error(ENOMEM, rbuf, rsize);

and TCP_MAX_PACKET_SIZE itself is defined in the same source file as:

#define TCP_MAX_PACKET_SIZE 0x4000000 /* 64 M */

thereby explaining our weird ENOMEM error.

Now, how to solve this conundrum? A possible approach would be to maintain some state in our receiver, optimistically read as much data as possible, and then try to reconstruct the logical messages, perhaps using something like erlang:decode_packet to take care of the book-keeping for us.

Taking a step back — and finding a clean solution

Before we jump to writing more code, let's consider our position. We're trying to read a framed message off of a TCP stream. It's been done thousands of times before. Surely the sagely developers whose combined experience is encoded in OTP have thought of an elegant solution to this problem?

It turns out that if you read the very long man entry for inet:setopts, you'll eventually come across this revealing paragraph:

{packet, PacketType}(TCP/IP sockets)

Defines the type of packets to use for a socket. Possible values:

raw | 0

No packaging is done.

1 | 2 | 4

Packets consist of a header specifying the number of bytes in the packet, followed by that number of bytes. The header length can be one, two, or four bytes, and containing an unsigned integer in big-endian byte order. Each send operation generates the header, and the header is stripped off on each receive operation.

The 4-byte header is limited to 2Gb.

Packets consist of a header specifying the number of bytes in the packet, followed by that number of bytes. Yes indeed they do! Let's try it out!

#!/usr/bin/env escript
recv(Sock) ->
  {ok, String} = gen_tcp:recv(Sock,0),
  io:format("Got string of length: ~p~n", [byte_size(String)]),
  recv(Sock).

main(_) ->
  {ok, L} = gen_tcp:listen(7777, [binary, {active, false}, {reuseaddr, true}, {packet, 4}]),
  {ok, Sock} = gen_tcp:accept(L),
  recv(Sock).

And the output is:

Got string of length: 1000
Got string of length: 1000000
Got string of length: 10000000
Got string of length: 100000000
escript: exception error: no match of right hand side value {error,closed}

Problem solved! (The last error is from a recv call on the socket after it has been closed from the Python side). Turns out that our TCP framing pattern is in fact so common, it's been subsumed by OTP as a mere option for gen_tcp sockets!

If you'd like to know why setting this option lets us sidestep the TCP_MAX_PACKET_SIZE check, I encourage you to take a dive into the OTP codebase and find out. It's suprisingly easy to navigate, and full of great code.

And if you ever find yourself fighting a networking problem using brute-force in Erlang, please consider the question: "Peraphs this was solved long ago and the solution lives in OTP?" Chances are, the answer is yes!

Ponylang (SeanTAllen)

Last Week in Pony - February 17, 2019 February 17, 2019 02:19 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, our users’ mailing list, our Slack community, or join us on IRC.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

Luke Picciau (user545)

Gnubee Install Guide February 17, 2019 01:51 AM

The GnuBee is an open source crowd funded nas system. It requires a fair bit of configuration to use and I found that the docs on the website to be fairly inadequet for setting up a working system so after a lot of messing around I have come up with a guide to get to a fully working debian system quickly. What you need The gnubee An SD card A 19v power supply A usb UART cable!

Kevin Burke (kb)

Going Solo, Successfully February 17, 2019 12:48 AM

Three years ago I quit my job and started consulting full time. It's worked out really well. I get to spend more time doing things I like to do and I've been able to deliver great products for clients. I wanted to go over some tips for starting a successful consulting business.

  • Charge more - Everyone says it and it's true. I started out charging a monthly rate that was close to my full time salary / 12. This is not a good idea because you have overhead that your employer is no longer covering - health care probably the biggest one, you don't have paid vacations, there may be unpaid downtime between contracts and also companies might not pay you. You need to be charging a lot more to just break even.

    I dread "what's your rate" conversations every time and they haven't gotten easier. Before I quote my rate I reread the details of the High Tech Employee Antitrust case to pump myself up - it reminds me that I'm negotiating with a company that couldn't care less really and I am the only one who's going to stand up for myself. If you think you don't need the extra money - get it anyway, and then donate more to charities at the end of the year/buy CD's/put it in the stock market/give it to the government. Amazon just made $11 billion and paid $0 in taxes; you are going to spend an additional dollar better than Amazon's executives will.

    If you are not sure how much to charge, quote each new client more than the last. Your quote is often a signal of your quality so it's not even really the case that demand slopes downwards as you charge more.

    If you are working with a client and they are very happy with your work and want to extend your contract consider asking for a higher rate. "Now that you know how good I am," etc.

  • Get the money - Signed contracts, work performed, don't matter until the money hits your bank account. I learned this the hard way. If a company is going under your invoices are worthless. You can hold onto the company IP but that's probably also worthless. You can sue but at the end you will win a judgment that you can collect money from a company that doesn't have any to pay you.

    Try to get as much of the contract as you can paid up front - I generally ask for half or more up front. If a company offers Net 30 ask if you can shorten it to Net 5 or 10 or submit your invoices in advance. Submit invoices on time - it's a very costly mistake and you won't learn its importance until it's too late.

    Try as hard as you can to figure out the financial health of the company - if you can do your homework in the press or ask questions to your primary point of contact, like how much cash they are burning, how many months of runway do you have. If a company is not forthcoming with this information it's a red flag that they may not be able to pay you.

    If you see any red flags - the company wants to cut the contract short, people start leaving, company suddenly cuts back on perks - tell your contact that you need to be paid upfront or you are not going to work anymore. If they push back on this they may not have the cash to pay you at all. It's a crappy situation but better to cut your losses than to work for someone that can't actually pay.

  • Don't charge by the hour - I have never actually done this so I can't speak to how bad it is but don't do this. You don't want a client to cut you loose at 3pm and suddenly you lose three hours you were counting on. Charge per week.

  • Get a lawyer - Get a lawyer to review every contract you sign. Read through them, flag concerning things to the lawyer. They will suggest language. Send the language to the company. You are not being difficult when you do this, the company does this all the time. Lawyers are expensive, expect to pay north of $400 per hour and contract review can take 30-60 minutes. This money is worth it.

    A good clause to try to push for is limitation of liability. You don't want to be in a situation where $2 million of damages occurred or a high value client left the company because of an error you pushed and the company is suddenly coming after everything you own. Similarly the company may want to protect against you trying to sue them for a high amount of damages to your reputation, future business etc. Limiting the total liability to the size of the contract, or a multiple of the size of the contract - on both sides - can be a good move.

  • Register as a Company - Consult with the lawyer you hired on what kind of company you want to be. Generally the more "company-like" you are the harder it is for companies to try to take your personal assets. I don't have employees or shareholders so I have a single member LLC that is disregarded for tax purposes — read this description from the IRS. Sometimes businesses are confused what this means when I tell them or try to sign up for things. Still, it is a good fit for me. It may not be for you - I am not a lawyer, you should talk with one, learn the tradeoffs and see what makes sense for your business.

  • Make Sure Contracts Are Signed With the Company - The contracts you sign should be between the client you are working with and your company NOT between the client and you personally. Discuss this with your lawyer.

  • Get an accountant - As a small business a lot of stuff is tax deductible - a home office, client travel, for example, even if it's just across town - and you want to make sure you are getting ~35% off on everything that you can. An accountant will help you with this.

  • Market yourself - Not necessarily ads or sponsorships, but: everyone you've worked with full time should know they can hire you now. If they don't then reach out to people and let them know. Put up a website that engineers can send to their boss. My website isn't fancy but it is effective. Order business cards - VistaPrint is garbage, order from moo.com. If you have a website or open source projects put a note at the bottom advertising that you're available for hire, like the one at the bottom of this post.

  • Set up separate accounts for everything - Open separate accounts for your business. Get a business credit card or just a separate cash back card on your personal account. I don't have a checking account registered for the business but I opened a second checking account that I call my "business account". Clients pay into that account and I pay the business credit card out of that account. I even have a separate Clipper card that I use for business travel.

    There are two reasons for this. It makes tax accounting a lot easier. I know that every transaction on the business Clipper card is work travel and can be expensed; I don't have to try to remember what I was doing when I paid $2 to SamTrans at 5:34pm on July 27.

    Second, if you don't keep good records for the business - if you "commingle" funds between your personal life and the business - it makes it much easier for clients to go after your personal assets, what's called "piercing the veil." Separate accounts (and discipline about transfers!) make it much easier to argue that your business income and spending and personal income and spending are separate even if you don't necessarily have the legal structures to back them up.

    I also set up a new Github account for every company I work with. This avoids any issues with emails going to the wrong place, or the need to grant/revoke permissions to any 3rd party tools a company uses. I use github.com/kevinburke/swish to swap SSH settings between my Github accounts:

    $ cat $(which notion-github)
    #!/usr/bin/env bash
    ${GOPATH}/bin/swish --identity-file ${HOME}/.ssh/github_notion_ed25519 --user kevinburkenotion
  • Balancing multiple clients: If you can do this or do things like charge retainers, great. I find it really hard to switch contexts so I work with one client at a time and treat it as a full time job. Do what works for you.

  • Give back to the tools that make you successful - I give a percentage of my earnings every year to support software tools that help me do my job - iTerm2, Vim, Go, Postgres, Node.js, Python, nginx, various other open source projects. You should consider doing this too. (If you are an open source maintainer reading this - tell me how I can pay you!!)

February 16, 2019

David Wilson (dw)

Threadless mode in Mitogen 0.3 February 16, 2019 10:00 PM

Mitogen has been explicitly multi-threaded since the design was first conceived. This choice is hard to regret, as it aligns well with the needs of operating systems like Windows, makes background tasks like proxying possible, and allows painless integration with existing programs where the user doesn't have to care how communication is implemented. Easy blocking APIs simply work as documented from any context, and magical timeouts, file transfers and routing happen in the background without effort.

The story has for the most part played out well, but as work on the Ansible extension revealed, this thread-centric worldview is more than somewhat idealized, and scenarios exist where background threads are not only problematic, but a serious hazard that works against us.

For that reason a new operating mode will hopefully soon be included, one where relatively minor structural restrictions are traded for no background thread at all. This article documents the reasoning behind threadless mode, and a strange set of circumstances that allow such a major feature to be supported with the same blocking API as exists today, and surprisingly minimal disruption to existing code.

Recap

Above is a rough view of Mitogen's process model, revealing a desirable symmetry as it currently exists. In the master program and replicated children, the user's code maintains full control of the main thread, with library communication requirements handled by a background thread using an identical implementation in every process.

Keeping the user in control of the main thread is important, as it possesses certain magical privileges. In Python it is the only thread from which signal handlers can be installed or executed, and on Linux some niche system interfaces require its participation.

When a method like remote_host.call(myfunc) is invoked, an outgoing message is constructed and enqueued with the Broker thread, and a callback handler is installed to cause any return value response message to be posted to another queue created especially to receive it. Meanwhile the thread that invoked Context.call(..) sleeps waiting for a message on the call's dedicated reply queue.

Latches

Those queues aren't simply Queue.Queue, but a custom reimplementation added early during Ansible extension development, as deficiencies in Python 2.x threading began to manifest. Python 2 permits the choice between up to 50 ms latency added to each Queue.get(), or for waits to execute with UNIX signals masked, thus preventing CTRL+C from interrupting the program. Given these options a reimplementation made plentiful sense.

The custom queue is called Latch, a name chosen simply because it was short and vaguely fitting. To say its existence is a great discomfort would be an understatement: reimplementing synchronization was never desired, even if just by leveraging OS facilities. True to tribal wisdom, the folly of Latch has been a vast time sink, costing many days hunting races and subtle misbehaviours, yet without it, good performance and usability is not possible on Python 2, and so it remains.

Due to this, when any thread blocks waiting for a result from a remote process, it always does so within Latch, a detail that will soon become important.

The Broker

Threading requirements are mostly due to Broker, a thread that has often changed role over time. Today its main function is to run an I/O multiplexer, like Twisted or asyncio. Except for some local file IO in master processes, broker thread code is asynchronous, regardless of whether it is communicating with a remote machine via an SSH subprocess or a local thread via a Latch.

When a user's thread is blocked on a reply queue, that thread isn't really blocked on a remote process - it is waiting for the broker thread to receive and decode any reply, then post it to the queue (or Latch) the thread is sleeping on.

Performance

Having a dedicated IO thread in a multi-threaded environment simplifies reasoning about communication, as events like unexpected disconnection always occur in a consistent location far from user code. But as is evident, it means every IO requires interaction of two threads in the local process, and when that communication is with a remote Mitogen process, a further two in the remote process.

It may come as no surprise that poor interaction with the OS scheduler often manifests, where load balancing pushes related communicating threads out across distinct cores, where their execution schedule bears no resemblance to the inherent lock-step communication pattern caused by the request-reply structure of RPCs, and between threads of the same process due to the Global Interpreter Lock. The range of undesirable effects defies simple description, it is sufficient to say that poor behaviour here can be disastrous.

To cope with this, the Ansible extension introduced CPU pinning. This feature locks related threads to one core, so that as a user thread enters a wait on the broker after sending it a message, the broker has much higher chance of being scheduled expediently, and for its use of shared resources (like the GIL) to be uncontended and exist in the cache of the CPU it runs on.

Runs of tests/bench/roundtrip.py with and without pinning.
Pinned? Round-trip delay
No 960 usec Average 848 usec ± 111 usec
782 usec
803 usec
Yes 198 usec Average 197 usec ± 1 usec
197 usec
197 usec

It is hard to overstate the value of pinning, as revealed by the 20% speedup visible in this stress test, but enabling it is a double-edged sword, as the scheduler loses the freedom to migrate processes to balance load, and no general pinning strategy is possible that does not approach the complexity of an entirely new scheduler. As a simple example, if two uncooperative processes (such as Ansible and, say, a database server) were to pin their busiest workers to the same CPU, both will suffer disastrous contention for resources that a scheduler could alleviate if it were permitted.

While performance loss due to scheduling could be considered a scheduler bug, it could be argued that expecting consistently low latency lock-step communication between arbitrary threads is unreasonable, and so it is desirable that threading rather than scheduling be considered at fault, especially as one and not the other is within our control.

The desire is not to remove threading entirely, but instead provide an option to disable it where it makes sense. For example in Ansible, it is possible to almost halve the running threads if worker processes were switched to a threadless implementation, since there is no benefit in the otherwise single-threaded WorkerProcess from having a distinct broker thread.

UNIX fork()

In its UNIX manifestation, fork() is a defective abstraction surviving through symbolism and dogma, conceived at a time long predating the 1984 actualization of the problem it failed to solve. It has remained obsolete ever since. A full description of this exceeds any one paragraph, and an article in drafting since October already in excess of 8,000 words has not yet succeeded in fully capturing it.

For our purposes it is sufficient to know that, as when mixed with most UNIX facilities, mixing fork() with threads is extremely unsafe, but many UNIX programs presently rely on it, such as in Ansible's forking of per-task worker processes. For that reason in the Ansible extension, Mitogen cannot be permanently active in the top-level process, but only after fork within a "connection multiplexer" subprocess, and within the per-task workers.

In upcoming work, there is a renewed desire for a broker to be active in the top-level process, but this is extremely difficult while remaining compatible with Ansible's existing forking model. A threadless mode would be immediately helpful there.

Python 2.4

Another manifestation of fork() trouble comes in Python 2.4, where the youthful implementation makes no attempt to repair its threading state after fork, leading to incurable deadlocks across the board. For this reason when running on Python 2.4, the Ansible extension disables its internal use of fork for isolation of certain tasks, but it is not enough, as deadlocks while starting subprocesses are also possible.

A common idea would be to forget about Python 2.4 as it is too old, much as it is tempting to imagine HTTP 0.9 does not exist, but as in that case, Python is treated not just as a language runtime, but as an established network protocol that must be implemented in order to communicate with infrastructure that will continue to exist long into the future.

Implementation Approach

Recall it is not possible for a user thread to block without waiting on a Latch. With threadless mode, we can instead reinterpret the presence of a waiting Latch as the user's indication some network IO is pending, and since the user cannot become unblocked until that IO is complete, and has given up forward execution in favour of waiting, Latch.get() becomes the only location where the IO loop must run, and only until the Latch that caused it to run has some result posted to it by the previous iteration.

@mitogen.main(threadless=True)
def main(router):
    host1 = router.ssh(hostname='a.b.c')
    host2 = router.ssh(hostname='c.b.a')

    call1 = host1.call_async(os.system, 'hostname')
    call2 = host2.call_async(os.system, 'hostname')

    print call1.get().unpickle()
    print call2.get().unpickle()

In the example, after the (presently blocking) connection procedure completes, neither call_async() wakes any broker thread, as none exists. Instead they enqueue messages for the broker to run, but the broker implementation does not start execution until call1.get(), where get() is internally synchronized using Latch.

The broker loop ceases after a result becomes available for the Latch that is executing it, only to be restarted again for call2.get(), where it again runs until its result is available. In this way asynchronous execution progresses opportunistically, and only when the calling thread indicated it cannot progress until a result is available.

Owing to the inconvenient existence of Latch, an initial prototype was functional with only a 30 line change. In this way, an ugly and undesirable custom synchronization primitive has accidentally become the centrepiece of an important new feature.

Size Benefit

The intention is that threadless mode will become the new default in a future version. As it has much lower synchronization requirements, it becomes possible to move large pieces of code out of the bootstrap, including any relating to implementing the UNIX self-pipe trick, as required by Latch, and to wake the broker thread from user threads.

Instead this code can be moved to a new mitogen.threads module, where it can progressively upgrade an existing threadless mitogen.core, much like mitogen.parent already progressively upgrades it with an industrial-strength Poller as required.

Any code that can be removed from the bootstrap has an immediate benefit on cold start performance with large numbers of targets, as the bottleneck during cold start is often a restriction on bandwidth.

Performance Benefit

Threadless mode tallies in well with existing desires to lower latency and resource consumption, such as the plan to reduce context switches.

.right-aligned td, .right-aligned th { text-align: right; }
Runs of tests/bench/roundtrip.py with and without threadless
Threaded+Pinned Threadless
Average Round-trip Time 201 usec 131 usec (-34.82%)
Elapsed Time 4.220 sec 3.243 sec (-23.15%)
Context Switches 304,330 40,037 (-86.84%)
Instructions 10,663,813,051 8,876,096,105 (-16.76%)
Branches 2,146,781,967 1,784,930,498 (-15.85%)
Page Faults 6,412 17,529 (+173.37%)

Because no broker thread exists, no system calls are required to wake it when a message is enqueued, nor are any necessary to wake the user thread when a reply is received, nor any futex() calls due to one just-woke thread contending on a GIL that has not yet been released by a just-about-to-sleep peer. The effect across two communicating processes is a huge reduction in kernel/user mode switches, contributing to vastly reduced round-trip latency.

In the table an as-yet undiagnosed jump in page faults is visible. One possibility is that either the Python or C library allocator employs a different strategy in the absence of threads, the other is that a memory leak exists in the prototype.

Restrictions

Naturally this will place some restraints on execution. Transparent routing will no longer be quite so transparent, as it is not possible to execute a function call in a remote process that is also acting as a proxy to another process: proxying will not run while Dispatcher is busy executing the function call.

One simple solution is to start an additional child of the proxying process in which function calls will run, leaving its parent dedicated just to routing, i.e. exclusively dedicated to running what was previously the broker thread. It is expected this will require only a few lines of additional code to support in the Ansible extension.

For children of a threadless master, import statements will hang while the master is otherwise busy, but this is not much of a problem, since import statements usually happen once shortly after the first parent->child call, when the master will be waiting in a Latch.

For threadless children, no background thread exists to notice a parent has disconnected, and to ensure the process shuts down gracefully in case the main thread has hung. Some options are possible, including starting a subprocess for the task, or supporting SIGIO-based asynchronous IO, so the broker thread can run from the signal handler and notice the parent is gone.

Another restriction is that when threadless mode is enabled, Mitogen primitives cannot be used from multiple threads. After some consideration, while possible to support, it does not seem worth the complexity, and would prevent the aforementioned reduction of bootstrap code size.

Ongoing Work

Mitogen has quite an ugly concept of Services, added in a hurry during the initial Ansible extension development. Services represent a bundle of a callable method exposed to the network, a security policy determining who may call it, and an execution policy governing its concurrency requirements. Service execution always happens in a background thread pool, and is used to implement things like file transfer in the Ansible extension.

Despite heavy use, it has always been an ugly feature as it partially duplicates the normal parent->child function call mechanism. Looking at services from the perspective of threadless mode reveals some notion of a "threadless service", and how such a threadless service looks even more similar to a function call than previously.

It is possible that as part of the threadless work, the unification of function calls and services may finally happen, although no design for it is certain yet.

Summary

There are doubtlessly many edge cases left to discover, but threadless mode looks very doable, and promises to make Mitogen suitable in even more scenarios than before.

Until next time!

Just tuning in?

Jan van den Berg (j11g)

Ten years on Twitter 🔟❤️ February 16, 2019 07:44 AM

Today marks my ten year anniversary on Twitter! There are few other web services I have been using for ten years. Sure, I have been e-mailing and blogging for longer, but those are activities — like browsing — and not specifically tied to one service (e.g. Gmail is just one of many mail services). And after ten years, Twitter is still a valuable and fun addition to life online. But it takes a bit of work to keep it fun and interesting.

TL;DR

  • Twitter is your Bonsai tree: cut and trim.
  • Use the Search tab, it’s awesome!
  • Stay away from political discussions.
  • Be nice! No snark.
  • Bookmark all the things.

Twitter, the protocol

Twitter, of course, provides short synchronous one-to-all updates. In comparison; mail and blogging are asynchronous. Their feedback loop is different and they’re less instant. And WhatsApp or messaging are forms of one-to-many communication and they’re not public (so not one-to-all). So Twitter takes a unique place among these communication options.

Effectively the service Twitter provides is it’s own thing. Because Twitter is more a protocol, or an internet utility if you like. And more often than not, protocols or utilities tend to get used in ways they weren’t supposed to. I’ve written about Twitter many times before. And I love blogging and RSS but Twitter for me is still the place for near real-time updates. This post is part celebration of Twitter and part tips how I, personally, use this protocol to keep it fun and interesting.

Bonsai

Twitter can be many things to many people. For some people it can be the number one place to get their news on politics. For others Twitter is all about comedy (Twitter comedy is certainly a fun place!) or sports (I do follow quite a bit of NBA news). And some people just jump in, enjoy the moment, not caring about what came before and logging off again. And that is fine, but that is just not how I roll. When I follow you, I care about what you have to say, so I make an effort to read it.

So I am careful about following people that tweet very often. When I check out a profile page, and see a user with 45,978 updates, that’s usually an indication that I will not follow that account. But, this is me. I see my Twitter timeline like a bonsai tree, cutting and trimming is an integral part of keeping things manageable. Because when you’re not careful, Twitter can become overwhelming. Certainly when you’re a strict chronological timeline user, like I am. But, sparingly following accounts can make you miss out on great stuff, right?

Search tab

My solution to this problem is the Search tab (available on the app and mobile). Because this tab is actually pretty good! Twitter knows my interests based on a cross-section of accounts I follow, and in this tab it makes a nice selection of tweets that I need to see. It is my second home, my second timeline. Usually I catch up on interesting things of otherwise loud Twitter accounts (i.e. lots of tech accounts that I don’t follow). So Twitter helps me to point out things that I still would like to see. I get the best of both worlds. It’s great!

Politics

There are few subjects as flammable as politics on Twitter. So I try to stay away from politics and try not to get involved in political discussions. That doesn’t mean I am not aware of things going on, or that I am not interested in politics. Quite the opposite! I just don’t think Twitter is the best place for political debate. The new 280 character limit was an improvement, but it’s still too short for real discussions or nuance (maybe this is true for the internet as a whole). Sure, certain threads can provide insight, and some people really know what they’re talking about. But I will think twice before personally entering a discussion. I do follow quite a bit of blogs/sites on politics and Twitter helps me to point to those things. These places usually help me more in understanding things that are otherwise hard to express in 280 characters.

Be positive

It is very easy to be dismissive or negative on Twitter. But very little good comes from that. So I always try to add something positive. I recently came across this tweet, and I think this sums it up rather well:

Pointer

Like stated Twitter can be many things to many people. But from day one, for me it has always been a place to point and get pointed to interesting things. The best Twitter for me is Twitter as a jump-off zone. My love for Twitter comes from the experience of being pointed to great books, movies, blogs, (music) videos and podcasts. And I am a heavy user of the bookmark option. (I tend to like very little on Twitter, which is more of a thank you nowadays.) But I bookmark all the things. Usually I scan and read my timeline on mobile, bookmark the interesting things and come back to it later in the day on a PC.

What’s next?

I had been blogging for a few years when Twitter came along. So I have never been able to shake the feeling of seeing Twitter as a micro blog for everyone. (Which is just one of its uses.) I am also aware of concepts like micro.blog, matrix.org or Mastodon. Services that, at the very least, have been inspired by Twitter, and build further on the idea of a communication protocol. But the thing is, Twitter was first, and Twitter is where everybody is. It’s part of the plumbing of the internet now, I don’t see it going away soon and that is all right by me! Cheers!

The post Ten years on Twitter 🔟❤️ appeared first on Jan van den Berg.

February 15, 2019

Pierre Chapuis (catwell)

Goodbye Lima February 15, 2019 07:40 PM

You may have heard it already: five years after I joined Lima, the company is shutting down.

Obviously, things did not go like we hoped they would. Customers are disappointed, and wondering what will happen now. Let me try to answer some of the questions I read online the best I can.

Please note that this is my personal take on things, and does not represent the views of anyone else but me (i.e. not Lima, not other employees...).

What happened to the company exactly?

Lima as a company no longer exists. It ran out of money. Its employees (including me) have all been fired, and its assets will be sold to pay its debts.

Regarding why the company died, it is a long story and it is not my place to tell it all. What I can say is that it ran into unexpected funding problems in early 2017, shortly after we started shipping the Lima Ultra. During most of 2017, there was strong demand for the product but we could not fulfill it because we did not have enough cash to pay for production and shipping (Remember the never-ending waiting list?) At the end of the year, we had to fire a large part of the team and we switched our business model to sell our software to other companies. We made a deal where we worked for another startup. The deal was good enough to keep the company afloat and the product alive for a year, but it forced us to stop selling Lima devices. What happened recently is that this deal eventually fell through, leaving us with no viable options.

This past year was not the best time of my life, or for any of the other employees who stayed. Many of us could have left for much better jobs at any time, some did and I cannot blame them. All those who stayed on board all this time did so hoping for a better end for the company and its customers.

What will happen to the devices?

Once Lima's servers shut down, Lima will keep working on your local LAN with the devices you have already paired with it. However, a lot of things will stop working.

First, it won't be possible to add new devices to the system. That's because, when you log a new device into Lima, you do so with an email and password. To find out which Lima those credentials belong to, the system asks a server, and that server won't answer anymore.

Second, it won't be possible to reset your password, because email confirmation will be broken. If you have forgotten your password, change it now while the servers are still up.

Third, the sharing feature will be broken, because it relies on sending HTTP requests to relay servers which will go down as well.

Finally, it won't be possible to access Lima from outside your home. This is a little harder to explain than the rest. Basically all communications between anything related to Lima (Lima devices, your devices, servers...) happen in a peer-to-peer VPN. To "locate" devices within the VPN (basically figure out how to talk to something), devices rely on a node which is called the ZVPN master. The IP address and public key of that node are hardcoded into every Lima client, and that node will go down as well. The use of that node is not needed on local networks because Lima devices and applications have a protocol to pair with other devices associated to the same account on a LAN without talking to any server.

Is there a risk for my personal data?

At that moment, not that I know of. Your data was never stored on Lima's servers, and all data traffic going through relay servers is end-to-end encrypted, which means that even if an attacker took control of one they couldn't decipher your data.

However in the long run there are two issues.

First, we won't be able to publish updates for the Lima firmware and applications anymore. If a security issue is found in one of the components they use, they may become vulnerable with no way to fix them.

Second, if someone was to acquire all the assets or Lima, including the domain and code signing certificate, they could theoretically do everything Lima was able to do, including publishing updates. That means they could publish malicious updates of the applications and firmware.

That second issue sounds scary but I do not think there is any chance it will happen. Potential acquirers will probably be more interested in Lima's technological IP, there are very few chances that an acquirer will get all the assets necessary for such an attack, and even if they do they probably won't have an interest in performing it. Even if it did happen, it would be easy to notice. Still, I have to mention it for transparency.

What I will personally do now, and what I advise users to do as well, is export all my data out of Lima, unplug the device and uninstall all the applications.

Note: If you have problems when trying to recover your data (due to e.g. a hardware issue with the USB drive), do not uninstall the applications. The data on your desktop might sometimes help recovering some of the files.

If you have an issue with the Decrypt Tool, check here for potential answers.

What can users replace Lima with?

It depends on the users. I don't know anything that is exactly like Lima. There was Helixee, which I have never tried out, but I just found out they are shutting down as well. I also learned that a project I had never heard about before called Amber had a special offer for Lima customers.

For technical people, you can probably do most of what you were doing with Lima with a Synology NAS, or a setup based on some small computer and Open Source software such as Nextcloud or Cozy Cloud.

However, Lima was never designed for technical customers. It was built for, marketed to and mostly bought by non-technical people. For them, I don't have a good answer. I heard that WD My Cloud Home had become a lot better than it once was, but I have not tried it personally.

Can you open-source the code?

To the best of my knowledge, there is no way that can happen. This makes me extremely sad, especially since I know there are parts of the code I would love to reuse myself, and that could be useful to other projects.

The reason why we cannot open-source is that the code does not belong to us, the employees, or the CEO. Intellectual property is considered an asset of a bankrupt company, and as such will be sold to the highest bidder to pay the company's debts.

That being said, Lima has contributed some source code to a few Open Source projects already. Most importantly we fixed the issues in OSXFUSE that prevented it from being used for something like Lima, and those fixes are now in the main branch.

Completely independently from the company, the former CTO of Lima has also released a project which looks a lot like a second, fully decentralized iteration of the Lima network layer ZVPN (using a DHT instead of a master node, and WireGuard instead of TLS). Let me be clear: this project contains no code or IP from Lima, it is a clean room implementation.

Can you give us root access to the device?

For Lima Original, no, I think that would be impossible (or rather, I can't see a solution that doesn't involve soldering...). The device is not worth much today anyway, its specs are so low I don't think you could run any other private cloud software on it.

For Lima Ultra, a few of us ex-Lima employees (and the CEO) are trying to figure out a way to let users get root access. We can't promise anything, but we will keep you informed if we do.

EDIT (2019-02-18): We did it, check this out!

Why does it say something different in the Kickstarter FAQ?

Some people have mentioned that what was happening was not in line with what had been said in the Kickstarter FAQ.

This FAQ has been written in 2013, before I or any other Lima developer joined the company. At the time Lima was a very small project with two founders trying to raise $70,000 to make their dream happen. Instead they raised $1,229,074, hired 12 people (including me), and the rest is history.

I do not think we have not communicated like that ever since, especially regarding decentralization. As far as what I know we have been transparent that our servers were needed for some major features of the product, as it was obvious the few times they went down. You may ask why we didn't amend this page then, and the answer is (I think) that it is technically impossible to edit it after the campaign is over.

Regarding Open Source, I sincerely believe the CEO of Lima would have done it if it was possible, but with the success of the Kickstarter the company had to take VC funding very early on (see below), and from that moment on I do not think it was in his hands.

Where did all that Kickstarter money go?

OK, let's address this last. What Kickstarter money?

Yeah, the founders raised over a million dollar. But do you remember how much the backers paid for those devices? From $59 to $79 each. Well, as bad as the hardware was, it was planned for about 1000 devices, not over 10,000. And it was pretty expensive.

I don't know the exact figures, but basically Lima did not make money on those devices, or no significant amount of money at least. Which is why it raised extra cash from VCs just afterwards, to pay the team that worked on the project, the production of more devices to sell, etc...

If you still think something shady went on with that money, rest assured: when a company like Lima goes bankrupt, its books are closely investigated by the state, which is one of its main creditors. So if you are right, the people responsible will end up in jail. (Spoiler: I really don't think it will happen.)

What are you going to do next?

Yes, I have plans.

No, they are not in any way related to Lima.

I will tell you more next month, probably.

Gustaf Erikson (gerikson)

Fotografiska, 14 Feb 2019 February 15, 2019 06:13 PM

Jonas Bendiksen - The Last Testament

Magnum photographer photographs seven people around the world who claim they are Jesus Christ. Great reportage.

Anja Niemi - In Character

Self-portraits (sometimes doubled), with that “2-stops overexposed Portra” aesthetics that the kids like so much these days. It’s nothing we haven’t seen before.

Kirsty Mitchell - Wonderland

Exceedingly lush tableaux, backed by a tragic backstory (the memory of the creator’s deceased mother) and a hugely successful Kickstarter campaign. There’s no denying the craftmanship nor the quality of the work, but somehow it feels a bit weird for an artist to so publicly involve crowdfunding in something so private. On the other hand the work of Niemi (above) struck me as very cold and solitary, so what do I know about how artists get inspiration from others.

Pierre Chapuis (catwell)

Software Architecture Principles February 15, 2019 10:20 AM

This is just a short post to share what I now consider, after 10 years in the industry (and almost twice as many writing code), my core software architecture principles.

You may or may not agree with them all, but if you design software or systems, you should have a similar list in your head; it really helps a lot when making decisions.

Without further ado, the principles are:

  • Separation of Concern often trumps not repeating oneself (DRY). In other words, avoiding duplication does not justify introducing coupling.

  • Gall's Law: "A complex system that works is invariably found to have evolved from a simple system that worked."

  • Conway's Law: "Organizations produce designs which are copies of their communication structures."

  • When writing code or designing, stop and think "consequences". What will be the impact of what you are doing on the rest of the systems? Could there be adverse side-effects?

  • Think about debuggability in production. There is nothing worse than having your software break and not being able to figure out why. Do not automate things you do not understand.

  • Write code that is easy to delete, not easy to extend.

Andreas Zwinkau (qznc)

The Spartan Web February 15, 2019 12:00 AM

Defining a label for websites I like to visit and would like to see more of.

Read full article!

February 14, 2019

Jan van den Berg (j11g)

Blue Bananas – Wouter de Vries jr. & Thiemo van Rossum February 14, 2019 06:36 PM

Blauwe Bananen (Blue Bananas) is a management book that was number one for 38 days on managementboek.nl. It is aimed at people who generally don’t read management books. So it sometimes tries to be unnecessarily funny, seemingly afraid to alienate the reader with otherwise dry concepts. Nonetheless the message itself is pretty solid. The theme being: how to become a blue banana. A blue banana is a business with a unique skill set or proposition.

Blue Bananas – Wouter de Vries jr. & Thiemo van Rossum (2012) – 94 pages

That the message carries merit is not a surprise. This book unabashedly builds on the famous organisational and business strategy theories laid out by Treacy & Wiersema and Hamel & Prahalad. The book introduces readers to a succinct and on-point summary of their concepts. It does so by guiding the reader through four steps: Pursuits, Promises, Perception, Proof (freely translated by me from the Dutch B letter words).

With these steps the book makes the theory practical and consequently is very direct. Which is a good thing. To further cement the theory it offers 29 exercises and practical thought experiments (Things like “write down what you think are unique talents of your organisation”). Overall it does a good job of landing one of the main messages: it does not matter what value you add, if your customer does not perceive it as such. Everything you do as an organisation should add value to your customers’ experience.

If you rarely read management books, Blue Bananas can be a good starting point and offers valid questions of how to add value to your organisation.

The post Blue Bananas – Wouter de Vries jr. & Thiemo van Rossum appeared first on Jan van den Berg.

February 13, 2019

Derek Jones (derek-jones)

Offer of free analysis of your software engineering data February 13, 2019 03:02 AM

Since the start of this year, I have been telling people that I willing to analyze their software engineering data for free, provided they are willing to make the data public; I also offer to anonymize the data for them, as part of the free service. Alternatively you could read this book, and do the analysis yourself.

What will you get out of me analyzing your data?

My aim is to find patterns of behavior that will be useful to you. What is useful to you? You have to be the judge of that. It is possible that I will not find anything useful, or perhaps any patterns at all; this does not happen very often. Over the last year I have found (what I think are useful) patterns in several hundred datasets, with one dataset that I am still scratching my head over it.

Data analysis is a two-way conversation. I find some patterns, and we chat about them, hopefully you will say one of them is useful, or point me in a related direction, or even a completely new direction; the process is iterative.

The requirement that an anonymized form of the data be made public is likely to significantly reduce the offers I receive.

There is another requirement that I don’t say much about: the data has to be interesting.

What makes software engineering data interesting, or at least interesting to me?

There has to be lots of it. How much is lots?

Well, that depends on the kind of data. Many kinds of measurements of source code are generally available by the truck load. Measurements relating to human involvement in software development are harder to come by, but becoming more common.

If somebody has a few thousand measurements of some development related software activity, I am very interested. However, depending on the topic, I might even be interested in a couple of dozen measurements.

Some measurements are very rare, and I would settle for as few as two measurements. For instance, multiple implementations of the same set of requirements provides information on system development variability; I was interested in five measurements of the lines of source in five distinct Pascal compilers for the same machine.

Effort estimation data used to be rare; published papers sometimes used to include a table containing the estimate/actual data, which was once gold-dust. These days I would probably only be interested if there were a few hundred estimates, but it would depend on what was being estimated.

If you have some software engineering data that you think I might be interested in, please email to tell me something about the data (and perhaps what you would like to know about it). I’m always open to a chat.

If we both agree that it’s worth looking at your data (I will ask you to confirm that you have the rights to make it public), then you send me the data and off we go.

February 11, 2019

Tobias Pfeiffer (PragTob)

Benchee 0.14.0 – Micro Benchmarks? Pah, how about Nano Benchmarks! February 11, 2019 03:30 PM

Long time since the last benchee release, heh? Well, this one really packs a punch to compensate! It brings you a higher precision while measuring run times as well as a better way to specify formatter options. Let’s dive into the most notable changes here, the full list of changes can be found in the […]

February 10, 2019

David Wilson (dw)

Mitogen v0.2.4 released February 10, 2019 11:59 PM

Mitogen for Ansible v0.2.4 v0.2.5 has been released. This version is noteworthy as it contains major refinements to the core libary and Ansible extension to improve its behaviour during larger Ansible runs.

Work on scalability is far from complete, as it progresses towards inclusion of a patch held back since last summer to introduce per-CPU multiplexers. The current idea is to exhaust profiling gains from a single process before landing it, as all single-CPU gains continue to apply in that case, and there is much less risk of inefficiency being hidden in noise created by multiple multiplexer processes.

Please kick the tires, and as always, bug reports are welcome!

Just tuning in?

Ponylang (SeanTAllen)

Last Week in Pony - February 10, 2019 February 10, 2019 04:03 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, our users’ mailing list, our Slack community, or join us on IRC.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

February 09, 2019

Unrelenting Technology (myfreeweb)

haha wow my SoundFixer addon is on the addons.mozilla.org front page February 09, 2019 04:53 PM

haha wow my SoundFixer addon is on the addons.mozilla.org front page

Nikita Voloboev (nikivi)

What problem did you encounter? February 09, 2019 03:59 PM

What problem did you encounter? I updated the article to include `:simlayers {:o-mode {:key :o}}` for launcher key blocks as it is needed and was missing.

I am using Mojawe and you can look at my config on GitHub for reference of a working config I use currently.

Alex Wilson (mrwilson)

Notes from the Week #18 February 09, 2019 12:00 AM

Monday was pretty meeting-focused.

We had a huddle on deriving a set of SLOs from our initial graphite SLIs. The outcome of the session was that our metrics needed further refinement — what we actually want is to have a response time bound for well-formed requests and a different threshold for number of queries that time out or are invalid rather than overall request latency.

Our second session was a retrospective on how we handle ‘walk-ups’ — Shift is pretty lucky that we surrounded by our customers which keeps feedback loops tight but we become overloaded by questions and distractions. We used index cards to keep track of the number of walkups broken down by subject, and decided to productise it into a Google Form for longer-term storage and analytics.

Tuesday

I paired with Narayan on Tuesday to make some efficiency improvements to our generated firewall configurations. We’ve been less than judicious with some of the templated rule-sets and this was an opportunity to smooth out our global Puppet runtimes. We did this by putting feature flags on our security-related puppet classes and started to turn off parts that weren’t being used.

I also went along to my first weeknotes meetup! :D

The first venue that Steve suggested turned out to be mostly booked by a speed-dating event so we ended up decamping to a cocktail bar nearby. I always like meeting new people (big shout-out to Dan and Giuseppe) and it’s a weird sensation to hang out for the first time with people who you only know from the Twitter-sphere, but good times were had by all even if I did forget to actually eat and had a delicious two-pint dinner instead.

Wednesday

As tradition dictates, it was 20% time day — I finally got around to releasing my ProtonMail DNS terraform module and pushing it to the terraform registry.

I set aside a bit of time for attempting an upgrade of part of our Puppet systems and it turns out that it’s going to be a fair bit more work than I thought — we’re using the open-source version and we’ve architected it in a way that worked when we first brought it up but makes it harder to incrementally scale.

Thursday and Friday

I did a fair bit of pairing this week, in total!

I paired with Petrut on AWS optimisations and with Seng on improving the state of our SSL certificates, both of which required doing a fair bit of Terraform-ing (I swear this is 90% of my development time now, the rest is Python). I miss TDD’d app development. :(

Speaking of app development, Stephen made an initial release of a small puppet-token Slack app that builds on the data that we started piping into DynamoDB from our Puppet runs.

We have a monolithic shared puppet codebase and because we practice trunk-based development we also use a physical mutex to make sure only one team is committing/deploying at a time. This token, an android plush, often requires developers to go and “search” for it to acquire the lock (a practice that worked when we were small but Shift is working to make more scale-appropriate).

We can type /puppet-token into our Slack and it will tell you where it thinks the token is!

An improvement to our process, but the next step will be using these events (START, FINISH) to determine whether a run is in progress and use that as a mutex instead of our token. I’m excited for more of these small human-centric improvements.

This weekend, I ordered a copy of Shoshana Zuboff’s The Age of Surveillance Capitalism — probably required reading for working in ad-tech? — and I’m looking forward to eating through it, I’m going to regenerate my GPG keys, and then quite probably bake a Lemon Drizzle cake (which will require me taking at least some of it into work).

Originally published at blog.probablyfine.co.uk on February 9, 2019.

February 08, 2019

Ponylang (SeanTAllen)

Pony Stable 0.2.0 Released February 08, 2019 04:00 AM

Pony-stable 0.2.0 is a recommended release. It fixes a couple bugs that could result in end user issues.

February 07, 2019

Grzegorz Antoniak (dark_grimoire)

C++: Shooting yourself in the foot #4 February 07, 2019 06:00 AM

C++11 has introduced a better way to generate random numbers than the immortal srand(time(NULL)) and rand() % N method. However, this family of functions sometimes may behave in a not very intuitive way, especially when it comes to multi-platform programming.

TL;DR: before commenting, please at least read …

February 06, 2019

Jeff Carpenter (jeffcarp)

Kaiser SF Half Race Report February 06, 2019 09:46 PM

Overall It went great, I PR’d by 10 minutes! The course is super fast and the light drizzle of rain didn’t really put a damper on things. Report t-0:20 I arrived and was able to use the bathroom – they did a great job of making sure there were enough port-a-potties. After that since it was drizzling I hid under a tree to the side of the start line with a bunch of other runners who looked like they were from a club and knew what they were doing.

February 05, 2019

Bogdan Popa (bogdan)

Google Groups Without Google Accounts February 05, 2019 06:00 PM

It turns out that when you delete your Google accounts, Google unsubscribes you from any (public and private) Google Groups you’re a member of. I found out about this only because my inbox traffic these past couple of days felt unusually light so I went and looked at racket-users and, lo and behold, there were a bunch of new posts I hadn’t received. I get it. They want to avoid sending emails to an address that, from their perspective, no longer exists.

February 04, 2019

Bogdan Popa (bogdan)

Bye, Bye, Google February 04, 2019 07:00 AM

I spent this past weekend de-Google-ifying my life and, despite my expectations, it wasn’t too hard to do. I started by moving all of my websites off of Google App Engine and onto a dedicated box that I had already owned. That was straightforward enough. Next, I removed any Google Analytics snippets from each of them and replaced those with my own analytics server that I had built a while back (it doesn’t store any PII, only aggregate information (and very little of that, too)).

Jan van den Berg (j11g)

Plato – R.M. Hare February 04, 2019 12:41 AM

Writing short introductions to classic philosophers are hard. This book tries, but falls a bit short as a true introduction.

Plato – R.M. Hare (1983) – 117 pages

Plato, the first documented, Western philosopher set the pace for 25 centuries of philosophy. This book explains the culture and setting where Plato developed his philosophy, and their interrelation. It also touches on the main aspects of his philosophy as well as you could possibly expect in a short book of around 100 pages. But I do have two issues with this book.

Firstly, as a reader you need to bring your a-game. There are quite a few names and concepts thrown at you. I assume that people who pick up this book know very little about philosophy so this seems like a mismatch. Secondly; it does not help that most of the language is highly academic (note, I did read a Dutch translation). Two or three chapters were decisively easier to read than the rest of the book, because the language was completely different.

So even if I picked up a few things I would not suggest this book as an introduction to Plato. (Reasons are similar to the Kierkegaard book.) In 2019, if you need an introduction I would suggest you read the Wikipedia page. It’s clearer in language and structure than this book from 1983. I expect somewhere there must be easier introductions to philosophy, geared towards true novices. If not, consider it an untapped market (or maybe we have it already and it’s called Wikipedia).

The post Plato – R.M. Hare appeared first on Jan van den Berg.

February 03, 2019

Jeff Carpenter (jeffcarp)

Building a Running Pace Calculator With AMP February 03, 2019 11:45 PM

Sometimes you need to know how fast you need to run to achieve a personal best time. Previously the way I did this was to search “running pace calculator” and follow and use one of the top results. However, I was doing this almost always on mobile and none of those results are very mobile friendly. There might be good native apps for this, but I’m a fan of the web and don’t want to download an extra app if I can avoid it.

How I Host Static Sites With Automatic Deploy on Green February 03, 2019 11:19 PM

This site, jeffcarp.com, is written in markdown and uses the Hugo static site generator. This post walks you through how I set automatic building, testing, and deployment to Firebase hosting. Project Setup I assume we’re starting from a working Hugo project. For more on how to set that up, see the Hugo docs. Testing Setup I want the site to be Deploy-on-Green (i.e. only if it passes the tests). The CI setup I use is GCP Cloud Build.

Stig Brautaset (stig)

Musical Goals January Update February 03, 2019 10:38 PM

The first of (hopefully) monthly posts with updates on my musical goals for 2019. I cover achievements in January, and new goals for February.

Ponylang (SeanTAllen)

Last Week in Pony - February 3, 2019 February 03, 2019 03:57 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, our users’ mailing list, our Slack community, or join us on IRC.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

Gergely Nagy (algernon)

NOP NOP NOP says the clock, on the bug-fuse falls a lock February 03, 2019 01:00 AM

Lately, I've been porting Kaleidoscope to keyboards that happened to land on my desk for one reason or the other. Keyboards such as the ErgoDox EZ, the Atreus, Splitography, and most recently, KBD4x. In almost every case, I ran into weird issues I couldn't immediately explain, where the symptoms weren't search-engine friendly. There wasn't anything obviously wrong with the ported code, either, because the same code worked on another board. Figuring out what went wrong and where was an incredibly frustrating process. I'd like to save others from having to do the same digging, debugging, hair-tearing I did, so we'll look at these three cases today.

A tale of fuses and woe

The first problem I ran into was during the Splitography port. It should have been a straightforward port, because it is built on top of ATMegaKeyboard, like the Atreus port, which has been working wonderfully. I prepared the port in advance, before the keyboard arrived, and was eagerly waiting the shipment to try it. I was confident it will work out of the box. It did not: the left half was dead.

I quickly flashed QMK back to verify that the hardware is fine, and it was, both halves worked with QMK. What am I doing wrong then? I verified that the pinout is correct, I checked with a simple key logger that the problem is not that we don't act on key presses, but the firmware doesn't even see them. This was the first clue, but I wasn't paying enough attention, and went comparing ATMegaKeyboard's matrix scanning code to QMK. I even copied QMK's matrix scanner verbatim - to no avail.

At this point, I looked at the pinout again, and noticed that the left half's columns are all on PINF. Why aren't we able to read from PINF? It works on the Atreus! At this point, I searched for "reading from PINF not working", but since PINF isn't a common search term, my search engine helpfully added results for "ping not working" too - which I did not notice because I've been fighting this for over an hour by that point. Eventually, once describing the problem to Jesse on Discord, he gave me the final clue: JTAG.

The ATMega32u4 MCU the Splitography has JTAG support enabled in fuses by default. Most vendors who ship the MCU to end-users disable this, but on the Splitography, it wasn't disabled. This meant that using PINF didn't work, because the MCU was expecting to use it for JTAG, not as an input pin to read from. Once this was clear, it didn't take much time to find the solution by looking at the datasheet, the following sections in particular:

  • 2.2.7 (PIN Descriptions; PIN F)
  • 7.8.7 (On-chip Debug System)
  • 26.5.1 (MCU Control Register – MCUCR)

In the end, the fix was these two lines in the constructor of the Splitography hardware plugin:

MCUCR |= (1 << JTD); MCUCR |= (1 << JTD);

What it does, is it writes to the JTD bit of MCUCR twice within four cycles, which disables JTAG at run-time, and makes it possible for us to use PINF for input.

Time is of the essence

The next problem I faced was when I started to work on underglow support for the KBD4x keyboard. As expected, it didn't quite work with my port, but ran flawlessly with QMK. So what do I do? Compare the code.

The code to drive the LEDs on the KBD4x (pretty common WS2812 leds), I used the same source library the QMK code is based on. This was strange, because the code I used for the KBD4x LED strips, I used before for the Shortcut port, and everything was working fine there. Nevertheless, I went and compared the code, down on the compiled, optimized assembly level. It was exactly the same.

Yet, even though the code was the same, with QMK, I was able to set the LED colors as I saw fit. With my Kaleidoscope port, no matter what data I sent its way, the LEDs always ended up being bright white. This is surprisingly hard to search for, and my searches yielded no useful results. At the end of the day, I let my frustration out in QMK's discord a bit, and got a little hint from there: the WS2812 strips are very picky about timing.

After a good night's sleep, I went looking into the QMK sources to see if there's anything there I do differently in Kaleidoscope, focusing on time-related things, such as the clock. And that was the key!

The ATMega32u4 has an option to divide its clock speed, to conserve power. Like in the case of JTAG, this can be set or unset in fuses. Thankfully, like in the JTAG case, we can disable this at run-time too, with the following magic words:

CLKPR = (1 << CLKPCE); CLKPR = (0 << CLKPS3) | (0 << CLKPS2) | (0 << CLKPS1) | (0 << CLKPS0);

If the MCU divides its speed to conserve power, it will run slower, and all the timing the library I work with uses will be totally wrong. No wonder poor LEDs lit up white!

With the magic incantation added to the keyboard's constructor, I was able to set colors properly. Why wasn't this a problem on the Shortcut? Because it had clock division disabled in fuses.

NOP, NOP, NOP

Many days later, I had a few spare minutes, so I figured I'll add support for the KBD4x to Chrysalis. This was a 15 minute task, and everything worked fine, yay! I figured I'll build a sketch for my own uses while there, and that's when I noticed that the first column wasn't working at all.

Quickly flashing QMK back verified that the issue is not with the hardware. Yay, I guess?

So the usual thing happens: what went wrong? The pinout is the same as in QMK, JTAG and clock division are disabled. The first column is on PIN_F0, so making sure JTAG was disabled was my first step. Some other columns were also on PINF, and those worked, so it's not JTAG.

Frustrating. I cry out on Discord, and Jesse tells me immediately he saw something similar on the Planck, and had a fix. We look into ATMegaKeyboard, and indeed, there appears to be a fix there:

uint16_t ATMegaKeyboard::readCols() { uint16_t results = 0x00 ; for (uint8_t i = 0; i < KeyboardHardware.matrix_columns; i++) { // We need to pause a beat before reading or we may read // before the pin is hot asm("NOP"); results |= (!READ_PIN(KeyboardHardware.matrix_col_pins[i]) << i); } return results; }

Emphasis on the asm("NOP"). That line is supposed to slow us down a bit so that the pin we're reading has a chance to settle. I had two questions at this point: why isn't the existing fix enough, and why do I remember my first column working before?

Turns out, this is related to the previous issue! You see, when I turned clock division off, the keyboard started to run a bit faster, which meant that a single NOP didn't slow us down enough for the pin to go hot. With clock division, we were running slow enough for a single NOP to be enough.

But how do we fix this? I checked QMK (checking other, similar projects for clues is such a great thing!), and they delay for 30µs. While that'd work for us too, we didn't want to add an explicit delay on the fast path. I looked at the assembly to see if we can do anything smart there, and noticed an interesting thing: the compiler inlined .readCols(), and unrolled the loop too.

What if we didn't inline it? We'd have a function call overhead then, which is faster than a delay, but slower than being inlined. Adding an attribute that disables inlining made my first column work. However, I wasn't satisfied, figured we can do better. What if we allowed inlining, but stopped unrolling the loop? Checking the loop condition is faster than calling a function, but still slower than being inlined and unrolled. Turns out, disabling loop unrolling was enough in this case:

__attribute__((optimize("no-unroll-loops"))) uint16_t ATMegaKeyboard::readCols() { // ... }

Summary

In the end, if I read documentation, and truly understood the hardware I'm working with, I wouldn't have faced any of these issues. But I'm not good at hardware, never will be. My brain just stops if I try to ingest too much hardware documentation in one sitting. Which means I'll run into similar issues again for everyone's benefit, but mine! Yay.

And to think I'm about to build a home security system soon... yikes. I'm utterly scared about that prospect. So many things will go wrong, that the next post about hardware-related things going awry will be even longer.

Alex Wilson (mrwilson)

Notes from the Week #17 February 03, 2019 12:00 AM

Oh, it’s a long one. I’m trying another new format, breaking down by day — I often forget highlights in trying to limit myself to 2 or 3 things.

Monday

Going faster

I had a two hour session with the rest of the Team Leads about ways to help us go faster, within the constraints of keeping the Unruly culture that makes us unique and not over-egging the process pudding (so to speak).

It feels a lot like a linear/combinatorial optimisation problem that I learned about in school and uni respectively (I’m shuddering at the memory of manually running the Simplex algorithm during exams). There are a bunch of different levers we are able to pull but every action has an effect on everything else.

This probably falls somewhere in what Cynefin calls the complex domain, and we improve by Probing -> Sensing -> Responding.

Demo time

Shift have also started doing huddles to demonstrate our little pet projects since we’ve been trialling a flexible working system. I’m a self-identified morning person and I like to tinker when I’m in the office on my own.

This week I demo’d a custom Terraform module to wrap up different resources between GitHub, AWS, and other sources. Modules support multiple providers being passed in as attributes since 0.11, so this is no longer problematic to wire up.

Stephen demo’d a spike of a Slack app to administer resources on AWS, as well as communicate to users when they have stale assets, building on his 20% from last week

Tuesday

Yes/Yes/No

I’m deliberately very free-and-easy with our team’s process — when someone wants to try something new, like a facilitation technique or a way of doing things, I try my best to take a leaf from performance improv and respond with “Yes, and …” (unless the idea would potentially have drastic negative consequences, of course).

The idea came from Stephen listening to the podcast Reply All which has a segment called “Yes/Yes/No”. The podcasters look at a particular tweet, and answer “Yes” or “No” to whether they understand what it means or not, before finally explaining it to each-other so that everyone can answer Yes.

We tried using this format to really dig into our network security model, the technologies we use, the way we provision it and team feedback was a unanimous “let’s do this again”.

My gut reaction to why this worked for us was that actively engaging with the content and explaining bits of it to each-other rather than someone doing a demo made our brains work differently and absorb the information better.

1-to-1s

My team and I have 1-to-1s every other Tuesday after lunch. I genuinely love these because I’m able to talk with my team members on a different level to when we’re in a group, and I get a lot of pleasure from engaging ith their thoughts and ideas.

It’s a great time to give and receive feedback, outside of our fortnightly “feedback and cake” sessions which focuses more on group feedback, so 1-to-1 feedback tends to be a lot more personal.

Wednesday

Coffee with Steve

Sometimes I’m a bit rubbish with times, but now that I’m in the office early every morning I no longer have an excuse for being late for my wednesday chat with Steve!

This week we talked about the differences between the Unruly and the GDS models for infrastructure and SREs. Our SREs have a team of their own but also act as enablers and pseudo-coaches to raise ProDev’s skill level across the board.

The world of SRE-ness is one that Shift has been dipping its feet into a lot over the last couple of months, so it’s great to hear how places other than e.g. Google, Facebook do SRE stuff.

20% Time

Wednesday is traditionally (but not always) the day I take my 20% time. It’s normally the day with fewest meetings and it breaks the week up quite nicely. I spiked the terraform module for managing ProtonMail DNS records that I spoke about in my last weeknotes, which will be released on GitHub and the Terraform Registry very soon.

(Is it obvious yet that I really like Terraform? I want to build a custom provider next, probably for some obscure web service)

Thursday

DIY Team Lunch

Sarah had the great idea to hold a spontaneous team lunch in our office — she wrote about it in her own weekly reflections. She brought a picnic blanket to one of our meeting rooms, we all brought along our own lunches, and we shared a bottle of Appletizer.

I hadn’t tasted Appletizer for years, and I felt undeniably classier drinking it out of champagne glasses.

There was plenty of non-shop talk but we had a brilliant idea over the course of the hour — we provide a bit of tooling to deploy and run our puppet code, but we still use a manual mutex (in the form of a slightly grubby android plush).

Could we steal a leaf from Hashicorp’s book and replace our manual lock with something like Terraform’s state locking?

This discussion escalated into how we could push events to DynamoDB and profile the workflow much like we would any other system, and show our slightly unkempt Python script a bit of love.

Tech talks

Ina and I presented the findings and progress that Shift had made towards implementing SLx and Error Budgets for the first of our mission critical systems, graphite.

We got some really great feedback on both the presentation and the content, and there were some great questions about how we might be using our Error Budget when we’ve finished making the calculations.

Farewell to Jahed

We said goodbye to Jahed this week who is, or … was :(, one of our developers — he’s been here long enough to feel part of the furniture and we’ll certainly miss what he brought to Unruly and ProDev as a whole.

He’s a big supporter of open source like myself so I personally will miss his voice in the blogging and open source group.

Friday

Team lunch followup

As the last Shift member in the office, I took 15 minutes and spiked a quick-and-dirty attempt at event pushing functionality in our Puppet workflow.

  • Terraform’d a DynamoDB instance on AWS.
  • Python’d pushing { session_uuid, date, commit_hash, event, hostname } at specific points in the workflow.

I’m excited for this to start accreting data over the next week — we’ll have better insight onto how many runs start but are abandoned, a better look at how long runs take, and many more.

This weekend I baked some easy Fork Biscuits as I slowly build up my baking rep, and I’ve not made biscuits for absolutely YONKS.

I’ve also been ill, but this felt more like a physical cold than the last one I had which seriously affected my usual levels of reasoning. Grotty, but mostly still able to operate at normal capacity.

Originally published at blog.probablyfine.co.uk on February 3, 2019.

Pepijn de Vos (pepijndevos)

LM13700: Voltage Controlled Everything February 03, 2019 12:00 AM

When making a modular synth, everything has to be voltage controlled. So how do you make a voltage controlled amplifier, a voltage controlled oscillator, or a voltage controlled filter? One way is with an operational transconductance amplifier.

The LM13700 is like a swiss army knife of voltage control. Its datasheet is completely packed with refference circuits for voltage controlled everything. To get familiar with its operation, I built a few of the circuits on breadboard.

Voltage controlled amplifier

Basically an OTA is like an opamp with current output, but it’s frequently used without feedback. To make the differential pair more linear, biasing diodes are used at the input. But the linear range is still limited to a few dozen millivolt. What makes it voltage controlled is that the current gain is controlled by IABC, which is the tail current of the differential pair.

For my test circuit I hooked the current gain up to a button with an RC network connected to it, so it does a nice attack and decay when pressed and released.

State variable filter

Then I fed the output of my VCA into this beautiful state variable filter. What is cool about state variable filters is that they can have low-pass, high-pass and band-pass outputs from the same signal. Each OTA basically forms a Gm-C filter. Put simply, a resistor’s current depends on the voltage you put on it, and so does the current of the OTA depend on its input voltage.

For the above video, I output white noise and a low-frequency sine from the MyDAQ. The white noise goes through the VCA controlled by my RC button envelope, and through the band-pass output of the state variable filter, controlled by the slow sine wave.

February 02, 2019

Pepijn de Vos (pepijndevos)

Microrack: A Small Modular Synthesizer February 02, 2019 12:00 AM

Inspired by the Modulin, I’ve been making my own synthesizer, starting with a Game Boy violin, adding pressure sensitivity, and adding analog delay.

Over the past weeks I’ve been thinking about how I want to connect everything together. I knew I wanted to make it modular, but also that it had to be small enough to become a portable instrument, and hopefully easy to prototype and not too expensive. So I came up with what I call Microrack, a compact mixed-signal bus that is electronically compatible with CV. I typed up a rough description here. In short, it uses a bus with analog multiplexers for audio, and an I2C bus for control signals.

I started by making the power supply and base board. Ideally you’d have something more efficient and powerful, but I started with a simple half-wave rectifier into linear regulators. The I2C lines are exposed to an external Arduino board that will control the user interface and the digital bus. Here is a rough schematic. One thing that is regrettably absent is any sort of current limit or fuse.

power supply schematic

Then I started working on the first module. I decided to start a little drum machine based on a noise source, a filter, and an envelope generator. The drum machine was mostly driven by the idea to make white noise in discrete logic. The heart of this module is a linear feedback shift register, implemented with two 74HC595 shift registers and a 4030 quad XOR gate.

linear feedback shift register

The shift clock of the registers is driven by an atmega328p. The output clock of the last shift register is driven by a NOT-wired XOR gate to close the feedback loop. The output clock of the first shift register is driven by the atmega at a lower rate, to sample the noise. The outputs of the first shift register are fed to a R-R2 resistor ladder.

resistor ladder

So by controlling the shift clock and the output clock, the bandwidth and randomness of the noise can be controlled. The DAC output is then fed into an opamp to translate from [0 5] V to [-5 +5] V, which is then output via the analog multiplexer. I’m pretty happy with the result.

microrack

Except then I fried the atmega.

Robin Schroer (sulami)

Building a Literal Library of Building Blocks February 02, 2019 12:00 AM

This postI know, insert the obligatory “I haven’t posted in a while” bit here.

is heavily inspired by a remark Zach Tellman made on the defn podcast, where he says:

Having been a professional programmer for a decade, I have a decade’s worth of experience at writing stuff from scratch, not a decade’s worth of tools in my toolbox. And that seems like a not optimal set of circumstances. [Quote at 57:45]

I have listened to this some time around Christmas, and this quote has kept me thinking over the past couple of months. What Zach is talking about is a project he is working on which would allow you to capture explorative programming in the branching fashion in which it happens. His example revolves around using a shell to perform some work, like extracting some specific values from a file.You should really go and listen to the episode, he has a lot of very great insights.

He explains how we work out these sequences of commands that to accomplish our goal, but never generalise them, but instead throw them away, just to write them from scratch the next time we encounter a similar problem. This rings true for all kinds of programming, not just shell scripting, though shell scripts are especially susceptible to this.

Like Zach, I believe this to be a suboptimal situation. Especially being a functional programmer, I believe in small, abstract building blocks, composition, and code reuse, rather than overly specific, bespoke solutions that have to be written from scratch every time. I am someone who tinkers a lot, and there is a lot of code I never commit anywhere. As a matter of fact, I have a habit of creating throw-away files or whole projects in /tmp just to play with something for anywhere between five minutes and a weekend. At the same time I also have a repository on my Github literally called playground, which contains all kinds of small things that I did not want to go through the hassle of creating a Github repository for.Interesting aside: while creating a local repository has so little friction that I do it all the time, only a fraction of them ever touch Github’s servers, as creating a repository through the web interface incurs so much friction.

This repository has allowed me to cannibalise some snippets of codes I used in the past, but it is not what I would call a comprehensive library of generalised solutions to problems I repeatedly face. And that has been hugely helpful already, for example I have written about path-finding using the A* algorithm before, so I had a working implementation ready when I needed it for another project.

Having a library, in the worldly sense of the word, of useful, generalised snippets of code would institutionalise the knowledge of them. You would not have to remember how to invert a binary tree, because if you have ever played with binary trees you would already have an implementation handy, and it would be tried and tested, and performance-optimised.

Practical Implementations

Having arrived at the decision of generalising and collecting useful snippets of code somewhere, we are now facing the question of where somewhere actually is, and how we distribute the snippets in a way that allows us to easily use them.

The simplest solution would be to maintain one or several collections of useful snippets, and just copy-pasting them into the code you are writing. While this is fast and simple, it does not facilitate innovation flowing in either direction. Updates to the generalised versions are not included in downstream products using them, and vice versa. The result would likely be a duplication of similar, but subtly different solutions to all kinds of problems, scattered over various projects. Bugs that have long been fixed in one of them might still be present in others.

The alternative solution is packaging your snippets, and using them as a library. Most of the practical implementation will depend on the programming language you are using, and what kind of projects you are usually working on. Zach Tellman himself has a Clojure library called Potemkin, which is a collection of “some ideas which are almost good”, and which he uses as a dependency for most of his other libraries.

While this incurs some overhead, namely the packaging of the library, it does come with a lot of advantages. Other people can benefit from your library. Depending on the scale of the overhead involved with building a library, splitting snippets by topic into “actual” libraries might make sense. It does require more abstraction, and more documentation, but that is not a bad thing. For a simple library with a handful of data structures or functions, writing a quick readme and some docstrings takes less than an hour.

There is still room for a default, catch-all library that is just for personal use and contains miscellaneous snippets without any particular topic, and it can be where new snippets end up first. If a section of it grows large enough, it can be extracted into its own library. The bottom line here is, if you write something that solves a problem, keep it somewhere, ideally where you can find it again. Even if it is not generalised or documented, it might come in handy in the future.

February 01, 2019

Unrelenting Technology (myfreeweb)

LLVM 8.0 release highlight (ha): LLDB now has syntax highlighting! February 01, 2019 11:27 PM

LLVM 8.0 release highlight (ha): LLDB now has syntax highlighting!

January 31, 2019

Bogdan Popa (bogdan)

Announcing north January 31, 2019 05:00 AM

A couple of days ago, I released north, a database schema migration tool written in Racket. It currently supports PostgreSQL and SQLite, but new adapters are extremely easy to add and my main reason for building it was because I wanted not only a CLI utility but also programmatic access to do migrations from Racket. I’m going to make that last part easy for everyone with the next release after I clean up some of the internals and write more docs.

Alex Wilson (mrwilson)

Debugging an outage without an internet connection January 31, 2019 12:00 AM

The Monday of this week, I was drafted in to help resolve a production incident on a system that I had helped build before I moved teams. What makes this unusual is that I had no way to actually debug it at the time. So here’s a small experience post about what I did and what I learned.

NB: These are my personal conclusions, YMMV

A small amount of scene-setting

  1. Unruly practices developers-on-call because we believe it makes us build better services.
  2. We practice primary, secondary, team-lead levels of escalation, but engineers with relevant experience can be drafted in if they are available.
  3. For REASONS, I had my laptop but no internet access, so I couldn’t see anything that was going on.
  4. I was on a phone-call with the lead of the team that owns the service, and he had all the usual tools at his disposal

For the next half hour, I was asking him questions and helping to debug remotely given what I knew of the system.

He couldn’t read my mind.

If I was debugging alone, I would likely be jumping back and forth between my hunches, trying to cross them off as quickly as possible. The nature of this new dynamic required a much slower and measured approach.

Don’t say “Can you read out the logs from 3 o’clock to 5 o’clock?”

We’re trying to identify possible causes of the issues, and these (caveat: in my opinion) are better phrased as questions rather than requests

Do say “There was a deploy at 3 o’ clock today, is there anything unusual in the app log?”

If you have the source code available to you, referring to files and line numbers is really helpful — this enabled us to identify particular lines of configuration that might be causing the issue.

Lesson 2: Say why you’re asking the question

I’m not going to treat my colleague like he’s just a pair of hands for me, and I felt it was really important to clarify why I was asking the question. It gives an opportunity to short-circuit the query if it’s something that’s already been eliminated.

Don’t say “Can you search the logs for AWS request errors?”

Bonus points: ask about the output, not the act. I don’t really mind how we get to the answer, more that we eliminate or prove a potential cause.

Do say “I think that the problem might be being caused by a lack of correct permissions for the AWS credentials. Are there AWS permission denied errors in the app log?”

Lesson 3: They are the system owners, and the experts. Treat them as such.

I was pulled in because I’d worked on the system before but well over a year ago. The system might have changed in ways that I don’t know about, so it was important for me to recognise that my mental model might be out of date and I needed to tailor my questions as such.

Do say “I remember it behaving like X due to Y. Is this still true?”

In this scenario, there were a number of things I could probably eliminate based on my previous knowledge of what failure causes would look like for network issues, datastore connections, etc, but I didn’t want to assume anything.

Lesson 4: On-call outages are inherently stressful. Don’t make it worse.

Feedback when debugging yourself is of the order of seconds.

The thought -> question -> action -> reply loop is significantly longer.

Given that we’re trying to solve the issue in the shortest timeframe available, this process can make things even more stressful if things get out of hand.

There are several things you can do but they depend on how the other person works — for example, are they someone who likes to talk a lot, or more measured with the way they speak?

In the former, I would try to keep a conversation going with my thought process to normalise the conversation, but I wouldn’t do this in the latter scenario.

I hope will be a useful read for people who do on-call and who might encounter something like this — tl;dr try your best to help, be clear and up-front with questions, show empathy and care to not make things more stressful.

Originally published at blog.probablyfine.co.uk on January 31, 2019.

January 30, 2019

Mark J. Nelson (mjn)

DeepMind AlphaStar (Starcraft II bot) roundup January 30, 2019 12:00 PM

The UK-based Google subsidiary, DeepMind, which brought big news to the AI & games world in 2016 with its bot AlphaGo defeating a world-class Go player, says they've done it again with AlphaStar, a bot that beat a high-level human player in the real-time strategy (RTS) game StarCraft II, previously considered to be a top-tier challenge.

I'm currently teaching an AI & Games special-topics course at American University, where I recently started in a new job. It's a seminar-style class where the first part of the course is largely based on jointly reading Julian Togelius's new book (so new, in fact, that it was officially published one day after our semester started), Playing Smart: On Games, Intelligence, and Artificial Intelligence.

The idea is that both I and the students bring some things to discuss to each class, within a framework broadly set by Togelius. But this AlphaStar news is pretty high-profile breaking news, at least as far as AI & games news goes. So I decided that we'd dedicate the next class or two to this event.

Some discussion that immediately ensued around the Google StarCraft bot: What does this show, does it show what it claims, and what does it mean for AI & Games, or AI in general?

* * *

I've collected relevant links below:

DeepMind StarCraft II Demonstration video

  • Commented recording of five human-vs-machine games
  • There's an explanation of the learning algorithm at 1:16:30

DeepMind blog post

  • Explanation of the algorithms and bot training league
  • Some interactive visualizations of strategies
  • Downloadable data

Reaction blog posts and articles:

Indrek Lasn (indreklasn)

How to set-up a powerful API with GraphQL, Koa, and MongoDB — CRUD January 30, 2019 12:00 AM

How to set-up a powerful API with GraphQL, Koa, and MongoDB — CRUD

This is a series where we learn how to set-up a powerful API with GraphQL, Koa, and Mongo. The primary focus will be on GraphQL. Check out part I if you haven’t yet.

Mutations

So far we can read our data, but there’s a big chance we need to edit our data records/documents. Any complete data platform needs a way to modify server-side data as well.

Okay, imagine this– a company launched a new gadget. How would we go on about adding the record to our database with GraphQL?

Mutations to the rescue!

Think of Mutations like POST or PUT REST actions. Setting up a mutation is quite straight-forward.

Let’s jump in!

Adding records to our database

Create a file graphql/mutations.js

Inside the file, we will place mutations.

  • We import the GraphQLObjectType and GraphQLObjectType Objects from the GraphQL library.
  • Import the GraphQL type for gadget
  • Import the gadget mongoose Model.

After importing the stuff we need, we can go on about creating the mutation.

A Mutation is just a plain GraphQLObjectType like the query we had before. It has two main properties we’re interested in.

The name of the mutation is what appears in the graphiql docs.

Fields are where we can place our mutation logic.

Notice I added a new object inside the fields object. It’s called addGadget and it would do exactly what it says it will do.

Inside the addGadget we have access to three properties, type, args, and resolve()

The addGadget type will be gadgetGraphQLType. The gadget can only have properties which are allowed in the gadgetGraphQLType type we declared earlier.

addGadget is a query which accepts arguments. The arguments are needed to specify which gadget we want to add to our database.

We declare up-front which arguments the query accepts and the types of the arguments.

Lastly–what happens with the query? Which precisely why we have the resolve() function.

Remember the resolve() function has two arguments– parent and args. We’re interested in the args since these are the values we pass to our query.

Inside the resolve we place the logic for creating a new mongo record.

We create a new instance of our Gadget mongoose model, pass the props we receive from GraphQL as new fields and finally save the record.

Here’s how the full mutation looks;

graphl/mutations.js

Voila! All we need to do is import the mutation to our schema.js file.

graphl/schema.js

If everything went fine, this is what we should see on our graphiql.

And if we click on it;

Notice how GraphQL creates automatically self-documentation. This is why we have such strong typing.

Firing off the mutation query

A mutation is just a plain graphQL query which takes our arguments, saves it to the Mongo database, and returns the properties we want.

Here’s the catch;

Every mutation needs to be marked as mutation

Voila! We successfully created and inserted a new gadget to our mongo database.

If you head over to mlab or whatever provider you’re using– you should see the new record.

Here’s the complete query for our mutation.

Good job!

Editing our records in the database

What if we want to edit pre-existing records? Surely we can’t rely on never making a typo, or what if the price changes.

Editing a record is also a mutation. Remember, every time we want to change/add a new record– it’s a graphql mutation!

Open the graphql/mutations file and create another mutation. A mutation is just a plain object.

Notice the new mutation called updateGadget. It’s pretty much a replica of the previous mutation. Notice the extra argument, the id– That’s because we need to find the existing gadget and change it. We can find the gadget by id.

The resolve() function is where it gets more interesting. Ideally, we want to find the gadget by id, change the props, and save it. How would we go on about doing this?

Mongoose gives us a method to do this, called findById.

This returns a promise. If we console.log the promise, we can see a huge blob of properties attached to it. What we can do with the promise is chain it with a then() method. If promises are a stranger, check out this article I wrote.

Like so; we find the Gadget, change the props, save it. But this returns another promise which we need to resolve.

.catch() for error handling incase we run into errors. Remember, you can monitor your pm2 logs via the pm2 logs command. If you run into errors, these will be logged to the pm2 logger.

That’s all! Query time! Look at your Mongo table and pick a random id from there and edit the corresponding gadget.

And if we inspect the database; we should see the edited record.

Voila! Success!

Here’s the query for the updateGadget mutation.

https://medium.com/media/7382914bb8bd13ff1d668cea6ca3e62f/href

Okay, good job for making it this far. So far we have the Create, Read, Update, but we’re missing the final d(elete).

Deleting a record from a mongo database is quite straight-forward. All we need is another mutation, since we are, in fact mutating the database.

For deleting records, Mongoose gives us a handy method called findOneAndDelete – more about findOneAndDelete

https://medium.com/media/f8d4d24ff095c6a0160d08d113f5849a/href

Deleting the record just takes one argument, the id. We find the gadget by ID, delete it, and return it. If there’s an error, we’ll log it.

And the query;

https://medium.com/media/19399028d64f306e689d94a6ee56dfee/href

note: Make sure the id is correct and exist in the database, otherwise, it won’t work.

If we head over to our database and inspect it — indeed the record got deleted from our database.

Well done, we have achieved basic CRUD functionality. Notice how GraphQL is a thin layer between our database and view. It’s not supposed to replace a database, but rather make it easier to work with data, fetching, and manipulating.

Here’s the source for the chapter

I’ll see you at part 3 where we’ll do more cool stuff!

Thanks for reading!

Originally published at strilliant.com on January 30, 2019.


How to set-up a powerful API with GraphQL, Koa, and MongoDB — CRUD was originally published in freeCodeCamp.org on Medium, where people are continuing the conversation by highlighting and responding to this story.

January 29, 2019

Jeff Carpenter (jeffcarp)

A Year of the Pomodoro Technique January 29, 2019 11:19 PM

The Pomodoro Technique is method for improving productivity by segmenting work into 25-minute intervals. You focus intensely on a task for 25 minutes, then take a break. Rinse, repeat. I began using the technique to study for the cryptography course I took last winter. The benefits were clear from the beginning. I enjoyed working in 25-minute segments and started using it at work as well. I want to share with you some of the things I learned along this year-long journey.

Derek Jones (derek-jones)

Modeling visual studio C++ compile times January 29, 2019 04:34 PM

Last week I spotted an interesting article on the compile-time performance of C++ compilers running under Microsoft Windows. The author had obviously put a lot of work into gathering the data, and had taken care to have multiple runs to reduce the impact of random effects (128 runs to be exact); but, as if often the case, the analysis of the data was lackluster. I posted a comment asking for the data, and a link was posted the next day :-)

The compilers benchmarked were: Visual Studio 2015, Visual Studio 2017 and clang 7.0.1; the compilers were configured to target: C++20, C++17, C++14, C++11, C++03, or C++98. The source code used was 100 system headers.

If we are interested in understanding the contribution of each component to overall compile-time, the obvious fist regression model to build is:

compile_time = header_x+compiler_y+language_z

where: header_x are the different headers, compiler_y the different compilers and language_z the different target languages. There might be some interaction between variables, so something more complicated was tried first; the final fitted model was (code+data):

compile_time = k+header_x+compiler_y+language_z+compiler_y*language_z

where k is a constant (the Intercept in R’s summary output). The following is a list of normalised numbers to plug into the equation (clang is the default compiler and C++03 the default language, and so do not appear in the list, the : symbol represents the multiplication; only a few of the 100 headers are listed, details are available):

                             Estimate Std. Error  t value Pr(>|t|)    
               (Intercept)                  headerany 
               1.000000000                0.051100398 
               headerarray             headerassert.h 
               0.522336397               -0.654056185 
...
            headerwctype.h            headerwindows.h 
              -0.648095154                1.304270250 
              compilerVS15               compilerVS17 
              -0.185795534               -0.114590143 
             languagec++11              languagec++14 
               0.032930014                0.156363433 
             languagec++17              languagec++20 
               0.192301727                0.184274629 
             languagec++98 compilerVS15:languagec++11 
               0.001149643               -0.058735591 
compilerVS17:languagec++11 compilerVS15:languagec++14 
              -0.038582437               -0.183708714 
compilerVS17:languagec++14 compilerVS15:languagec++17 
              -0.164031495                         NA 
compilerVS17:languagec++17 compilerVS15:languagec++20 
              -0.181591418                         NA 
compilerVS17:languagec++20 compilerVS15:languagec++98 
              -0.193587045                0.062414667 
compilerVS17:languagec++98 
               0.014558295 

As an example, the (normalised) time to compile wchar.h using VS15 with languagec++11 is:
1-0.514807638-0.183862162+0.033951731-0.059720131

Each component adds/substracts to/from the normalised mean.

Building this model didn’t take long. While waiting for the kettle to boil, I suddenly realised that an additive model was probably inappropriate for this problem; oops. Surely the contribution of each component was multiplicative, i.e., components have a percentage impact to performance.

A quick change to the form of the fitted model:

log(compile_time) = k+header_x+compiler_y+language_z+compiler_y*language_z

Taking the exponential of both side, the fitted equation becomes:

compile_time = e^{k}e^{header_x}e^{compiler_y}e^{language_z}e^{compiler_y*language_z}

The numbers, after taking the exponent, are:

               (Intercept)                  headerany 
              9.724619e+08               1.051756e+00 
...
            headerwctype.h            headerwindows.h 
              3.138361e-01               2.288970e+00 
              compilerVS15               compilerVS17 
              7.286951e-01               7.772886e-01 
             languagec++11              languagec++14 
              1.011743e+00               1.049049e+00 
             languagec++17              languagec++20 
              1.067557e+00               1.056677e+00 
             languagec++98 compilerVS15:languagec++11 
              1.003249e+00               9.735327e-01 
compilerVS17:languagec++11 compilerVS15:languagec++14 
              9.880285e-01               9.351416e-01 
compilerVS17:languagec++14 compilerVS15:languagec++17 
              9.501834e-01                         NA 
compilerVS17:languagec++17 compilerVS15:languagec++20 
              9.480678e-01                         NA 
compilerVS17:languagec++20 compilerVS15:languagec++98 
              9.402461e-01               1.058305e+00 
compilerVS17:languagec++98 
              1.001267e+00 

Taking the same example as above: wchar.h using VS15 with c++11. The compile-time (in cpu clock cycles) is:
9.724619e+08*3.138361e-01*7.286951e-01*1.011743e+00*9.735327e-01

Now each component causes a percentage change in the (mean) base value.

Both of these model explain over 90% of the variance in the data, but this is hardly surprising given they include so much detail.

In reality compile-time is driven by some combination of additive and multiplicative factors. Building a combined additive and multiplicative model is going to be like wrestling an octopus, and is left as an exercise for the reader :-)

Given a choice between these two models, I think the multiplicative model is probably closest to reality.

Jan van den Berg (j11g)

High Output Management – Andrew S. Grove January 29, 2019 12:00 PM

This classic management book starts off with two confusing and underwhelming chapters, but ends up being one of the best three management books I have ever read. And I would highly recommend it to any manager.

High Output Management – Andrew S. Grove (1983) – 243 pages

If you run into a book recommendation thread on Hacker News or somewhere else involving tech savvy people, chances are, High Output Management will be mentioned. Published in 1983, updated in 1995 and revived in 2015 with high prise from Ben Horowitz, this book seems to have an enduring allure and impact on tech managers and the tech industry in general.

The 47 year old, down to earth, polymath Andy Grove was already an experienced and successful leader at Intel when he wrote this book. But nobody, probably not even Andy, could have predicted what was yet to come for Intel. Namely the explosion of personal computing and the internet, catapulting Intel into becoming one of the most valuable technology companies in the world. And consequently confirming Andy’s written down theories and practices. This would explain part of the lasting appeal to tech people.

But most of the appeal is in the words itself. Andy has a very direct and practical approach to dissecting hard managerial themes and is crystal clear on what is important and where (and what) the leverage is. Many, many management books have been written since, and even when I think there is room for editing in this book, I still have never read a comparable comprehensive collection of valuable management lessons in such a clear and concise manner.

The post High Output Management – Andrew S. Grove appeared first on Jan van den Berg.

January 28, 2019

Gustaf Erikson (gerikson)

Fotografiska, 27 Jan 2019 January 28, 2019 09:13 PM

Arvida Byström - Inflated Fiction

Byström’s sui generis blend of selfie photography, feminist critique, and social media chops doesn’t translate well into one single room exhibit. It’s a shame but I’m glad she’s getting the recognition see deserves.

STHLM Forever

Every large city no doubt has its fair share of introspection but as a (relative) outsider to Stockholm I’ve always felt Stockholm’s has been a bit over the top. Probably because the entirety of Sweden’s population is hardly bigger than a “real” big city like London or Paris, the capital has to do double duty as something for locals and the focus of the nation.

Most libraries here have a section called Stockholmiana where books on street names etc are shelved.

So exhibits of photographs of Stockholm are usually a big hit. Here were the greatest hits from greats like Christer Strömholm and Lennart Nilsson, along with newer works. Weirdly there’s very little color. The genius loci of Stockholm is apparently only visible in B&W.

Under the bridges

This photo was taken under Skanstullsbron, the construction of which was subject of one of Nilsson’s images from the exhibit. It’s a nice connection I think!

January 27, 2019

Siddhant Goel (siddhantgoel)

Off by 1 (Day) January 27, 2019 11:00 PM

One of the most common bugs when writing software is the classic "off by 1" error. In this post, I'll talk about a similar bug I found in some code I maintain, and how I fixed it.

Backstory

I maintain beancount-dkb, which is a Python package that provides helper classes for converting DKB CSV exports to the Beancount format. In Beancount's terminology, these "helper" classes are called "importers".

If you're not familiar with Beancount, it's a plain-text accounting tool which lets you keep track of all your finances using plain text files. The idea is that you maintain all your bank transactions in one text file, and then use the tools that Beancount provides to run reports over all that data. The transactions in this file follow the Double Entry Accounting method, and are written in a DSL strictly specified by Beancount.

The way this works in practice is that every few weeks, you download transactions from your bank (often this is a simple CSV export), and run them through an importer to convert them into a data format that Beancount expects. You then append the resulting data to a .beancount file you maintain which contains all your transactions, going all the way back to stone age. Finally, you use the suite of tools that Beancount provides to run all sorts of analysis on your financial data.

It's actually much less complicated than it sounds.

I've used it to import my financial history from the last three years and the whole process has been quite smooth, except for one hiccup. Balance assertions. And that's what this post is about.

Balance Assertions

Here's a short code snippet that represents the history of a single bank account, written in the Beancount DSL. The bank account is named Assets:DKB and starts out with an opening balance of €100.

For simplicity, the history here consists of a single "going to the supermarket" transaction.

;; -*- mode: beancount -*-

; Date format - YYYY-MM-DD

option "title" "Max Mustermann"
option "operating_currency" "EUR"

2019-01-01 open Assets:DKB
2019-01-01 open Equity:Opening-Balances
2019-01-01 open Expenses:Supermarket

2019-01-15 * "Initialize Assets:DKB"
    Assets:DKB                           100.00 EUR
    Equity:Opening-Balances

2019-01-15 * "Going to the supermarket"
    Assets:DKB                          -30.00 EUR
    Expenses:Supermarket                 30.00 EUR

2019-01-16 balance Assets:DKB            70.00 EUR

The transaction with the description "Going to the supermarket" shows that the owner went to some supermarket and spent €30, which means that the account has €70 left at the end.

The interesting bit here is the last line.

2019-01-16 balance Assets:DKB            70.00 EUR

This line instructs Beancount to assert that the balance of the given account is the given amount at the beginning of the given date. And in case that's not true, Beancount should refuse to process things any further because there's obviously something wrong with the data.

Such assertions are not completely necessary, but having them gives you the peace of mind that the data you're working with is not wrong. This can happen in case of something like duplicate transactions. Often, when you have two accounts and you transfer money from one account to the other one, the same transaction is going to show up in both the account summaries. For Beancount they are two different transactions, but practially speaking that's not true. They are two legs of the same transaction. Left unmerged, these would result in wrong numbers on both the accounts. This is why balance assertions come in handy.

The Bug

When I first started out with Beancount, there were no balance assertions in my data. I knew the concept, but the initial versions of beancount-dkb I released didn't output any balance directives.

After a few months of regular Beancount usage and realizing how useful these assertions can be, I decided to implement support in beancount-dkb. This was not too much work since the documentation is pretty clear on how to do this.

So something like this,

"01.01.2019";"";"";"";"Tagessaldo";"";"";"100,00";

... becomes this

2019-01-01 balance Assets:DKB            100.00 EUR

What turned out to be more work was testing the whole thing.

After the initial implementation, I noticed the numbers just weren't adding up. The test cases were fine, but the test cases were made-up data anyway. The output on the actual data just didn't add up. After a few hours of trying to figure things out, I found this little gem in the documentation.

Note that a balance assertion, like all other non-transaction directives, applies at the beginning of its date (i.e., midnight at the start of day). Just imagine that the balance check occurs right after midnight on that day.

The difference is subtle, but can easily lead to numbers not adding up.

What's happening here is that Beancount is expecting the balance amount to be valid at the beginning of the day, while the balance values from the DKB output correspond to the amount at the end of the day. Note that the DKB behavior is not documented anywhere (and if it is, I couldn't find the relevant docs), but from all the data I saw, this makes the most sense.

Conclusion

The fix in this case was easy. Just set the date of the balance directive to 1 day after what's in the CSV. This shifts the time of the assertion to midnight at the start of the next day, which in turn makes the numbers all look good.

Admittedly, this bug wasn't too hairy and the fix wasn't that tricky either. But often in cases like this where things are rather subtle, it can take you anywhere between a few minutes to a few hours to find a fix. For me, it was somewhere in between.

The latest release of beancount-dkb includes this patch. I have (more or less completely) rewritten my Beancount data using the latest code with the correct balance assertions and so far things have been smooth. Apologies if an older version affected your data!

Ponylang (SeanTAllen)

Last Week in Pony - January 27, 2019 January 27, 2019 04:47 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, our users’ mailing list, our Slack community, or join us on IRC.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

Gonçalo Valério (dethos)

Channels and Webhooks January 27, 2019 01:04 PM

Django is an awesome web framework for python and does a really good job, either for building websites or web APIs using Rest Framework. One area where it usually fell short was dealing asynchronous functionality, it wasn’t its original purpose and wasn’t even a thing on the web at the time of its creation.

The world moved on, web-sockets became a thing and suddenly there was a need to handle persistent connections and to deal with other flows “instead of” (or along with) the traditional request-response scheme.

In the last few years there has been several cumbersome solutions to integrate web-sockets with Django, some people even moved to other python solutions (losing many of the goodies) in order to be able to support this real-time functionality. It is not just web-sockets, it can be any other kind of persistent connection and/or asynchronous protocol in a microservice architecture for example.

Of all alternatives the most developer friendly seems to be django-channels, since it lets you keep using familiar django design patterns and integrates in a way that seems it really is part of the framework itself. Last year django-channels saw the release of it second iteration, with a completely different internal design and seems to be stable enough to start building cool things with it, so that is what we will do in this post.

Webhook logger

In this blog post I’m gonna explore the version 2 of the package and evaluate how difficult it can be to implement a simple flow using websockets.

Most of the tutorials I find on the web about this subject try to demonstrate the capabilities of “channels” by implementing a simple real-time chat solution. For this blog post I will try something different and perhaps more useful, at least for developers.

I will build a simple service to test and debug webhooks (in reality any type of HTTP request). The functionality is minimal and can be described like this:

  • The user visits the website and is given a unique callback URL
  • All requests sent to that callback URL are displayed on the user browser in real-time, with all the information about that request.
  • The user can use that URL in any service that sends requests/webhooks as asynchronous notifications.
  • Many people can have the page open and receive at the same time the information about the incoming requests.
  • No data is stored, if the user reloads the page it can only see new requests.

In the end the implementation will not differ much from those chat versions, but at least we will end up with something that can be quite handy.

Note: The final result can be checked on Github, if you prefer to explore while reading the rest of the article.

Setting up the Django project

The basic setup is identical to any other Django project, we just create a new one using django_admin startproject webhook_logger and then create a new app using python manage.py startapp callbacks (in this case I just named the app callbacks).

Since we will not store any information we can remove all database related stuff and even any other extra functionality that will not be used, such as authentication related middleware. I did this on my repository, but it is completely optional and not in the scope of this small post.

Installing “django-channels”

After the project is set up we can add the missing piece, the django-channels package, running pip install channels==2.1.6. Then we need to add it to the installed apps:

INSTALLED_APPS = [
    "django.contrib.staticfiles", 
    "channels", 
]

For this project we will use Redis as a backend for the channel layer, so we need to also install the channels-redis package and add the required configuration:

CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {"hosts": [(os.environ.get("REDIS_URL", "127.0.0.1"), 6379)]},
    }
}

The above snippet assumes you are running a Redis server instance on your machine, but you can configure it using a environment variable.

Add websocket’s functionality

When using “django channels” our code will not differ much from a standard django app, we will still have our views, our models, our templates, etc. For the asynchronous interactions and protocols outside the standard HTTP request-response style, we will use a new concept that is the Consumer with its own routing file outside of default urls.py file.

So lets add these new files and configurations to our app. First inside our app lets create a consumer.py with the following contents:

# callbacks/consumers.py
from channels.generic.websocket import WebsocketConsumer
from asgiref.sync import async_to_sync
import json


class WebhookConsumer(WebsocketConsumer):
    def connect(self):
        self.callback = self.scope["url_route"]["kwargs"]["uuid"]
        async_to_sync(self.channel_layer.group_add)(self.callback, self.channel_name)
        self.accept()

    def disconnect(self, close_code):
        async_to_sync(self.channel_layer.group_discard)(
            self.callback, self.channel_name
        )

    def receive(self, text_data):
        # Discard all received data
        pass

    def new_request(self, event):
        self.send(text_data=json.dumps(event["data"]))

Basically we extend the standard WebsocketConsumer and override the standard methods. A consumer instance will be created for each websocket connection that is made to the server. Let me explain a little bit what is going on the above snippet:

  • connect – When a new websocket connection is made, we check which callback it desires to receive information and attach the consumer to the related group ( a group is a way to broadcast a message to several consumers)
  • disconnect – As the name suggests, when we lose a connection we remove the “consumer” from the group.
  • receive – This is a standard method for receiving any data sent by the other end of the connection (in this case the browser). Since we do not want to receive any data, lets just discard it.
  • new_request – This is a custom method for handling data about a given request/webhook received by the system. These messages are submitted to the group with the type new_request.

You might also be a little confused with that async_to_sync function that is imported and used to call channel_layer methods, but the explanation is simple, since those methods are asynchronous and our consumer is standard synchronous code we have to execute them synchronously. That function and sync_to_async are two very helpful utilities to deal with these scenarios, for details about how they work please check this blog post.

Now that we have a working consumer, we need to take care of the routing so it is accessible to the outside world. Lets add an app level routing.py file:

# callbacks/routing.py
from django.conf.urls import url

from .consumers import WebhookConsumer

websocket_urlpatterns = [url(r"^ws/callback/(?P<uuid>[^/]+)/$", WebhookConsumer)]

Here we use a very similar pattern (like the well known url_patterns) to link our consumer class to connections of certain url. In this case our users could connect to an URL that contains the id (uuid) of the callback that they want to be notified about new events/requests.

Finally for our consumer to be available to the public we will need to create a root routing file for our project. It looks like this:

# <project_name>/routing.py
from channels.routing import ProtocolTypeRouter, URLRouter
from callbacks.routing import websocket_urlpatterns

application = ProtocolTypeRouter({"websocket": URLRouter(websocket_urlpatterns)})

Here we use the ProtocolTypeRouter as the main entry point, so what is does is:

It lets you dispatch to one of a number of other ASGI applications based on the type value present in the scope. Protocols will define a fixed type value that their scope contains, so you can use this to distinguish between incoming connection types.

Django Channels Documentation

We just defined the websocket protocol and used the URLRouter to point to our previous defined websocket urls.

The rest of the app

At this moment we are able to receive new websocket connections and send to those clients live data using the new_request method on the client. However at the moment we do not have information to send, since we haven’t yet created the endpoints that will receive the requests and forward their data to our consumer.

For this purpose lets create a simple class based view, it will receive any type of HTTP request (including the webhooks we want to inspect) and forward them to the consumers that are listening of that specific uuid:

# callbacks/views.py

class CallbackView(View):
    def dispatch(self, request, *args, **kwargs):
        channel_layer = get_channel_layer()
        async_to_sync(channel_layer.group_send)(
            kwargs["uuid"], {"type": "new_request", "data": self._request_data(request)}
        )
        return HttpResponse()

In the above snippet, we get the channel layer, send the request data to the group and return a successful response to calling entity (lets ignore what the self._request_data(request) call does and assume it returns all the relevant information we need).

One important piece of information is that the value of the type key on the data that is used for the group_send call, is the method that will be called on the websocket’s consumer we defined earlier.

Now we just need to expose this on our urls.py file and the core of our system is done.

# <project_name>/urls.py

from django.urls import path
from callbacks.views import CallbackView

urlpatterns = [
    path("<uuid>", CallbackView.as_view(), name="callback-submit"),
]

The rest of our application is just standard Django web app development, that part I will not cover in this blog post. You will need to create a page and use JavaScript in order to connect the websocket. You can check a working example of this system in the following URL :

https://webhook-logger.ovalerio.net

For more details just check the code repository on Github.

Deploying

I not going to explore the details about the topic of deployments but someone else wrote a pretty straightforward blog post on how to do it for production projects that use Django channels. You can check it here.

Final thoughts

With django-channels building real-time web apps or projects that deal with other protocols other than HTTP becomes really simple. I do think it is a great addition to the current ecosystem, it certainly is an option I will consider from now on for these tasks.

Have you ever used it? do you any strong opinion about it? let me know on the comments section.

Final Note: It seems based on recent messages on the mailing list that the project might suspend its developments in its future if it doesn’t find new maintainers. It would definitely be a shame, since it has a lot of potential. Lets see how it goes.

Gokberk Yaltirakli (gkbrk)

Rendering GPS traces January 27, 2019 02:10 AM

If you ask a bunch of people to upload GPS traces when they walk/drive and you combine those traces, you can get a rudimentary map. In fact, this is one of the primary data sources of OpenStreetMap. The data for those is freely available, so we can use it in a small project.

To draw simple outlines, iterating over the GPS track points and putting them on an image should be enough. It will give us the main roads and the general city structure, and will be possible to recognize when compared to an actual map.

To begin, let’s get the coordinates of the place we’ll be mapping. In my case, this will be Sheffield. If you go to OpenStreetMap and hit Export, it will let you select and area with a bounding box and get the coordinates of it. We’ll get the coordinates and write it to to our script.

# Area format is left, bottom, right, top
AREA = [-1.4853, 53.3730, -1.4557, 53.3893]

The other thing we should get out of the way is the output size. We should go with a nice 720p picture.

WIDTH  = 1280
HEIGHT = 720

Getting the GPS data

OpenStreetMap provides an API that we can use in order to fetch GPS track data. It gives us the data for a given region in the XML format.

A small disclaimer about the API. It’s normally meant for editing, which means you should try to keep your usage very light. While making this project and iterating on the code, I kept all my API calls in a local cache. I’ll write about this in the future.

You can find the documentation for the API here. Here’s the gist of it.

bbox = ','.join(AREA)
url = 'https://api.openstreetmap.org/api/0.6/trackpoints'
xml = requests.get(url, params={'bbox': bbox}).text

This should get an XML document with the GPS trackpoints. Let’s parse it and get the latitude/longitude pairs. The coordinates are held in <trkpt> tags.

root = ET.fromstring(xml)
selector = './/{http://www.topografix.com/GPX/1/0}trkpt'

for trkpt in root.findall(selector):
  print(trkpt.attrib['lat'], trkpt.attrib['lon'])

As you can see, this is relatively straightforward. The XML selector might look weird. It just means get all the trkpt elements that belong to the namespace of that URL.

Plotting the data

Let’s start with a small example, creating an empty image and drawing a pixel in it.

img = Image.new('L', (WIDTH, HEIGHT), color='white')
img.putpixel((5, 5), 1)

This code will draw a single black pixel on an empty image. The rest of the way should be looking pretty clear now, go through the lat/lon pairs and plot them as pixels. But before we get to that step, there is one more hurdle to get through. And that is mapping the GPS coordinates to image coordinates.

Mapping coordinates for our map

The problem is, we have a 1280x720 image. And we can’t ask Python to put a pixel on (52.6447, -8.6337). We already know the exact area of the map we’re drawing, and the size of our output. What we need to do is get those two ranges and interpolate where a given coordinate falls on our image. For this, we can use the interp function from numpy.

y, x = point
x = int(interp(x, [AREA[0], AREA[2]], [0, WIDTH]))
y = int(interp(y, [AREA[1], AREA[3]], [HEIGHT, 0]))

try:
    img.putpixel((x, y), 1)
except:
    # In case math goes wrong
    pass

Drawing the map

We are know able to get GPS trackpoints and know how to map them to image coordinates. So let’s loop through everything and put the pixels on our map. Since each page has a limit of 5000 points, we should also iterate through the pages.

for page in range(15):
  for point in get_points(AREA, page):
    y, x = point
    x = int(interp(x, [AREA[0], AREA[2]], [0, WIDTH]))
    y = int(interp(y, [AREA[1], AREA[3]], [HEIGHT, 0]))
    try:
      img.putpixel((x, y), 1)
    except:
      pass

Getting points with pagination

Here’s a generator function to return OpenStreetMap trackpoints with pagination.

def get_points(area, page=0):
  bbox = ','.join(map(str, area))
  xml = sess.get('https://api.openstreetmap.org/api/0.6/trackpoints',
    params={'bbox': bbox, 'page': page}).text
  root = ET.fromstring(xml)

  for trkpt in root.findall('.//{http://www.topografix.com/GPX/1/0}trkpt'):
    yield trkpt.attrib['lat'], trkpt.attrib['lon']

Results

Rendering of Tokyo Rendering of Limerick Rendering of Sheffield

If you come up with any cool-looking renders, or better ways to plot this data, either leave a comment about it or send me an email about.

Indrek Lasn (indreklasn)

How to setup a powerful API with GraphQL, Koa and MongoDB January 27, 2019 12:00 AM

Building an API is super fun! Especially when you can leverage modern technologies such as Koa, GraphQL and MongoDB.

Koa is a Node framework, just like Express is a Node framework. We’ll replace Express with Koa since Koa uses async/await syntax over callbacks.

Getting started

The prerequisites for building our API are the following:

  • Node installed
  • Text Editor; I pick Visual Studio Code
  • Terminal
  • Browser

If you have everything you need, please proceed — if not, please install them.

Open your terminal and create a node project, like so:

So far we created our project folder, and initialized a fresh Node project. Now we have the NPM packages available which we can use to install Koa, Mongo, and GraphQL.

Let’s install koa with NPM.

npm i koa

Starting a new Koa server is very simple. All we need is a server.js file and with the following contents:

https://medium.com/media/095658543e077a5a771abf0c652125a6/href

Start the project with Node:

Installing GraphQL

We need two packages to setup GraphQL with Koa: koa-mount and koa-graphql

npm i koa-mount koa-graphql

GraphQL requires that we pass our initial schema to the GraphQL server. Let’s create one.

We will place the graphQL schema at graphql/schema.js

https://medium.com/media/3f978f3b968834ad804149cbc4d73219/href

We pass our initial Query to the buildSchema function

Note: Notice how the argument for the buildSchema is a template literal. I encourage you take a look at this article if this is unfamiliar.

And now we can pass the initial schema to our GraphQL server.

https://medium.com/media/acc392def89e474d7aaccf1f2c3563e3/href

Don’t forget to import koa-mount, koa-graphql, and finally the schema.js.

https://medium.com/media/5a810901eb7a62b937162e6b5b97f8fb/href

And then, if we head over to localhost:9000/graphql:

Graphiql — a powerful tool for querying GraphQL on the frontend

Voilà! Initial setup is done. It’s not very useful yet. Ideally, we would like to query GraphQL to save data to our mongodb and read from there.

Setting up MongoDB

In order to read and write with GraphQL, we need a place to read from. This is where Mongo will come in handy. We’ll save and read our data from there.

To make things simpler, we’re gonna use a cloud instance for Mongo. Head over to mlab.com and create a user and a mongo database.

Creating the mongoDB database

Once you’ve created the database, you’ll need a user for the database.

Creating the MongoDB user

Click on the users tab and create a new username with password.

Now you can use the mongoDB paired with Mongoose. The remote url for your database will be something like this:

mongodb://:@ds213615.mlab.com:13615/koa-graphql

Installing mongoose

npm i mongoose

Creating database.js file

We create a dedicated file for database connection.

https://medium.com/media/269060d1c4b5ebc88e0ed86d4312b1cf/href

Note: Make sure you use the username and credentials for your database.

This block of code will try to connect to the remote mongodb. We need to call it somewhere now.

Open server.js and require and call the initDB method.

https://medium.com/media/16367e579e6c3ef606a0d31685f8d70e/href

If we did everything correctly, our console should tell us we connected successfully.

Bravo!

Notice how annoying it is to constantly refresh the server? Let’s solve this with a package called pm2.

PM2 is a production process manager for Node.js applications with a built-in load balancer. It allows you to keep applications alive forever, to reload them without downtime, and to facilitate common system admin tasks.

npm install pm2 -g

Add a script called start to our package.json:

https://medium.com/media/8141dc101802090c90a1c20131402135/href

Pm2 runs in the background, which frees up our terminal. If you ever want to stop the process, just run pm2 kill. Now we don’t have to restart our server all the time, pm2 does it automatically.

Note: pm2 logs returns the console log statements to the terminal.

MongoDB models

If you ever worked with Mongo, you’re aware that mongoDB lets us create Models for our data. This is a neat way for us to structure how our data will look like.

Create a folder models and inside a file gadgets.js enter:

https://medium.com/media/aef429035f57829d443911cd4a4ac6f7/href

Note: There is no ID field in our schema. That’s because Mongoose will assign
an ID by default to all schemas.

Great. Let’s also add a collection and some dummy data. The collection name has to map our gadget name in plural, gadgets in this case.

After creating the collection, insert the document in JSON format like so:

That’s all for Mongo. Let’s fetch the data with GraphQL.

GraphQL Queries

GraphQL requires us to create types as well. Think of it like instructions for computers.

graphql/gadgetType.js

https://medium.com/media/c87c4c1651d1ac7cdca4a27d02cdd385/href

Notice we created a graphql type. Inside the fields we can specify the properties of the given type.

https://medium.com/media/757a3216f6a2a164cacf673e8184371b/href

Notice the GraphQLObjectType and GraphQLObjectType types we deconstruct from graphQL. These are the primitive types for graphQL.

Creating the graphQL types also grants type-hinting which we’ll use when creating our queries.

Last thing we need to do is refactor our schema.js. We want to query a gadget by id.

Import the Gadget model, gadgetGraphQLType graphql type and GraphQLSchema, GraphQLObjectType, GraphQLString from graphQL to schema.js.

https://medium.com/media/f37aad4f4d3cd6b1e2988b363c45e840/href

Next we need a root query. Every GraphQL query starts with curly brackets {}:

https://medium.com/media/39d488de8116585dc544000f9bb876eb/href

Voilà! Inside the fields we can specify the gadget query.

https://medium.com/media/af1b1cb66dc3eac20b055787e080495d/href

Notice the three properties inside the gadget query:

  • type — this is the type of the query, gadgetGraphQLType in this case.
  • args — we can provide arguments to graphql queries, like: gadgets(id: "1")
  • resolve — how do we want to resolve the query? What should happen once the query is being made? Here we return the Gadget model by id.

And finally export it.

https://medium.com/media/1b417932d88d70e7f4c3f758edd0c501/href

The file schema.js file should look like this:

https://medium.com/media/f7a1297059a672f0087a86d8875e2afe/href

Now head over to http://localhost:9000/graphql and make the query.

https://medium.com/media/121fbe68c859321542279bf69130f9fc/href

This it what we should end up with!

Thanks for reading! ❤ The source code can be found here.

Part II — we’ll discover mutations, and how we can create, delete, update, and read records from our MongoDB with GraphQL.

Keep up to date with me on Twitter

Originally published at strilliant.com on January 27, 2019.


How to setup a powerful API with GraphQL, Koa and MongoDB was originally published in freeCodeCamp.org on Medium, where people are continuing the conversation by highlighting and responding to this story.

Noon van der Silk (silky)

Designing Functional Clothes with Haskell! January 27, 2019 12:00 AM

Posted on January 27, 2019 by Noon van der Silk

So I gave my talk at the Art-and-Tech miniconf at Linux Conf AU earlier this week.

The video of my talk is here:

Other materials are here:

Thanks to Mark for having me at the miniconf!

Errata

Note that there’s a typo on the types of the somethingFun function that has since been corrected in the slides.

Wesley Moore (wezm)

My Rust Powered linux.conf.au e-Paper Badge January 27, 2019 12:00 AM

This week I attended linux.conf.au (for the first time) in Christchurch, New Zealand. It's a week long conference covering Linux, open source software and hardware, privacy, security and much more. The theme this year was IoT. In line with the theme I built a digital conference badge to take to the conference. It used a tri-colour e-Paper display and was powered by a Rust program I built running on Raspbian Linux. This post describes how it was built, how it works, and how it fared at the conference. The source code is on GitHub.

The badge in its final state after the conference. The badge in its final state after the conference

Building

After booking my tickets in October I decided I wanted to build a digital conference badge. I'm not entirely sure what prompted me to do this but it was a combination of seeing projects like the BADGEr in the past, the theme of linux.conf.au 2019 being IoT, and an excuse to write more Rust. Since it was ostensibly a Linux conference it also seemed appropriate for it to run Linux.

Over the next few weeks I collected the parts and adaptors to build the badge. The main components were:

The Raspberry Pi Zero W is a single core 1Ghz ARM SoC with 512Mb RAM, Wi-FI, Bluetooth, microSD card slot, and mini HDMI. The Inky pHAT is a 212x104 pixel tri-colour (red, black, white) e-Paper display. It takes about 15 seconds to refresh the display but it draws very little power in between updates and the image persists even when power is removed.

Support Crates

The first part of the project involved building a Rust driver for the controller in the e-Paper display. That involved determining what controller the display used, as Pimoroni did not document it. Searching online for some of the comments in the Python driver suggested the display was possibly a HINK-E0213A07 from Holitech Co. Further searching based on the datasheet for that display suggested that the controller was a Solomon Systech SSD1675. Cross referencing the display datasheet, SSD1675 datasheet, and the Python source of Pimoroni's Inky pHAT driver suggested I was on the right track.

I set about building the Rust driver for the SSD1675 using the embedded HAL traits. These traits allow embedded Rust drivers to be built against a de facto standard set of traits that allow the driver to be used in any environment that implements the traits. For example I make use of traits for SPI devices, and GPIO pins, which are implemented for Linux, as well as say, the STM32F30x family of microcontrollers. This allows the driver to be written once and used on many devices.

The result was the ssd1675 crate. It's a so called no-std crate. That means it does not use the Rust standard library, instead sticking only to the core library. This allows the crate to be used on devices and microcontrollers without features like file systems, or heap allocators. The crate also makes use of the embedded-graphics crate, which makes it easy to draw text and basic shapes on the display in a memory efficient manner.

While testing the ssd1675 crate I also built another crate, profont, which provides 7 sizes of the ProFont font for embedded graphics. The profont crate was published 24 Nov 2018, and ssd1675 was published a month later on 26 Dec 2018.

The Badge Itself

Now that I had all the prerequisites in place I could start working on the badge proper. I had a few goals for the badge and its implementation:

  • I wanted it to have some interactive component.
  • I wanted there to be some sort of Internet aspect to tie in with the IoT theme of the conference.
  • I wanted the badge to be entirely powered by a single, efficient Rust binary, that did not shell out to other commands or anything like that.
  • Ideally it would be relatively power efficient.

An early revision of the badge from 6 Jan 2019 showing my name, website, badge IP, and kernel info. An early revision of the badge from 6 Jan 2019

I settled on having the badge program serve up a web page with some information about the project, myself, and some live stats of the Raspberry Pi (OS, kernel, uptime, free RAM). The plain text version of the page looked like this:

Hi I'm Wes!

Welcome to my conference badge. It's powered by Linux and
Rust running on a Raspberry Pi Zero W with a tri-colour Inky
pHAT ePaper dispay. The source code is on GitHub:

https://github.com/wezm/linux-conf-au-2019-epaper-badge


Say Hello
---------

12 people have said hi.

Say hello in person and on the badge. To increment the hello
counter on the badge:

    curl -X POST http://10.0.0.18/hi


About Me
--------

I'm a software developer from Melbourne, Australia. I
currently work at GreenSync building systems to help make
better use of renewable energy.

Find me on the Internet at:

   Email: wes@wezm.net
  GitHub: https://github.com/wezm
Mastodon: https://mastodon.social/@wezm
 Twitter: https://twitter.com/wezm
 Website: http://www.wezm.net/


Host Information
----------------

   (_\)(/_)   OS:        Raspbian GNU/Linux
   (_(__)_)   KERNEL:    Linux 4.14.79+
  (_(_)(_)_)  UPTIME:    3m
   (_(__)_)   MEMORY:    430.3 MB free of 454.5 MB
     (__)


              .------------------------.
              |    Powered by Rust!    |
              '------------------------'
                              /
                             /
                      _~^~^~_
                  \) /  o o  \ (/
                    '_   -   _'
                    / '-----' \

The interactive part came in the form of a virtual "hello" counter. Each HTTP POST to the /hi endpoint incremented the count, which was shown on the badge. The badge displayed the URL of the page. The URL was just the badge's IP address on the conference Wi-Fi. To provide a little protection against abuse I added code that only allowed a given IP to increment the count once per hour.

When building the badge software these are some of the details and things I strived for:

  • Handle Wi-Fi going away
  • Handle IP address changing
  • Prevent duplicate submissions
  • Pluralisation of text on the badge and on the web page
  • Automatically shift the text as the count requires more digits
  • Serve plain text and HTML pages:
    • If the web page is requested with an Accept header that doesn't include text/html (E.g. curl) then the response is plain text and the method to, "say hello", is a curl command.
    • If the user agent indicates they accept HTML then the page is HTML and contains a form with a button to, "say hello".
  • Avoid aborting on errors:
    • I kind of ran out of time to handle all errors well, but most are handled gracefully and won't abort the program. In some cases a default is used in the face of an error. In other cases I just resorted to logging a message and carrying on.
  • Keep memory usage low:
    • The web server efficiently discards any large POST requests sent to it, to avoid exhausting RAM.
    • Typical RAM stats showed the Rust program using about 3Mb of RAM.
  • Be relatively power efficient:
    • Use Rust instead of a scripting language
    • Only update the display when something it's showing changes
    • Only check for changes every 15 seconds (the rest of the time that thread just sleeps)
    • Put the display into deep sleep after updating

I used hyper for the HTTP server built into the binary. To get a feel for the limits of the device I did some rudimentary HTTP benchmarking with wrk and concluded that 300 requests per second was was probably going to be fine. ;-)

Running 10s test @ http://10.0.0.18:8080/
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   316.58ms   54.41ms   1.28s    92.04%
    Req/Sec    79.43     43.24   212.00     67.74%
  3099 requests in 10.04s, 3.77MB read
Requests/sec:    308.61
Transfer/sec:    384.56KB

Mounting

When I started the project I imagined it would hang around my neck like a conference lanyard. By the time departure day arrived I still hadn't worked out how this would work in practice (power delivery being a major concern). In the end I settled on attaching it to the strap on my backpack. My bag has lots of webbing so there were plenty of loops to hold it in place. I was also able to use the Velcro covered holes intended for water tubes to get the cable neatly into the bag.

At the Conference

I had everything pretty much working for the start of the conference. Although I did make some improvements and add a systemd unit to automatically start and restart the Rust binary. At this point there were still two unknowns: battery life and how the Raspberry Pi would handle coming in and out of Wi-Fi range. The Wi-Fi turned out fine: It automatically reconnected whenever it came into range of the Wi-Fi.

Badge displaying a count of zero. Ready for day 1

Reception

Day 1 was a success! I had several people talk to me about the badge and increment the counter. Battery life was good too. After 12 hours of uptime the battery was still showing it was half full. Later in the week I left the badge running overnight and hit 24 hours uptime. The battery level indicator was on the last light so I suspect there wasn't much juice left.

Me with badge display showing a hello count of 1. Me after receiving my first hello on the badge

On day 2 I had had several people suggest that I needed a QR code for the URL. Turns out entering an IP address on a phone keyboard is tedious. So that evening I added a QR code to the display. It's dynamically generated and contains the same URL that is shown on the display. There were several good crates to choose from. Ultimately I picked one that didn't have any image dependencies, which allowed me to convert the data into embedded-graphics pixels. The change was a success, most people scanned the QR code from this point on.

Badge display now including QR code. Badge display showing the newly added QR code

On day 2 I also ran into E. Dunham, and rambled briefly about my badge project and that it was built with Rust. To my absolute delight the project was featured in their talk the next day. The project was mentioned and linked on a slide and I was asked to raise my hand in case anyone wanted to chat afterwards.

Photo of E. Dunham's slide with a link to my git repo. Photo of E. Dunham's slide with a link to my git repo

At the end of the talk the audience was encouraged to tell the rest of the room about a Rust project they were working on. Each person that did so got a little plush Ferris. I spoke about Read Rust.

Photo of a small orange plush crab. Plush Ferris

Conclusion

By the end of the conference the badge showed a count of 12. It had worked flawlessly over the five days.

Small projects with a fairly hard deadline are a good way to ensure they're seen through to completion. They're also a great motivator to publish some open source code.

I think I greatly overestimated the number of people that would interact with the badge. Of those that did, I think most tapped the button to increase the counter and didn't read much else on the page. For example no one commented on the system stats at the bottom. I had imagined the badge as a sort of digital business card but this did not really eventuate in practice.

Attaching the Pi and display to my bag worked out pretty well. I did have to be careful when putting my bag on as it was easy to catch on my clothes. Also one day it started raining on the walk back to the accommodation. I had not factored that in at all and given it wasn't super easy to take on and off I ended up shielding it with my hand all the way back.

Would I Do It Again?

Maybe. If I were to do it again I might do something less interactive and perhaps more informational but updated more regularly. I might try to tie the project into a talk submission too. For example, I could have submitted a talk about using the embedded Rust ecosystem on a Raspberry Pi and made reference to the badge in the talk or used it for examples. I think this would give more info about the project to a bunch of people at once and also potentially teach them something at the same time.

All in all it was a fun project and excellent conference. If you're interested, the Rust source for the badge is on GitHub.

January 25, 2019

Ponylang (SeanTAllen)

0.26.0 Released January 25, 2019 07:27 PM

Pony 0.26.0 is a breaking release. It is however a pretty small breaking change. Most of the release is a series of bug fixes. As such, upgrading as soon as possible is recommended.

January 24, 2019

Jan van den Berg (j11g)

My music discoveries of 2018 part 2 January 24, 2019 11:30 AM

This is part 2 of my best music discoveries of 2018. Earlier editions are here: 2015, 2016, 2017 and 2018 (part 1).

Hands down my favorite tracks of 2018 are still One. Sentence Supervisor with Yelena, and City Calm Down with Blood. I have played these tracks more than any other. Both were featured in part 1. But, make no mistake, the second part of 2018 also presented some amazing discoveries.

In no particular order.

Sam Fender has the looks and the licks. Seemingly brimming with talent, this young gun already dropped a handful of shiners. But this track especially jumped from my screen. It’s easy to see where he gets his inspiration and that is not a bad thing. Not at all. I hope to hear a lot more from this guy.

This video from Castle Pines is one of my favorite, clever, visuals of 2018 (just watch it). But above all, this is a killer track! I can not believe it only has 242 views on Youtube!! That is just wrong.

The best EP from 2018 came from Australia. They’re called the Rolling Blackouts Coastal Fever. Quite a mouthful, but what a band. And this track in particular, Talking Straight, is different. On first listen it didn’t click. But a couple of listens in, this is a mesmerizing song. If you don’t like it: you’re not talking straight.

You can’t really discuss 2018 without at least mentioning Greta van Vleet. The multi Grammy nominated band made quite a splash in 2018. A classic rock band setup, three brothers and a real screamer. Sure, we’ve all seen the long read think pieces discussing their sound. It’s tiring. Fact is, I’d rather listen to this SG than a lot of other things out there now.

Ok, we need to talk about Idles. As punk as they come. But there is substance there too. And they’re probably the best live ticket in town at the moment. I mean, just look at this. The music seems to suffer by the performance, but overall it does not take anything away from what they’re trying to say. If anything, it adds to it.

Kurt Vile has been on my list before. The former founding member of The War on Drugs does his own thing and I love him for it. Listen to his singing lines. He does not follow usual singing conventions nor timing and makes lazy ears work. I park for free!

They’re called The Horrors for a reason, so you shouldn’t be surprised that this is a rather unsettling video. But at the core this is actually just a tender love song. I say this, because I think this track would also work well acoustic. But the futuristic synth sound they’ve chosen here, works equally well.

If my taste was expressed by an algorithm which you could feed videos to calculate the probability of what I like, this next song would score very high. Maybe it’s the name, maybe it’s the tense intro, maybe it’s the chorus, maybe it’s the rolling R’s. Maybe it’s all of that. The Twilight Sad.

The Decemberists is one of those bands I always thought I should pay closer attention to. This next gorgeous track underscores that thought most definitely.

Amen Dunes. Fragile sounding voice, heartfelt lyrics. Can you feel it, tonight?

I don’t know where the drums come from and I think that this is probably not a real live take, even though it’s made to look like it. But I’m game. You can never have enough violins. Darlingside with Hold Your Head Up High.

What, no shirt and only a leather vest? I am intrigued. Tell me more. The main melody is straight from Fleetwood Mac, and the guitar solo is one of the worst that I have ever heard. But this song has a certain honesty to it, and that goes a long way.

I don’t know anything about Linn Koch Emmery, but if I had to guess I think she probably owns a few Dinosaur Jr. records. And that is just fine! Videos that subtitle the lyrics need special attention. Attention well deserved in this case.

Bonus. Since it is 2019 already, here is a brand new track from Death Cab for Cutie. Laden with 80s nostalgia and keyboards, melancholy lyrics and a lot of guitar delay. Yes! I think I will probably listen to this song a lot in 2019. I must admit I think the mix on this song could be better. Why? Just listen to this live performance.

The post My music discoveries of 2018 part 2 appeared first on Jan van den Berg.

January 22, 2019

Alex Wilson (mrwilson)

Notes from the Week #15 January 22, 2019 12:00 AM

I’ve spent more time pairing this week than I have all year — granted that’s just over 3 weeks, but still, I often miss the tight collaboration of pairing when I have a fragmented week due to discussions and huddles across the business.

Where’s O11y?

This week’s main event was a kick-off session to make sure the Product Development is aligned around one of our explicit tech goals for the next quarter: Observability. As Shift team-lead, and the owner of the observability strand of work within Shift, I’ve been quite knee-deep and hands-on with this saga.

I ran a quick non-scientific poll in our Slack channel to get a rough idea of how people currently felt about it — my suspicion was that people had ideas about what observability meant but those ideas are probably at best inconsistent or at worst contradictory.

A lot of the work to improve our observability foundations will be driven through the Shift team, so members of each ProDev team got together and worked in groups to come up with user stories (as developers) for features that would improve their ability to observe their systems in production.

I’m (internally) working from the definition of Observability below:

… the ability to infer the internal state and behaviour of a system from its outputs

In the context of software development, outputs could be metrics, logs, or actual system outputs — Shift will be sharing their recent experiences with SLx and error budgets with the wider team,

Load balancing between teams

We have two teams in ProDev that are ostensibly have infrastructure as their primary concern:

  1. Shift, the Shared Infrastructure Team, lead by myself, accountable to the core development teams and the CTO.
  2. SREs, the Site Reliability Engineers, lead by the VP of Architecture, accountable to the team leads and the CTO.

The work that we might be doing tends to overlap a lot — how do we define the boundaries between the two such that neither team is stepping on the other’s toes but we don’t let things “fall between the cracks”?

SREs help the core development teams achieve excellence in their team-specific infrastructure concerns — Shift doesn’t have the size or the bandwidth to be across all of these concerns. Our SREs are also not part of the on-call system, and are sources of expertise for ProDev.

Shift on the other hand is responsible for operational stability of shared systems and their explicit mission is to build a solid foundation for systems consumed by all teams such as metrics, alerting, and configuration management.

But we work together!

SREs are often prototyping ideas and technologies that Shift doesn’t have capacity to do, and we collaborate on bringing them into the production environment in a state that Shift is happy to be on call for. On the flipside, Shift benefits from their expertise much in the same way as the core development teams.

The result is a quite harmonious relationship, and I’m eagerly awaiting upcoming advances that we’ll be collaborating on.

I paired a lot more than usual this week, with Petrut who is one of our SREs. We’ve been fine-tuning lifecycle policies on our S3 buckets and encouraging each team to create new buckets rather than use shared buckets.

When we were much smaller, having a single shared bucket for e.g. backups was fine, but as we’ve grown we’ve had to create numerous policies on the same bucket to cope with each separate prefix.

It’s hard to grok even for someone who’s been at Unruly for a while (like me) so we’re in the process of decommissioning our shared S3 buckets in favour of team-owned single-responsibility buckets.

It feels a lot like refactoring a piece of code with too many responsibilities, or breaking apart a monolithic application into smaller services — doing a bulk data copy is neither fast nor cheap, and we want to avoid breaking changes during switch-overs too.

Counting S3 Vastness

We’ve been using it at Unruly pretty much since it was released, for reporting aggregation and building our data lake, but I thoroughly enjoy the experience of using AWS Athena — I get into a workflow of:

  1. Discover process that takes a long time or outputs a lot of data
  2. Write a Python script to generate smaller CSV files from (1)
  3. Upload the CSVs to S3
  4. Query with Athena

As part of the work I was doing with Petrut, we were investigating digital assets from ad campaigns that have long since concluded — videos, images, etc. There were quite of these so we uploaded almost 10,000 CSV files to S3 containing metadata about these assets and could then run sub-5s queries over it all.

The more I do this kind of work, the more I realise the sheer utility of a simple standard format like CSV because it can go pretty much anywhere. If we wanted to do some more fun stuff with the data, we could write some ETL jobs in e.g. Python because it has CSV/JSON support out of the box.

I’m going to be leveraging the Athena learnings with my investigations into commit data using conventional commits in the near future, so watch this space for a blogpost full of graphs and fun facts!

Originally published at blog.probablyfine.co.uk on January 22, 2019.

January 21, 2019

Jan van den Berg (j11g)

Bad Blood – John Carreyrou January 21, 2019 10:49 PM

Everything you read and hear about Bad Blood by John Carreyrou is true. It is an incredible — TRUE — story, told exceptionally well, about the lethal cocktail of greed, ambition and narcissism.

Bad Blood: Secrets and Lies in a Silicon Valley Startup – John Carreyrou (2018) – 339 pages

If this wasn’t non-fiction I would have dismissed parts of this book as contrived. We are used to Silicon Valley success stories, but reading about such a supposed success, which raised 900 million dollars at a 10 billion dollar valuation, and which manages to completely fool people such as George Shultz and Henry Kissinger and which, eventually, turned out to be one big decade long lie, is insane.

But not only is the story in itself extraordinary, two other things make this book stand out.

First, the investigation about how this book came to be is part of the story itself. Which makes for a riveting read.

And secondly this book is a celebration of the freedom of the press. With the dazzling number of lawyers, lawsuits and NDA’s described in the book, I was sometimes worried it would be snatched from my hands as I was reading it. The pressure put on people NOT to reveal the truth was enormous. If people like Elizabeth Holmes and David Boies would have had their way, the world would look a lot different. So you get a real sense how these things can go the other way, and where we will never know the truth. But the truth has prevailed (this time) through the freedom of the press. Bad Blood gives hope!

The Steve Jobs book by Walter Isaacson (my favorite biographer) is mentioned a few times as a (misplaced) inspiration to some of the main character’s actions. Let’s hope Bad Blood can be such an inspiration of how not to do things.

Go read it.

The post Bad Blood – John Carreyrou appeared first on Jan van den Berg.

Bogdan Popa (bogdan)

Announcing forms January 21, 2019 07:00 PM

Today marks the first public release of forms, a Racket library for web form validation. Racket’s formlets module from the standard library already does something similar, but, unfortunately, it lacks any facilities for easily showing validation errors to end users which is a big part of what I want from this kind of library. Another nice thing about this new library is it’ll be able to validate things other than forms – like JSON – soon!

Chris Aumann (chr4)

Make keepalived play nicely with netplan/ systemd-network January 21, 2019 12:48 PM

Saturday night, 06:00:
I was torn from my sleep by the lovely voice of my colleague, uttering the words dreaded among the ops community: “We’re down, Chris.”

After opening the lid of my notebook, I started digging into the issue.

We’re running several high availability services with the help of keepalived. When a service goes down, keepalived switches over services to a standby instance. This is usually done by assigning a dedicated virtual IP address (VIP) to a service which can then be migrated on the fly between instances. This is achieved by keepalived using the Virtual Router Redundance Protocol (VRRP) as described in RFC5789, and has proved itself to be very reliable in the last years.

Not so today. The primary server somehow lost its VIP, without any notice in the log files. Just like that. I reassigned the VIP manually, verified that we’re back up and went back to sleep.

We then did a post-mortem on Monday. Obviously, we wanted to know what the heck had happened, and how we could prevent it from happening again in the future.

DHCP

Our OpenStack hoster relies on DHCP to configure the attached network interfaces. I usually prefer static configured interfaces on servers, and eyed the configuration with some suspicion for a while. We’ve had some minor issues with static routes and DHCP in the past, but the environment was pretty stable, so we went with the hoster’s original base images and their configuration.

Ubuntu upgrade

Keeping this in mind, we’ve been implementing one bigger change in our infrastructure: We’ve upgraded some instances (including the database node that had the outage) from Ubuntu 16.04 LTS (xenial) to Ubuntu 18.04 LTS (bionic). One of the major changes between the two releases is the change of the default networking stack from ifupdown to netplan. Could this have caused the issue?

netplan

After digging around a bit more, I found out that calling netplan apply (to apply a network configuration) resulted in dropped VIPs.

Apparently, keepalived doesn’t monitor the VIP assigned to it, and neither initiates a failover, nor tries to reinstate the dropped VIP after it was removed from the interface. This is a known issue, and was fixed by this commit which was released with keepalived-2.0.0.

Unfortunately, Ubuntu ships with keepalived-1.3.9, and the keepalived developers do not provide an official repository for more recent versions. After considering providing packages myself, I came to the conclusion that this actually wouldn’t fit the actual problem, as keepalived would just note the removed VIP and failover to another machine. But I wanted to fix the underlying problem itself, instead of just coping with the symptoms.

systemd-networkd

The default renderer for netplan on Ubuntu is systemd-networkd, a systemd component. On every netplan apply, the systemd-networkd is restarted, applying the new configuration. This apparently also happens when a DHCP lease is renewed and results in the VIPs being removed, because systemd-networkd is unaware of them, as they are assigned by keepalived and are not configured in /etc/netplan.

This seems to be a feature, and I kind of agree why: If you have a messy interface configuration and you connect to a network, it makes sense to make sure the interface is in a defined state.

But having a downtime every time a DHCP lease is renewed obviously also is not an option.

Possible solutions

So, how do I fix the problem permanently in a clean way? I was thinking about the following approaches:

  1. Migrating to a static IP configuration

As I was critical of using DHCP anyway, I considered migrating to static IPs. While this would fix the issues with DHCP lease renewals, we’d still have the same problem on other occasions of systemctl restart systemd-networkd (e.g. automatic security updates). I was reluctant to change that much of the networking stack, as I believe this should be done by the underlying hypervisor.

  1. Consider ditching netplan for the old ifupdown

We never had problems while using ifupdown with our configuration so far. I quickly decided against this though, as I really didn’t want to completely exchange fundamental network stacks on every new instance.

The final, clean solution

Before actually messing around with the underlying network stack, fortunately I was hinted at RFC1122. The Linux network stack defaults to “Weak ES (End System) Model”, enabling it to handle incoming packages for IP addresses configured on another interface than the physical interface the package came in.

This basically meant, that we could create a dummy interface and let keepalived manage the VIP on that interface. Then, systemd-networkd can configure the original interface at will, without hurting the dedicated interface for the virtual IP.

Here’s an example: On our system, the primary network card is ens3, and we’ve configured keepalived to assign the VIP as an alias on the same interface in /etc/keepalived/keepalived.conf:

1
2
3
virtual_ipaddress {
    10.1.0.1/24
}

I quickly hopped on a staging instance, creating a dummy interface keepalived0 and assigned existing VIP with a /32 netmask.

1
2
ip link add keepalived0 type dummy
ip address add 10.1.0.1/32 dev keepalived0

After that, I pinged the instance from another machine and then removed the original VIP:

1
ip address del 10.1.0.1/24 dev ens3

The pings are still answered!

Applying the configuration

This was the perfect solution! It was easily implemented without any downtime, got rid of the ancient IP aliasing (apparently deprecated since linux-2.4) and allowed us to keep the default network stack including netplan and systemd-networkd.

Bonus: The interface (incl. the master VIP) are also now displayed in Ubuntu’s default message-of-the-day (motd) upon login.

Edit: An earlier version of this blog post suggested to persist the keepalived0 dummy interface using netplan. This is not sufficient to solve the problem of the purged addresses on netplan apply.

Just add the keepalived0 interface by deploying the following file to /lib/systemd/network/90-keepalived.netdev:

1
2
3
[NetDev]
Name=keepalived0
Kind=dummy

Then run systemctl restart systemd-networkd to apply the configuration.

One thing I noted with this configuration is that the keepalived0 interface is always in DOWN state:

1
2
3
4
3: keepalived0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether ab:cd:00:00:ff:ff brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/32 scope global keepalived0
       valid_lft forever preferred_lft forever

This seems to be intended behaviour, as the dummy interface doesn’t implement many functions of a real interface.

So the last thing left was adjusting the keepalived configuration in /etc/keepalived/keepalived.conf:

1
2
3
virtual_ipaddress {
    10.1.0.1/32 dev keepalived0
}

I hope this ensures I can sleep in on Saturdays.

Derek Jones (derek-jones)

Teaching basic data analysis to programmers: summer internship January 21, 2019 12:28 AM

Software engineering is one of the topics in this year’s summer internships being sponsored by R-Studio. The spec says: “Data Science Training for Software Engineers – Develop course materials to teach basic data analysis to programmers using software engineering problems and data sets.”

It’s good to see interest in data analysis of software engineering data start to gain traction.

What topics might basic data analysis for programmers include? I have written about statistical techniques that I think are useful in software engineering, but I don’t think this list would be regarded as basic. Techniques that are think are basic are:

  • a picture is worth a thousand words, so obviously visualization is a major topic,
  • building regression models is good for helping to understand what is going on.

Anything else? Well, I don’t know.

An alternative approach to teaching basic data analysis is to give examples of the kind of useful things it can be used to do. Software developers are fast learners, and given the motivation have the skills needed to find and learn techniques that they think are of use. In a basic course, I would put the emphasis on motivating developers to think that data analysis can help them do a better job.

I would NOT, repeat, not, include any material on machine learning. Software engineering data sets tend to be too small to obtain reliable results from machine learning, and I don’t want to encourage clueless button pushers.

What are the desirable skills in the summer intern? I would say that being able to write readable material is the most important, with statistical knowledge ranked second; the level of software engineering knowledge is unimportant. Data analysis tends to follow the same pattern whatever the subject; so it’s important to get somebody who knows about data analysis.

A social science major is the obvious demographic for this intern (they do lots of data analysis); the last people to consider are students majoring in a computing subject.

Geoff Wozniak (GeoffWozniak)

Adventures in account deletion January 21, 2019 12:27 AM

For the past couple of years I’ve been cleaning up my digital presence a bit by deleting accounts with services or sites I no longer use. The list is pretty long. I’ve been hanging out on the Internet now for over 20 years and have been signing up for things since I started (my Slashdot user id is less than 5 digits). My list of known accounts, both deleted and active, is about 200. There are probably more since I didn’t start keeping track until about 2007.

The act of deleting happens in fits and spurts, but there is one unsurprising constant: deletion of accounts is, in almost all cases, a lot more work than creating them.

At this point I have deleted, or attempted to delete, nearly 120 accounts. Here is a list of some observations about the process as I’ve encountered it.

  • Sites that offer an option to close or delete an account directly have not been as common as I would like. The ability to do this usually resides with more recent services. You have to work through a bunch of confirmation dialogues, but that’s understandable. My guess it that I’ve only been able to delete somewhere between 20-40% of the accounts this way. (If it’s more than that, it sure doesn’t feel like it.) It’s a blessed relief when this option exists.
  • My Firefox account was probably the simplest to delete. I have nothing against them, but I don’t need the account, so I got rid of it. Kudos for a simple and obvious process.
  • Any account that is associated with payment or credit cards is probably going to require you to run through the customer service gauntlet. More often than not there is no option in your account preferences to close the account and you end up either in some silly pop-up chat window getting nothing but canned responses, or you end up getting emails with stuff like “a Customer Service Representative will respond to your email within 24 business hours”. It’s clearly meant to be a barrier, and it is.
  • Corollary to the previous point: these sites will often have no information on how to delete an account in their customer help pages.
  • Some sites don’t even ask to confirm the account deletion. A few times I’ve sent a message via a “Contact Us” form and gotten a message back that says, “Your account has been deleted.” Good thing it was me making the request.
  • Amazon made me go through an online chat, wherein the representative apologized for not being able to do anything then sent me a link to another page that blared grave warnings, upon which I could actually delete the account. The chats were pretty terrible because it was clear the reps bots were using canned responses they didn’t care about. This same sequence was repeated for both the .com and .ca versions of the account.
  • Don’t sign up for an IBM account if you don’t like chasing your tail. It was nothing but dark patterns there. (That account was a vestige of the time they were my employer and I tried out their web services for an experiment that did not go anywhere. Here’s hoping the $0.00 invoices stop coming.)
  • Edmodo (one of those accounts forced on me by circumstance) wouldn’t let me delete the account unless my course instructor approved it. Of course, by the time I was doing this, the course had ended about four years prior. I think it was shortly after they leaked 77 million records that I was able to delete the account without supervision.
  • Shout out to Apress for being unresponsive jerks.
  • Fuck Facebook, the ultimate purveyors of assholery.
  • Forums are probably the least enjoyable of all. phpBB and vBulletin are everywhere and come with an interface almost as friendly as z/OS. Deleting an account usually means getting in touch with an administrator, which may or may not be done via email. It’s clear the concept of getting rid of an account wasn’t even considered in the design of these things.

As I audited my accounts I realized just how often I had to sign up for a site to do something benign, like purchase an album or make a donation. Most of my accounts are throwaway junk. A lot of them were with sites that are now defunct. My habit these days is to delete the account as soon as I accomplish whatever it is I need to do. (PayPal holds the record for shortest account so far: about a minute. The deletion process is pretty smooth, actually.)

You probably have way more accounts than you need. Regrettably, if you intend to go through your accounts and delete the detrius, expect to spend a lot of time on it.

January 20, 2019

January 20, 2019

Ponylang (SeanTAllen)

Last Week in Pony - January 20, 2019 January 20, 2019 03:03 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, our users’ mailing list or join us on IRC.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

Jeff Carpenter (jeffcarp)

Book Review: Inspired January 20, 2019 01:07 AM

Inspired is a great introduction on how to be a Product Manager by Marty Cagan, a former engineer turned product expert. Here’s one of my favorite themes of the book: Product: build the right product Engineering: build the product right Here are the most poignant things I learned from this book organized by category. Product team structure It’s important to have somebody between product marketing and engineering (i.

January 19, 2019

Joe Nelson (begriffs)

Inside the C Standard Library January 19, 2019 12:00 AM

Cover of the Standard C Library After diving into the C language through K&R, and then studying portability (see C Portability Lessons from Weird Machines), my next challenge was to take a systematic look at the standard library. To do this I worked through P. J. Plauger’s book The Standard C Library (ISBN 978-0131315099) where he examines an implementation of all the functions. It has a chapter for each header, with background information, an excerpt from the C89 standard, tips on use, and full implemention with tests. The author was on the X3J11 committee that defined ANSI C.

As I worked through the book – trying first to write the examples myself, then comparing his code to mine, and finally running the examples – I kept notes with questions about portability, rationale, and C behavior. By cross-referencing the following books, asking questions on IRC, and browsing StackOverflow and the comp.lang.c archives, I found satisfactory answers.

  • “The C Standard: Incorporating Technical Corrigendum 1” by The British Standards Institution (ISBN 978-0470845738). This is the C99 standard itself (rather than C89 like Plauger’s book), and it includes an entire first half devoted to the rationale behind language and library choices. This is helpful for understanding C semantics.
  • “Portable C Software” by Mark R. Horton (ISBN 978-0138680503). Written after ANSI C was standardized, but early enough where it wasn’t fully adopted. He provides early history of each standard library function, as well as some functions that are now defunct.
  • “Portable C” by Henry Rabinowitz (ISBN 978-0136859673). Great for illustrating the design decisions of the language as it relates to diverse hardware.
  • “The CERT C Coding Standard” by Robert C. Seacord (ISBN 978-0321984043). Illustrates potential insecurity with, among other things, the standard library. Lists real code that caused vulnerabilities.
  • “C Programming FAQs” by Steve Summit (ISBN 978-0201845198). I can see why these were historically the most frequently asked questions. I asked many of them myself.

This article is not a comprehensive explanation of the standard by any means. It’s just things that were new or interesting to me. Some of it may be old news to you, and conversely I may have omitted something that seemed basic to me but would have been useful to mention. The focus is C89, with comparisons to the later standards C99 and C11 when relevant.

(Note: updated several sections based on feedback from Lobste.rs, HN, and Reddit. Thank you for the review, everyone.)

Brief History of the Library

Functions in the library grew organically from communities of programmers sharing ideas and implementations. Many groups of people used C on Unix throughout the 70s, across multiple architectures. They wrote compilers with extra features, and experimented with additions to Unix. By February 1978 core C practice had stabilized to the point where Kernighan and Ritchie codified it in the first edition of their book The C Programming Language (ISBN 978-0131101630).

By 1980 C users formed the “/usr/group” organization to combine their library experience into an informal standard, which they released in 1984. Meanwhile in 1983, the American National Standards Institute (ANSI) formed a committee, X3J11, to establish a standard specification of C and officially standardize the library. The committee reviewed the work of /usr/group, K&R 1st edition, and various compiler extensions. They deliberated from 1983 to 1989 to produce the C89 standard (“ANSI C”).

“Design by committee” may not have pleasant associations for some people, but in this case the committee drew on a lot of experience, and often declined to speculatively innovate, working to clarify existing practice instead. The result was a small, tight language and standard library.

Compared with libraries in other languages, the standard C library is lean. It doesn’t have much in the way of general algorithms or containers. This helped the language port easily and more widely. The library has basic facilities for time, math, I/O etc, and operates on simple types. It also provides portable facilities to do non-portable things, like variadic arguments, and non-local gotos.

We will restrict attention to the C standard library only, and not extensions such as POSIX. Thus no functions from unistd.h, fcntl.h etc.

assert.h

A simple way to halt a program with debugging information if an assumption doesn’t hold:

#include <assert.h>
int main(void) { assert(1 == 0); }

/* outputs
Assertion failed: (1 == 0), function main, file assert.c, line 2.
Abort trap: 6
*/

Because it echoes the statement under test and includes a filename and line number, it’s useful for a quick and dirty test suite. As a hack to include a little extra information, logically conjoin a string:

assert(1 == 0 && "Checking this because...");

Furthermore, assert() calls abort() rather than exit(), which causes the program to quit and dump core if permitted by the operating system. If the binary was compiled with debugging support you can load that core file in a debugger and inspect the program’s full state at the time the assertion was made:

gdb -c /path/to/corefile

# ^^^ inspect variables, backtrace, etc in the debugger

This means that littering functions liberally with assertions can be a good way to debug problems. It provides richer information than debug print statements in situations where it’s OK to terminate the program. It also adds no overhead to the final release, because the assertions will be removed by the preprocessor when compiled with NDEBUG defined. This can be done by adding -DNDEBUG to CFLAGS or by adding a regular #define NDEBUG in the code.

All headers in the standard library are idempotent, except assert.h. By including it multiple times in a file you can enable or disable the assert macro.

#define NDEBUG
#include <assert.h>

/* Now assert() won't do anything... */

#undef NDEBUG
#include <assert.h>

/* Now assert() works again */

It’s OK to include assert.h twice in a row because it causes what C calls a “benign” redefinition of the assert macro.

/* no harm, this is a benign redefinition: */
#define FUN 1
#define FUN 1

/* not benign and not allowed: */
#define FUN 1
#define FUN 2

A properly designed assert macro should work in any context, even somewhere weird like for (i = 0, assert(n<10); i < n; ++i). Because I didn’t think of this, my own attempt at writing assert used an if statement. In fact this is what Mark Horton shows in his Portable C Software book:

/* incorrect definition */

#ifndef NDEBUG
#define assert(p) (if(!(p)) ... )
#else
#define assert(p) ;
#endif

This can be improved. Plauger uses the ternary operator, and substitutes the result ((void)0) when the predicate holds. He also keeps the code (to print the error and exit) in a real function which is defined in a separate source file. That’s because headers in the standard library are not allowed to include each other. It’s a self-imposed discipline.

The last trick I found interesting was how to delay preprocessor evaluation into two steps using these helpers:

/* a "thunk" to evaluate __LINE__ */
#define _STR(x) _VAL(x)
#define _VAL(x) #x

The string can then be built as

"Assertion failed: " #p ", file " __FILE__ ", line " _STR(__LINE__) "\n"

Finally, some implementations tolerate an arbitrary scalar expression as the argument to assert, but the ANSI committee decided to require int expressions for correct operation.

Given that it was the first header I saw in Plauger’s book, this is where I learned that the standard reserves names like _Foo starting with underscore and a capital letter, for the standard library. Don’t use that naming convention in regular code.

ctype.h

Although I learned a few things from this header, it was ultimately kind of a letdown. It’s not sufficient for international text processing because, while the functions operate on the int type, they specify that its value should fall in the range of unsigned char or the negative value EOF. So while non-English speakers can use a new codepage for their locale, it will be unable to hold more than 255 symbols (plus \0, plus EOF), which is too few for east asian languages. C99 introduced wctype.h to operate on wide characters, but that has its own problems. (More about that in the stdlib.h section below.)

Even for languages which fit in an 8-bit codepage, the ctype functions don’t always suffice. For example the German letter ß uppercases into two letters, SS. The tupper() function in ctype.h can’t handle it, replacing one character with another. The greek letter Σ ordinarily lowercases to σ, except at the end of a word where it should be ς. The tolower() function doesn’t have enough context to pick the correct form.

I still learned some interesting techniques by studying this header. Plauger implements all the isxxxxx() functions with a lookup table of bit-packed shorts. The table itself is specific to EOF (from stdio.h) being -1, and relies on a certain size of unsigned char. The code uses a nice trick to fail early on a system where this assumption is incorrect:

#include <limits.h>
#include <stdio.h>

#if EOF != -1 || UCHAR_MAX != 255
#error WRONG TABLES IN CTYPE.H
#endif

Although the code is non-portable, it fails in the most honest and upfront way possible at compile time. Also as a historical note, the isxxxxx() functions used to be defined only for 7-bit characters, but ANSI requires them to handle all values for unsigned char.

One advantage of using a lookup table is that the value to be looked up requires only one evaluation. Thus the lookup function can be a macro without danger of executing code with a side effect more than once.

/* evaluates c at least twice
 * (also ASCII-specific, but don't focus on that)
 */
#define isalpha(c)  \
  (((c) >= 'a' && (c) <= 'z') || ((c) >= 'A' && (c) <= 'Z'))

/* evaluates c only once */
#define isalpha(c)  (_Ctype[(int)(c)] & (_LO|_UP))

/* consider how this would behave */
if (isalpha(c = getchar()))
  ...

Using macros for these little functions is good for performance. However, every library function in the standard library (unless specifically noted otherwise) must be represented as an actual function too, in case a program wishes to pass its address as a parameter to another function. How can we define a function called “isalpha” if that name is also recognized by the preprocessor? I learned you can just enclose the name in parens:

int (isalpha)(int c) {
  return (_Ctype[(int)(c)] & (_LO|_UP));
}

/* now &isalpha will be defined if needed */

The trick is used throughout the standard library but I first saw it in ctype. Conversely, every library function is a candidate for redefinition as a macro, provided that the macro evaluates each of the arguments exactly once and parenthesizes them thoroughly. (Well, getc is an exception but more on that later.)

Another trick is to perform an array lookup using a pointer that is shifted forward in the original array.

static const short ctyp_tab[257] = {0, /* ... */};
const short *_Ctype = &ctyp_tab[1];

The shifted pointer allows EOF (-1) to be looked up easily without undefined behavior. The expression _Ctype[EOF] means _Ctype[-1] which is the same as *(ctyp_tab+1-1), which does not attempt to dereference – or even point to – memory before a primary data object. Pointers are allowed to be assigned only addresses either inside, or one space to the right, of a data object. (Data objects are arrays, structures, or regions allocated from the heap. See Rabinowitz’s book for a good discussion of this.)

Character codes 128-255 are interpreted as negative numbers when considered as signed char. C does not specify whether char is signed or unsigned (it varies by platform). When char is signed, beware of converting it to int for the ctype functions. The integral promotion will “sign-extend,” creating a negative int value. This is not suitable for ctype functions, which require int values that are either storable in unsigned char or are the special EOF value. To avoid sign extension, cast char values to unsigned char in calls to ctype functions.

errno.h

Errno is the mechanism everyone loves to hate. The X3J11 committee wanted to remove it but decided not to make such a radical innovation on existing practice. In the early drafts of the standard it was kept it in stddef.h, but they decided that stddef should exist on freestanding environments and they split errno.h off into its own header.

Not much to say about it. The errno global variable is set to zero at the start of program execution. Library functions can set it to nonzero values but will never set it to zero themselves. To use it, explicitly set it to zero yourself, call a library function, then check whether errno changed.

One interesting tidbit is that errno is sometimes not a global variable at all, but a macro for (*_Error()). This allows different program threads to see a different value in a multi-threaded application. Even before threading considerations, having to set a real data object immediately after performing hardware floating point ops would break the FPU pipeline. Allowing the check to be deferred until requested with this _Error() function doesn’t break the pipeline.

float.h, limits.h, and math.h

Both float.h and limits.h are inventions of the committee. You can generate them with the enquire program written by Steven Pemberton. It performs runtime checks on the data types to find information about them, then generates the desired header file. It’s very (but not completely) portable, as it “only works if overflows are ignored by the C system or are catchable with signal().” It also detects and outputs information about the representation of base types, like endianness.

I decided not to slow down to study these headers in depth because I lack the knowledge of floating point representation necessary to understand the internals of the functions inside. I do know the definitions and parameters in float.h were recommended by numerical analysts on the Committee. The set was chosen so as not to prejudice an implementation’s selection of floating-point representation.

The math functions are written carefully to avoid overflow and underflow. I’ll revisit the topic after studying Michael Overton’s short book, “Numerical Computing with IEEE Floating Point Arithmetic.”

locale.h

Changing a program’s locale tells it how to handle local customs. There are multiple locale categories that can be adjusted independently. They control different things, like the codeset used by ctype, the date or monetary formatting desired, or alphabetical sort order. Typically all categories are set to the same locale. The alternative – a so called “mixed locale” – is less common.

Some of the settings are meant for interpretation by your own program, and others automatically affect the C standard library. For example, when parsing a double, strtod checks what the current locale uses for a decimal point symbol. Even if you want to avoid ctype and use a third-party Unicode library, some of the locale information is still useful for your program.

By default C programs use the “C” locale, which is ASCII text and American formatting. The most respectful thing, though, is to accept the locale in all categories as set by system environment variables. This is indicated by the empty string for locale name.

#include <locale.h>
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
  if (setlocale(LC_ALL, "") == NULL)
  {
    fputs("Unable to select system locale", stderr);
    return EXIT_FAILURE;
  }

  /* ... */
}

To see what locales are known on your local system, run

locale -a

There are 203 installed on MacOS. The list begins with:

  • en_NZ
  • nl_NL.UTF-8
  • pt_BR.UTF-8
  • fr_CH.ISO8859-15
  • eu_ES.ISO8859-15
  • en_US.US-ASCII
  • af_ZA

They are in the format [language[_territory][.codeset]]. OpenBSD has chosen to support only the “C” (aka “POSIX”) and “UTF-8” codesets, but supports many languages and territories in the other locale categories. Unicode subsumes all those partial character encodings, so BSD just wanted to eliminate a source of complexity.

One way to see locales in action is by setting environment variables and using Unix tools. We can change sort order using LC_COLLATE.

cat <<EOF >cn.txt
我
喜
歡
整
理
單
詞
EOF
LC_COLLATE=C sort <cn.txt LC_COLLATE=zh_CN.UTF-8 sort <cn.txt

Setlocale() doesn’t play well with threads because it updates an internal global variable. Set the locale before spawning threads.

One other random nugget of wisdom from the book, unrelated to locales but mentioned in that chapter, is to use a tool to visualize the call tree in a C program. This can help you understand a new codebase. Try cflow. See also Steve Summit’s C FAQ question 18.1.

setjmp.h

The setjmp/longjump functions create a “saveable goto” statement to return to places you’ve already been. It allows jumping from one function into another. They are like the C programmer’s version of exception handling. Setjmp and longjmp are very tricky inside, as they have to save and restore variables, arguments, registers etc to resume execution in another location.

#include <setjmp.h>
#include <stdio.h>

void dostuff(void);

jmp_buf target;

int main(void)
{
  if (setjmp(target) == 0)
  {
    puts("Saved the target, continuing on.");
    dostuff();
  }
  else
    puts("I feel like I've been here before...");
  return 0;
}

void dostuff(void)
{
  longjmp(target, 42);
}

To indicate that the jmp_buf was set, setjmp returns 0. When execution is jumped back to this point, setjmp returns value passed as the second argument to longjmp, for us 42. The same if statement is evaluated again but with a different result, like waking up in an alternate universe.

As simple as this looks, it’s easy to break. The statement containing setjmp must be very simple. Calling setjmp in an if statement or switch statement are fine, but you should not save the return value like n = setjmp(...). An assignment statement is too complicated and can disturb the sensitive machinery.

The function containing setjmp should be as simple as possible. Only variables declared as volatile are guaranteed to be restored. Generally it’s best to execute the real processing in another function called when setjmp returns 0.

Longjmp should not be called from an exit handler (i.e., a function registered with the atexit function). Finally it’s undefined behavior to attempt to longjmp to a function that has since returned. Thus longjmp’s usual use case is like exceptions in other languages, going back up the call chain, skipping intermediate functions.

The jmp_buf type is actually an array behind a typedef, which is why it’s typically passed to setjmp without an ampersand. The standard forbids jmp_buf to be implemented as a scalar or struct.

Given all these caveats, the committee considered requiring that compilers recognize calling setjmp as a special case. Then the function could work in all types of statements. However they decided against it for consistency because they don’t require any other function to be a special case (although they allow compiler writers to make special cases as desired).

signal.h

Signals are a UNIX technique for interprocess communication that causes a process to asynchronously call a handler function, or else take a default action. Programs also receive “synchronous” signals for their own logical exceptions like division by zero, segmentation faults, or floating point problems.

The ANSI committee decided to standardize a weakened portable version of signal functionality. Portable signal handlers can do very little. Here’s a typical example of what a handler can do safely:

/* a global */
volatile sig_atomic_t intflag = 0;

/* SIGINT handler */
void field_int(int sig)
{
  signal(SIGINT, &field_int);
  intflag = 1;
  return;
}

It does as little as possible, simply setting a global variable which regular code can check at leisure. One thing to note is that it re-installs itself in the very first line with the signal() function. That’s because on some platforms (like Linux), as soon as a signal is handled it reverts to its default handler, which in the case of SIGINT terminates the program. Other systems such as BSD leave a handler installed when called.

On Linux this would be an unwise thing to do:

void field_int_badly(int sig)
{
  /* open a window where a repeat signal could
   * hit the default handler before we reinstall */
  sleep(1);

  signal(SIGINT, &int_catch);
  intflag = 1;
  return;
}

Even our earlier technique of calling signal() right away in the handler isn’t completely safe. The CERT C Coding Standard warns that this leaves a tiny window open for a race condition. They suggest not to use the C library signal functionality at all, but the equivalent POSIX functions instead. POSIX allows you to specify persistence of the handler during initial registration.

Another thing to note is the type declaration of the shared flag (that we called intflag in the example). Exception handlers should read and write only volatile variables. For asynchronous exceptions, volatile alone isn’t even enough. The variable should be small enough that it can be read or written atomically by the processor.

The C standard provides the sig_atomic_t typedef for this. Each standard library implementation defines it as an alias for a suitably small integral underlying type. If you’re writing portable code, don’t assume that sig_atomic_t is anything bigger than a char, and don’t assume its signedness. Thus the portable value range is 0…127, although C99 added macros to determine its min and max values.

The standard says not to call any standard library functions from a signal handler except abort(), exit(), longjmp() or signal(). Certainly avoid any functions that interact with state, like those performing I/O or else stdio streams can become corrupted.

Even though the C standard says it’s OK to call longjmp from a signal handler, CERT gave an example where doing so caused a vulnerability [VU #834865] in Sendmail because it allowed attackers to time a race condition in main() by timing signals.

A program can raise signals for itself with the raise() function. (It used to be called kill.) However longjmp() is less tricky than raising signals for yourself and should be preferred.

The standard library defines this limited list of signals:

  • SIGABRT abnormal termination, such as is initiated by the abort function
  • SIGPPE an erroneous arithmetic operation, such as zero divide or an operation resulting in overflow
  • SIGILL detection of an invalid function image, such as an illegal instruction
  • SIGINT receipt of an interactive attention signal
  • SIGSEGV an invalid access to storage
  • SIGTERM a termination request sent to the program

Using other signals makes a program less portable.

stdarg.h

On the PDP-11 it was easy to walk function arguments with a pointer. The memory layout of arguments was well known, the size of pointers was the same as the size of int, and the original C language could not pass structures by value.

However when spreading to other architectures, C benefited from creating a portable way to access variable numbers of arguments. Non-PDP architectures had complex calling conventions. Using a library for variadic arguments makes code clearer too, whether or not portability is an issue.

Pre-ANSI C on UNIX used <varargs.h>, which required a final “dummy argument.” ANSI C got more picky about arguments matching a declaration, and introduced the “…” token to take the place of the dummy argument, which breaks varargs. The “…” also signals to a compiler that it may want to change the function calling convention.

The committee turned varargs.h into stdarg.h, and generalized the macros to the extent that all known C implementations would be able to handle them with little modification. This functionality was important for other functions in the standard library like printf and scanf.

Using the library is pretty easy, just initialize the list based on the final fixed argument, loop through the args, and release the list.

/* add a list of n numbers */
int sum(int n, ...)
{
  va_list ap;
  int s;

  va_start(ap, n);

  for (s = 0; n > 0; --n)
    s += va_arg(ap, int);

  va_end(ap);
  return s;
}

It is undefined behavior to call va_arg() more times than there are arguments, so the function will need to determine the number of arguments by other means. Our function above consults the n parameter for that information.

Speaking of the n variable, we’re not actually passing its value to va_start(), much as it may look. The va_start macro manipulates “n” as merely a name, so it can calculate the address of the next argument. Both va_start and va_arg must be implemented as macros, not as functions.

These are the portable assumptions for using stdarg.h:

  • The variadic function must declare at least one fixed argument
  • The function must call va_end before returning (for cleanup on some architectures)
  • va_arg can deal only with those types that become pointers by appending “*” to them. Thus register variables, functions, and arrays can’t be returned by va_arg
  • If a type widens with default argument promotions, then va_arg should request the widened type

The last point requires some explanation. When functions had no prototypes in pre-ANSI C, the compiler would promote smaller types to wider ones when sending them to functions. That’s because doing so is essentially free – it costs more to put a byte in a register than to put a word in.

Although ANSI requires functions to have prototypes, the promotion rule still applies to variadic arguments. Char and short get expanded to int, and float gets promoted to double. Thus to accept a char argument, ask for its value with va_arg(ap, int) and then cast to char. Don’t do va_arg(ap, char).

To pass a va_list to another function for continued processing either a) memcpy it if you want to consume all arguments in the current function or b) pass a pointer to it if the called function should consume some or all of the arguments. C99 added va_copy() for the first scenario.

The implementation of stdarg.h macros is gnarly and entirely platform specific. On my system they resolve to builtin compiler functions.

stddef.h

Stddef.h is a catchall place for definitions, sort of like stdlib.h. Why make two headers rather than combine them into one? It’s because C can be compiled in either a “hosted” or a “freestanding” environment. The latter is for embedded programming where there isn’t enough room for the entire standard library. An implementation must include all the standard library headers to be considered a hosted environment, while a freestanding environment must include only float.h, limits.h, stdarg.h, and stddef.h.

The committee deliberated about putting the things in stddef into the C language itself, but decided the need to extend C is not quite there.

Stddef provides ptrdiff_t, size_t, wchar_t, NULL, and offsetof(structname, attrib). Only ptrdiff_t and offsetof are unique to this header. Other headers usually contain duplicates of the other definitions.

I used to think that NULL was a reserved word in C, but have come to learn that the constant 0 is actually the crux. The compiler treats 0 specially in a pointer context and transforms it to whatever value represents the NULL pointer on the given architecture (which needn’t be bitwise zero). Thus NULL is typically a macro for ((void *)0). Ordinarily function pointers and data pointers cannot be mixed, but the standard makes an exception for the null pointer constant, allowing it to be assigned to even a function pointer: int (*foo)(void) = NULL.

The typedef size_t is an unsigned integral type big enough to hold the size of the largest possible object in memory. In some systems that might not be very large, for instance only 64k in the segmented memory model on the Intel 80286. Related rule of thumb: if a variable is going to index an array, it should be type size_t.

The typedef ptrdiff_t is a signed integral type of the result of a pointer subtraction. It’s signed because if (p - q) is positive then (q - p) will be negative. Note that if size_t is already the largest integer type, then ptrdiff_t can be no larger, yet the latter loses one bit to hold the sign. So it’s possible to make an array with cells too far apart for ptrdiff_t to measure. (Assuming there is room in memory for such a large single array.)

C99 provides a macro SIZE_MAX with the maximum value possible in size_t. C89 doesn’t have it, although you can obtain the value by casting (size_t)-1. This works in any number representation (C89 section 6.2.1.2, C99 section 6.3.1.3 signed and unsigned integers), because the value (-1) is converted by “repeatedly adding or subtracting one more than the maximum value that can be represented in the new type until the value is in range of the new type.”

The offsetof() macro can determine the offset in bytes of a member within its structure. This cannot easily be determined otherwise due to structure padding. The C99 rationale talks about using it to provide “generic” routines that work from descriptions of the structures, rather than from the structure declarations themselves. On many platforms it is defined as:

#define offsetof(type, field) ((size_t)(&((type *)0)->field))

That’s undefined behavior, but that’s what the standard library is for: a portable way to do sometimes non-portable operations.

stdio.h

UNIX I/O was clean and simple compared with other systems of its day. It was 8-bit clean, and used a consistent line terminator. At the edges ioctl() would translate the simple I/O streams for the idiosyncrasies of attached devices. The kernel would hold file control state internally and give programs a simple integer descriptor for to use when reading and writing.

Leaving UNIX, C ran up against the complexity of I/O on other systems. The X3J11 committee talked with vendors and came away with a sharper understanding of the I/O model they wanted to support. They had to distinguish text and binary modes. While UNIX used ‘’ for line endings, DOS used \r\n and Mac classic used ‘’. These endings have to be normalized to ‘’ in text mode but not binary mode. UNIX ignores binary mode, but you better enable it for portability when necessary.

UNIX also had an unusually faithful representation of files. You could put bytes in and expect to read them out unchanged. When doing fully portable I/O keep these caveats in mind:

  • A final line without a terminating newline (in text mode) can confuse some systems, and they may drop the line or append a newline.
  • Don’t count on the system preserving trailing space in a line. Some systems strip it out. Conversely, some systems add a space to a blank line so the line “has something in it.”
  • The maximum fully portable line length is 254 characters.
  • Implementations are free to pad the end of binary files with a block of NUL characters to make the files match certain disk block sizes.

Another difference between the C standard I/O and UNIX is buffering. In UNIX, people often wrote their own buffering code to reduce the number of relatively costly I/O system calls. The X3J11 committee decided to include this buffering functionality in stdio.

Buffering is an optimization that can be tailored to expected patterns of I/O. The standard library provides the setvbuf() function to change the size and location of a stream’s buffer, as well as choosing between line or block buffering. By default, stdin and stdout are line and block buffered respectively, and stderr is unbuffered. Setvbuf() must be called immediately after a stream is opened, before I/O happens, to have any chance of working.

(stdio.h) opening and closing

Perhaps surprisingly, there is a lot to learn about just opening files. First there may be a limit on how long a filename can be on a system. Stdio provides the FILENAME_MAX macro with this limit. If the system imposes no practical limit then the macro is just a suggested size. This value could be both too short, or paradoxically too long. If it is set very large then you might end up wasting memory or causing problems if allocating on the stack.

Similarly L_tmpnam is the size needed for an array large enough to hold a temporary filename generated by tmpnam(). This function is a security hazard (though it can be useful for generating entropy). It introduces a Time of Check, Time of Use (TOCTOU) race condition because another program or thread could obtain the same temporary file name and create the file first. Use the tmpfile() function instead which actually creates the file, and registers it for removal on normal program exit().

Another common TOCTOU happens with fopen() when trying to create but not replace a file. Programs first check existence, right after which an evildoer can create a symlink of the same name in time for the fopen with “w” mode to overwrite another file with possibly elevated permissions.

/* dangerous */

FILE *fp = fopen("foo.txt","r");
/* <-- attacker gets busy here */
if (!fp)
{
  fp = fopen("foo.txt","w");
  ...
  fclose(fp);
}
else
  fclose(fp);

C11 fixes this with an “x” (exclusive) mode modifier. If exclusive mode is specified (“wx”), the fopen fails if the file already exists or cannot be created. In C89 you can either go beyond the standard library, using the POSIX open() function with the O_CREAT | O_EXCL flags, or just try to keep the time between check and write as small as possible.

Once you have opened a file to your liking, or have been given a FILE pointer, treat the pointer as totally opaque. Don’t even try to make a copy of the FILE structure because some implementations rely on magic memory addresses for some of them. The CERT standard (FIO38-C) says that the following can cause a crash on some platforms if you try to use my_stdout:

/* don't do this */
FILE my_stdout = *stdout;

There’s a related function called freopen(), but it’s not used very often. The main use is converting a big program from reading stdin to reading a named file. It’s the simplest way to do that, whereas a new program should just directly fopen whatever file it wants.

During a normal program exit, all open files will be closed. Still, it’s useful to explicitly call fclose() on file handles. It helps avoid exceeding the FOPEN_MAX limit of files that can be open at once. Also, failing to properly close files may allow an attacker to exhaust system resources and can increase the risk that file buffers will not be flushed in the event of abnormal program termination.

Speaking of flushing buffers, fflush() might force items in the buffer to be processed, but there is no guarantee. For convenience, fflush(NULL) flushes all streams, which is useful in preparation for possible loss of program control, like going into a dangerous section, or telling the user to turn off the computer.

Two other quirks. Some operating systems will not actually create a file that you fopen() and fclose() unless you write something. Also you can close stdout or stderr, and there are sometimes reasons to do so!

(stdio.h) file navigation

Stdio.h has two similar pairs of functions to move around in a file: fgetpos/fsetpos and ftell/fseek. Why the duplication? The second pair represents position as a long integer. When the file is opened in binary mode, this long is the number of bytes from the start of file. This is useful because you can do arithmetic on the integer to jump to particular places. The drawback is that on some systems a long is only 32 bits, so cannot support large files.

The fgetpos/fsetpos pair works using a special structure that can represent positions in huge files. You must treat treat this structure as a magic bookmark. It’s only obtainable from a call to fgetpos, you cannot construct your own to point to a position you haven’t already been.

Stdio also includes a rewind() function, but don’t use it. It actively clears the error indicator for a stream. Instead do a fseek(stream, 0L, SEEK_SET).

These navigation commands were interesting to me, so I created a project called randln to experiment with different ways of picking a random line from a text file. You might find it interesting to look at the pros and cons of each method as explained in the readme.

You can actually inspect I/O as it happens. The trick is walk the program through a debugger while tracing its system calls in another terminal. To use that randln program as an example, First start it in the debugger. Enable tracing for the debugger’s I/O and that of its children.

ktrace -i -ti gdb randln

Put a breakpoint in the line-finding function and start the program. In another terminal find the PID of randln that was launched by gdb, and start tailing the ktrace dump. This command will show the first 20 bytes of data in each I/O request:

 kdump -m 20 -l -p <pid>

Now step through the program in the debugger and watch the consequences of each statement.

A final note about stream navigation and the shell. I noticed when doing foo <bar the program can perform fseeks on bar, but cat bar | foo cannot. It’s another reason not to abuse cat.

(stdio.h) reading and writing

The foundation of all stream input is fgetc(). The other standard library input functions must be implemented as if they call it repeatedly, even if they don’t. It pulls a character out of a stream (or a character that was pushed back by ungetc() if such exists), and refills the stream buffer if needed. Some platforms allow ungetc() to push back a whole stack of characters, but the portable assumption is that it can store only one.

While fgetc is a function, getc() is a macro that avoids incurring a function call just to get a character. The downside is that getc() is allowed to evaluate its argument more than once, so don’t do anything with a side effect there. Now, fgetc() is allowed to be a macro too, but may not be as efficient because it has less freedom in its implementation. It is not permitted to evaluate its argument twice.

Moving up the food chain we come to gets(), which reads characters into a string until it encounters the NUL character – and it’s a buffer overflow waiting to happen. It was removed in C11. The fgets() function is better because you can specify a max length. However if it fails, the contents of the array being written is indeterminate. It is necessary to reset the string to a known value to avoid errors on subsequent string manipulation functions.

The fread/fwrite functions work in chunks. My only notes about them are

/* prefer this */
fread(buf, 1,    size*n, stream)
/* over this   */
fread(buf, size, n,      stream)

The second form is worse because you can’t detect whether it read an extra size-1 characters past what it reports. Also some implementations of fread (fwrite) simply call fgetc (fputc) in a loop, whereas others are more optimized. Doing a straight UNIX read (write) can be faster.

The standard library doesn’t allow alternating reads and writes without intervening operations to flush or explicitly position the stream:

  • after writing, call fseek(), fsetpos(), rewind(), or fflush() before reading
  • after reading, call fseek(), fsetpos(), or rewind() before writing unless the file is at EOF: a read that hits EOF can be followed immediately by a write

The last thing I wanted to mention about reading and writing is a freak consequence of weird machines. On some digital signal processors (or more generally on the DeathStation 9000), both char and int are 32 bits. This causes a vulnerability in the common pattern:

int c;
while ((c = getchar()) != EOF)
  ...;

There is no extra room in the int type to hold EOF as distinct from a valid character code, so a valid character can make the loop stop early. It’s a high severity bug, a variation of which caused a nasty vulnerability in the bash shell, CA-1996-22.

Fine, you say, you don’t plan to target such machines!

#include <limits.h>

#if UCHAR_MAX == UINT_MAX
#error "Your machine is weird."
#endif

Also you’ll be careful not to indicate a parse failure with an unsigned code that casts to a signed value of EOF. Well the same logic can still trick you in C99 with wide characters if you’re not careful.

It works like this: the fgetwc(), getwc(), and getwchar() functions return a value of type wint_t. This value can represent the next wide character read, or it can represent WEOF, which indicates end-of-file for wide character streams. On most implementations, the wchar_t type has the same width as wint_t, and these functions can return a character indistinguishable from WEOF. For this situation be sure to check after the loop for feof() and ferror(). If neither happened then you’ve received a wide character that resembles WFEOF.

It’s yet another place where wide character implementations are half baked.

(stdio.h) formatting

Scanf and printf have formatting options I hadn’t seen before. First of all they can limit the length of strings they read or write.

/* limit to 10 characters */
printf("%.10s", big_string);

/* or limit to n characters */
printf("%.*s", n, big_string);

/* limit to 10 characters */
scanf("%10s", input);

I used to think scanf was very unsafe, but this limiting helps. Scanf also supports “scansets” to match strings containing specific characters. Here is how to match up to ten vowels: %10[aeiou]. Scanf also allows you to match but not capture, using *. E.g. %*d. The %n option saves the number of characters read so far in the scan.

Finally, in C99 printf has modifiers for size_t (%zu) and ptrdiff_t (%td). Because they are typedefs which change by architecture there is otherwise no way to specify them portably.

stdlib.h

Stdlib is a hodgepodge. It has six categories of functions inside:

  • algorithms (search, sort, rand)
  • integer functions
  • number parsing
  • multibyte conversions
  • storage allocation
  • environmental interactions

(stdlib.h) random numbers

Let’s start with an interesting topic: random numbers. The standard library provides a rand() function to generate a pseudorandom sequence starting from a seed specified by srand(). The numbers range from 0 to RAND_MAX. The first problem is that RAND_MAX can be very small (~ 65535) on some platforms. Second problem is that the quality of rand() is not generally very good on most platforms. On Mac OS it is horrible. Changing the random seed by only a small amount (such as seeding by the epoch), leads to similar initial random values.

seed 1st rand
1500000000 1189467867
1500000001 1189484674
1500000002 1189501481
1500000003 1189518288
1500000004 1189535095
1500000005 1189551902
1500000006 1189568709
1500000007 1189585516
1500000008 1189602323
1500000009 1189619130

Rather than rely on whatever implementation of rand you get on a given architecture, it’s just as easy to define your own xorshift rand function.

static unsigned long g_rand_state = 0;

unsigned long defensive_rand()
{
  uint32_t x = g_rand_state;
  x ^= x << 13;
  x ^= x >> 17;
  x ^= x << 5;
  /* restrict to 32-bits */
  x &= 0xFFFFFFFF;
  return (g_rand_state = x);
}

The constants 13, 17, 5 are specific to 32-bit numbers, so we mask the result to keep it within that range. There are also good 64-bit versions you can use with the “long long” type in C99. See also Chris Wellons’ discussion of creating good rand() functions through an exhaustive search.

Now we have a good rand() function, but how should we seed g_rand_state with an initial value? The typical way is to gather the epoch from time(NULL), but it will mean the application will use the same seed if run more than once per second.

There are two other sources of entropy available from the standard library. One is hashing the path generated by tmpnam(), which in many implementations consults the process ID or a higher precision clock. The second is hashing the address of the main function, which is often fairly unpredictable due to address space layout randomization (ASLR). Do not rely on ASLR alone, however, as not all platforms use it. The address of main() should always be mixed with other sources of entropy.

Using the address of main numerically is a little tricky. C99 has a type called intptr_t which can hold pointer values as an integer, but this type is for data pointers only, not function pointers which on some architectures have a different size. We might consider casting a pointer to main as (char *) and reading the bytes, but for the same size reason this isn’t feasible.

What we can do is create a function pointer to main, point another pointer at it and read the value out byte by byte. A pointer-to-function-pointer is just a data pointer and can be cast to (void *).

int (*p)(int, char**) = &main;
unsigned char bytes[sizeof(p) + 1] = { 0 };

memcpy(bytes, (void*)&p, sizeof(p));

The bytes array is a NUL terminated string and can be hashed. To see these techniques in action and how to combine the entropy, check out rand.c.

If you’re willing to use functions beyond the C standard library, POSIX provides a random() function that is higher quality, and the BSDs/macOS/illumos (or Linux with libbsd) provide arc4random() which returns crypto-grade randomness.

(stdlib.h) integer functions

In C89 the rounding of integer division isn’t fully specified. When one of the numerator or denominator are negative it may either round toward zero or downward. It matches whatever the underlying hardware does. The committee didn’t want to introduce overhead on any system going against the hardware convention. In C99 they changed their mind, reasoning that Fortran (known for numerical programming) defines the rounding.

C99 rounds toward zero. To match this behavior in C89, use the div and ldiv functions. They return structures div_t and ldiv_t that contain both the quotient and remainder of the division. The only reason to use these functions in C99 is for efficiency, because the functions may be implemented with a compiler builtin that can compute the quotient and remainder together in a single assembly instruction.

A note about the ato{i,l,f} functions – their behavior is undefined if the input cannot be parsed correctly. These functions need not set errno even. Except for behavior on error, atoi is equivalent to (int)strtol(nptr, (char **)NULL, 10). The strto{d,l,ul} functions should always be preferred, because they provide proper error reporting and choice of base.

The implementation of strtod in Plauger’s book was very careful about floating point overflow. It processed digits in groups of eight and then combined them later into a final result. It also consulted the decimal_point attribute set by the LC_MONETARY part of the current locale.

(stdlib.h.) memory

An interesting bit of history: malloc used to be considered a low-level UNIX function, while calloc (“C alloc”) was conceived as the portable C version. However the committee standardized malloc too because it has less overhead (doesn’t zero the memory). Nowadays it seems malloc is more popular. There also used to be a cfree, but it was identical to free and didn’t make the cut for ANSI C.

The most flexible allocation function is realloc because it can simulate both malloc and free. When passed NULL for the original pointer it behaves like malloc, and when passed 0 for requested size it acts like free.

(stdlib.h) termination and execution

Stdlib contains the EXIT_FAILURE and EXIT_SUCCESS macros with implementation defined integer values to indicate success or failure on exit. These values would be returned by main or passed to exit(). The C standard actually treats return 0 in main or exit(0) specially, and maps it to whatever the system specific success code is. (Similar to how 0 in a pointer context is treated specially as the NULL value the way we talked about earlier.) Thus you don’t need to include stdlib.h just to return successfully.

EXIT_FAILURE is necessary for portably indicating failure, though. The raw value 1 is considered successful on some platforms.

Speaking of exit(), it causes “normal” termination. It closes all open file handles, deletes files created by tmpfile(), and calls any handlers registered by atexit() in reverse order of their registration. The abort() function, on the other hand, causes immediate and “abnormal” termination. It needn’t flush buffers, remove temp files, or close open streams. It can be canceled by catching SIGABRT. Aborting can be useful because it will cause a core dump if the OS is configured to save one. Thus the assert() function calls abort() to produce a core file to debug assertion failure.

Stdlib.h provides the system() function to run a command in the shell. If the command is a NULL pointer, system() will return non-zero if the command interpreter is available, and zero if it is not. The CERT standard says system() is a security violation and flat out says not to call it. It’s easy to make a mistake with system(). Commands that are not fully pathed can be overridden, and even relative paths can be tricky if the attacker can control the current directory. And of course unsanitized input can attack through the shell. CERT suggests using execve() in POSIX to call fully pathed executables.

(stdlib.h) wchar_t

OK, this is going to get complicated. The takeaway is that the C standard library cannot portably handle Unicode. You’ll need a good third-party library. Let’s see why this is.

Before locales and all that, C was designed for 7-bit ASCII text. The committee endorsed the use of locales to specify a codepage that reassigned the meaning of extra characters using all eight bits (the minimum allowed size of char). However some languages use more than 255 symbols (+ NUL). There were two ways to handle bigger alphabets: use bigger chunks of memory to hold each character code, or define special sequences of single byte characters to stand for extended characters.

These approaches are called “wide characters” and “multibyte characters.” Generally because networking equipment and disk storage is byte oriented, programs use multi-byte character encoding for communication with the outside world and storage. However for in-memory use people felt that using wide characters would be cleaner, because each cell in an array would map to exactly one character, like in the old ASCII string days.

The C89 standard is not very helpful with multibyte or wide character encoding. It merely set the stage with a wchar_t type for wide characters, and functions in stdlib.h to convert multibyte to wchar_t (mbstowcs) and the reverse (wcstombs) based on the locale. The committee was waiting to see how people wanted to work with international characters before standardizing it. K&R 2nd edition does not list these functions but Plauger and the C89 spec have them.

The whole idea was that locale-specific multibyte to wide character conversion was supposed to be more general than any particular encoding system. However, nowadays Unicode has proven to be more popular than other encodings. The multibyte UTF-8 encoding is the standard interchange on the web, and the UCS-4 character set is up to the task of handling all the world’s languages and then some. Should be no problem, right?

Well, at the time C99 was being standardized, the Unicode consortium (actually the contemporaneous European ISO committee) was endorsing the more limited Universal Coded Character Set UCS-2. The codepoints for UCS-2 are 16 bits, so that’s the minimum width that C99 requires for wchar_t. Sadly the committee made their decision shortly before four byte UCS-4 was proposed (ISO 10646).

Vendors like Microsoft jumped into Unicode right away and implemented wchar_t as 16 bits. It’s deep in their APIs and they’re now stuck with that size for backward compatibility. Even Mac OS and iOS use 16 bit wchar_t for whatever reason.

UTF-16 combines the worst of multibyte characters and wide characters. a) Characters outside the “base multilingual plane” require two UTF-16 codepoints (called a surrogate pair) to represent them. This breaks the one-character-one-codepoint assumption. b) It’s wasteful of memory because even ASCII characters take two bytes.

C99 provides wide character versions of the ctype functions, in <wctype.h>, but they simply cannot work properly with surrogate pairs. For example (as pointed out here):

  • 0xD800 0xDF1E = U+1031E is a letter (iswalpha should be true)
  • 0xD800 0xDF20 = U+10320 is not a letter (iswalpha should be false)
  • 0xD834 0xDF1E = U+1D31E is not a letter (iswalpha should be false)
  • 0xD834 0xDF20 = U+1D320 is not a letter (iswalpha should be false)
  • 0xD835 0xDF1E = U+1D71E is a letter (iswalpha should be true)
  • 0xD835 0xDF20 = U+1D720 is a letter (iswalpha should be true)

Neither the first nor the second element of the pair alone can predict whether the resulting Unicode character is alphabetic. There is no way that a system can provide this information through a function ‘iswalpha’ that takes a single wchar_t argument.

C99 does guarantee a macro will be present when the current environment is ISO 10646 compliant, meaning wchar_t can hold every UCS-4 codepoint. We can blow up for the other platforms:

#ifndef __STDC_ISO_10646__
#error "Your wide characters suck."
#endif

Even assuming we restrict the code to ISO 10646 systems only, the wctype functions are too crude to deal with the subtleties of international languages. Because of how Unicode characters can join together, it’s infeasible to use pointer arithmetic to calculate “string length” or parse “words” with iswspace() robustly.

Some parts of programs can continue to operate on text as an opaque series of bytes. However for other parts that must inspect the characters themselves, you should use a sophisticated Unicode library like ICU or utf8proc. This has the advantage of working with C89, so you won’t be forced to upgrade to C99 just because of text processing.

With a good Unicode library we don’t need wide characters. We can use UTF-8 everywhere, even in program memory. That’s the school of thought behind utf8everywhere.org.

(stdlib.h) lessons from the code

While reading Plauger’s implementation for stdlib, I noticed some tricks worth sharing.

The comma operator can be used to group multiple small assignments together under an if statement without needing to add curly braces.

if (condition)
  foo = 1, bar = 2;

Also an if statement with no statements can be used to shrink a big expression in another if statement:

if (cond)
  ;
else if (big cond)
  foo;

C89 allows bare blocks inside a function to segregate variable declarations near the code that uses them. The variables are not accessible outside the block. It can help readability to know when variables are no longer needed, although you might argue that it suggests the function as a whole is too big.

#include <stdio.h>

int main(void)
{
  puts("Hello.");

  {
    int i = 0;
    printf("%d\n", i);
  }
  /* error: use of undeclared identifier 'i' */
  printf("%d\n", i);
}

The trick is even useful in C99 to reuse and reinitialize variables across repetitive test cases in a single function.

Speaking of brackets, Plauger uses Whitesmiths style indentation, with brackets indented to the level of their code:

if (cond)
  {
  foo();
  while (cond)
    {
    bar();
    baz();
    }
  }

I find it difficult to read (probably just unfamiliar), but it does have an internal consistency. The brackets are wrapping up multiple statements into a single unit, and this unit is indented the same way that a single statement would be. Still not going to indent this way, but just saying.

Another interesting trick is negating an unsigned number. Plauger does that to consolidate code for signed and unsigned numbers in the same function.

unsigned long _Stoul(const char *, char **, int);

#define atoi(s)                  (int)_Stoul(s, NULL, 10);
#define atol(s)                  (long)_Stoul(s, NULL, 10);
#define strtoul(s, endptr, base) _Stoul(s, endptr, base);

This _Stoul() function negates its unsigned long value if the string it is parsing has a negative sign. This operates bitwise on an unsigned value the same as it would on signed, and after casting for atoi and atol it will be negative as expected. I didn’t know it was “allowed,” but C doesn’t care.

string.h

This header is divided between functions starting with str- and those starting with mem-. The former work with NUL terminated strings, and the latter operate with explicit lengths. Some of the str- functions have a modifier to limit length (strncat, strncmp, strncpy) or a modifier to work backward (strrchr). However the header doesn’t have all permutations. For instance no strnlen or memrchr.

(string.h) widened types

Why does strchr take an int rather than a char for its search character? Same with memset, it takes an int value, but converts it to unsigned char internally. The spec dictates this in section 7.21.6.1. That misleading int signature is for backward compatibility with pre-ANSI code which lacked function prototypes and promoted char arguments to int. Under default argument promotions any integral type smaller than int (or unsigned int) always converts to int.

All standard library functions are specified in terms of the “widened” types. It’s just that string.h contains many functions where it is especially apparent. The widened types ensure that most library functions can be called with or without a prototype in scope. Legacy code doesn’t use prototypes, and ANSI C did not want to break backward compatibility. Even if people were willing to update all the legacy code, any legacy modules distributed as compiled object files rather than source would not link properly against functions with changed argument types.

Similar rationale lies behind the standard’s guarantee that char pointers and void pointers share the same representation and alignment requirements. Relying on this guarantee allows old code that to work with the void* malloc in place of the original char* malloc. (See the C99 Rationale section 7.1.4, Use of Library Functions.)

(string.h) signed vs unsigned

We’re not done with the type mysteries in string.h. Why is it that strchr casts its int argument to char internally and memchr casts to unsigned int? Well given that strchr is searching through char*, it makes sense to match the type. But memchr is searching through void*. Why not compare each memory location with char rather than unsigned char?

It turns out that the C standard makes special guarantees about unsigned char that make it an ideal type to represent arbitrary binary information.

  1. Unsigned char is guaranteed to have no padding bits. All bits contribute to the value of the data. Other types on some architectures include things like parity bits which don’t affect the value itself but do use space.
  2. No bitwise operation starting from an unsigned char value, when converted back into that type, can produce overflow, trap representations or undefined behavior. It can be freely manipulated. Trap representations are certain bit patterns that are reserved for exceptional circumstances, like the NaN value in floating point numbers.
  3. Accessing parts of a larger data object ith an unsigned char pointer will not violate any “aliasing rules.” The unsigned char pointer will be guaranteed to see all modifications of the data object.

In “Portable C,” Rabinowitz says the char type has a few distinct uses: codeset characters, units of storage, small numbers, and small bit patterns. He recommends regular char for codepoints and small numbers, and unsigned for the others. (In C99 wchar_t might be considered the proper way to represent codeset characters, but we’ve already seen the difficulty there.)

Note that Rabinowitz doesn’t specify signed char and unsigned char, but rather plain char and unsigned char. That’s because C does not specify whether plain char is signed or unsigned. We would get a warning trying to pass a definitively signed char* to a function in string.h if the default was unsigned char for that platform.

(string.h) mem- implementations

The next functions that taught me something are memcpy and memmove. The first one blasts bytes from one part of memory to another, possibly taking advantage of machine instructions to do large block copies. It doesn’t check for an overlap between the source and destination, in which situation the results are undefined. C99 marks the source and destination pointers with the restrict qualifier to allow the compiler to optimize under that assumption.

Memmove is the slower more careful brother. It works correctly even if the source and destination areas overlap. It is specified to act as if the source memory is first copied to a separate buffer, then copied into the destination. When I tried writing it that is exactly what I did, but Plauger has a faster way:

void *(memmove)(void *d, const void *s, size_t n)
{
  unsigned char *ds = d;
  const unsigned char *ss = s;

  if (s > d)
    while (n-- > 0)
      *ds++ = *ss++;
  else
    for (ss += n, ds += n; 0 < n; --n)
      *ds-- = *ss--;

  return d;
}

This compares the pointer positions to see which occurs before the other in memory, then does the copy from left to right or right to left depending on which comes before the other. This is very fast and uses no extra space.

However, isn’t comparing random pointers undefined behavior? C allows you to compare pointers within the same primary data object (like the addresses of different cells in the same array), but not any random pointers. Objects could be in different segments of a segmented memory architecture, or even in totally different memory banks such as in a Harvard architecture.

But with memmove why would you call it when copying one data object into a totally different one? There would be no danger that they overlap, unless you are copying too many bytes, which is already its own big problem. Thus we can assume that the pointers to memmove are in the same data object and thus comparable. What this also tells me is not to indiscriminately use memmove all the time in order to be “safe” or something, because some implementations like Plauger’s would then cause undefined behavior.

One other thing to note about the memmove implementation is that it uses unsigned char pointers to do the work rather than void pointers. Doing pointer arithmetic on void* is a GNU-ism not permitted in portable C.

(string.h) strncpy

No discussion of string.h is complete without talking about the dangers of that “hybrid” function strncpy(). I call it a hybrid because in some ways it acts like a mem- function operating on blocks of memory. Let’s look at the history.

The function was originally used to copy filenames into fixed width fields. Two fields were considered equal when each and every character of one matched the other. The comparison didn’t stop at \0 but blindly compared the full length of both memory regions. To represent a short file name, the remainder of the memory was padded with \0 just to normalize it. A long filename that used every available character wouldn’t have a \0 at all because the length was already known.

So while strncpy might look like a “safe” strcpy that is guaranteed not to overflow a destination buffer, it’s actually pretty weird and not safe. Consider this snippet:

char dst[BUFSZ], *src = /* ... */;

strncpy(dst, src, BUFSZ);

It copies the first BUFSZ characters of src into dst. If BUFSZ is shorter than src (meaning \0 is not one of the first BUFSZ characters) then \0 doesn’t get written to dst! The result is not NUL terminated. Also, strncpy does a bunch of potentially useless work adding extra NUL padding if BUFSZ is larger than strlen(src).

The proper way to handle strncpy depends on how you want to treat string truncation. In some cases it would be an error to truncate the source string, such as if it holds a URL or filename. In this case the code should check for truncation.

if (strlen(src) < BUFSZ)
  strcpy(dst, src);  /* a regular strcpy is safe */
else
  /* handle truncation error */

If truncation is permissible, then be sure to include a NUL character in the destination.

/* either of these techniques work in C89: */

/* adding the NUL character manually */
strncpy(dst, src, BUFSZ);
dst[BUFSZ-1] = '\0';

/* or truncating dst and using (safe) strncat */
dst[0] = '\0';
strncat(dst, src, BUFSZ);

/* C99 provides snprintf that can do this in one line */

snprintf(dst, BUFSZ, "%s", src);

The C11 standard in Annex K contains functions with “bounds-checking” interfaces. This includes strncpy_s which acts more safely. The functions in Annex K are safe for other reasons as well, but covering this goes too far afield from the C89 focus of this article. One thing to note is that it provides a memset_s which is guaranteed not to be removed by an optimizing compiler. Sometimes memset is optimized out if it operates on memory that the compiler can tell is not read afterwards.

Those are the notes I wanted to share for this header. Also C99 introduces wide character versions of these functions in wchar.h.

time.h

First, the C standard is very lenient about this header. It has functions to do all kinds of conversions, but the bigger picture is that implementations are allowed to make their “best approximation” to the date and time. Some of them might do a bad job and yet still conform to the standard. Many C environments do not support the concepts of daylight savings or time zones. Both notions are defined geographically and politically, and thus may require more knowledge about the real world than an implementation can support.

There are a lot of details in this header that I don’t want to regurgitate, but it is useful to see how the functions convert between the data types. I made a graph to make it clearer.

functions in time.h

functions in time.h

One thing I didn’t know was that the header provides a clock() function to measure the CPU time elapsed since program start. Also that it provides a resolution of CLOCKS_PER_SEC which is often higher than one. The rest of the library is limited to nothing smaller than seconds of precision.

January 18, 2019

Derek Jones (derek-jones)

Choosing between two reasonably fitting probability distributions January 18, 2019 02:40 AM

I sometimes go fishing for a probability distribution to fit some software engineering data I have. Why would I want to spend time fishing for a probability distribution?

Data comes from events that are driven by one or more processes. Researchers have studied the underlying patterns present in many processes and in some cases have been able to calculate which probability distribution matches the pattern of data that it generates. This approach starts with the characteristics of the processes and derives a probability distribution. Often I don’t really know anything about the characteristics of the processes that generated the data I am looking at (but I can often make what I like to think are intelligent guesses). If I can match the data with a probability distribution, I can use what is known about processes that generate this distribution to get some ideas about the kinds of processes that could have generated my data.

Around nine-months ago, I learned about the Conway–Maxwell–Poisson distribution (or COM-Poisson). This looked as-if it might find some use in fitting software engineering data, and I added it to my list of distributions to keep in mind. I saw that the R package COMPoissonReg supports the fitting of COM-Poisson distributions.

This week I came across one of the papers, about COM-Poisson, that I was reading nine-months ago, and decided to give it a go with some count-data I had.

The Poisson distribution involves count-data, i.e., non-negative integers. Lots of count-data samples are well described by a Poisson distribution, and it is one of the basic distributions supported by statistical packages. Processes described by a Poisson distribution are memory-less, in that the probability of an event occurring are independent of when previous events occurred. When there is a connection between events, the Poisson distribution is not such a good fit (depending on the strength of the connection between events).

While a process that generates count-data may not meet the requirements needed to be exactly described by a Poisson distribution, the behavior may be close enough to give good-enough results. R supports a quasipoisson distribution to help handle the ‘near-misses’.

Sometimes count-data has a distribution that looks nothing like a Poisson. The Negative-binomial distribution is the obvious next choice to try (this can be viewed as a combination of different Poisson distributions; another combination is the Poisson inverse gaussian distribution).

The plot below (from a paper analyzing usage of record data structures in Racket; Tobias Pape kindly sent me the data) shows the number of Racket structure types that contain a given number of fields (red pluses), along with lines showing fitted Negative binomial and COM-Poisson distributions (code+data):

Number of Racket structure types containing a given number of fields.

I’m interested in understanding the processes that are generating the data, and having two distributions do such a reasonable job of fitting the data has given me more possible distinct explanations for what is going on than I wanted (if I were interested in prediction, then either distribution looks like it would do a good-enough job).

What are the characteristics of the processes that generate data having each of the distributions?

  • A Negative binomial can be viewed as a combination of Poisson distributions (the combination having a Gamma distribution). We could create a story around multiple processes being responsible for the pattern seen, with each of these processes having the impact of a Poisson distribution. Sounds plausible.
  • A COM-Poisson distribution can be viewed as a Poisson distribution which is length dependent. We could create a story around the probability of a field being added to a structure type being dependent on the number of existing fields it contains. Sounds plausible (it’s a slightly different idea from preferential attachment).

When fitting a distribution to data, I usually go with the ‘brand-name’ distributions (i.e., the one with most name recognition, provided it matches well enough; brand names are an easier sell then the less well known names).

The Negative binomial distribution is the brand-name here. I had not heard of the COM-Poisson distribution until nine-months ago.

Perhaps the authors of the Racket analysis paper will come up with a theory that prefers one of these distributions, or even suggests another one.

Readers of my evidence-based software engineering book need to be aware of my brand-name preference in some of the data fitting that occurs.

January 17, 2019

Jan van den Berg (j11g)

Kierkegaard – Patrick Gardiner January 17, 2019 07:39 PM

Kierkegaard – Patrick Gardiner (1988) – 154 pages

In general, philosophy has a reputation of being hard to understand, and Kierkegaard certainly does nothing to lessen this reputation. He was largely misunderstood in his lifetime, but also very aware that he was ahead of his time. Maybe that’s why he was an enormously prolific writer. His works are often complex and paradoxical in tone and plentiful enough to ponder on its meaning for many lifetimes.

This book has the almost ungrateful job of trying to give an introduction to this rebel amongst philosophers. No easy task. And the writer does not help himself by choosing overly complex phrasings for already complex matters. So this is not an easy read. And, being an introduction, I expected a different writing style.

But I did enjoy the biographical background information about Kierkegaards’ life in the first chapters. After that, Gardiner picks out 4 seminal works to explore main Kierkegaard themes. He does so by offsetting these to other philosophers, especially Kant en Hegel. But without necessary background this can quickly get complicated. (I had to keep notes.) Nevertheless, I especially enjoyed the Abraham paradox as a thinking exercise, and this book also provides nice starting points on Kierkegaards’ thoughts on subjectivity, immorality, individualism and the leap of faith concept.

The post Kierkegaard – Patrick Gardiner appeared first on Jan van den Berg.

January 16, 2019

Jeff Carpenter (jeffcarp)

Build a Markov Chain Sentence Generator in 20 lines of Python January 16, 2019 06:44 PM

A bot who can write a long letter with ease, cannot write ill. —Jane Austen, Pride and Prejudice This post walks you through how to write a Markov Chain from scratch with Python in order to generate completely new sentences that resemble English. The text we’ll be using to build the Markov Chain is Pride and Prejudice by Jane Austen. You can follow along here or grab a runnable notebook version of this post on Colab.

January 14, 2019

Ponylang (SeanTAllen)

Last Week in Pony - January 14, 2019 January 14, 2019 08:29 PM

Last Week In Pony is a weekly blog post to catch you up on the latest news for the Pony programming language. To learn more about Pony check out our website, our Twitter account @ponylang, our users’ mailing list or join us on IRC.

Got something you think should be featured? There’s a GitHub issue for that! Add a comment to the open “Last Week in Pony” issue.

Jan van den Berg (j11g)

The Sun Also Rises – Ernest Hemingway January 14, 2019 06:47 AM

On my honeymoon in 2009 I asked the store clerk at the Hemingway house on Key West: “what’s a good book to start with?”. She recommended The Sun also Rises.

The Sun also Rises – Ernest Hemingway (1926) – 222 pages

And I understand why. Because this novel is probably the most accessible summary of the typical Hemingway writing style. Very much set in the 1920s — yet timeless — it is a story about classic themes such as friendship, love, morality and searching for meaning, told through sparse dialogue and with minimal background. And where the main characters drink, a lot. A whole lot.

At the surface this book is, at most, an enjoyable adventure about a group of restless friends, who do a bit of travelling and go to see the bullfights. But when you dig a bit deeper: a lot more is going on. Which underscores the iceberg theory for which Hemingway is famous. There are many sources where you can read about what Hemingway supposedly meant. And while these can be a welcome addition to reading, I am sceptical about too much analysis. Sometimes maybe it just is what it is. Either way, one of the features of great art is that there is no definitive meaning and it is what it is to you.

I read a book about flawed and memorable characters fleeing and navigating through the complexities of life and love. And they drank. A lot.

The post The Sun Also Rises – Ernest Hemingway appeared first on Jan van den Berg.

January 13, 2019

Wesley Moore (wezm)

Goals and Directions for Rust in 2019 January 13, 2019 09:00 PM

This is my response to the call for 2019 roadmap blog posts proposing goals and directions for 2019 and future editions. See also Read Rust, where I've collected all the #Rust2019 posts.

2018 was a very busy year for the Rust project. A new edition was released, progress on stabilising Rust's asynchronous I/O story was made, a new website was launched, and so much more! In 2019 I'd like to see the language and wider crates community become more sustainable, address some common gothca's that newcomers experience, and promote more platforms/architectures to tier 1 status.

2018 Retrospective

Before diving into 2019 goals I think it's worth tracking how the project went on my ideas from last year:

  1. Become a better option for building network daemons and HTTP micro-services.
  2. Continue to improve the discoverability and approachability of crates and Rust's web presence in general.
  3. Gain wider, more diverse tier-1 platform support (especially on servers).
  4. Start delivering on the prospect of safer system components, with fewer security holes.

Network Services

A lot of progress was made on Futures async/await in 2018. The keywords were reserved in the 2018 edition but are not yet usable on a stable release. Are we async yet? shows there's still some work to do. Hyper saw more major changes to keep up with the Futures work and added HTTP/2 support!

Improve Rust's Web Presence

The Rust web presence was improved with the release of the new website and blog. The rest of the ecosystem remains much the same. Content from crates.io is still largely invisible to non-Google search engines such as DuckDuckGo (my primary web search tool), and Bing. The Rust Cookbook remains in the nursery.

Platform Support

Tier 1 platform support remains unchanged from last year. There were a number Tier 2 additions/promotions.

System Components and Increased Safety

The oxidisation of librsvg continued in 2018 to the point where almost all the public API is done in terms of Rust. I'm not aware of many other projects following this path at the moment:


Rust 2019

In 2019 I'd like to see the Rust community focus on three areas:

  1. Sustainable development
  2. Make is easier for newcomers to write fast code / don't surprise people
  3. More portability

Sustainable Development

Recently Murphy Randle and Jared Forsyth were discussing the event-stream compromise on the Reason Town podcast. Jared commented:

The problems of having infrastructure that’s based on unpaid labour that has a high degree of burnout.

Reason Town podcast episiode 13 @ 19:29

This is a pretty succinct summary of the problem with our industry. Rust hasn't shied away from tackling hard problems before and taking on the sustainability of open source doesn't feel out of the question. There's evidence that many of the folks deeply involved with the Rust project are already feeling the pressure and we don't want to lose them to burnout. Such as these posts:

Part of this revolves around culture. The Rust community generally values quality, correctness, performance, and treating each other with respect. I think it would be possible to make it normal to contribute financially, or other means (equipment, education) to Rust language and crate developers (where people are in a position to do so). A simple first step might be allowing for a donate badge, akin to CI badges to be added to crate meta data and have this shown on the Crate page.

Michael Gattozzi covered some similar thoughts in his, Rust in 2019: The next year and edition, post.

Naïve Code Is Fast Code

People hear that Rust is fast and lean, they try it out converting something from a language they already know and are surprised to find that it's slower and/or a much larger binary.

There are many many many many examples of this in the Rust Reddit. Things that frequently seem to trip newcomers up are:

  1. Not compiling with --release
  2. Stdio locking
  3. Binary size

It would be good to apply the principle of least surprise here. I think the current defaults are inspired by the behaviours expected from C/C++ developers, Rust's original target audience. However the Rust audience is now much wider than that. With that in mind it might worth reevaluating some of these things in terms of the current audience. These need not require API changes, perhaps they could be clippy lints. Perhaps they could be slight changes to language. For example, cargo currently says:

Finished dev [unoptimized + debuginfo] target(s) in 0.11s

Perhaps a second line could be added that says:

Compile with --release for an optimized build

to help guide people.

More Tier 1 Platforms

This one is inherited from last year. I do all my server hosting with FreeBSD and up until recently used it as my desktop OS as well. It's not uncommon to uncover portability bugs or assumptions when using such a platform. Portability is type of diversity of software. It makes it stronger and useful to more people.

Rust is already appealing to people because of its portability. I was recently talking to a veteran developer at the Melbourne Rust meetup's hack night about what got them into Rust. It was the combination of modern language features, native binary and portability that drew them to Rust.

To this end I'd like to see more platforms and CPU architectures promoted to tier 1 status. Up until recently one thing that made this difficult was the lack of hosted CI services with support for anything other than Linux, macOS, and Windows. Recently two options have become available that make it possible to test on other systems, such as FreeBSD. There is Cirrus CI which includes a FreeBSD image in their hosted option, as well as the ability to create custom images. Secondly there is sr.ht, a completely open source (but hosted) option that supports a variety of Linux distributions, and FreeBSD, with more planned.

Pietro Albini suggested in his post that the Rust infrastructure team is already planning to start the discussion on CI options. I think this would be a perfect opportunity to integrate more platforms into the CI infrastructure:

One of the biggest one is switching away from Travis CI for the compiler repository. In the past year we had countless issues with them (both small and big), and that's not acceptable when we're paying (a lot) for it. The infra team is already planning to start the discussion on where we should migrate in the coming weeks, and we'll involve the whole community in the discussion when it happens.

Conclusion

After an intense 2018 it sounds like the Rust project needs to focus on making the project sustainable over time. I'd love to see some improvements to address issues newcomers often experience and push more platforms to tier 1 status. Rust is still very exciting to me. I can't wait to see what 2019 brings!

For more great #Rust2019 posts check out readrust.net.

January 11, 2019

Geoff Wozniak (GeoffWozniak)

Timestamps, GNU autotools, and repositories January 11, 2019 04:07 AM

If you include the distributed source for software that uses the GNU autotools in a source repository, beware of timestamps on the files when you checkout the source.

Consider, for example, the source distribution of the GNU MPFR library. The source distribution includes the files aclocal.m4 and Makefile.in. In the autotools world, Makefile.in is dependent on aclocal.m4. When you run configure, the Makefile is derived from Makefile.in.

This all fine and not surprising. What will be surprising is that when you run make, you may immediately get an error like this.

> gmake
/home/woz/tmp/testing/mpfr/missing: automake-1.15: not found
WARNING: 'automake-1.15' is missing on your system.
         You should only need it if you modified 'Makefile.am' or
         'configure.ac' or m4 files included by 'configure.ac'.
         The 'automake' program is part of the GNU Automake package:
         <http://www.gnu.org/software/automake>
         It also requires GNU Autoconf, GNU m4 and Perl in order to run:
         <http://www.gnu.org/software/autoconf>
         <http://www.gnu.org/software/m4/>
         <http://www.perl.org/>
gmake: *** [Makefile:404: ../mpfr/Makefile.in] Error 1

The distribution may create a Makefile whose first course of action is to check that the build files are up-to-date. This means checking files such as aclocal.m4 and Makefile.in. If the timestamps on those files are such that Makefile.in is ever-so-slightly older than aclocal.m4, for example, the build will attempt to regenerate Makefile.in, which requires automake to be installed. It may even complain about having a specific version, such as in the example above. Perhaps more insidiously, if you do have the correct version installed, it will regenerate the file in the source directory. You may be really confused to see a modified Makefile.in (or .info files, or perhaps machine descriptions) after doing a clean checkout and build. (It’s also bad because, technically, the build is based on different source than what is in the repository.)

I’ve run into this where the distributed source for libraries such as MPFR are included in large source repositories. When cloning them, you never know what order the files will be written and when. If you try to do this with a small repository, there’s a very good chance the timestamps of the files will be the same.

This likely won’t happen when the source is unpacked from the distribution because TAR files preserve the timestamps.

If there’s an option to configure to prevent this, such as --disable-maintainer-mode, you may want to use that. (Why would “maintainer mode” be the default in the first place?) You could also touch the troublesome files in the correct order, although that will probably bring about its own maintenance headaches.

January 10, 2019

Alex Wilson (mrwilson)

Notes from the Week #14 January 11, 2019 12:00 AM

Getting back into the habit of blogging after stopping for almost a month is a bit of a wrench, but it means there’s more juicy stuff for me to talk about!

new year, new me

I’ve been digging really hard into reliability and SLXs this year — we’re already well down the road to better understanding the existing reliability for our Graphite metrics-collection system, and are hoping to take these learnings into building SLXs and Error Budgets for our other core systems.

We’ve already gained a lot of insight into how our customers use the metrics collection API to build dashboards:

  • Setting harder timeouts at the load balancer level has exposed just how many dashboards have panels with too many metrics (as Grafana bunches them into a single request)
  • Measuring our error rates highlighted the number of dashboards relying on metrics that are no longer being collected and cause 5xx responses at the backend.

I want Shift to set a good example for ProDev as a whole — as we scale out a service-oriented approach to architecture being able to have concrete discussions for reliability is going to become paramount.

Keeping records of our decisions

Shift built out a new Puppet module under our open-source repository. The nrpe_custom_check module wraps several different configuration files to provide a clean interface to build NRPE checks for production machines.

We designed it to have as few configurable parts as possible (and indeed only has 3 inputs — name, content, and whether the script needs sudo privileges) but hit a major snag on an architectural point.

There’s an existing module base which has some default NRPE plugins. Given base is designed to be completely standalone and “batteries included”, should it depend on and use the nrpe_custom_check module rather than using static files?

There was a lot of back-and-forth debate about the merits composing small well-defined modules together versus jumping into an abstraction too quickly rather than letting the design evolve “naturally”.

In the end, we decided that the points made were too valuable to lose in the mists of time and resolved to adopt Architecture Decision Records. The implementation we chose was the Nygard format supported by Nat Pryce’s excellent adr-tools toolchain.

You can check out our record here: 2. Standalone NRPE Custom Check module

Historically, we’ve been not as good as we could be at recording not just the decision but the context in which it was made. A key part of Norm Kerth’s Prime Directive is

Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.

Code, documentation, and domain models all succumb to rot over time and it’s important to remember that choices made are probably correct given the information available at the time.

I’m hopeful that we continue to adopt this, especially for shared projects like our monolithic codebases, so that knowledge about why we do certain things is not purely contained in the anecdotes of people who were there at the time.

What does value actually mean?

New year, and I’m having a bit of a new think on this topic. For a team like Shift whose role is primarily that of support:

  1. What does ‘value’ mean?
  2. How do we measure it effectively?

A (paraphrased) aphorism from our CTO has been rattling around in my head for a while:

At any given point in time, there is a good argument to not do X (usually in favour of feature development). Not doing X causes more problems, and will take longer to heal, the longer we leave it.

Where X can be anything from dependency upgrades, platform investments, even refactoring.

Shift are fundamentally an enabling team — the systems we build support the other teams in ProDev and we are also a source of extra hands for doing work related to our area of expertise.

We frequently pair with other teams to share the knowledge and help guide them around pitfalls that we have experienced, all while improving the state of our own infrastructure by learning about the needs of our teammates.

We don’t often get the quick highs of rapid feature delivery like a focused product development team but we are steadily building out our own tooling and systems to aid ProDev.

2019 is a year full of potential and Shift are going to seize as much of it as possible.

Originally published at blog.probablyfine.co.uk on January 11, 2019.

January 10, 2019

Derek Jones (derek-jones)

Wanted: 99 effort estimation datasets January 10, 2019 01:24 AM

Every now and again, I stumble upon a really interesting dataset. Previously, when this happened I wrote an extensive blog post; but the SiP dataset was just too big and too detailed, it called out for a more expansive treatment.

How big is the SiP effort estimation dataset? It contains 10,100 unique task estimates, from ten years of commercial development using Agile. That’s around two orders of magnitude larger than other, current, public effort datasets.

How detailed is the SiP effort estimation dataset? It contains the (anonymized) identity of the 22 developers making the estimates, for one of 20 project codes, dates, plus various associated items. Other effort estimation datasets usually just contain values for estimated effort and actual effort.

Data analysis is a conversation between the person doing the analysis and the person(s) with knowledge of the application domain from which the data came. The aim is to discover information that is of practical use to the people working in the application domain.

I suggested to Stephen Cullum (the person I got talking to at a workshop, a director of Software in Partnership Ltd, and supplier of data) that we write a paper having the form of a conversation about the data; he bravely agreed.

The result is now available: A conversation around the analysis of the SiP effort estimation dataset.

What next?

I’m looking forward to seeing what other people do with the SiP dataset. There are surely other patterns waiting to be discovered, and what about building a simulation model based on the charcteristics of this data?

Turning software engineering into an evidence-based disciple requires a lot more data; I plan to go looking for more large datasets.

Software engineering researchers are a remarkable unambitious bunch of people. The SiP dataset should be viewed as the first of 100 such datasets. With 100 datasets we can start to draw general, believable conclusions about the processes involved in software effort estimation.

Readers, today is the day you start asking managers to make any software engineering data they have publicly available. Yes, it can be anonymized (I am willing to do that for people who are looking to release data). Yes, ‘old’ data is useful (data from the 1980s could have an interesting story to tell; SiP runs from 2004-2014). Yes, I will analyze any interesting data that is made public for free.

Ask, and you shall receive.

January 09, 2019

Jan van den Berg (j11g)

Joy Division and the making of Unknown Pleasures – Jake Kennedy January 09, 2019 09:36 PM

I picked up this book in the bargain bin of a HMV in Manchester in 2006, when I was on a — sort of — pilgrimage. But I left it on my bookshelf for 12 years, thinking I probably knew most of it already. But books are meant to be read, so I had to get to it eventually.

Joy Division and the making of Unknown Pleasures – Jake Kennedy (2006) – 214 pages

I think I understand why this book was already in the bargain bin the same year it was released. Not that it is a terrible book, but it is just a really hard sell. In depth reviews of every Warsaw/Joy Division song, demo or recording session are something only hard hardcore fans have a thirst for. But if you do, you will find a chronological and remarkable tale of four guys who went from sloppy generic punk music to producing a timeless, unique and legendary album within two short years. With each page turn, they improve and get closer to reaching that point. Which was a natural conclusion of everything prior, and didn’t drop out of the sky. But even so, the progression and quality of a debut album is still unmatched (maybe only by that other band).

The book falls somewhere between a biography and an encyclopedia. In other words, just my cup of tea! I certainly picked up a few new things. Mick Middles gets quoted often, he is another authority on Joy Division because of his book. I also own this book, and I definitely need to read it now!

The post Joy Division and the making of Unknown Pleasures – Jake Kennedy appeared first on Jan van den Berg.

Indrek Lasn (indreklasn)

Thanks. January 09, 2019 11:49 AM

Thanks.

Gergely Nagy (algernon)

One hat less January 09, 2019 11:45 AM

Almost a week ago, on a snowy night of January 3, I hung up my Debian Developer hat, and sent my retirement letter in. This has been a long time coming, and I contemplated doing this many times over the past year or two, but never got the courage and the willpower to retire. It wasn't easy. Debian has been part of my life for the past two decades, and I've been a developer for about eighteen years. I considered - and to some extent, still do - Debian my second family. There have been times when Debian was my escape hatch, something I could focus on to help me pass the days by. I've made friends, I've met people, I learned humility, I learned empathy. If it weren't for Debian, I wouldn't be the person I am today. Retiring felt like admitting defeat for a long time. But it isn't.

It's not defeat when you admit that your circumstances changed. It's not defeat when you let others take better care of the stuff you've been neglecting for way too long. It's not defeat when you're taking a break. It's not defeat when you're reducing the amount of stress that keeps you awake at night. It's not defeat when you do the right thing. It's not defeat when you gracefully retire: rather, it is a start of something new.

I originally joined Debian for two reasons: I was a Debian user for a while by then, and felt that I should be giving back, as I received so much from Debian. The second reason wasn't this noble, but after eighteen years, I guess I could admit it: the other reason I applied was vanity. Having a @debian.org email-address was something.

At the time, I was... a different person. I was full of myself at times, I was angry, perhaps a tad too elitist at times too. I'm not proud of old me, but it's part of me. I grew, and became a better person, there's no shame in being able to grow - quite the contrary. And Debian helped immensely. I've had role models in the project, who I look up to even to this day, who helped shape me one way or the other.

There are two people I need to mention specifically: Martin Michlmayr and Rhonda D'Vine.

Martin was my Application Manager when I applied, I considered him a mentor, a friend. The example he set were instrumental in shaping me too. Rhonda helped me get to my first ever conference: I got on a train to Vienna, and she took me to LinuxTag in her car from there, and then back again. That first LinuxTag, the path that led there, the conference itself, was formative, and both Martin and Rhonda had a big part in it being so. Thank you again - so many years later, I still smile when I think back. Those years we were in touch, meant a lot to me.

I often feel nostalgic about the times way back then, when I was active on IRC. I miss the old bunch (oh, so many people I haven't talked to in years!). I miss being active in Debian. I miss talking to fellow developers. But life took me elsewhere, and there's only so many hours in a day. Over the years Debian and me, we grew apart. There was always something more important, so my Debian contributions dropped (though, I had some ups too, while I was ftp-master assistant for example). By 2018, I barely did anything, and one of my resolutions for the new year was that I'll be taking better care of my health, mental and physical alike. Part of this is reducing my workload, reducing the stress levels, and things I feel guilty about.

It's a strange feeling. A mix of sadness and relief. On the flip side, as Lars put it, I'm now part of the retired DD club, and that's one illustrious club to be part of. First I joined the Debian Parents Club, now the Retired Debian Developers Club. This is a good start of a new chapter.

At some point though, I will be back. Debian is part of my life, and always will be. One way or the other, there will be a way back. It may take years, or a decade, but I'll put on that swirly hat again.

January 07, 2019

Indrek Lasn (indreklasn)

Thanks for the kind words. January 07, 2019 09:35 PM

Thanks for the kind words. I don’t have any monetization plans yet but I’m thankful for every reader I have.

Jan van den Berg (j11g)

Faith – Jimmy Carter January 07, 2019 06:22 PM

I literally received this book from the hands of Jimmy Carter himself at a book signing in New York. I never met a president before — even if only for a few seconds — so I was keen to read his book!

Carter always struck me as an interesting person. A multifaceted outlier: deeply religious but pro science and evolution, anti-NRA but pro guns, peanut farmer by trade and a nuclear submarine operator by education. But apart from all that, any 93 year old who has been married for over 70 years, probably has some interesting experiences to share.

Faith – Jimmy Carter (2018) – 174 pages

But this is not a biography. This is a personal creed about his definition of faith in Jesus Christ, interwoven with experiences about engaging with world leaders or dealing with personal or global crises of the past. In the last chapter, Carter, without naming names, demonstrates awareness by exactly pinpointing current problems. And you learn that his general approach to past and current problems are often the same, which I think come down to one powerful word: inclusion.

At first I was confused who this book was for. I expected it to be for either Christians or deliberately for non-religious people. But none of that. Carter just writes from his own personal experiences and deeply held beliefs, for anyone who wants to listen and learn something. And he does so with authenticity, honesty and integrity.

Bonus!

I was also lucky enough to be at the taping of this interview on the same day as the book signing. He was a surprise extra guest. The interview offers a great recap of what to expect of the book.

The post Faith – Jimmy Carter appeared first on Jan van den Berg.

Mark J. Nelson (mjn)

New city, new job January 07, 2019 12:00 PM

A few days ago I moved to Washington, DC, to start a new job as Assistant Professor in the Computer Science department of American University. The CS department here is small but growing, and I'm looking forward to being a part of that. Thanks especially to profs. Mike Treanor and Nathalie Japkowicz for their part in hiring me and helping me to get oriented here.

For the past 2½ years I was Senior Research Fellow at the MetaMakers Institute, a grant-funded European Union research initiative attached to the Games Academy of Falmouth University, which is located in the remote but pretty region of Cornwall, in the far southwestern United Kingdom. Moving from a 20,000-person seaside village to the capital of the United States is a big change! A change for both good and bad naturally, but I'm optimistic overall.

In addition to my home in Computer Science, I'll be affiliated with the AU Game Lab. I'll teach a mixture of game-oriented courses and more standard computer science courses as a result. For Spring 2019 I'm teaching one of each: a special-topics course on AI & Games, and a core CS course, Programming Languages. I'm designing both with mostly new syllabi, which means things are pretty busy at the moment, but it promises to be interesting. I'll try to share some more on those courses once everything is worked out (classes start next week). It's nice that American University has small classes (~20 students), which allows for a little more experimentation with these things – I've got some more ideas for the coming years if things go well.

January 06, 2019

Indrek Lasn (indreklasn)

How I got average 200k monthly views on Medium January 06, 2019 10:34 PM

This came as a total surprise.

My Medium stats (As of 2019, January)

I never expected so many smart people to read my articles. At the time when I started blogging — I was happy enough to let [at that time] my girlfriend read my articles. I never sent her the link, but she managed to read them anyway. I thought it was cute.

Times have moved on, and the audience has grown. With great audience comes great responsibility and scrutiny. I’ve been writing consistently for about 2 years and this is what I’ve learned.

Photo by Nick de Partee on Unsplash

So here’s my giving back to you. My mission has always been to share as much knowledge as possible. I believe in no tricks and shortcuts; only in hard work and becoming the best as you can.

So without further ado, here are some of my personal habits to keep writing articles people enjoy reading;

Assume your audience is smarter than you

It’s true. I’m not the smartest person in the room. But before you close this tab, let me explain why I’m not the smartest person in the room.

I’m aware of what I know, and I’m aware of what I don’t know; I’ve met some extraordinarily smart people in my life, I’m thankful for this. Unfortunately, this is a two-sided blade. The smarter people you hang out with, the less dumb you feel. I’ve been hanging around with some of the smartest guys.

Put yourself in a situation where you can always nourish and flourish. Being in the teacher shoes should never discourage you from learning new stuff. It’s expected that at some point, the master becomes the student, and the student becomes the master.

Humility is key, keep your head low and your hunger for new information high.

Me taking a cheerful selfi

Persistence, Persistence, Persistence!

Nothing great will happen if you won’t stick to it. There are people who train martial arts from pre-teens to late senior years. That is true dedication to the art.

Photo by Leslie Jones on Unsplash

How many have you started doing something and realized you were terrible at it and quit right after?

Everyone starts at the same point and has the same feelings and questions.

“Is this for me?”

Sure, you might find it hard, that’s perfectly sane and reasonable. Quitting is not. Never give up on your dreams and goals. If your dream is to have a big audience reading your articles — so be it. But you got to but the effort, sweat and, work in.

I have a habit of minimum one article per month and if I can — one article per week. This habit keeps the consistency in check on my terms.

Love what you do with passion burning in your heart

I love it so much when people come to me saying how useful and how much my article helped them. I find that extremely rewarding and satisfying. Thank you, everyone, for the kind words!

Writing has always been my passion. I guess I inherited it from my father, who is also a passionate novel writer.

Writing definitely helps me clear my mind, put my thoughts on paper and connect with people around the globe.

Photo by Nick Fewings on Unsplash

Pick topics which interest you the most but also have a broader audience

Picking topics are tricky — pick something too niche and not many people will read and pick something too specific and many people have written about it already.

Finding the balance is key, and I believe passion is the most important ingredient.

If you’d like to write about coding, I’d recommend building a couple of apps and putting some experience under the belt. This way you can be sure you feel confident on what you’re writing about and make it fun.

Readers are only interested in interesting and useful content. It’s your job to find out which content and how to make it sound interesting.

Photo by Sophie Elvis on Unsplash

Thanks for reading and I hope you succeed! I hope you found this article useful.

If you have any questions, I’d be more than happy to discuss on Twitter.

Indrek Lasn (@lasnindrek) | Twitter

Here are some of my previous article you might enjoy;

This story is published in The Startup, Medium’s largest entrepreneurship publication followed by +411,714 people.

Subscribe to receive our top stories here.


How I got average 200k monthly views on Medium was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

Patrick Louis (venam)

Adding Glue To a Desktop Environment January 06, 2019 10:00 PM

glue

In this article we will put some light on a lot of tools used in the world of Unix desktop environment customization, particularly regarding wmctrl, wmutils, xev, xtruss, xwininfo, xprop, xdotools, xdo, sxhkd, xbindkeys, speckeysd, xchainkeys, alttab, triggerhappy, gTile, gidmgr, keynav, and more. If those don’t make sense then this article will help. Let’s hope this can open your mind to new possibilities.

NB: A lot of the questions regarding those tools and what they are usually come from people that don’t actually grasp what the purpose of a window manager is or what desktop environments are. If this is the case please read the following before starting:

However, even without reading the previous content, this article will try to explain concepts as simply as possible.

With that in mind we can wonder if what’s actually needed from a window manager, presentation and operation, can be split up and complemented with other tools. We can also start thinking laterally, the communication and interaction between the different components of the environment. We have the freedom to do so because the X protocol is transparent and components usually implement many standards for interfacing between windows. It’s like gluing parts together to create a desktop environment.

The tools we’ll talk about fall into one of those categories:

  • Debugging
  • Window manipulation
  • Simulation of interaction
  • Extended manipulation
  • Hotkey daemon
  • Layout manager

overview

Window Manipulation

Windows are objects living in the X server that have a visual representation when mapped on the screen (viewable and visible). Windows, like any other objects, have attributes and properties attached to them. Another interesting object within the X Server is the pointer aka cursor.

The tools in this section manipulate those bare objects using the X11 protocol to speak to the X server. They do things such as list the identifiers of the window objects within the X server and their basic properties, they let you do basic tasks on them such as moving them, resizing them, changing their stack order (which one is above the other), labeling them, hiding/unmapping them (or the opposite by mapping them), changing the focus from one to the other, and as an extra some also allow to manipulate the pointer position and cursor shape.

Two popular tools that offer those functionality are wmutils core and the window utility from jessies x11-extras softwares. Along with those come some other utilities used for window listing such as winselect also from jessies, xlsw from baskerville, and the one included within wmutils core lsw. We could also mention single usage tools such as xkill.

The philosophy behind wmutils is more Unix-like, portrayed as the “Coreutils for Xorg” it’s less monolithic than a lot of similar tools we’ll see which renders it useful on the pipeline. Moreover it’s written using the XCB which is more of a direct correspondence with the X protocol.

We can manipulate the windows basic object but how do we actually verify that the change has been done properly. This can be done using debugging utilities such as xwininfo or xwinfo.

We should also mention a particular one that is p9-like, the x11fs that does the manipulation though a virtual filesystem API.

Simulation of interaction

So far so good, we’re able to do some basic actions on windows but what about simulating inputs such as keyboard or mouse, in the same way as winmacro on Windows platform or headless browser for the javascript world.

This is possible if the X Server implementation comes with the XTEST extension which as you would’ve guessed was principally added to test the server without having user interaction.

Most popular implementations builds (X.org Server) nowadays come with this extension however you can make sure that it’s there by querying the list:

xdpyinfo -display :0 -queryExtensions | awk '/opcode/'

Which should return a line similar to this:

...
    XFree86-VidModeExtension  (opcode: 153, base error: 172)
    XINERAMA  (opcode: 141)
    XInputExtension  (opcode: 131, base event: 66, base error: 129)
    XKEYBOARD  (opcode: 135, base event: 85, base error: 137)
    XTEST  (opcode: 132)
    XVideo  (opcode: 151, base event: 93, base error: 155)

Thus a set of utilities such as xdotool and xdo use this extension to simulate inputs. Most of them also include the behavior of the previous tools, especially the listing of window identifiers, this is a bit redundant.

Another interesting one is keynav, the “retire the mouse” tool that let you control all mouse functions from the keyboard in an efficient way, splitting the screen into sections.

Extended manipulation

A standard called icccm (Inter-Client Communication Conventions Manual) was devised a long time ago between 1988 and 1989, and another one named EWMH (Extended Window Manager Hints) not long after builds upon it. Both are used for lateral communication between windows. Their main purpose, which you can guess by taking a glance at them here and here , is that they emphasize on giving information about components and creating the interface between the standard components that usually make up a desktop environment such as bars, panels, desktop and all that is related (managed windows for instance), viewports, state of the window, title of window, the current focus, and much more.
A window manager or a client can choose to implement all or parts of those standard (respond to events), for example you can find the list that Openbox supports here.

Naturally, there’s yet again a set of tools letting you manipulate those properties however keep in mind that they only have an effect if the EWM/NetWM atoms (it’s the name of the properties) are respected. wmctrl accesses almost all the EWMH features, xtitle as the name implies accesses only the title, xdotool previously seen can also access some of the features (moving between desktops).

Lastly, the debug command line utility that is useful to display such window properties is xprop.

Hotkey daemon

What if instead of having the window manager catching the keys you press you had another application listening to your keys and performing actions whenever it encounters certain combinations. This is the job of the hotkey daemon. The explanation couldn’t be simpler, assign keys to actions.
Examples of such are xbindkeys, speckeysd again from jessies x11-extras, sxhkd a popular one and yet another tool from baskerville, triggerhappy, xchainkeys which works by creating a chain of keys, devilspie2 which takes action on certain EWMH events, and alttab which is for task switching (keybind mixed with the window manipulation).

But how do you make sure, if there’s an issue, that your hotkey manager responds to the right keys or which keys to set in its configuration. The xev let you debug X input events in general.

Moreover, it’s good to note that you can debug anything seen previously in more depth using xscope, xtruss, or x11vis. However most of them are a hassle to set up, xtruss is the easiest to compile and use in my opinion.

Layout manager

In normal cases it’s the job of the window manager to choose where to place windows on the screen and their layout but there are separate programs that can be used for this. Generally the layout manager program add tiling feature to a window manager that doesn’t have tiling.
This can be achieved using a software that specializes in this such as gTile for GNOME or gridmgr for others. Or it could be achieved by combining a hotkey daemon along with a window manipulation tool, so let’s say sxhkd with wmutils for instance.

This puts into question the role of the window manager if you can have all of its behavior into separate entities.

Summary

  • Debugging the protocol in general
    • Action: xtruss, xscope, x11vis
  • Window manipulation
    • Action: wmutils, window, x11fs, xkill, ..
    • Listing: xlsw, lsw, ..
    • Debug: xwininfo, xwinfo
  • Simulation of interaction
    • Server Extensions: xdpyinfo
    • Action: xdotool, xdo, keynav
  • Extended manipulation
    • Action: wmctrl, xtitle, xdotool
    • Debug: xprop
  • Hotkey daemon
    • Action: xbindkeys, speckeysd, sxhkd, triggerhappy, xchainkeys, alttab, devilspie2
    • Debug: xev
  • Layout manager
    • Action: gTile, gridmgr, any combination of the above ..

Examples

What can be done with all this, let’s push some ideas.

  • Starting your graphical session with a specific layout everytime regardless of the window manager
  • Automate a user action inside your program, in test units for example
  • Beautify your desktop with conky and make it clickable like a panel
  • Automate captcha
  • Dump your window manager entirely and have pieces in separate programs instead

… And much more.

There’s really nothing you can’t do that a window manager can do.

Conclusion

We went over a lot of tools in this post, we covered basic manipulation, to extensions, to simulation of inputs, to listening for your own inputs to take action. The glue thinking should make sense now, the freedom to build and customize your environment the way you want.

Let’s close this post with something that might be confusing for some. In the reddit community called /r/unixporn users tag their posts with the window manager they are using. Some of the entries are tagged as wmutils but after reading this article you’ll realize that in fact wmutils is not a window manager in itself. Those post should really be tagged “No WM” however that would be confusing as the line is thin between the definition of a window manager and the glue that could possibly be created using the tools in this post.
Another window manager that is missing the operation part is bspwm and it’s usually accompanied by sxhkd, but that sxhkd (hotkey daemon) can be used with any other lightweight window manager such as monsterwm, dwm, catwm, or no window manager at all.

There’s no limit to what you can do, enjoy playing with glue.

Indrek Lasn (indreklasn)

How to improve your asynchronous Javascript code with async and await January 06, 2019 09:37 PM

Photo by Max Nelson on Unsplash

If you’ve had the chance to observe modern Javascript code — chance are high you have seen the async and await syntax somewhere.

Async/await is arguably one of the best-received new additions to the language. Async/await makes asynchronous code appear and behave like synchronous code. Async/await are promise-based.

Before we jump into async/await, we must understand what are promises and the role they play.

Note; This article was originally published under strilliant.com

How to improve your asynchronous Javascript code with async and await

Promises

A Promise is an object representing the eventual completion or failure of an asynchronous operation.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Using_promises

Why do we need asynchronous code, anyway?

Javascript is a single threaded language — this means Javascript can only do one thing at once. Imagine calling our API synchronously and blocking the entire thread for the API call duration — our users would have to wait 30 seconds or as long it takes for the network request to resolve — a big no-no!

In case you’re interested to learn more — here’s a more in-depth explanation about asynchronous programming with Javascript.

Asynchronous Programming :: Eloquent JavaScript

The way we used to handle asynchronous Javascript code was via callbacks. Chances are high you’ve come across callbacks.

What are callbacks?

A callback function, also known as a higher-order function, is a function that is passed to another function. Functions are first class citizens in Javascript — this means they can be passed as arguments to functions.

https://medium.com/media/0b6bc451058d66f87260592b710c89ca/href

You might have seen jQuery specific code like this.

The following code attaches an event listener on our button and calls the alert once it’s triggered.

Where is the callback? Can you tell?

You probably can — it’s the anonymous function inside the click function parentheses.

https://javascriptissexy.com/understand-javascript-callback-functions-and-use-them/

Callbacks in a nutshell: we trigger a function, do some stuff, then call another function.

Callbacks are not all bad, they worked — they still do. But, what happens if we have a callback inside a callback inside a callback — you get the point. It gets really messy and unmaintainable really quick.

This problem above is named “callback hell.”

http://callbackhell.com/

One more example to seal the case for callbacks.

We have a melon!

We check if we have a melon inside the array — if we do, we chew the melon, after chewing we throw away the melon. We’re also handling exceptions with the err callback.

Hey — we have a banana instead!

Note: the err callback is the first argument always in the Node world — best practices!

Just to throw you off a little — I made the previous code as readable as I can, here’s how it might look more often:

mini callback hell; technically, having anonymous arrow functions would be the remedy but still is not the perfect solution.

You can imagine, couple more callbacks and we’re on the highway to [callback] hell — pun intended!

Promises to the rescue!

Promises are a clean way to write asynchronous code. The promise has a single argument, which is a callback.

The callback has two arguments, the reject and resolve conditions.

And if we use the arrow function to shorten the code:

Inside the promise block, we can decide when to resolve and when to reject the promise.

Inside the promise we check if we have a melon. If we do, let’s resolve the function with the value we pass inside resolve — we can pass literally any value to the resolve.

Promises are immediately invoked/resolved, thus why we see the console.log without calling the promise.

Promises are called and chained with the then and catch methods. Think of it like this: the promise has resolved with a value — what do we do with it?

This is where the then and catch syntax comes in. They are both callbacks which accept one argument, which is the return value passed inside the promise.

Here’s how we could handle a promise rejection:

Handling a promise rejection with catch()

Right, now our promise is not very dynamic — why?

Well, because we have an if statement that isn’t dynamic. Promises are extremely powerful when wrapped inside a function. We call them higher order functions.

Wrapping our promise inside a function

Do you notice the small change? We wrapped our promise inside a function which takes one argument in this case. This gives our promises huge flexibility. We can pass any condition to our promise and based on that condition, it will get either rejected or resolved.

Here’s the fully working promise which gets resolved.

Resolved promise

And the rejected promise.

Rejected promise

Starts to look familiar? You might have seen Axios API calls like the one below.

Axios + React API call example (promise based)

https://medium.com/media/dccf0bed60056c668286137318baffcc/href

Or the fetch API call;

Fetch + react API call example (promise based)

https://medium.com/media/1b85a617edc829c5356df6342a49be57/href

What do they have in common?

Well, for starters they’re both a promise. They’re fundamentally using promises under the “hood”. Just like we wrapped our promise inside a function, so do both of these examples.

Secondly, they’re both asynchronous code. Promises are naturally asynchronous.

If you find promises confusing — please read this article and come back ❤

https://gist.github.com/domenic/3889970

Here’s how an asynchronous API call looks like:

It’s safe to say promises are much better than callbacks. Promises have their own flaws although — promises can get out of hand quite quickly.

What if there’s a better way, an even cleaner way. Async/wait to the rescue!

https://codesandbox.io/s/p9mr3jzwp0?autoresize=1&expanddevtools=1&hidenavigation=1

We mark the function as async — inside the function we mark the asynchronous code with await — Javascript will resolve the promise and then move on to the next line. In a nutshell, we change the asynchronous code to read like synchronous code while still running asynchronously.

Notice how we don’t call the promise constructor anymore and how there are much less then() and catch() methods.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function

What about error handling?

Our current async/await doesn’t handle exceptions. This can lead to catastrophic bugs which might crash your app.

Try…catch to the rescue!

Wrapping our async function inside try catch block

The try catch block attempts to execute the code and if it runs into any exceptions/problems, it passes the error object to the catch block and executes the code inside the catch block.

try...catch

Here’s how our async/await fetch would look with error handling.

This is just one of many examples, there are many ways to leverage async/wait. If you’re curious — here’s a pretty good example StackOverflow answer exploring the alternatives.

try/catch blocks with async/await

Bonus: Check out this great article on misconceptions about async/await.

A common misconception about async/await in JavaScript

Thanks for reading and I hope you found this useful!

I’d love to connect with you directly on Twitter and pass on more knowledge in the future. ❤

Indrek Lasn (@lasnindrek) | Twitter

More cool articles here;

Strilliant

About the author:

Indrek Lasn - Medium

Here are some of my previous articles:


How to improve your asynchronous Javascript code with async and await was originally published in freeCodeCamp.org on Medium, where people are continuing the conversation by highlighting and responding to this story.

Gustaf Erikson (gerikson)

December January 06, 2019 09:33 AM

January 05, 2019

Gokberk Yaltirakli (gkbrk)

Reverse Engineering the Godot File Format January 05, 2019 04:09 PM

I’ve been messing around with the Godot game engine recently. After writing some examples that load the assets and map data from files, I exported it and noticed that Godot bundled all the resources into a single .pck file. It was packing all the game resources and providing them during runtime as some sort of virtual file system.

Of course; after I was finished for the day with learning gamedev, I was curious about that file. I decided to give myself a small challenge and parse that file using only hexdump and Python. I opened the pack file with my hex editor and I was met with a mixture of binary and ASCII data. Here’s the beginning of the file from hexdump -C game.pck.

00000000  47 44 50 43 01 00 00 00  03 00 00 00 01 00 00 00  |GDPC............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000050  00 00 00 00 1e 00 00 00  40 00 00 00 72 65 73 3a  |........@...res:|
00000060  2f 2f 2e 69 6d 70 6f 72  74 2f 61 6d 67 31 5f 66  |//.import/amg1_f|
00000070  72 31 2e 70 6e 67 2d 32  32 66 30 32 33 32 34 65  |r1.png-22f02324e|
00000080  63 62 39 34 32 34 39 37  62 61 61 31 38 30 37 33  |cb942497baa18073|
00000090  37 36 62 37 31 64 30 2e  73 74 65 78 00 3d 00 00  |76b71d0.stex.=..|
000000a0  00 00 00 00 28 04 00 00  00 00 00 00 de a0 23 30  |....(.........#0|
000000b0  cf 7c 59 5c fb 73 5d f6  a7 f8 12 a7 40 00 00 00  |.|Y\.s].....@...|
000000c0  72 65 73 3a 2f 2f 2e 69  6d 70 6f 72 74 2f 61 6d  |res://.import/am|
000000d0  67 31 5f 6c 66 31 2e 70  6e 67 2d 62 33 35 35 35  |g1_lf1.png-b3555|
000000e0  34 66 64 31 39 36 37 64  65 31 65 62 32 63 64 31  |4fd1967de1eb2cd1|
000000f0  32 33 32 65 32 31 38 33  33 30 32 2e 73 74 65 78  |232e2183302.stex|

Immediately we can see that the file starts with some magic bytes, and before our ASCII filename begins there’s a lot of zeros. They might be keeping that space as a padding or for future extensions, or maybe it can even contain data depending on your settings, but for our immediate goal it doesn’t matter.

What looks interesting here is the two integers right before our path. 1e 00 00 00 and 40 00 00 00, probably lengths or counts since they are before real data. Saying these two are little-endian unsigned integers would be a good assumption, because otherwise they would be huge numbers that have no business being the length of anything.

The first number is 30 and the second one is 64. Now, what’s the name of that file? res://.import/amg1_fr1.png-22f02324ecb942497baa1807376b71d0.stex. Exactly 64 bytes. That means we now know that the paths are prefixed by their length.

If we look at the next path, we can see that a similar pattern of being length prefixed still applies. The first integer we found, 30, is most likely the number of files we have. And a rough eyeballing of the file contents reveals that to be the case.

Let’s get a little Python here and try to read the file with our knowledge so far. We’ll read the first integer and loop through all the files, trying to print their names.

pack = open('game.pck', 'rb')
pack.read(0x54) # Skip the empty padding

file_count = struct.unpack('<I', pack.read(4))[0]

name_len = struct.unpack('<I', pack.read(4))[0]
name = pack.read(name_len).decode('utf-8')

print(f'The first file is {name}')

Running this code produces the following output, success!

The first file is res://.import/amg1_fr1.png-22f02324ecb942497baa1807376b71d0.stex

Now let’s try to loop file_count times and see if our results are good. One thing we should notice is the data following the ASCII text, before the new one begins. If we miss that, we will read the rest of the data wrong and end up with garbage. Let’s go back to our hexdump and count how many bytes we need to skip until the next length. Looks like we have 32 extra bytes. Let’s account for those and print everything.

for i in range(file_count):
  name_len = struct.unpack('<I', pack.read(4))[0]
  name = pack.read(name_len)
  pack.read(32)
  print(name)
...
b'res://MapLoader.gd.remap'
b'res://MapLoader.gdc\x00'
b'res://MapLoader.tscn'
b'res://Player.gd.remap\x00\x00\x00'
b'res://Player.gdc'
...

Much better, the only issue is the trailing null bytes on some file names. This shouldn’t be a huge issue though, these are probably random padding and they don’t even matter if we consider the strings null-terminated. Let’s just get rid of trailing null bytes.

name = pack.read(name_len).rstrip(b'\x00').decode('utf-8')

After this change, we can get a list of all the resource files in a Godot pack file.

Getting the file contents

Sure, getting the list of files contained in the file is useful. But it’s not super helpful if our tool can’t get the file contents as well.

The file contents are stored separately from the file names. The thing we parsed so far is only the file index, like a table of contents. It’s useful to have it that way so when the game needs a resource at the end of the file, the game engine won’t have to scan the whole file to get there.

But how can we get find where the contents are without going through the whole thing? With offsets of course. Every entry we read from the index, along with the file name, contains the offset and the size of the file. It’s not the easiest thing to explain how you discover something after the fact, but it’s a combination of being familiar with other file formats and a bunch of guesswork. Now I’d like to direct your attention to the 32 bytes we skipped earlier.

Since we already have the file names, we can make assumptions like text files being smaller than other resources like images and sprites. This can be made even easier by putting files with known lengths there, but just for the sake of a challenge let’s pretend that we can’t create these files.

After each guess, we can easily verify this by checking with hexdump or plugging the new logic in to our Python script.

The 4 byte integer that follows the file name is our offset. If we go to the beginning of the file and count that many bytes, we should end up where our file contents begin.

offset = struct.unpack('<I', pack.read(4))[0]

This is followed by 4 empty bytes and then another integer which is our size. Those 4 bytes might be used for other purposes, but again they are irrelevant for our goal.

pack.read(4)
size = struct.unpack('<I', pack.read(4))[0]

The offset is where the file contents began, and the size is how many bytes it is. So the range between offset and offset + size is the file contents. And because we ended up reading 12 bytes from the file, we should make our padding 20 instead of 32.

Reading a File

To finish up, let’s read the map file I put in my game. It’s a plaintext file with JSON contents, so it should be easy to see if everything looks complete.

pack = open('game.pck', 'rb')
pack.read(0x54) # Skip the empty padding

file_count = struct.unpack('<I', pack.read(4))[0]

for i in range(file_count):
  name_len = struct.unpack('<I', pack.read(4))[0]
  name = pack.read(name_len).rstrip(b'\x00').decode('utf-8')

  offset = struct.unpack('<I', pack.read(4))[0]
  pack.read(4)
  size = struct.unpack('<I', pack.read(4))[0]
  pack.read(20)

  print(name)

  if name == 'res://maps/map01.tres':
    pack.seek(offset)
    content = pack.read(size).decode('utf-8')
    print(content)
    break
...
res://Player.gd.remap
res://Player.gdc
res://Player.tscn
res://block.gd.remap
res://block.gdc
res://block.tscn
res://default_env.tres
res://icon.png
res://icon.png.import
res://maps/map01.tres
{
    "spawn_point": {"x": 5, "y": 3},
    "blocks": [
        {"x": 5, "y": 5, "msg": "Welcome to Mr Jumpy Man", "jumpLimit": 0},
        {"x": 7, "y": 5, "msg": "We're still waiting on the trademark"},
        {"x": 9, "y": 5, "texture": "platformIndustrial_001.png"},
        {"x": 11, "y": 6, "msg": "If you fall down,\nyou will be teleported to the start point"},
...

Everything looks good! Using this code, we should be able to extract the resources of Godot games.

Finishing words

Now; before the angry comments section gets all disappointed, let me explain. I am fully aware that Godot is open source. Yes, I could’ve looked at the code to know exactly how it works. No, that wouldn’t be as fun. Kthxbye.

Stig Brautaset (stig)

My Musical Goals for 2019 January 05, 2019 04:00 PM

I list my musical goals for 2019, and touch briefly on achievements in 2018.

Geoff Wozniak (GeoffWozniak)

Book review: Retro debugging January 05, 2019 01:00 AM

This is the fifth in a series of reviews of debugging books.

  • The Frozen Keyboard: Living With Bad Software (Boris Beizer, TAB Books, 275 pp., 1988)
  • Secrets of Software Debugging (Truck Smith, TAB Books, 276 pp., 1984)
  • How to Debug Your Personal Computer (Jim Huffman and Robert C. Bruce, Reston Publishing, 157 pp., 1980)
  • Software Debugging for Microcomputers (Robert C. Bruce, Reston Publishing, 351 pp., 1980)
  • Program Style, Design, Efficiency, Debugging, and Testing, 2nd Ed. (Dennie Van Tassel, Prentice-Hall, 323 pp., 1978)
  • Program Debugging (A.R. Brown and W.A. Sampson, American Elsevier Computer, 166 pp., 1973)
  • Debugging Techniques in Large Systems (edited by Randall Rustin, Prentice-Hall, 148 pp., 1971)

Part of the reason I started this series of book reviews was to get a feel for how approaches to debugging have changed over the years. Similarly, I’m interested in seeing what has stayed the same, perhaps parading around in different guises.

To that end, I collected some debugging books that were published before 1990. I used the following methodology, that is totally and completely based on rigorous techniques practised in academic settings.

  1. Search for books published before 1990 with “debug” in the title.
  2. If the combination of title, cover, and author seemed interesting—and the price was not ridiculous—obtain a copy and read it.

There isn’t any theme to the content aside from that, save for Beizer’s book, which was a recommendation.

As it happens, the books fall into two different computing eras: mainframe computing and personal/home computing. These also happen to line up with the publication dates: 1970s for mainframe, 1980s for personal. You won’t glean much about debugging for the current day and age if you read them, but you will find some enlightening contrasts and parallels.

(A note about Beizer’s book: it’s really a general book for the home computer user and not a book about the process of software development or debugging. It should not be considered part of the home computing books. Although it does discuss how software is developed, particularly how it is (or should be) tested, it is more about the state of software and computers at the time. I’ve included it in this review because it is both a good book and helps provide context for the era.)

Aside from the books’ specific content—which is amusingly out of date in almost all respects—the primary difference they demonstrate from today’s world is speed. Every single aspect of development comes across as slower. Making a change in the code and just running it, let alone deploying it, could take on the order of days. Brown and Sampson’s book describes a case study, run in 1971, that contains the specification for a program that today would be, at best, an assignment for a first year student. Competent developers would probably be able to write it in under a day, with unit tests. In the case study it takes two professional programmers, each with 6-7 years experience, nearly two weeks of effort to get a working program.

As ludicrous as that may seem, a big reason for it is the lack of machine time. The case study contains a detailed program development log for one of the programmers. That programmer logged 13.75 days of development time and only 37 minutes of machine time. It takes him days to get to the point where he even starts writing code, which in all likelihood ends up on cards or some other tedious, error-prone input mechanism. All the mainframe era books allude to the fact that CPU time cost money and that “the computer” is often used for many other things. In other words, developers almost never had a computer to work on and initially wrote most of the code at their desk on paper. There were certainly terminals and time sharing operating systems in the 1970s so it is not accurate to assume that all programmers were using punch cards to submit programs in batch to the computer. Nevertheless, there was significantly less computing power and storage space than today and it was much more expensive. Hence, the mainframe era books consistently push the notion that you should aim to “get it right the first time” so as to avoid wasting CPU cycles.

The books for home computing don’t go to this extent for the obvious reason that you had a computer right in front of you, although it would be a stretch to say that development was a speedy affair. Truck Smith’s book contains the programming journals he recorded while writing three different programs on an Apple II. One of them is for a simple diff-like program written in BASIC that ends up being a little under 300 lines, including comments. Here he is reflecting on the effort.

Reading the blow by blow descriptions [about 50 pages worth] gives the impression that it took a long time to write this program and get it running. Actually it took me about a week, working nights (but not every night of the week).

The home computing books are devoid of discussion on software tools for development and debugging, including something as rudimentary as a text editor. They assume the use of the BASIC interpreter and primitive (that is, non-multitasking) operating system that comes with the computer to input and manage a program. Smith also has journals for a Pascal and assembly language program, but there is no hint as to how to go about actually using those languages on the computer. The assumption is that the user has the knowledge and can obtain the necessary software to use a different programming language. Bruce’s books centre entirely on the limited capabilities of BASIC on the Sol 20 microcomputer. The home computing world comes with more immediate computing power and very little in the way of taking advantage of it.

This lack of speed goes hand-in-hand with debugging being talked about as a distinct stage of software development. It is almost exclusively portrayed as an offline activity. This is explicitly stated in the mainframe books. It is also true in the home computing books, although the definition shifts slightly.

When you look at the debugging tools and techniques the books describe there is one item that rules them all: printed output. Nearly everything is geared toward making program listings or diagnostic output more efficient and useful. Like most things in these books, this will strike readers in this day and age as ridiculous. Still, it is entirely reasonable given the conditions of the time. Recall that in the mainframe domain CPU time was precious: running automated tests or using tools to query output (such as something like grep) may have been difficult or impossible. In the world of home computing merely inspecting your program on a monitor, without obtaining extra software, was likely a chore. There was very limited memory (be it disk or RAM) for holding output so even if there was a pager program, only so much could be stored. The cheapest, most abundant source of “memory” for storing output for the purposes of inspection was paper.

Given that paper is characterized as the preferred method for debugging it’s not surprising that debugging is talked about as something separate from programming. This is probably the starkest difference from today’s methods, where program output is easily stored and examined making debugging practically synonymous with development. Bruce talks about the “program development and shakedown” stages. Van Tassel says that one should plan for debugging to take four times as long as any of planning, writing, and testing. Brown and Sampson’s book starts out decrying the fact that so much time is spent on debugging. Practically every paper in the collection edited by Rustin assumes debugging is independent from the act of programming. This is not surprising: when output is removed from the runtime environment, the development process feels more delineated and less fluid.

The debugging techniques, however, are not markedly different. They’re just smaller in scope to the point where in modern environments we don’t really think of them as techniques. Everything that is prescribed for instrumenting a program to tease out information about its state is still applicable, although it may not be necessary.

The first is desk checking, that is, reading the program listing, possibly making a flow chart, and running through examples manually. Exactly when this step is recommended depends on whether you are reading the mainframe books (before you write the code) or the home computing books (after you run the code). The need to make a flow chart is pretty rare these days since we can quickly navigate code online to understand its control flow. Due to an abundance of computing power and storage space it’s also easy to execute example code and store it (automated testing, anyone?). Desk checking is a form of last resort today and a primary technique of the past.

The other technique, talked about at great length, is careful and judicious placement of print statements. When it comes to tools used in the online computing environment, print statements are the only one mentioned in the home computing books. There is no online debugger: no breakpoints, no program halting, and no memory inspection. There are also no system tools since the whole computer is devoted to running the program you’re writing. The mainframe books do talk about system-level monitors and online debugging, but it is done in a way that suggests it is an exotic luxury. It’s more or less “use ’em if you got ’em.” Any such tools are described as system specific, so a treatment of it would be out of place in a book about general debugging. The use of memory dumps is talked about by the mainframe books, saying that they are hard to work with (you probably wouldn’t examine them online) and that you’ll probably want to avoid them. In the end, the only reliable technique that you can use anywhere is a well-placed print statement. The advice on how to place them is largely the same as today, with more attention paid to avoiding extensive output since this would use a lot of paper.

As mentioned, the mainframe books see debugging as a stage of software development. The home computing books, by contrast, barely discuss the notion of process at all. Bruce’s books completely ignore how a program got written and dive into how you can fix an existing one; any programs provided come into existence through a contrived narrative for the sake of presentation. Smith’s book spends limited space on any development formalities and buries all the advice in enjoyable, haphazard experience reports. Debugging BASIC programs is something that gets done in fast edit-run cycles, the caveat being that the paucity of features in the environment makes it less appealing than it sounds. This is probably why the home computing books push offline debugging so much.

The lack of rigour in the home computing books carries over into testing. While testing is not the same as debugging, they are closely related and are frequently discussed together. So it’s a bit surprising that there is no serious mention of it in the home computing book set. Smith does touch on it, but doesn’t talk much about how to test; Bruce and Huffman completely ignore the subject. It’s embarrassing that Beizer provides more insight on how to test software, even superficially, than books directly aimed at programmers. Even more shameful, Beizer goes into some detail—with wonderful, witty diatribes—and demonstrates a deep knowledge of the subject. The lack of testing discussion in the home computing books relegates them to the realm of amateurs.

The mainframe books, on the other hand, are serious about testing and how it acts as a “debugging aid”. There is a clear emphasis on isolating parts of the program and testing them on a regular basis, especially after a change has been made. The practice of “testing units” or testing modules is mentioned often. This process is very drawn out in the description from Brown and Sampson as it involves multiple people: those who write the program, those who type it, and those who run it. Van Tassel says that using automated testing is the only way to handle large code bases and that storing test input so it can be reused is a necessity. Keeping a log of tests and test results is recommended if you desk check a routine. Testing plays a key role in managing how one debugs a program. (Included in the testing discussions is a lot of hopefulness on one day proving programs correct, complete with a healthy dose of skepticism and no serious expectation of it happening soon.)

Again, what differs from today is the scope. For example, merely checking types at module/function boundaries, let alone within a function, is referred to as “runtime overhead” that may require multiple compilers since optimizing compilers probably wouldn’t verify types. There is an emphasis on getting the compiler to check as much as possible if the resources are available (such as having a compiler that checks those things in the first place!). As usual this is because the resources to run tests are constrained and compilation itself took a significant amount of time. Descriptions of what is tested by the programmer is generally limited to a single module, which is something on the order of implementing a few equations or file operations.

All in all, debugging on mainframes in the 1970s and home computers in the early 1980s comes across a tedious exercise. In the mainframe books you can practically feel the authors pining for more power. There is all this potential and nothing to realize it. It’s mostly today’s development process played out in super-slow motion, but at least it is methodical. The home computing books, on the other hand, have a severe lack of discipline. This is probably in large part due to the use of BASIC as the grounding point, although Truck Smith tries hard to branch out a little.

This collection of books shows that, in the past, a lot more attention was paid to things that we now mostly ignore. And those things are found in the periods of time that emerge from having to wait for anything to happen. There is a clear advantage to having a wealth of computing power available when it comes to debugging. Being able to capture gigabytes of output from any number of tools and analyze it in short order, all on the same machine on which the program is running, is significantly better than worrying about whether a print statement will send a hundred pages of output to the line printer. The introspection tools that our current execution environments offer is a realization of what was yearned for in the mainframe books and we get to reap the rewards. What is revelatory is the seeming lack of formality in home computing. It’s as if everyone was so enamoured by the fact there was a computer in their home that they forgot about the computers at work. The complaints in Beizer’s book may very well be attributed to the lessons found in Smith, Bruce and Huffman’s work. There isn’t enough in the books to explain this discrepancy in thoroughness, but there’s certainly something to explore.

One thing is clear: the more things change the more they stay the same, they just get faster.

Capsule reviews

For those interested in the specific books I’ve included short reviews. If you’re really keen on a book and want to read it be forewarned: unless you want to study the historical context, you won’t learn much from the actual content, except in one case. Appreciating the content requires some research into, and possibly experience with, the actual computing environments referenced in the books.

The Frozen Keyboard
Five stars. Beizer is a pleasure to read. His clear prose, pointed metaphors, and backhanded compliments elegantly excoriate the tech industry for its hyped promises and general lack of empathy for the user. A lot of what he says—perhaps aside from the buying guide and the extensive backup procedure using a printer—still applies. He’s amusingly colourful: when talking about programming and why bugs exist, his description starts with “Programming is a bitch.” If I were teaching software testing or usability, I’d put this book on the supplemental reading list. Recommended for discussion over drinks. (Shout out to John Regehr for pointing me to Beizer’s work.)
Secrets of Software Debugging
This book is mentioned in another book I reviewed (Debugging by Thinking, by Metzger) where it is described as targeting “an obsolete language/environment and is of historical interest only.” The hell with that. This book is actually fun and somewhat insightful, despite its dearth of rigour. Truck Smith is clearly a programmer writing for aspiring programmers. The content is thin and there’s a lot of grasping at formality, but the discussion about the mindset and attitude you need to tackle tough debugging problems still resonates. There’s also three long chapters that are his journals from developing three different programs (one of which is his first Pascal program) that helps you appreciate what went into developing on home computers. As an added bonus, the edition I have comes with a cover whose genesis demands an explanation.
How to Debug Your Personal Computer
This is a condensed version of Software Debugging for Microcomputers containing select chapters from it, with an extra chapter devoted to describing hardware of the time. The hardware chapter is dull and probably wouldn’t help if you had to debug hardware. The rest is the same as the original book, save for the fact it doesn’t tell you what personal computer the code is for. (It’s a Sol 20 microcomputer.)
Software Debugging for Microcomputers
Review in less than ten words: 300+ pages of PRINT statement examples. The programs are all BASIC, the designs are questionable, and the discussion drags more than experimental avant-garde cinema. Each concept could be explained in a few pages. Instead, they are weaved into the development of medium sized programs that come with grandiose descriptions and pedestrian implementations. Furthermore, the names of known, studied concepts at the time are given strange names: “block debugging” (subroutines), “forcing” (testing edge cases), “snapshots” (subroutines, again), just to name a few. I pity anyone who learned from this book. (The same goes for its condensed cousin mentioned above.)
Program Style, Design, Efficiency, Debugging, and Testing
Van Tassel’s book was probably a good one to have on your shelf if you were a professional programmer or a student of programming in the late 1970s and into the 1980s. Topics germane to programming at the time are presented seriously and with care. By now all the advice has been superseded or obsoleted, but you won’t be guided in the wrong direction by reading it. There are lots of programming exercises provided that would still be enjoyable to tackle. Some descriptions are strange and dated, feeling out of place in a modern text. And through an incredible coincidence, I found a passage plagiarized from Program Debugging, a book he does at least reference.
Program Debugging
A strange little book that describes a formal approach to debugging that the authors call “The Method”. This method is based on a problem solving technique used by managers and is this: study the specification, look for the differences from it, hypothesize why they exist, look at the program to determine if you’re correct, and verify that you are correct. It’s like reading a business book that claims scientific management is why science is so successful. What is interesting is the case study of using The Method in a real programming environment. It’s well documented and fascinating to read. There is also this sexist passage on page 37 that Van Tassel rips off: “…and assume that his program has to survive all the sabotage attempts of cross-eyed punch girls, cretinous data control clerks, and idiotic operators.” Van Tassel (page 240) excises the sexism and shifts the insult: “A robust program should be able to survive cross-eyed keypunchers, cretinous data control clerks, and retarded operators.”
Debugging Techniques in Large Systems
This is a collection of papers and talk summaries from a symposium held in 1970. It’s also the earliest book about debugging that I could obtain. It mostly provides historical context, making it clear that interactive debugging was a good idea but difficult to pull off given the lack of resources and diversity of hardware at the time. There is a fair bit of discussion of using the compiler to prevent bugs, things like typographical errors and basic type checking. A lot of what is talked about here is taken for granted today. It did provide a view into the world of programming in very limited environments. I also learned about the unfortunately named debugging system AIDS.

If you know of some home computing books from the microcomputer era that talk about program development or debugging, or you know of popular references for programming on home computers in the early to mid-1980s, I’d appreciate an email pointing me to them.

January 4, 2019

January 04, 2019

Derek Jones (derek-jones)

Changes in the shape of code during the twenties? January 04, 2019 11:42 PM

At the end of 2009 I made two predictions for the next decade; Chinese and Indian developers having a major impact on the shape of code (ok, still waiting for this to happen), and scripting languages playing a significant role (got that one right, but then they were already playing a large role).

Since this blog has just entered its second decade, I will bring the next decade’s predictions forward a year.

I don’t see any new major customer ecosystems appearing. Ecosystems are the drivers of software development, and no new ecosystems has several consequences, including:

  • No major new languages: Creating a language is a vanity endeavor. Vanity project can take off if they are in the right place at the right time. New ecosystems provide opportunities for new languages to become widely used by being in at the start and growing with the ecosystem. There is another opportunity locus; it is fashionable for companies that see themselves as thought-leaders to have their own language, e.g., Google, Apple, and Mozilla. Invent your language at the right time, while working for a thought-leader company and your language could become well-known enough to take-off.

    I don’t see any major new ecosystems appearing and all the likely companies already have their own language.

    Any new language also faces the problem of not having a large collection packages.

  • Software will be more thoroughly tested: When an ecosystem is new, the incentives drive early and frequent releases (to build a customer base); software just has to be good enough. Once a product is established, companies can invest in addressing issues that customers find annoying, like faulty behavior; the incentive change results in more testing.

    There are other forces at work around testing. Companies are experiencing some very expensive faults (testing may be expensive, but not testing may be more expensive) and automatic test generation is becoming commercially usable (i.e., the cost of some kinds of testing is decreasing).

The evolution of widely used languages.

  • I think Fortran and C will have new features added, with relatively little fuss, and will quietly continue to be widely used (to the dismay of the fashionista).
  • There is a strong expectation that C++ and Java should continue to evolve:
  • I expect the ISO C++ work to implode, because there are too many people pulling in too many directions. It makes sense for the gcc and llvm teams to cooperate in taking C++ in a direction that satisfies developers’ needs, rather than the needs of bored consultants. What are Microsoft’s views? They only have their own compiler for strategic reasons (they make little if any profit selling compilers, compilers are an unnecessary drain on management time; who cares what happens to the language).
  • It is going to be interesting watching the impact of Oracle’s move to charging for runtimes. I have no idea what might happen to Java.

In terms of code volume, the future surely has to be scripting languages, and in particular Python, Javascript and PHP. Ten years from now, will there be a widely used, single language? People have been predicting, for many years, that web languages will take over the world; perhaps there will be a sudden switch and I will see that the choice is obvious.

Moore’s law is now dead, which means researchers are going to have to look for completely new techniques for building logic gates. If photonic computers happen, then ternary notation may reappear again (it was used in at least one early Russian computer); I’m not holding my breath for this to occur.

Maxwell Bernstein (tekknolagi)

Bytecode compilers and interpreters January 04, 2019 02:15 AM

Two fun things happened recently and the proximity of the two made something go click in my head and now I think I understand how bytecode interpreters work.

  1. I went to a class at work called “Interpreters 101” or something of the sort. In it, the presenter walked through creating a dead-simple tree-walking Lisp interpreter. Then he ended by suggesting we go out and re-implement it as a bytecode interpreter or even a JIT.
  2. I joined a new team at work that is working on something Python related. Because of my new job responsibilities, I have had to become rather closely acquainted with CPython 3.6 bytecode.

Since learning by writing seems to be something I do frequently, here is a blog post about writing a small bytecode compiler and interpreter in small pieces. We’ll go through it in the same manner as my lisp interpreter: start with the simplest pices and build up the rest of the stack as we need other components.

Some definitions

Before we dive right in, I’m going to make some imprecise definitions that should help differentiate types of interpreters:

  • Tree-walking interpreters process the program AST node by node, recursively evaluating based on some evaluation rules. For a program like Add(Lit 1, Lit 2), it might see the add rule, recursively evaluate the arguments, add them together, and then package that result in the appropriate value type.
  • Bytecode interpreters don’t work on the AST directly. They work on preprocessed ASTs. This simplifies the execution model and can produce impressive speed wins. A program like Add(Lit 1, Lit 2) might be bytecode compiled into the following bytecode:

    PUSH 1
    PUSH 2
    ADD
    

    And then the interpreter would go instruction by instruction, sort of like a hardware CPU.

  • JIT interpreters are like bytecode interpreters except instead of compiling to language implementation-specific bytecode, they try to compile to native machine instructions. Most production-level JIT interpreters “warm up” by measuring what functions get called the most and then compiling them to native code in the background. This way, the so-called “hot code” gets optimized and has a smaller performance penalty.

My Writing a Lisp series presents a tree-walking interpreter. This much smaller post will present a bytecode interpreter. A future post may present a JIT compiler when I figure out how they work.

Without further ado, let us learn.

In the beginning, there was a tree-walking interpreter

Lisp interpreters have pretty simple semantics. Below is a sample REPL (read-eval-print-loop) where all the commands are parsed into ASTs and then evaled straightaway. See Peter Norvig’s lis.py for a similar tree-walking interpreter.

>>> 1
1
>>> '1'
1
>>> "hello"
hello
>>> (val x 3)
None
>>> x
3
>>> (set x 7)
None
>>> x
7
>>> (if True 3 4)
3
>>> (lambda (x) (+ x 1))
<Function instance...>
>>> ((lambda (x) (+ x 1)) 5)
6
>>> (define f (x) (+ x 1))
None
>>> f
<Function instance...>
>>> +
<function + at...>
>>> (f 10)
11
>>>

For our interpreter, we’re going to write a function called compile that takes in an expression represented by a Python list (something like ['+', 5, ['+', 3, 5]]) and returns a list of bytecode instructions. Then we’ll write eval that takes in those instructions and returns a Python value (in this case, the int 13). It should behave identically to the tree-walking interpreter, except faster.

The ISA

For our interpreter we’ll need a surprisingly small set of instructions, mostly lifted from the CPython runtime’s own instruction set architecture. CPython is a stack-based architecture, so ours will be too.

  • LOAD_CONST
    • Pushes constants onto the stack.
  • STORE_NAME
    • Stores values into the environment.
  • LOAD_NAME
    • Reads values from the environment.
  • CALL_FUNCTION
    • Calls a function (built-in or user-defined).
  • RELATIVE_JUMP_IF_TRUE
    • Jumps if the value on top of the stack is true.
  • RELATIVE_JUMP
    • Jumps.
  • MAKE_FUNCTION
    • Creates a function object from a code object on the stack and pushes it on the stack.

With these instructions we can define an entire language. Most people choose to define math operations in their instruction sets for speed, but we’ll define them as built-in functions because it’s quick and easy.

The Opcode and Instruction classes

I’ve written an Opcode enum:

import enum


class AutoNumber(enum.Enum):
    def _generate_next_value_(name, start, count, last_values):
        return count


@enum.unique
class Opcode(AutoNumber):
    # Add opcodes like:
    # OP_NAME = enum.auto()
    pass

Its sole purpose is to enumerate all of the possible opcodes — we’ll add them later. Python’s enum API is pretty horrific so you can gloss over this if you like and pretend that the opcodes are just integers.

I’ve also written an Instruction class that stores opcode and optional argument pairs:

class Instruction:
    def __init__(self, opcode, arg=None):
        self.opcode = opcode
        self.arg = arg

    def __repr__(self):
        return "<Instruction.{}({})>".format(self.opcode.name, self.arg)

    def __call__(self, arg):
        return Instruction(self.opcode, arg)

    def __eq__(self, other):
        assert(isinstance(other, Instruction))
        return self.opcode == other.opcode and self.arg == other.arg

The plan is to declare one Instruction instance per opcode, naming its argument something sensible. Then when bytecode compiling, we can make instances of each instruction by calling them with a given argument. This is pretty hacky but it works alright.

# Creating top-level opcodes
STORE_NAME = Instruction(Opcode.STORE_NAME, "name")
# Creating instances
[STORE_NAME("x"), STORE_NAME("y")]

Now that we’ve got some infrastructure for compilation, let’s start compiling.

Integers

This is what our compile function looks like right now:

def compile(exp):
    raise NotImplementedError(exp)

It is not a useful compiler, but at least it will let us know what operations it does not yet support. Let’s make it more useful.

The simplest thing I can think of to compile is an integer. If we see one, we should put it on the stack. That’s all! So let’s first add some instructions that can do that, and then implement it.

class Opcode(AutoNumber):
    LOAD_CONST = enum.auto()

I called it LOAD_CONST because that’s what Python calls its opcode that does something similar. “Load” is a sort of confusing word for what its doing, because that doesn’t specify where it’s being loaded. If you want, you can call this PUSH_CONST, which in my opinion better implies that it is to be pushed onto the VM stack. I also specify CONST because this instruction should only be used for literal values: 5, "hello", etc. Something where the value is completely defined by the expression itself.

Here’s the parallel for the Instruction class (defined at the module scope):

LOAD_CONST = Instruction(Opcode.LOAD_CONST, "const")

The argument name "const" is there only for bytecode documentation for the reader. It will be replaced by an actual value when this instruction is created and executed. Next, let’s add in a check to catch this case.

def compile(exp):
    if isinstance(exp, int):
        return [LOAD_CONST(exp)]
    raise NotImplementedError(exp)

Oh, yeah. compile is going to walk the expression tree and merge together lists of instructions from compiled sub-expressions — so every branch has to return a list. This will look better when we have nested cases. Let’s test it out:

assert compile(5) == [LOAD_CONST(5)]
assert compile(7) == [LOAD_CONST(7)]

Now that we’ve got an instruction, we should also be able to run it. So let’s set up a basic eval loop:

def eval(self, code):
    pc = 0
    while pc < len(code):
        ins = code[pc]
        op = ins.opcode
        pc += 1
        raise NotImplementedError(op)

Here pc is short for “program counter”, which is equivalent to the term “instruction pointer”, if you’re more familiar with that. The idea is that we iterate through the instructions one by one, executing them in order.

This loop will throw when it cannot handle a particular instruction, so it is reasonable scaffolding, but not much more. Let’s add a case to handle LOAD_CONST in eval.

def eval(code):
    pc = 0
    stack = []
    while pc < len(code):
        ins = code[pc]
        op = ins.opcode
        pc += 1
        if op == Opcode.LOAD_CONST:
            stack.append(ins.arg)
        else:
            raise NotImplementedError(op)
    if stack:
        return stack[-1]

Note, since it will come in handy later, that eval returns the value on the top of the stack. This is the beginning of our calling convention, which we’ll flesh out more as the post continues. Let’s see if this whole thing works, end to end.

assert eval(compile(5)) == 5
assert eval(compile(7)) == 7

And now let’s run it:

willow% python3 ~/tmp/bytecode0.py
willow%

Swell.

Naming things

My name, and yours, and the true name of the sun, or a spring of
water, or an unborn child, all are syllables of the great word that
is very slowly spoken by the shining of the stars. There is no other
power. No other name.


-- Ursula K Le Guin, A Wizard of Earthsea

Now, numbers aren’t much fun if you can’t do anything with them. Right now the only valid programs are programs that push one number onto the stack. Let’s add some opcodes that put those values into variables.

We’ll be adding STORE_NAME, which takes one value off the stack and stores it in the current environment, and LOAD_NAME, which reads a value from the current environment and pushes it onto the stack.

@enum.unique
class Opcode(AutoNumber):
    LOAD_CONST = enum.auto()
    STORE_NAME = enum.auto()
    LOAD_NAME = enum.auto()

# ...

STORE_NAME = Instruction(Opcode.STORE_NAME, "name")
LOAD_NAME = Instruction(Opcode.LOAD_NAME, "name")

Let’s talk about our representation of environments. Our Env class looks like this (based on Dmitry Soshnikov’s “Spy” interpreter):

class Env(object):
    # table holds the variable assignments within the env. It is a dict that
    # maps names to values.
    def __init__(self, table, parent=None):
        self.table = table
        self.parent = parent

    # define() maps a name to a value in the current env.
    def define(self, name, value):
        self.table[name] = value

    # assign() maps a name to a value in whatever env that name is bound,
    # raising a ReferenceError if it is not yet bound.
    def assign(self, name, value):
        self.resolve(name).define(name, value)

    # lookup() returns the value associated with a name in whatever env it is
    # bound, raising a ReferenceError if it is not bound.
    def lookup(self, name):
        return self.resolve(name).table[name]

    # resolve() finds the env in which a name is bound and returns the whole
    # associated env object, raising a ReferenceError if it is not bound.
    def resolve(self, name):
        if name in self.table:
            return self

        if self.parent is None:
            raise ReferenceError(name)

        return self.parent.resolve(name)

    # is_defined() checks if a name is bound.
    def is_defined(self, name):
        try:
            self.resolve(name)
            return True
        except ReferenceError:
            return False

Our execution model will make one new Env per function call frame and one new env per closure frame, but we’re not quite there yet. So if that doesn’t yet make sense, ignore it for now.

What we care about right now is the global environment. We’re going to make one top-level environment for storing values. We’ll then thread that through the eval function so that we can use it. But let’s not get ahead of ourselves. Let’s start by compiling.

def compile(exp):
    if isinstance(exp, int):
        return [LOAD_CONST(exp)]
    elif isinstance(exp, list):
        assert len(exp) > 0
        if exp[0] == 'val':  # (val n v)
            assert len(exp) == 3
            _, name, subexp = exp
            return compile(subexp) + [STORE_NAME(name)]
    raise NotImplementedError(exp)

I’ve added this second branch to check if the expression is a list, since we’ll mostly be dealing with lists now. Since we also have just about zero error-handling right now, I’ve also added some asserts to help with code simplicity.

In the val case, we want to extract the name and the subexpression — remember, we won’t just be compiling simple values like 5; the values might be (+ 1 2). Then, we want to compile the subexpression and add a STORE_NAME instruction. We can’t test that just yet — we don’t have more complicated expressions — but we’ll get there soon enough. Let’s test what we can, though:

assert compile(['val', 'x', 5]) == [LOAD_CONST(5), STORE_NAME('x')]

Now let’s move back to eval.

def eval(code, env):
    pc = 0
    stack = []
    while pc < len(code):
        ins = code[pc]
        op = ins.opcode
        pc += 1
        if op == Opcode.LOAD_CONST:
            stack.append(ins.arg)
        elif op == Opcode.STORE_NAME:
            val = stack.pop(-1)
            env.define(ins.arg, val)
        else:
            raise NotImplementedError(ins)
    if stack:
        return stack[-1]

You’ll notice that I’ve

  • Added an env parameter, so that we can evaluate expressions in different contexts and get different results
  • Added a case for STORE_NAME

We’ll have to modify our other tests to pass an env parameter — you can just pass None if you are feeling lazy.

Let’s make our first environment and test out STORE_NAME. For this, I’m going to make an environment and test that storing a name in it side-effects that environment.

env = Env({}, parent=None)
eval([LOAD_CONST(5), STORE_NAME('x')], env)
assert env.table['x'] == 5

Now we should probably go about adding compiler and evaluator functionality for reading those stored values. The compiler will just have to check for variable accesses, represented just as strings.

def compile(exp):
    if isinstance(exp, int):
        return [LOAD_CONST(exp)]
    elif isinstance(exp, str):
        return [LOAD_NAME(exp)]
    # ...

And add a test for it:

assert compile('x') == [LOAD_NAME('x')]

Now that we can generate LOAD_NAME, let’s add eval support for it. If we did anything right, its implementation should pretty closely mirror that of its sister instruction.

def eval(code, env):
    # ...
    while pc < len(code):
        # ...
        elif op == Opcode.STORE_NAME:
            val = stack.pop(-1)
            env.define(ins.arg, val)
        elif op == Opcode.LOAD_NAME:
            val = env.lookup(ins.arg)
            stack.append(val)
        # ...
    # ...

To test it, we’ll first manually store a name into the environment, then see if LOAD_NAME can read it back out.

env = Env({'x': 5}, parent=None)
assert eval([LOAD_NAME('x')], env) == 5

Neat.

Built-in functions

We can add as many opcodes as features we need, or we can add one opcode that allows us to call native (Python) code and extend our interpreter that way. Which approach you take is mostly a matter of taste.

In our case, we’ll add the CALL_FUNCTION opcode, which will be used both for built-in functions and for user-defined functions. We’ll get to user-defined functions later.

CALL_FUNCTION will be generated when an expression is of the form (x0 x1 x2 ...) and x0 is not one of the pre-set recognized names like val. The compiler should generate code to load first the function, then the arguments onto the stack. Then it should issue the CALL_FUNCTION instruction. This is very similar to CPython’s implementation.

I’m not going to reproduce the Opcode declaration, because all of those look the same, but here is my Instruction declaration:

CALL_FUNCTION = Instruction(Opcode.CALL_FUNCTION, "nargs")

The CALL_FUNCTION instruction takes with it the number of arguments passed to the function so that we can call it correctly. Note that we do this instead of storing the correct number of arguments in the function because functions could take variable numbers of arguments.

Let’s compile some call expressions.

def compile(exp):
    # ...
    elif isinstance(exp, list):
        assert len(exp) > 0
        if exp[0] == 'val':
            assert len(exp) == 3
            _, name, subexp = exp
            return compile(subexp) + [STORE_NAME(name)]
        else:
            args = exp[1:]
            nargs = len(args)
            arg_code = sum([compile(arg) for arg in args], [])
            return compile(exp[0]) + arg_code + [CALL_FUNCTION(nargs)]
    # ...

I’ve added a default case for list expressions. See that it compiles the name, then the arguments, then issues a CALL_FUNCTION. Let’s test it out with 0, 1, and more arguments.

assert compile(['hello']) == [LOAD_NAME('hello'), CALL_FUNCTION(0)]
assert compile(['hello', 1]) == [LOAD_NAME('hello'), LOAD_CONST(1),
                                    CALL_FUNCTION(1)]
assert compile(['hello', 1, 2]) == [LOAD_NAME('hello'), LOAD_CONST(1),
                                    LOAD_CONST(2), CALL_FUNCTION(2)]

Now let’s implement eval.

def eval(code, env):
    # ...
    while pc < len(code):
        # ...
        elif op == Opcode.CALL_FUNCTION:
            nargs = ins.arg
            args = [stack.pop(-1) for i in range(nargs)][::-1]
            fn = stack.pop(-1)
            assert callable(fn)
            stack.append(fn(args))
        else:
            raise NotImplementedError(ins)
    # ...

Notice that we’re reading the arguments off the stack — in reverse order — and then reading the function off the stack. Everything is read the opposite way it is pushed onto the stack, since, you know, it’s a stack.

We also check that the fn is callable. This is a Python-ism. Since we’re allowing raw Python objects on the stack, we have to make sure that we’re actually about to call a Python function. Then we’ll call that function with a list of arguments and push its result on the stack.

Here’s what this looks like in real life, in a test:

env = Env({'+': lambda args: args[0] + args[1]}, None)
assert eval(compile(['+', 1, 2]), env) == 3

This is pretty neat. If we stuff lambda expressions into environments, we get these super easy built-in functions. But that’s not quite the most optimal + function. Let’s make a variadic one for fun.

env = Env({'+': sum}, None)
assert eval(compile(['+', 1, 2, 3, 4, 5]), env) == 15
assert eval(compile(['+']), env) == 0

Since we pass the arguments a list, we can do all sorts of whack stuff like this!

Since I only alluded to it earlier, let’s add a test for compiling nested expressions in STORE_NAME.

env = Env({'+': sum}, None)
eval(compile(['val', 'x', ['+', 1, 2, 3]]), env)
assert env.table['x'] == 6

Go ahead and add all the builtin functions that your heart desires… like print!

env = Env({'print': print}, None)
eval(compile(['print', 1, 2, 3]), env)

You should see the arguments passed in the correct order. Note that if you are using Python 2, you should wrap the print in a lambda, since print is not a function.

Conditionals

Without conditionals, we can’t really do much with our language. While we could choose to implement conditionals eagerly as built-in functions, we’re going to do “normal” conditionals. Conditionals that lazily evaluate their branches. This can’t be done with our current execution model because all arguments are evaluated before being passed to built-in functions.

We’re going to do conditionals the traditional way:

(if a b c)

will compile to

[a BYTECODE]
RELATIVE_JUMP_IF_TRUE b
[c BYTECODE]
RELATIVE_JUMP end
b:
[b BYTECODE]
end:

We’re also going to take the CPython approach in generating relative jumps instead of absolute jumps. This way we don’t need a separate target resolution step.

To accomplish this, we’ll add two the opcodes and instructions listed above:

RELATIVE_JUMP_IF_TRUE = Instruction(Opcode.RELATIVE_JUMP_IF_TRUE, "off")
RELATIVE_JUMP = Instruction(Opcode.RELATIVE_JUMP, "off")

Each of them takes an offset in instructions of how far to jump. This could be positive or negative — for loops, perhaps — but right now we will only generate positive offsets.

We’ll add one new case in the compiler:

def compile(exp):
    # ...
    elif isinstance(exp, list):
        # ...
        elif exp[0] == 'if':
            assert len(exp) == 4
            _, cond, iftrue, iffalse = exp
            iftrue_code = compile(iftrue)
            iffalse_code = compile(iffalse) + [RELATIVE_JUMP(len(iftrue_code))]
            return (compile(cond)
                    + [RELATIVE_JUMP_IF_TRUE(len(iffalse_code))]
                    + iffalse_code
                    + iftrue_code)
        # ...

First compile the condition. Then, compile the branch that will execute if the condition passes. The if-false branch is a little bit tricky because I am also including the jump-to-end in there. This is so that the offset calculation for the jump to the if-true branch is correct (I need not add +1).

Let’s add some tests to check our work:

assert compile(['if', 1, 2, 3]) == [LOAD_CONST(1), RELATIVE_JUMP_IF_TRUE(2),
                                    LOAD_CONST(3), RELATIVE_JUMP(1),
                                    LOAD_CONST(2)]
assert compile(['if', 1, ['+', 1, 2], ['+', 3, 4]]) == \
        [LOAD_CONST(1),
         RELATIVE_JUMP_IF_TRUE(5),
         LOAD_NAME('+'), LOAD_CONST(3), LOAD_CONST(4), CALL_FUNCTION(2),
         RELATIVE_JUMP(4),
         LOAD_NAME('+'), LOAD_CONST(1), LOAD_CONST(2), CALL_FUNCTION(2)]

I added the second test to double-check that nested expressions work correctly. Looks like they do. On to eval!

This part should be pretty simple — adjust the pc, sometimes conditionally.

def eval(code, env):
    # ...
    while pc < len(code):
        # ...
        elif op == Opcode.RELATIVE_JUMP_IF_TRUE:
            cond = stack.pop(-1)
            if cond:
                pc += ins.arg  # pc has already been incremented
        elif op == Opcode.RELATIVE_JUMP:
            pc += ins.arg  # pc has already been incremented
        # ...
    # ...

If it takes a second to convince yourself that this is not off-by-one, that makes sense. Took me a little bit too. And hey, if convincing yourself isn’t good enough, here are some tests.

assert eval(compile(['if', 'true', 2, 3]), Env({'true': True})) == 2
assert eval(compile(['if', 'false', 2, 3]), Env({'false': False})) == 3
assert eval(compile(['if', 1, ['+', 1, 2], ['+', 3, 4]]), Env({'+': sum})) == 3

Defining your own functions

User-defined functions are absolutely key to having a usable programming language. Let’s let our users do that. Again, we’re using Dmitry’s Function representation, which is wonderfully simple.

class Function(object):
    def __init__(self, params, body, env):
        self.params = params
        self.body = body
        self.env = env # closure!

The params will be a tuple of names, the body a tuple of instructions, and the env an Env.

In our language, all functions will be closures. They can reference variables defined in the scope where the function is defined (and above). We’ll use the following forms:

((lambda (x) (+ x 1)) 5)
; or
(define inc (x) (+ x 1))
(inc 5)

In fact, we’re going to use a syntax transformation to re-write defines in terms of val and lambda. This isn’t required, but it’s kind of neat.

For this whole thing to work, we’re going to need a new opcode: MAKE_FUNCTION. This will convert some objects stored on the stack into a Function object.

MAKE_FUNCTION = Instruction(Opcode.MAKE_FUNCTION, "nargs")

This takes an integer, the number of arguments that the function expects. Right now we only allow positional, non-optional arguments. If we wanted to have additional calling conventions, we’d have to add them later.

Let’s take a look at compile.

def compile(exp):
    # ...
    elif isinstance(exp, list):
        assert len(exp) > 0
        # ...
        elif exp[0] == 'lambda':
            assert len(exp) == 3
            (_, params, body) = exp
            return [LOAD_CONST(tuple(params)),
                    LOAD_CONST(tuple(compile(body))),
                    MAKE_FUNCTION(len(params))]
        elif exp[0] == 'define':
            assert len(exp) == 4
            (_, name, params, body) = exp
            return compile(['lambda', params, body]) + [STORE_NAME(name)]
    # ...

For lambda, it’s pretty straightforward. Push the params, push the body code, make a function.

define is a little sneaker. It acts as a macro and rewrites the AST before compiling it. If we wanted to be more professional, we could make a macro system so that the standard library could define define and if… but that’s too much for right now. But still. It’s pretty neat.

Before we move on to eval, let’s quickly check our work.

assert compile(['lambda', ['x'], ['+', 'x', 1]]) == \
    [LOAD_CONST(('x',)),
     LOAD_CONST((
         LOAD_NAME('+'),
         LOAD_NAME('x'),
         LOAD_CONST(1),
         CALL_FUNCTION(2))),
     MAKE_FUNCTION(1)]
assert compile(['define', 'f', ['x'], ['+', 'x', 1]]) == \
    [LOAD_CONST(('x',)),
     LOAD_CONST((
         LOAD_NAME('+'),
         LOAD_NAME('x'),
         LOAD_CONST(1),
         CALL_FUNCTION(2))),
     MAKE_FUNCTION(1),
     STORE_NAME('f')]

Alright alright alright. Let’s get these functions created. We need to handle the MAKE_FUNCTION opcode in eval.

def eval(code, env):
    # ...
    while pc < len(code):
        # ...
        elif op == Opcode.MAKE_FUNCTION:
            nargs = ins.arg
            body_code = stack.pop(-1)
            params = stack.pop(-1)
            assert len(params) == nargs
            stack.append(Function(params, body_code, env))
        # ...
    # ...

As with calling functions, we read everything in the reverse of the order that we pushed it. First the body, then the params — checking that they’re the right length — then push the new Function object.

But Functions aren’t particularly useful without being callable. There are two strategies for calling functions, one slightly slicker than the other. You can choose which you like.

The first strategy, the simple one, is to add another case in CALL_FUNCTION that handles Function objects. This is what most people do in most programming languages. It looks like this:

def eval(code, env):
    # ...
    while pc < len(code):
        # ...
        elif op == Opcode.CALL_FUNCTION:
            nargs = ins.arg
            args = [stack.pop(-1) for i in range(nargs)][::-1]
            fn = stack.pop(-1)
            if callable(fn):
                stack.append(fn(args))
            elif isinstance(fn, Function):
                actuals_record = dict(zip(fn.params, args))
                body_env = Env(actuals_record, fn.env)
                stack.append(eval(fn.body, body_env))
            else:
                raise RuntimeError("Cannot call {}".format(fn))
        # ...
    # ...

Notice that the function environment consists solely of the given arguments and its parent is the stored environment — not the current one.

The other approach is more Pythonic, I think. It turns Function into a callable object, putting the custom setup code into Function itself. If you opt to do this, leave CALL_FUNCTION alone and modify Function this way:

class Function(object):
    def __init__(self, params, body, env):
        self.params = params
        self.body = body
        self.env = env # closure!

    def __call__(self, actuals):
        actuals_record = dict(zip(self.params, actuals))
        body_env = Env(actuals_record, self.env)
        return eval(self.body, body_env)

Then eval should call the Function as if it were a normal Python function. Cool… or gross, depending on what programming languages you are used to working with.

They should both work as follows:

env = Env({'+': sum}, None)
assert eval(compile([['lambda', 'x', ['+', 'x', 1]], 5]), env) == 6

eval(compile(['define', 'f', ['x'], ['+', 'x', 1]]), env)
assert isinstance(env.table['f'], Function)

Hell, even recursion works! Let’s write ourselves a little factorial function.

import operator
env = Env({
    '*': lambda args: args[0] * args[1],
    '-': lambda args: args[0] - args[1],
    'eq': lambda args: operator.eq(*args),
}, None)
eval(compile(['define', 'factorial', ['x'],
                ['if',
                ['eq', 'x', 0],
                1,
                ['*', 'x', ['factorial', ['-', 'x', 1]]]]]), env)
assert eval(compile(['factorial', 5]), env) == 120

Which works, but it feels like we’re lacking the ability to sequence operations… because we are! So let’s add that.

One teensy little thing is missing

It should suffice to write a function compile_program that can take a list of expressions, compile them, and join them. This alone is not enough, though. We should expose that to the user so that they can sequence operations when they need to. So let’s also add a begin keyword.

def compile_program(prog):
    return [instr for exp in prog for instr in compile(exp)]  # flatten

And then a case in compile:

def compile(exp):
    # ...
    elif isinstance(exp, list):
        # ...
        elif exp[0] == 'begin':
            return compile_program(exp[1:])
        else:
            args = exp[1:]
            nargs = len(args)
            arg_code = sum([compile(arg) for arg in args], [])
            return compile(exp[0]) + arg_code + [CALL_FUNCTION(nargs)]
    raise NotImplementedError(exp)

And of course a test:

import operator
env = Env({
    '*': lambda args: args[0] * args[1],
    '-': lambda args: args[0] - args[1],
    'eq': lambda args: operator.eq(*args),
}, None)
assert eval(compile(['begin',
    ['define', 'factorial', ['x'],
        ['if',
        ['eq', 'x', 0],
        1,
        ['*', 'x', ['factorial', ['-', 'x', 1]]]]],
    ['factorial', 5]
]), env) == 120

And you’re done!

You’re off to the races. You’ve just written a bytecode interpreter in Python or whatever language you are using to follow along. There are many ways to extend and improve it. Maybe those will be the subject of a future post. Here are a few I can think of:

  • Make a JIT compiler — generate native code instead of bytecode
  • Re-write this in a language whose runtime is faster than CPython
  • Choose a file format and write the compiler and interpreter so that they eject to and read from the disk
  • Add a foreign-function interface so that the user can call functions from the host language
  • Expose compile and eval in the environment to add metaprogramming to your language
  • Add macros
  • Learn from CPython and store local variables in the frame itself instead of in a dict — this is much faster (see LOAD_FAST and STORE_FAST)
  • Target an existing VM’s bytecode so you can compile to, say, actual CPython or Lua or something
  • Similar to above: target OPy so you can run compiled programs with Andy Chu’s VM
  • Implement this in a statically-typed language like OCaml, SML, Haskell, Rust, Swift, etc

Pepijn de Vos (pepijndevos)

Bucket Brigade Delay January 04, 2019 12:00 AM

So far in my quest to build a synthesizer, I’ve controlled a Game Boy synth with a linear potentionmeter and made a pressure sensing circuit for the SoftPot. Today I want to make an analog delay using a bucket brigade device.

To do this, I used a BL3207 bucket brigade device, and a BL3102 clock generator, which are clones of the MN3207 and MN3102, respectively. The datasheets for the former are a single page and not of much use, but since they are clones, the original datasheets provide more useful information. Most importantly, this reference circuit.

MN3207 reference schematic

I did not build that exact circuit, but kind of used it as a hookup guide for the two chips. The MN3102 basically uses the RC filter to generate two opposing clock signals. It also generates a special voltage that’s 15/16 of Vdd for some reason. The MN3207 then takes an analog input, and with some pull-down resistors, produces a delayed output signal. In the above circuit there is a lot of filtering going on, while my circuit uses a simple first order RC filter for now. Both the delayed output and the input signal are fed into an opamp adder to make a nice feedback loop. In the above video I’m simply turning the feedback gain.

ball of wires

To be honest, the current circuit sounds quite bad and is not very flexible. It just adds a fixed delay and loops it back on itself, going from inaudible to metallic to oscillating in the blink of an eye. The best sound I got out of it is the click track when it’s almost oscillating and becomes this funny tone by itself. Maybe all of the filtering in the reference circuit make it a lot better, but I have some other ideas.

I have like 5 of these delay chips, so it’d be fun to chain them together for a longer delay. The other thing is the clock generator: You can disconnect the RC and drive the oscillator externally. I’m thinking I could create an external clock, and modulate that with an LFO to create a vibrato effect.

Update: Vibrato works like a charm. I simply drove the clock with an Arduino as follows. Then I played a sine wave and some acoustic guitar into it for demonstration.

void setup() {
}

void loop() {
  int i;
  for(i=0; i<2000; i+=30) {
    tone(2, 40000+i);
    delay(1);
  }
  for(i=2000; i>0; i-=30) {
    tone(2, 40000+i);
    delay(1);
  }
}

Noon van der Silk (silky)

Van der Silk Fashion at Linux Conf 2019 - New Zealand January 04, 2019 12:00 AM

Posted on January 4, 2019 by Noon van der Silk

Excitingly, this year I’m going to Linux Conference, and giving a talk at the Art+Tech MiniConf!

To celebrate, I redesigned the Van der Silk website:

Also, very coincidentally, PAOM re-designed their website for the new year, so it’s much easier to find all my designs: noonvandersilk on PAOM.

Hope to see you there!

January 03, 2019

Jan van den Berg (j11g)

To the Lighthouse – Virginia Woolf January 03, 2019 08:47 PM

To the Lighthouse – Virginia Woolf (1927) – 254 pages

This was an exhausting book to read. Because it reads like a prolonged poem with a stream of thoughts about everything and anything and with very little dialogue. Two things are very clear though: Woolf went very deep and therefore this book can not be anything else than autobiographical. Feeling already exhausted as a reader, I couldn’t even imagine what the process must have been for a writer to put this on paper.

It’s not my favorite kind of fiction but I am still glad I read it. It’s remarkable how Woolf is able to capture moments of human thought and emotion in an identifiable way: tempestuous and full of unfulfilled intent. And where love and hate are often not as indistinguishable as we think.

The post To the Lighthouse – Virginia Woolf appeared first on Jan van den Berg.

Pepijn de Vos (pepijndevos)

Sensing pressure with a SoftPot January 03, 2019 12:00 AM

In my previous post I connected a SoftPot to an Arduino and used a Game Boy as the synth. In this post I’m focusing more on the SoftPot, and specifically on reading pressure.

The SoftPot consists of a resistive bottom layer and a conductive top layer. When pressed together, the bottom resistor forms a voltage divider with the top conductor. When the conductive top side is pressed down, it shorts part of the bottom track, creating a lower total resistance. The harder you press down, the larger the area of the bottom track that is shorted, the lower the resistance.

So to measure both pressure and position, we’ll need to simultaneously read the voltage on the top track, and the resistance of the bottom track. To get an accurate position, we’d like to have the full voltage swing available, without any changes in voltage due to pressure. Meanwhile, we’d like to sense small DC variations in resistance of a large total resistance. This is my best attempt so far.

circuit

It uses a current mirror to maintain an almost fixed voltage on the pot (R1) and only lose 0.6V of the full swing. The current mirror then feeds into a biased transimpedance amplifier. The rightmost voltage divider is used to set the DC output voltage. (I later added a capacitor in parallel to the TIA feedback resistor to reject noise)

In reality the voltage divider on the positive TIA input is a trim pot. The changes in resistance of the SoftPot are smaller than the process variations. For example, this particular pot changes from 20.7k to 20.3k when pressed, so the bias voltage has to be adjusted for this.

sensing pressuer and pitch

In the above figure, the blue line is me sliding my finger up and down, while the yellow line is the pressure of my finger, with the green/red lines being the max/min. This seems to work just fine, except after letting it run for a while the pressure value starts to drift. Remember, we need the constant, absolute value, not some high-pass filtered version.

I was really confused, and tried various things to reproduce the problem and improve the situation. Then I blew on the circuit.

temperature sensitivity

OMG, look at that! The current mirror goes completely crazy when it gets warm. One solution is to add more degeneration, which adds voltage drop. Another is to use a Wilson current mirror, which also adds voltage drop. And finally, you could get two integrated transistors on the same die, which makes them the same temperature and improves matching. I guess that’s what I’ll have to do. To be continued.

const int pitchPin = A0;
const int pressurePin = A1;
const int pwmPin = 12;

const int pressureThreshold = 20;

int pitchValue = 0;        // value from SoftPot
int pressureValue = 0;        // value from TIA
int minPressureValue = INT16_MAX;
int maxPressureValue = 0;
int pressure = 0;
int frequency = 0;
int polarity = 1;

void setup() {
  Serial.begin(9600);
}

void loop() {
  delay(1);
  pitchValue = analogRead(pitchPin);
  delay(1);
  pressureValue = analogRead(pressurePin);
  delay(1);
  minPressureValue = min(pressureValue, minPressureValue);
  maxPressureValue = max(pressureValue, maxPressureValue);
  pressure = map(pressureValue, minPressureValue, maxPressureValue, 0, 10);
  frequency = map(pitchValue, 0, 1023, 440, 1320);
  if (pressureValue < minPressureValue+pressureThreshold) {
    pitchValue = 0;
    noTone(pwmPin);
  } else if (pressure > 6) {
    tone(pwmPin, frequency+pressure*polarity);
    polarity *= -1;
  } else {
    tone(pwmPin, frequency);
  }
  Serial.print(pitchValue);
  Serial.print(", ");
  Serial.print(minPressureValue);
  Serial.print(", ");
  Serial.print(maxPressureValue);
  Serial.print(", ");
  Serial.println(pressureValue);
  delay(50);
}

Noon van der Silk (silky)

2018 Books January 03, 2019 12:00 AM

Posted on January 3, 2019 by Noon van der Silk
div.book { display: flex; flex-direction: row; padding: 10px; } div.book img { margin-right: 10px; } div.alt { background: #eaeaea; }

I’ve seen a few people post the books that they’ve read over the last year; so I thought I’d also get in on the action. Here are the books that I read (or partially read), in no particular order, last year:

Non-Fiction

The Oregon Experiment by Christopher W. Alexander, Murray Silverstein, Shlomo Angel, Sara Ishikawa, Denny Abrams

Anyone that knows me, knows that Christopher Alexander is basically my personal Jesus. This book is amazing, to me in it’s demonstration of Alexander’s (and collaborators) standard approach to architecture; i.e building via small local activities in an endless cycle according to a specific and developing pattern language. It also contains very important insights, in my mind, in terms of project management and budgeting.

Educated by Tara Westover

I learned about this one from Bill Gates. It’s amazing. I found it really fascinating to read about Tara’s life, and how she understood her upbringing. It makes you think quite a lot about the things you tell yourself to make sense of the situations you’re in.

Dark Emu by Bruce Pascoe

Soon to be (if not already) an Australian classic; it’s very interesting to learn an entirely different history about our country and the Aboriginal people. I found it to be reasonably hard reading because there are some solid descriptions of various items and processes that it would’ve been amazing to have pictures of. In any case, necessary reading.

Radical Candor by Kim Malone Scott

I basically hated this book. I think, on the one hand, it does have useful information in it; but for me it came across too much like standard Silicon Valley propoganda. I do think there is good stuff in here, for leaders that struggle with feedback.

Cities for People by Jan Gehl

Recommended to me by Ryan, who was working as a Barista at from Streat, this book is, apparently, a classic of urban planning. It’s amazing. It has a incredibly readable style; with lots of pictures and summaries of each paragraph. Jan has had a lot of impact in designing Melbourne, so it’s really interesting to read this book and see and understand the impact he and his team have had.

Factfulness by Hans Rosling, Ola Rosling, Anna Rosling Rönnlund

Quite good. From this book I learned about the website Dollar Street, which is incredible and well worth a visit. Hans has many questions in the book; almost all of which I answered wrong, much like most of his audience that he’s asked the questions to. I learned a lot about how to think about world-wide progress, and that there are interesting organisations out there doing interesting work. I also learned, much to my surprise, that the World Bank isn’t a bank in the sense that I would’ve assumed; it’s main goal is to end poverty!

Enlightenment Now by Steven Pinker

Not bad. This is the first book of Steven Pinker I’ve read. I’m unlikely to read more, after learning about some sexist views he’s had from Delusions of Gender (not a book I’ve read yet; but heard thoughts from my partner).

I found this book to be reasonably good, and make some solid arguments for continuing to fight against people that don’t think progress is real. Unfortunately, to paraphrase a joke from the Simposns, Steven seems like the kind of person that can use graphs to prove anything that is even remotely true. It contains too many anecdotes and cherry-picked examples.

I’d recommend Factfulness over this one.

The Hidden Life of Trees by Peter Wohlleben

Really great. I’ve always been a bit of a hippie, and am mostly vegetarian, so when my partner found out I’d be reading this one, she started to get a bit concerned that I’d soon not even eat greens! Now, I haven’t quite gone that far, but I do really love the ideas and science in this book; and I think it’s really interesting to think of trees and their root-systems as being some giant entity. The one downside to this book is that it occasionally lacks pictures of the concepts that are being described.

Surfaces and Essences by Douglas R. Hofstadter, Emmanuel Sander

Amazing. Probably not for people feeling a bit impatient. The book goes into many examples of the ideas; and makes a strong argument for the idea that all thinking is based on analogies. I’ve always loved analogies, so I’m naturally a big fan.

As someone presently working in AI, I found this book full of many incredibly interesting and relevant ideas; one of which we discussed previously.

If you’re at all interested in when computers might be able to get to think like humans, definitely have a read!

Why Buddhism is True by Robert Wright

Amazing. Recommended to me by my friend Julia. Probably one of the best books I’ve ever read, really; right up there with Christopher Alexander’s “Nature of Order” and “The Timeless Way of Building”. This book is particularly neat, of course, because it takes a scientific approach to making sense of buddhist ideas. Overall, I find lots of agreement between this book, and the ideas of Christopher Alexander, and Surfaces and Essences.

Inferior by Angela Saini

Pretty interesting. In this book Angela debunks many myths about the supposed biological differences between men and women. This is the first book of this kind that I’ve read; it won’t be the last.

How Adam Smith Can Change Your Life by Russ Roberts

I read this because my partners dad had it in the house. It’s pretty good; it’s really interesting to see the thoughts from someone from so far back in history. For me, this book resonated with the ideas I read in “Why Buddhism is True”. Worth reading if you happen upon it.

Siddartha by Hermann Hesse

I was not a fan of this. Recommended to me by my partner; I found it to be way too “on the nose” about enlightenment. I felt like I was being dragged around to understand certain things, in an obliquely mystical way. Maybe it’s because I read it after I read Why Buddhism is True; which is much more direct and scientific.

The Last Lecture by Randy Pausch

Quite good; but quite “American”. If you don’t like too much american-style dreams of success and culture, you might not like this. There’s also the video of the talk that inspired it that is well worth watching.

I watched the video way back in the day; and thought I’d finally read the book. I wasn’t disappointed, but I’m not sure I got much new out of the book that I didn’t get out of the talk. Still, worth grabbing if you got something out of the talk.

The Big Picture by Sean Carroll

Not great. Before I’d read this, I really had thought I liked Sean Carroll. But, this book comes across to me as very arrogant. As someone who’s read way too many popular physics books, it contained a fair amount that wasn’t new to me (but that’s okay), but also a fair amount of just plain arrogance about what is known, where we can expect progress and what areas are accessible to science.

Lost Connections by Johann Hari

I really enjoyed this. I’ve also suffered (mild, I think) depression, and can really relate to the ideas in this book; namely making connections with people. Probably not the solution for everyone, but interesting reading.

Made by Humans by Ellen Broad

Not bad. Fairly introductory thoughts on the topic, but from someone working in policy on this area. Nice to see an Australian view on these issues. Worth reading it you want a broad overview.

Reality is Not What it Seems by Carlo Rovelli

This is okay. I get very frustrated these days by books that are implicitly sexist; using language like “when man discovered this”, etc, and this book does that a fair bit. I think I was hoping for this to answer more questions than it did; or to be a bit more detailed. I think I probably just misjudged the intended audience.

Your Brain Is a Time Machine by Dean Buonomano

Pretty good. I haven’t read much about neuroscience, and I am really mostly confused about the physics of time, and problems like the so-called “arrow of time”, and such. So I found this to be pretty interesting, and have some good ideas in it that I hadn’t thought about, such as different “clocks” in the body for different purposes.

I Feel You by Cris Beam

Interesting, but odd. I found this book to be worthwhile reading; in particular because it lead me to some interesting ideas around the ethics of AI. Cris goes into weird details of empathy in use in businesses, and life. Overall, worth taking a look at.

How to Change Your Mind by Michael Pollan

Great. Never having taken psychadelics myself, and basically having the view that they are a bit dangerous, this book totally changed my mind. I found it totally fascinating, alongside Why Buddhism is True, in terms of a potential path towards enlightenment. It’s certainly interesting to learn about the history of psychadelic drugs, and think about the connection to mindful meditation.

The Nature of Order - Book 1 - The Phenomenon of Life by Christopher Alexander

Incredible. As we learned at the start, Christopher Alexander is my personal Jesus and this set of books is essentially his bible. As a result, this is amazing reading for me. He describes what it means for something to “have life”; he goes into detail by way of examples, a mathematical model, and a practical, hands-on theory. He breaks it down into what he refers to as “centers”; things that build on each other.

The Value of Everything by Mariana Mazzucato

Amazing. I’m only part of the way through this book, but I love it. We saw Mariana talk at some event a few weeks ago; and she’s an amazing speaker with a incredibly ability to talk with enthusiasm about complicated ideas.

This book is all about “value” vs “price”; and what industries should be considered “productive” in how they contributed to the economoy. In particular it’s really opened my eyes about the finance industry. Amazing reading.

How To Sit, How To Walk by Thich Nhat Hanh

Amazing, and useful. I’ve found these two (small) books really good to read when I’m feeling stressed or overwhelmed. They contain many little thoughts that help ground and relax you.

Fiction

The Power by Naomi Alderman

Pretty good. Best read without too much knowledge of what it’s about. Worth reading.

The Martian by Andy Weir

Awesome. Instant classic. Great fun.

All the Birds in the Sky by Charlie Jane Anders

Pretty fun. A nice and simple read. I enjoyed it.

The Broken Earth Series: The Fifth Season, The Obselisk Gate, The Stone Sky, by N. K. Jemisin

Incredible. The first series of books I read by N. K. Jemisin. I couldn’t put these down, and as soon as I’d finished the first one I bought the remaining two and couldn’t wait to read them.

The Inheritance Trilogy by N. K. Jemisin

Really great. N. K. Jemisin is easily my favourite Sci-Fi author at the moment and I’ll probably read anything she writes. So interesting, unique, and innovative.

The Dreamblood Duology by N. K. Jemisin

Really great. Quite different, really, from The Inheritance Trilogy, but still amazing.

Less by Andrew Sean Greer

Not bad. I picked it up off the table in the bookshop because it had won the Pulitzer Prize, and I figured it might be good. I found it to be pretty good; but utlimately not quite my style.

The Stormlight Archive The Way of Kings, Words of Radiance, Oathbringer, by Brandon Sanderson

Awesome. I discovered Brandon after looking for something new after finishing The Wheel of Time. I’m really enjoying this series, and can’t wait for the next one!

Hope you enjoyed this list! If you have any suggestions of books, I’d be really interested!

January 02, 2019

Indrek Lasn (indreklasn)

Here are some of the best resources to improve your coding skills January 02, 2019 12:49 PM

Photo by Ash Edmonds on Unsplash

There are lots of high-quality coding tutorials, books, and overall resources on the web. I’ve gathered the very best for you and put them in a convenient list. Without further ado, let’s dive in!

Note; The original article was published under strilliant.com — give some love to the original! ❤

Here are some of the best resources to improve your coding skills

Devhints

Devhints has the biggest collection of cheat-sheets on the Internet. Big time saver and a great way to familiarize yourself with all the concepts.

Udemy

Udemy is by far the most popular place to learn to code in my opinion. Dropping $10 to learn new skills which yield huge returns is a no-brainer.

Egghead

Egghead has many courses by industry leaders. Definitely a top-notch bump that’ll take your coding skills to a new level. Courses are both free and paid.

Exercism

Exercism has tons of exercises in most programming languages. Understanding how to approach and solve critical problems is a very valuable trait to have in programming.

freeCodeCamp

freeCodeCamp.org is a tiny 501(c)(3) non-profit that’s helping millions of people learn to code for free. Their curriculum will prepare you to become a great coder. freeCodeCamp teaches both the necessary fundamentals and also makes you code real projects.

Mozilla Developer Network (MDN) Web Docs

If you are a complete beginner, web development can be challenging. The Mozilla Developer Network offers a hold-your-hand approach and provides enough detail for you to feel comfortable and learn the topics properly.

You should feel at home whether you are a student learning web development (on your own or as part of a class), a teacher looking for class materials, a hobbyist, or someone who just wants to understand more about how web technologies work.

Reactdom

Reactdom provides a platform allowing you to learn from the best programming tutorials & books.

Khan Academy

Khan Academy does what it states in its mission. To provide a free, world-class education to anyone, anywhere. Khan Academy is a 501(c)(3) nonprofit organization and offers various web development tutorials.

Eloquent JavaScript

This is a book about JavaScript, programming, and the wonders of the digital. You can read it online for free here.

This should be a great start to your coding journey. Now go out there and start coding!

Here are some project ideas you could build to improve your coding skills;

If you have any questions, I’m actively answering questions on Twitter.

Indrek Lasn (@lasnindrek) | Twitter

Here are some of my previous articles.


Here are some of the best resources to improve your coding skills was originally published in freeCodeCamp.org on Medium, where people are continuing the conversation by highlighting and responding to this story.

January 01, 2019

Nikita Voloboev (nikivi)

Activating things instantly by pressing two keys will be faster than activating Alfred and typing… January 01, 2019 11:48 PM

Activating things instantly by pressing two keys will be faster than activating Alfred and typing something.

December 31, 2018

Átila on Code (atilaneves)

Comparing Pythagorean triples in C++, D, and Rust December 31, 2018 01:16 PM

EDIT: As pointed out in the Rust reddit thread, the Rust version can be modified to run faster due to a suble difference between ..=z and ..(z+1). I’ve updated the measurements with rustc0 being the former and rustc1 being the latter. I’ve also had to change some of the conclusions. You may have recently encountered […]

December 30, 2018

Nikita Voloboev (nikivi)

Karabiner God Mode December 30, 2018 03:29 PM

Karabiner is a low level macOS kernel extension that gives you a virtual keyboard you can heavily customize to your own liking.

I found out about the power of Karabiner by reading an old article on how you could remap caps lock key into a key that’s actually useful. That is you could remap caps lock to act as escape key when pressed alone and some four/five key modifier when held together. With that article, the notion of a hyper (dual purpose) key was born but most people seemed to have stopped there. This article is about how to take this idea of a hyper key and apply it to every key on your keyboard using Karabiner and a special DSL named Goku that makes editing your Karabiner configuration a breeze.

Karabiner Elements uses JSON to define rules which lets you program keys in various ways. The most powerful of which is the ability to create custom hyper keys. Like the useful caps lock I mentioned earlier but applied to every other key on the keyboard. It is not enough to talk about this and praise how game changing this concept is in using a computer until you try it though. So let’s do that.

The problem with using JSON to define Karabiner rules is that it gets unwieldy complex and nearly impossible to edit by hand. Here is a gist of what my Karabiner configuration looks like in JSON format. It’s a 22,692 lines long file. Inside it there are definitions for how to make all of my QWERTY keyboard (minus some letters and delete/return keys because those are too far from the hjkl hand position) into custom hyper keys. There is no way a sane person would open this file and try to modify it in any meaningful way. It contains too much ‘boilerplate’ that Karabiner actually needs to make its magic. When as all you really need is to just point to a key that you want to make a hyper key, then point to a key that when inside that hyper key would do some custom action (run applescript/shell-script/open-app/..).

Fortunately this problem was solved by creating a special DSL that has the minimum amount of syntax needed to define these kind of hyper key definitions and more. Goku tool actually supports all of Karabiner Elements features but abstracts away the JSON mess so you don’t have to deal with it. Here is my 600 line Karabiner configuration written with Goku. 600 line file that generates 20,000+ lines of JSON that not only is a joy to edit but has the nice feature of auto reloading config. Meaning that as soon as I press save on that Goku config file, I have my keyboard modifications applied instantly.

Defining our first hyper key

If this sparked your interest and you are reading this article on a mac. Let’s try this magic in action and you can see for yourself if that’s something you may fancy doing to your computer.

Let’s install all the stuff we need first.

  1. Install Karabiner Elements (Press the big blue button to install latest version)
  2. Install Goku (Karabiner DSL) (brew install yqrashawn/goku/goku)
  3. Run goku service to watch over Goku file and instantly apply changes (brew services start goku)
  4. Open Karabiner Elements (with Alfred/Spotlight) and create a Goku profile

More instructions on how Goku works are in the README alongside its short tutorial that goes over all its features. There is also a Telegram chat where you can ask for help in any Karabiner modification you like to make.

Now that we have everything we need installed and running on our system, we can get to work.

Goku’s config file by default lives inside ~/.config/karabiner.edn. Since I like to keep all my useful config files in my dotfiles dir, I only keep a symlink there and original Goku config is placed inside ~/.dotfiles/karabiner/karabiner.edn.

Make caps lock great again

I have my caps lock mapped to escape key when pressed alone (VIM is life) and ⌥ ⌃ when held. This one is actually quite easy, especially since macOS already provides you with ability to map caps lock to escape key in System Preferences.

Of course we could have done that in Goku configuration itself too but since macOS gives us ability to do it in settings, let’s use that.

Assuming you’ve done that and everything worked, you can now press caps lock key and instead of going into caps lock mode, you will trigger escape instead. Pretty useful given that new macbooks don’t even ship with a physical escape key (although it was too far away anyway and TouchBar is genuinely useful). But holding onto the caps lock key now does nothing useful. Let’s change that.

Open ~/.config/karabiner.edn file in your favorite editor (create it if doesn’t exist). I edit my config with neovim. Then Paste this inside and save the file:

{:simlayers {:caps_lock-mode {:key :caps_lock}},
:main [{:des "capskey",
:rules [:caps_lock-mode
[:open_bracket :!TOopen_bracket]
[:close_bracket :!TOclose_bracket]
[:semicolon :!TOsemicolon]
[:quote :!TOquote]
[:comma :!TOcomma]
[:period :!TOperiod]
[:slash :!TOslash]
[:a :!TOa]
[:b :!TOb]
[:c :!TOc]
[:d :!TOd]
[:e :!TOe]
[:f :!TOf]
[:g :!TOg]
[:h :!TOh]
[:i :!TOi]
[:j :!TOj]
[:k :!TOk]
[:l :!TOl]
[:m :!TOm]
[:n :!TOn]
[:o :!TOo]
[:p :!TOp]
[:q :!TOq]
[:r :!TOr]
[:s :!TOs]
[:t :!TOt]
[:u :!TOu]
[:v :!TOv]
[:w :!TOw]
[:x :!TOx]
[:y :!TOy]
[:z :!TOz]
[:1 :!TO1]
[:2 :!TO2]
[:3 :!TO3]
[:4 :!TO4]
[:5 :!TO5]
[:6 :!TO6]
[:7 :!TO7]
[:8 :!TO8]
[:9 :!TO9]
[:0 :!TO0]]}]}

Goku is still a tool that is in beta so unfortunately the only way to check if your Goku configuration worked without any errors is to type in goku in the terminal. When I do that, I get:

~
❯ goku
Done!

Which means success. If there is a mistake anywhere in the configuration, you will get a nasty stack trace which usually means you forgot to close out some bracket.

If you did get Done! like me printed, congratulations! You now have yourself your first hyper key. Now if you hold caps lock and press another key, you will trigger ⌥ ⌃ + another-key. ⌃ ⌥ + keys are by default empty keys so you have to map them to something. Let’s map one of these keys to one of my favorite mappings on macOS.

Go to System Preferences -> Keyboard -> Shortcuts -> Services -> Find Search With Google action -> Add shortcut -> Press and hold caps lock briefly and with caps lock still held press g key. It should show the same hotkey as I have in screenshot below.

Now select some text in any app (not VS Code because electron) and press caps lock (hold), then press g key. If everything worked, it should make a Google query with the selected text. Pretty damn useful to have this action at your finger tips.

Hope you now got the idea of what these hyper keys are. You have to play a bit with it to get comfortable because if you hold onto your hyper key for long, it will trigger repeating of escape key and not leave you in the hyper key mode. You have to press caps lock key, hold it and instantly (whilst caps lock key is held) press on the activation key (any other key on the keyboard). In the case of our caps lock key, the activation keys will simply trigger our ⌥ ⌃ hotkeys. But macOS and Karabiner is a lot more awesome and powerful than that. Let’s take this power to its limits.

Actual God Mode

To fully appreciate what Karabiner can do for you, you need to buy and use these two very powerful macOS apps. Keyboard Maestro and Alfred.

Alfred is a programmable launcher for macOS. Like Spotlight but better because you can search anything with it and use a wide array of powerful workflows people have built and freely shared for anyone to use. I also wrote an article on how to start writing these workflows in Go.

Keyboard Maestro is an automation tool that lets you create macros and execute them. A macro is a series of actions to do things. It’s like shell scripting for macOS but better because it just makes sense and is intuitive. Here is a simple macro that activates Safari app on macOS.

I ran this macro this many times:

And that’s because I have this macro mapped nicely to w + k keys. So w key is my hyper key and k my activation key. Only instead of some boring hotkey, I now run a KM macro. I have 1,000’s of KM macros I’ve made and use and if you look into my Goku config you will see on 1st line, this code:

{:templates {:alfred "osascript -e 'tell application \"Alfred 3\" to run trigger \"%s\" in workflow \"%s\" with argument \"\"'"
:km "osascript -e 'tell application \"Keyboard Maestro Engine\" to do script \"%s\"'"}

Specifically this line:

:km "osascript -e 'tell application \"Keyboard Maestro Engine\" to do script \"%s\"'"

Creates an alias :km that will call a KM macro by name I pass into it. What it’s doing is actually calling an AppleScript code via osascript shell command (can man osascript if you like). The same thing happens here:

:alfred "osascript -e 'tell application \"Alfred 3\" to run trigger \"%s\" in workflow \"%s\" with argument \"\"'"

Only instead of KM macro, it calls an Alfred Workflow. To call Alfred workflows with AppleScript, you need 2 things. The external trigger name:

And the bundle.id of the workflow in which this trigger lives which can be found in the description of the workflow here:

And here is an example of calling this specific Alfred workflow in Goku:

[:n [:alfred "search lists" "nikivi.learn.anything"]]

Although you can’t just paste this in into Goku config. This just defines what happens when n key is pressed in some hyper key. What hyper key? Well that is defined as a block of code.

For example in my keyboard configuration I activate this Alfred Learn Anything workflow by pressing o key and then pressing n key inside my o key layer. In code that looks like this:

:simlayers {:o-mode {:key :o}}
{:des "okey (alfred)"
:rules [:o-mode
[:n [:alfred "search lists" "nikivi.learn.anything"]]]}

Goku syntax is odd, I agree, it’s optimized for typing less (thus iterating on config faster), not necessarily clarity. But that short snippet actually defines a hyper key only now this hyper key lives on the actual keys you type on. The letter o. So I can still type things away (like this article I am writing) but if I ever want to search curated lists with a workflow, I press o key briefly and whilst not releasing it, press n key. And I get an Alfred prompt show up just as I programmed my keyboard to do this:

Of course my actual o key definition doesn’t just have a binding for :n key. In reality it looks like this:

{:des "okey (alfred)"
:rules [:o-mode
[:1 :!C1]
[:2 :!C2]
[:3 :!C3]
[:4 :!C4]
[:5 :!C5]
[:6 :!C6]
[:w :!CObackslash]
[:v :!CT4]
[:spacebar [:alfred "search PDF files" "nikivi.search.files"]]
[:g [:alfred "search snippets" "com.renfei.SnippetsLab.AlfredWorkflow"]]
[:grave_accent_and_tilde [:alfred "search processes" "com.vitorgalvao.alfred.processcontrol"]]
[:e [:km "Make 2Do task"]]
[:r [:km "Make 2Do task with Safari title as name"]]
[:q [:alfred "search downloads" "recentdownloads.ddjfreedom"]]
[:t [:km "2Do with current url as note (read)"]]
[:b [:km "2Do task (ask)"]]
[:tab [:km "2Do with current url as note (check)"]]
[:h [:km "2Do with current url as note (watch)"]]
[:a [:alfred "search files" "nikivi.manage.wiki"]]
[:y [:alfred "search trash" "com.vitorgalvao.alfred.directories"]]
[:c [:alfred "search emoji" "com.github.jsumners.alfred-emoji"]]
[:i [:alfred "search tty sessions" "net.isometry.alfred.tty"]]
[:f [:alfred "search repos" "net.deanishe.alfred-git-repos"]]
[:n [:alfred "search lists" "nikivi.learn.anything"]]
[:m [:alfred "search lists" "nikivi.awesome.lists"]]
[:s [:alfred "search clones" "com.vitorgalvao.alfred.directories"]]
[:d [:alfred "search desktop" "com.vitorgalvao.alfred.directories"]]
[:j [:alfred "search km macros" "iansinnott.keyboardmaestro"]]
[:caps_lock [:alfred "search folders" "nikivi.search.folders"]]
[:z [:alfred "search workflows" "org.jeef.workflowdirectory"]]
[:k [:alfred "search menu bar" "com.tedwise.menubarsearch"]]]}

Now the description okey (alfred) makes more sense. My o key layer is responsible for calling various Alfred workflows. Making layer mappings by theme makes things easier to remember. For example my w key is used to activate my most used applications (VSCode/Safari/iTerm/2Do/..). Here is how it looks in code:

{:des "wkey (apps)"
:rules [:w-mode
[:k [:km "open: Safari"]]
[:h [:km "open: Activity Monitor"]]
[:i [:km "open: Chrome Canary"]]
[:g [:km "open: Bee"]]
[:t [:km "open: Trello"]]
[:spacebar :!CSO7]
[:comma [:km "open: Spotify"]]
[:caps_lock [:km "open: Finder"]]
[:m [:km "open: Tower"]]
[:b [:km "open: BetterTouchTool"]]
[:r [:km "open: Fantastical"]]
[:e :!OSC1] ; Open Dash
[:semicolon [:km "open: Xcode"]]
[:period [:km "open: SnippetsLab"]]
[:f [:km "open: 2do"]]
[:j [:km "open: iTerm"]]
[:n [:km "open: Sublime Text"]]
[:l [:km "open: VS Code"]]
[:o [:km "open: Keyboard Maestro"]]]}

In this layer, instead of Alfred workflows, I am calling KM macros. Like the one I showed above:

I share all these KM macros I made and use here alongside all the Alfred workflows.

Go wild

I’ve been using Karabiner with custom modifier keys for over 3 years now. I am still in awe that we have this ability to modify our computers to this degree. It’s beyond powerful what you can do with it and I am sad that this isn’t known to the public. Aside from mine and creator’s of Goku configuration, there seems to be no one doing this! And that’s a shame. 😞

Hopefully this article changes it. Because I want to live in a world where everyone pushes the boundaries of what computers can do for us.

If you enjoyed reading this, give it a 👏. And look into other articles I wrote. You can support me on Patreon and get updates on next articles I write on Twitter.💜