Planet Crustaceans

This is a Planet instance for lobste.rs community feeds. To add/update an entry or otherwise improve things, fork this repo.

April 21, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — Adapting: Free Market, Let It Solve Itself April 21, 2021 09:00 PM

The

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 5 — Adapting
  • Section 1 — Free Market, Let It Solve Itself
Table Of Content
  • The Market Will Decide
  • Promoting Fair Competition
  • Privacy as a Feature
  • Changing The Economic Dynamics

In this last part of the series we’ll go over the adaptations we are undergoing to remove the frictions we have with the internet — anything to make it better suited for us. We’ll take multiple perspectives, primarily the ones of users, societies, and others that are encountering the issues we’ve seen related to this new communication channel.

Let’s start by taking the point of view of the market and economy as ways, in themselves, to provide solutions to problems we’ve had.
Keep in mind that these heavily depend on the locality, culture, and practices of the tech companies at the epicenter of the internet drastic changes that are taking over the world. Most of these giants embody a neoliberal mindset and originate from the USA.

A market solution is a solution based on trusting the free market before all else. It hinges on not having to intervene with regulations and laws but to believe the behavior of consumers, their choices and decisions, will eventually lead to an ethical and moral balance.
This laissez-faire attitude is anchored in what we’ve seen in the cultural ambiguity and truth and trust crises sections. They rely on the assumption that individuals understand the dynamics, are well-educated on the subject matter, and will always make the best (economic) decisions overall.
This line of thinking is contested, but let’s consider the adjustments and innovations that can be driven by users.

Market solutions are hardly international as they can only work by relying on fair competition and transparency between actors. Non-intuitively, at least in the eye of some, this can only be achieved through some regulations that keep the market in checks, and such regulations are geographically bound.

When it comes to the online world, because a lot of the money is made from ads and tracking, it means regulating the whole digital advertising tech supply chain, and data brokers along with their partnerships.

If this isn’t done properly, there could be conflicts of interests, some companies self-preferencing advertisements for the services they themselves provide. This is relevant because many of the ad-tech providers also offer other services, which they can promote through their ads to compete with other players.

This problem is exacerbated when only a few entities, which as we know is the case today, have the upper hand (hear by that almost monopoly) in the ad tech sector and data broker space. These companies can then thwart competitors by buying them or competing directly with their services.
The companies will then accrue larger datasets and it’ll be harder to compete with them. This is what we’ve seen in the new economies section. They can control the pricing and quality of services on this digital supply chain.

To promote fair competition, to make it a free market, many things can be done.
One of them is to break the large datasets into subsets to allow new smaller entities to compete instead of having systems of interlocking monopolies.
Another thing that can be done is to manage conflicts of interests, which is what some lawmakers are doing these days. This includes institutions such as the European commission, the Competition and Markets Authority in the UK, the Australian government, and the USA Congress that are now checking for risks of collusions, antitrust, and what can be done for fair competition. We’ll see more of the law and regulatory concerns next.

Additionally, one thing that can foster good competition is to have open standards describing what can be done with ad tech, the format of the data traveling on the pipeline. This includes implementing a common transaction and user ID for consumer and making them portable and interoperable across ad providers.
It would then be easier to port the data between rivals and might even allow consumers to regain control over their data, selecting with whom they’d like to share it. We’ll see more of that when discussing digital identity.
Apple uses something called IDFA, Google uses an analytics and advertising tracking ID, Facebook uses a Pixel ID and others.

Standardization also goes along openness such as the right to repair, to allow products to be repairable to last longer, and decentralization.

However, openness can also be abused for profit. We’ve seen countless times that standards that are initially open can be hijacked. The big players will in turn drive the standard towards more complexity and growth, consequentially kicking out any small player that can’t follow.
This is arguably what happened in the “browser war”, and is still happening today with web standards. The ones contributing the most will control and subdue them, while defending their actions on the basis of openness, improvement, and progress.

History also shows that the same companies that emerged thanks to openness tend to reject these principles when they undermine their strategic position. After gaining a substantial amount of users, after being rooted deep enough, they’ll use the network effect to justify changing their tactic and stance.
A too big to fail scenario.

Obviously, companies can also spend money to defend their interests, for example by lobbying the USA Congress, as we’ve seen in the new economies section.

There are many ways to make money online, on the internet market.
To make profit, as we’ve seen in the new economies section, the most popular models are the ones that go along with what users expect, namely zero-price models where data is exchanged instead of money. This goes under different names such as “free model”, “data-as-payment model”, “online advertisement-based business model”, “ad-supported model”, “freemium model”, etc..
There are other models that are possible, as we’ll see, such as a flat-rate model with unlimited or limited usage, a bundle-like model, and a per-usage or subscription based model where users are enrolled and charged recurrently.

The names for the free models actually have separate particular meanings that reflect how each view and make money from the data, personal data, or attention, which become the currencies of exchange. These models also provide insights into manners of approaching privacy.
The consumers are then free to select the model that they find the most suited to their needs, interests, and ethical concerns.

The ad-supported model, one in which companies make money from selling ads or from gathering data for ad-tech providers, poses the most privacy concerns. It is either targeted advertising, such as microtargeting, or context-based advertising.

A freemium model is one in which the product is offered free of charge and money, a premium, can be paid to access additional features or services.
The free of charge part can be combined as a hybrid with advertisement.

A data-as-payment model, aka PDE or Personal Data Economy, is a model in which individuals deliberately sell their personal data as an asset to pay for services and products. This model turns the surveillance economy upside down.
Similarly, a privacy-discount model is a model in which the users relinquish their privacy to obtain a discount on a product or service.
These two models are user-centric, letting the consumers take back a semblance of control over their data: with whom they share or disclose it, the price for it, and on which terms.

Each model has a different perception of what privacy is, either as a commodity/product/feature, as a luxury/premium, as a right, or as a utility.
In the market view it is primarily a feature or premium that can be transacted.

What we described as data-as-a-payment or privacy-discount are instances of perceiving privacy as a commodity. In these scenarios neither citizens nor governments care about privacy regulations but freely exchange and trade it. In this case, people are powerless about what information is being transferred, while tech companies can make huge profits by selling the data.

Privacy can also be seen as a luxury or something you get for a premium, aka PFP or pay-for-privacy, aka primium aka privacy-as-a-premium. In this scenario, consumers are encouraged to pay extra to gain back their privacy, or to acquire new products that will fill their desire for more privacy and security.
This is a case where companies profit from privacy by charging users for it. Transparency, awareness, and privacy have become competitive features delivered at a premium.
We’ll see that awareness and transparency are two things that can be enforced through laws in the next section.

On the other side, privacy can also be seen as a utility or universal right. Either something the governments regulate, control, and monitor, just like water, a tool — this is what we’ll see later with digital identity. Either as a universal right that needs to be protected by law, which is something we’ll see next too.

The debate over privacy can be confusing for consumers, they may feel torn between all these options. Some of them wanting to disengage with the virtual world to maintain their anonymity instead of putting themselves at risks. Without privacy, we are subject to social cooling, feeling tracked, caught in the polarization, as we’ve seen before.
This confusion is indeed what sparked the whole branch of privacy as a luxury, companies are capitalizing on it, for them it’s a new business opportunity.

Privacy is not only a feature in itself but also a stand, a brand, a show of force for publicity and marketing differentiation.
Many companies sole existence rely on this need: selling something at a higher price mark because of this or having products, devices, services, and tools, that will offer control back to users.
The market size of this type of business, according to Arthur D. Little, is estimated to be, as of 2020, around $5 to $6 billion.

VPN providers are one category of these flourishing businesses. They sell a service to avoid the continuous online tracking and data collection. A VPN being a proxy to internet requests, essentially hiding where they are originating from. Some commentators have suggested that all consumers should use a VPN.
In general, their model is based on offering a subscription, a pay-for-privacy product with recurrent payment. Though, ironically, some VPN companies also provide their services for free through the hybrid “ad-supported” freemium model.

Another company profiting from privacy is the device and OS manufacturer called Apple that sells products at a premium. It has recently (2021) stirred uproar from American companies that make money from the ad tech infrastructure and pipeline when it announced a recent update to their phone, tablet, and TV OS. Apple is a popular product in the USA among high earners, thus good ad targets.
The recent update would notify users when applications want to access location, track them across apps and websites, or others. The user would have the choice to allow or disallow such action.

The move is arguable, as Apple makes a lot of money from the ~30% cut (the Apple tax) it takes from in-app purchases, disallowing any other type of external purchases linked to apps. Such a change would force app developers relying on the ad model to switch to another model such as subscription or in-app purchases, which they’d take a percentage of.

As can be seen, the free market and laissez-faire approach brings with it a lot of conflicts. When big players can enforce business models on their platforms, after using the network-effect to acquire users, it enrages other players.
The dynamics at play are interesting.

When privacy is offered at a premium it raises concern about the inequality of acquiring it, a socioeconomic digital divide. “Advertising is a tax on the poor”, as Scott Galloway said. It creates incentives for the companies offering privacy and security-centered products to keep the status-quo, to make more money from it.
There’s also the lack of consumer knowledge and understanding on these issues and their implications. They could either be unaware that these are issues, or be misled by marketing schemes portraying products as privacy-aware while not providing any proof to back it up.
Privacy can be advertised on the front while in the back tracking is done. The tracking, the telemetric gathering, is not limited to advertisement purposes but could also be used for partnerships with other entities that would allow companies to gain an edge. For example, a partnership with state actors, internet providers, or mobile operators.

Let’s remember that companies are not moral agents, nor arbiters of truth, and that their decisions are both restricted by, on one side, the legislative geography they are operating on, and on the other side, the profit made from the appearance and satisfaction they offer customers. Ethics is only a cosmetic feature.
This can be hard to grasp for the neoliberal mind that tries to find their meaning through the purchase of products, as an extension of themselves. Especially when these products only provide that in return of profit.

One solution for consumers is to go toward models that remove data from the equation, thus changing the economic incentives for online businesses. These alternative models rely on putting real value on what we consume and use online, going back to transacting money for intangible assets.

While the free models were disruptive when they came out, it turns out, unsurprisingly, that consumers don’t like being harassed by constant ads.
Tech aware users will often install adblockers on their machines, rely on one of the above paid solutions to not be tracked, use browsers that block third-party cookies, and other mechanisms. We’ll dive more into the tech solutions later. Statistics show that in the USA 47% of users use adblockers.

This kills companies that rely mainly on the ad model. Either because they aren’t able to attract enough users to generate revenue, or because it damages the brand image.
Additionally, companies don’t want to be at the mercy of the ad tech monopoly, which puts them at risks when they decide on the budget and price they allocate for specific ads and on which platforms they’ll run them. They could demonitize them at any time or disallow them on the ad platform for arbitrary reasons.
Furthermore, even companies paying for ads are starting to see the limits of this model because of the lack of good insights, bots, and fake information that return to them.

For these reasons, we’re seeing the rise on the internet of the subscription economy, micro-transaction economy, passion economy, and the economy of patronage.

We already discussed the passion and patronage economy in part 2 section 2 so let’s focus on the others. All these are about putting the consumers in power, letting them make decisions through their wallet. In this case we’ll have a consumer-controlled world, thus a working free-market solution.

There are different types of subscription models, either based on usage volume, or based on time via a billing cycle. It stands in contrast with a pay-per-product model where users would pay only once, in subscription the payment is recurrent.
Consequentially, this works well for products that either perish quickly, need to be replaced, e-commerce loyalty programs, or intangible goods such as the ones provided by SaaS (Software as a Service) companies.
These include Netflix, Amazon Prime, Hulu, YouTube, Spotify, SirusXM, Pandora, VPN services, among others.

The subscription model has been estimated in 2017 to have 28k companies worldwide and had more than a 100% yearly growth. The largest retailers in the category generated more than $2.6 billion in sales in 2016, up from $57 million in 2011.

The model has the advantage that it is flexible and provides customer satisfaction. Buyers aren’t burdened by large fees and can stop the subscription at any time.
The companies providing the services are then forced to innovate, improve, and provide quality to be able to attract and keep customers. There are many instances that show that switching from an ad-based model to a subscription model has helped increase revenue.

Here the customer is at the center and their expectations are king. For this model to work the company has to build a long-term relationship with them, often focusing on the relationship more than the product or the transaction.

Nonetheless, it isn’t an easy task. A minimum number of subscribers is needed for the business to subsist, and it’s not easy to attract them when netizens are used to getting things for free.
This has especially been hard for newspapers, as we’ve said in the past. Statistics show that in the USA in 2017, only 11% of consumers were paying for online news. The rest of the world isn’t as generous either, the statistics have been replicated in other places with an average of 10% paying for news. Yet, recently we are now seeing an unexpected resurgence, a new growth. It has been reported that in 2017 the American newspapers The New York Times gained 130k subscribers and the WSJ gained 300%, the British newspaper the Guardian has also announced that it had seen profits for the first time after 20 years with more than 300k people making contributions.

The most successful stories of subscription based businesses are those that relied heavily on partnerships — between brands, consumers, affiliates, and others.
Despite all this, companies relying on the subscription model can still rely on product placement and advertisements, which puts us back at square one.

Another method to increase the number of customers, to persuade them of contributing, is to lower the barrier of entry. This could mean making it more affordable, or even to change the value of the product based on the purchasing power of the individuals or their locality.

The new disruptive approach is to make the payment infinitesimally small, a micro-payment, a micro-transaction, which can be fractions of cents, there’s almost no line of demarcation.
This can create a dynamic in which ads and trackers are unnecessary because users would pay directly with these monetary tokens the online content creators. The tokens essentially being the monetary form of the attention of users.

The word micro-transaction refers to purchasing virtual goods through micro-payments, however it also has a historical meaning attached to the gaming world.
Initially micro-payments acted as a sort of virtual currency that existed only within games.
Later on, some game developers used a model in which they provide virtual desirable assets, such as loot boxes, in exchange for micro-payments. Micro-transactions have thus gained a connotation with gambling and addiction, which brings some countries’ legal matters and regulations in the mix especially when it involves kids.

The main issue with micro-transactions is that they can only be possible if there is a monetary technology that allows real-time transfer, no fees, and privacy.
Cryptocurrencies on a blockchain seem to be the go-to solutions for this. They are far away from the traditional pipeline that is also linked to data brokers, as we’ve previously seen.

Unfortunately, some of the cryptocurrencies have issues regarding scalability and volatility, which makes them ill-suited for such endeavor. Cryptocurrencies are now more attached to investment and gambling than an internet free of ads.
Recently, payment gateways and banks have taken a particular interest in trying to gain power in this space. Namely, VISA and PayPal are now allowing certain cryptocurrencies to be used and converted on their processing platform. This would then bring information related to cryptocurrency to data brokers, removing the privacy aspect.

Still, the applications for micro-payments open avenues and unlock an entire economy between subscriptions and ads.
We’re slowly starting to see this appear in specialized browsers that allow to distribute attention tokens to websites and content creator, and newspaper that want to allow micro-transactions along with their other revenue streams.

This concludes our review of how the free market can be used as a method to adapt to the internet. First we’ve seen what this idea implies: well-informed customers making the right choices to balance the status and set things right. Then we’ve explored how this cannot be true if there aren’t regulations set in place for fair competition to happen between companies. Next, we’ve discussed some of the possible regulations and things that could be put in place to achieve this such as collusion checks and open standards. After that we’ve set ourselves on looking at different business practices, ways of making money online. Some of them are related to the ad business, others subscriptions. We’ve tackled how these models offer a glimpse into what privacy is considered: product, luxury, tool, universal right. When privacy is a product it’s traded freely. When privacy is offered at a premium, which we called primium, it’s done because of a need and not because of moral or ethical standards. We’ve listed some businesses that base their marketing or product around privacy. Finally, we’ve seen what removing data from the equation implies: having customers pay again to be able to choose, either through subscriptions or micro-payments.

Table Of Content

References








Attributions: S. Trismosin, Splendor solis, London, 16th century

April 20, 2021

Sevan Janiyan (sevan)

HFS “Incorrect number of thread records” error April 20, 2021 11:28 PM

Note: this post doesn’t provide a solution to address the “Incorrect number of thread records” file system issue but documents what I went through to see if I could when I faced it. I think your best bet would probably be to try ALSOFT’s Disk Warrior if you need to fix this issue. I was …

April 19, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Big Picture: Mass Hypnosis or Mass Psychosis April 19, 2021 09:00 PM

The creation of the world

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 4 — The Big Picture
  • Section 3 — Mass Hypnosis or Mass Psychosis
Table Of Content
  • Relations With Well-Known Dystopia
  • Attention, Awakening
  • Mind Control and Mass Hypnosis
  • Alienation and Mass Psychosis

The internet brings with it technological advancements that remind us of dystopia that sci-fi writers have talked about. So let’s go beyond what we’ve tackled so far and project ourselves in hypothetical futures to posit bigger consequences.

The two archetypical stories of dystopian futures are Brave New World by Aldous Huxley and 1984 by George Orwell.

In Brave New World, Huxley presents a hedonistic future that is governed by techno-science and eugenics. The people conform and adhere to the societies rules for the following reasons:

  • The future of people is predetermined based on their genetics, their place is chosen and what makes them happy too.
  • The people are conditioned in their sleep, while unaware, to find certain perspectives more enticing and to always be happy and satisfied of their position.
  • It is a future in which pleasure and happiness are everywhere, and it makes everyone prone to accept the status-quo. Who will rebel against pleasure?

We can see some parallel with neoliberalism, consumerism, and the passivity or paralysis we’ve seen earlier. However, we’ve also seen how people can’t deal with such world of templatized lives, even when pushed by the internet.

Mass production, homogeneity, predictability, and consumption of disposable consumer goods.

In 1984, Orwell presents a world of constant surveillance by Big Brother. This surveillance is extended through self-surveillance between individuals with a thought police. The world described is believed to be in perpetual war (forever war), historical facts and news are rewritten or negated to fit the accepted narrative, and language is used in a powerful way to accompany it.
The fictitious world employs many of the propaganda techniques and the eunemics we’ve encountered.

We can see some parallel with the previous sections related to the surveillance economy, the return of the collectivist narrative, and the internet wars of memeplexes that use censorship, call-out, canceling, and social cooling to keep everyone in check.

In one of these worlds facts are eliminated and in the other facts don’t matter. Today we see both of them, censorship and speech itself as a censorial weapon. Two techniques used to make people docile and well-behaved, two techniques that are omnipresent on any communication channel but even more on the internet. We’re in the center of an information warfare.

Beyond dystopian worlds, the attention economy on the internet carries with it questions related to the very nature of consciousness.

When we talk of attention we also indirectly refer to themes such as awareness and awakening. These are then tied to spiritual practices. For instance, how we pay attention is an inherent part of Buddhist practices.
Hence, attention can be said to be the outermost expression of our consciousness. Losing control of it would be the equivalent of losing oneself or being detached from reality and be brought into a fabricated universe.

In an attention economy, we would then live in memeplexes of ideas, packaged, wrapped, and opened effortlessly to let us access these collapsible concepts, these portable philosophical ideas, stories, emotions, narratives, and meanings.
We are what we focus on, we live what we are aware of. Our reality is limited by our perception of it.

We all think that the way we live is the best way to live. We all think that we see the world as it is, that we’ve come to the best possible conclusion about our direction in life. However, that’s just our definition of normality. And this normality isn’t even ours as most of it happens without us being there. We often can’t justify why we do what we do or why we like what we like.

When we spend so much time on the internet, and will possibly spend much more in the future, we’re indirectly unaware that it is defining us.
Looking around us, for some it might appear more explicit than others as they get caught in apparent internet cults or are memeoids combatant of their memeplex, but aren’t we all?
Isn’t most of the content that is fed back to us a reflection of who we already are? And as the internet takes a bigger part of our lives, will everyone get stuck in loops of paralysis like we’ve seen before?

One question we can ask is whether it’s possible to engineer these wants, to use mind control or hypnosis to make people more inclined to give their attention or perform specific actions.

A far-fetched, or not so far-fetched, concept from sci-fi is a psychotronic weapon or electronic harassment. This abstract weapon is a representation of the totality of all possible methods and means, be them suggestive, pharmacological, paranormal, technogenic, or others that influence the psyche of a person for the purpose of modifying their consciousness, behavior, or health.
While this idea is often associated with lunatic conspiracy theories, it’s interesting to tinker about the possibility that the internet could indirectly be used in the future, or now, for such ends.

One way to be able to achieve this would be to get a direct interface to the brain. The neurotechnology for brain scanning, which used to be limited to laboratories, is slowly getting more accessible, transportable, and less invasive. There are instances using electrical activity circuits, rhythmic sound or flashing light, ultrasonic or magnetic simulation, all to modify brain waves and patterns of electrical activities in the brain. These can work both as reading devices or output devices.

Today, these are still used mainly in labs to treat neurological and mental illness, and improve mood and cognition. The USA FDA has approved in 2008 transcranial magnetic stimulation to treat depression, pain, and migraine.

However, we can imagine that when this becomes more mainstream, when brain-related data starts to be common on the internet, that ad-tech companies will jump on the opportunity to access whatever comes out of these new body sensors.
There is nothing more private than thoughts. This can give the ability to know and predict what someone’s mental dispositions are, with whatever this implies: from knowing their cognition style, to identifying cognitive strengths and weaknesses, to perceiving personality traits, and to determine their aptitudes or likeliness for certain information.
When in the wrong hands this could be called a neuroweapon.

Having this type of information in the wild would be the most advanced form of social cooling, full-on paralysis, a forced “psychocivilized” society, not far from Huxley’s brainwashing dreams.
This is why researchers and human rights activists are starting to think about adding new rights: cognitive liberty, mental privacy, mental integrity, and psychological continuity.

Another method to alter minds that has been tried in the past is by using chemicals and pharmaceuticals drugs. The research has shown, at least as far as the public knows, that they aren’t very effective and their applicability comes with a lot of technical difficulties.

Some mind-altering and stimulating chemicals are already readily available on the market of most countries. Others such as hallucinogenic mushrooms, marijuana, heroin, LSD, and truth serums aren’t as available everywhere but the intelligence community showed that they weren’t effective when it comes to applying a particular military objective or operational deployment. This is because their effects isn’t reliable nor the information that people give when under them — people could become drowsy and spurt out fictitious stories.
Yet, we can imagine that these can be used without a particular goal but with a more generic approach. This is why some of these are now being introduced and tested as psychiatric drugs to treat anxiety, depression, or mania. They force people to enter new states of minds and emotions. Many of them work by returning overactive or underactive neural networks and signal to more “normal” levels — a reset button on the brain.
These drugs can also engender the opposite effect and actually increase confusion, anxiety, and depression. Some military have been using them for torture to force the person to cooperate or surrender.

An additional topic in relation to mind control is hypnosis.

Hypnosis is a state of awareness where someone detaches their of attention from their environment and get absorbed by their inner experience, their feelings, imagery, and cognition. These inner experiences and imageries are induced by a clinician that will guide the process.
The imagined feels real in that state, bridging the gap between the left and right hemispheres, the hypnotic reality. Similar to a trance state or meditative state, an obliteration of the ego.

Three things happen in the brain of the hypnotized person:

  • The activity in their salience network decreases, a place in the brain related to worry, so they get absorbed in the moment.
  • The brain-body connection increases.
  • There’s a reduction, even a disconnect, between the actions and the awareness of them.

As we said, this can be seen abstractly as a link — by using imagery — between the left-intentional brain (logical, rational, abstract, critical), and right-involuntary-state brain (emotions, feelings, instinctive, intuitive).

In practice hypnosis is studied and used along other forms of therapy such as CBT (Cognitive Behavioral Therapy) to fix misbehavior. It is used to increase the effectiveness of suggestion, give access to the mind-body link and unconscious processing. Alone it is not very useful, but it facilitates other kinds of therapies.

Hypnosis can be self-taught but it is most often administered by a clinician. It starts with a relaxing phase where the patient focuses their attention, giving it away, an induction step. This could be done in multiple ways, be it a visual focus or auditory one. The next step is a suggestion phase which consists of the clinician guiding the patient by helping them create a visualization of events and scenarios, be it verbally or using imagery, to help them address or counteract unhelpful behaviors and emotions.

In a sense, this is very like guided meditation where someone sits in a relaxing position, calmed by a mantra or sound, and is guided by a master that will help them address their inner woes.
“In hypnosis, you’re so absorbed that you’re not worrying about anything else.”

However, studies show that only a minority of the population is hypnotisable, about 10-20% are highly receptive to it. It has been shown that hypnotisability is a genetic trait and follows a Gaussian or bell-shaped distribution. So most research related to hypnosis focuses on this part of the population.

Now, related to our main topic, this doesn’t seem like mass hypnosis would be possible, nor very actionable either. Yet, we can imagine a world in which daily activities, wandering online, becomes a trance state and where imageries are transmitted using compacted packages such as memes.
Still, that is far from the truth as most studies show that it is almost impossible to make someone do something against their will. Moreover, only an extremely small part of the population would be both willing to do “evil” biddings and also be susceptible to hypnosis. But wouldn’t these fringe people be easily findable on the internet and band together anyway?

Homo homini lupus.
Men is a wolf to men.

When the psychology of a whole population changes it can create a deleterious unstoppable snow-ball effect.

Indeed, it is becoming ever more obvious that it is not famine, not earthquakes, not microbes, not cancer but man himself who is man’s greatest danger to man, for the simple reason that there is no adequate protection against psychic epidemics, which are infinitely more devastating than the worst of natural catastrophes. — Carl Jung

When a society loses the grip on reality, becomes driven by uncontrollable fear, when mental illness becomes the norm rather than the exception, social alienation takes over, a psychic epidemic.

All one’s neighbours are in the grip of some uncontrolled and uncontrollable fear… In lunatic asylums it is a well-known fact that patients are far more dangerous when suffering from fear than when moved by rage or hatred. — Psychology and Religion

Mass psychosis happens when a large portion of society descends into delusion, similar to what happened during the European witch hunt.
It is a paranoia taken to the extreme caused by a rise in anomie, societal stress, and a flood of negative emotions.
“A mysterious, contagious insanity”.

We can already see how this can be amplified on the internet where these types of emotions, ambiguity, confusion, fear, and doubts thrive. We’ve discussed amply the cultural insecurities and cognitive dissonance in the previous sections.

We can also wonder if these can deliberately be induced in another population through PSYOPs.
Don’t we also see the rise of this phenomenon through the memeoids and their memeplexes, taken over by ideas so powerful that they possess them, consume them, even destroy them? Mobs and self-justice carnage and destruction.
Moral absolutism, and categorical imperatives are cognitive distortions because most of reality and the living of everyday life takes place in the gray area, between the extremes.

Those who control the flow of information in a society, the ideas we accept as true or false exert a great power over the course of civilization.

This concludes our review of future societal meltdown, be them hypothetical sci-fi scenarios or realistic ones. We’ve started by taking a look at the relation with well-known dystopian stories such as Brave New World and 1984. We discussed the parallel with today’s internet, censorship and speech itself as a censorial weapon. Then we’ve talked about the deep inter-relation between attention, awareness, and consciousness. We’ve said that a control of attention is a control of reality but that online we’re often only given back what we already want. Next, we’ve wondered about different ways to engineer the “wants” of a person, from mind-control weapons, to getting access to data taken directly from the brain, to drugs and chemicals, to hypnosis. Finally, we’ve discusses mass-psychosis, the state of a society where mental illness becomes the norm and reality is distorted to fit an invented narrative.

Table Of Content

References








Attributions: Hieronymus bosch, outer wings of the Garden of Delights, c. 1510

Marc Brooker (mjb)

Tail Latency Might Matter More Than You Think April 19, 2021 12:00 AM

Tail Latency Might Matter More Than You Think

A frustratingly qualitative approach.

Tail latency, also known as high-percentile latency, refers to high latencies that clients see fairly infrequently. Things like: "my service mostly responds in around 10ms, but sometimes takes around 100ms". There are many causes of tail latency in the world, including contention, garbage collection, packet loss, host failure, and weird stuff operating systems do in the background. It's tempting to look at the 99.9th percentile, and feel that it doesn't matter. After all, 999 of 1000 calls are seeing lower latency than that.

Unfortunately, it's not that simple. One reason is that modern architectures (like microservices and SoA) tend to have a lot of components, so one user interaction can translate into many, many, service calls. A common pattern in these systems is that there's some frontend, which could be a service or some Javascript or an app, which calls a number of backend services to do what it needs to do. Those services then call other services, and so on. This forms two kinds of interactions: parallel fan-out, where the service calls many backends in parallel and waits for them all to complete, and serial chains where one service calls another, which calls another, and so on.

Service call graph showing fan-out and serial chains

These patterns make tail latency more important than you may think.

To understand why, let's do a simple numerical experiment. Let's simplify the world so that all services respond with the same latency, and that latency follows a very simple bimodal distribution: 99% of the time with a mean of 10ms (normally distributed with a standard deviation of 2ms), and 1% of the time with a mean of 100ms (and SD of 10ms). In the real world, service latencies are almost always multi-modal like this, but typically not just a sum of normal distributions (but that doesn't matter here).

Parallel Calls

First, let's consider parallel calls. The logic here is simple: we call N services in parallel, and wait for the slowest one. Applying our intuition suggests that as N increases, it becomes more and more likely that we'll wait for a ~100ms slow call. With N=1, that'll happen around 1% of the time. With N=10, around 10% of the time. In this simple model, that basic intuition is right. This is what it looks like:

The tail mode, which used to be quite rare, starts to dominate as N increases. What was a rare occurrence is now normal. Nearly everybody is having a bad time.

Serial Chains

Serial chains are a little bit more interesting. In this model, services call services, down a chain. The final latency is the sum of all of the service latencies down the chain, and so there are a lot more cases to think about: 1 slow service, 2 slow services, etc. That means that we can expect the overall shape of the distribution to change as N increases. Thanks to the central limit theorem we could work out what that looks like as N gets large, but the journey there is interesting too.

Here, we're simulating the effects of chain length on the latency of two different worlds. One Tail world which has the bimodal distribution we describe above, and one No Tail world which only has the primary distribution around 10ms.

Again, the tail latency becomes more prominent here. That relatively rare tail increases the variance of the distribution we're converging on by a factor of 25. That's a huge difference, caused by something that didn't seem too important to start with.

Choosing Summary Statistics One way that this should influence your thinking is in how you choose which latency statistics to monitor. The truth is that no summary statistic is going to give you the full picture. Looking at histograms is cool, but tends to miss the time component. You could look at some kind of windowed histogram heat map, but probably won't. Instead, make sure you're aware of the high percentiles of service latency, and consider monitoring common customer or client use-cases and monitoring their end-to-end latency experience.

Trimmed means, winsorized means, truncated means, interquartile ranges, and other statistics which trim off some of the tail of the distribution seem to be gaining in popularity. There's a lot to like about the trimmed mean and friends, but cutting off the right tail will cause you to miss effects where that tail is very important, and may become dominant depending on how clients call your service.

I continue to believe that if you're going to measure just one thing, make it the mean. However, you probably want to measure more than one thing.

April 18, 2021

Ponylang (SeanTAllen)

Last Week in Pony - April 18, 2021 April 18, 2021 11:07 PM

The supported version of FreeBSD has switched to 13.0 for ponyc and corral. The Roaring pony folks met again.

Derek Jones (derek-jones)

Another nail for the coffin of past effort estimation research April 18, 2021 09:32 PM

Programs are built from lines of code written by programmers. Lines of code played a starring role in many early effort estimation techniques (section 5.3.1 of my book). Why would anybody think that it was even possible to accurately estimate the number of lines of code needed to implement a library/program, let alone use it for estimating effort?

Until recently, say up to the early 1990s, there were lots of different computer systems, some with multiple (incompatible’ish) operating systems, almost non-existent selection of non-vendor supplied libraries/packages, and programs providing more-or-less the same functionality were written more-or-less from scratch by different people/teams. People knew people who had done it before, or even done it before themselves, so information on lines of code was available.

The numeric values for the parameters appearing in models were obtained by fitting data on recorded effort and lines needed to implement various programs (63 sets of values, one for each of the 63 programs in the case of COCOMO).

How accurate is estimated lines of code likely to be (this estimate will be plugged into a model fitted using actual lines of code)?

I’m not asking about the accuracy of effort estimates calculated using techniques based on lines of code; studies repeatedly show very poor accuracy.

There is data showing that different people implement the same functionality with programs containing a wide range of number of lines of code, e.g., the 3n+1 problem.

I recently discovered, tucked away in a dataset I had previously analyzed, developer estimates of the number of lines of code they expected to add/modify/delete to implement some functionality, along with the actuals.

The following plot shows estimated added+modified lines of code against actual, for 2,692 tasks. The fitted regression line, in red, is: Actual = 5.9Estimated^{0.72} (the standard error on the exponent is pm 0.02), the green line shows Actual==Estimated (code+data):

Estimated and actual lines of code added+modified to implement a task.

The fitted red line, for lines of code, shows the pattern commonly seen with effort estimation, i.e., underestimating small values and over estimating large values; but there is a much wider spread of actuals, and the cross-over point is much further up (if estimates below 50-lines are excluded, the exponent increases to 0.92, and the intercept decreases to 2, and the line shifts a bit.). The vertical river of actuals either side of the 10-LOC estimate looks very odd (estimating such small values happen when people estimate everything).

My article pointing out that software effort estimation is mostly fake research has been widely read (it appears in the first three results returned by a Google search on software fake research). The early researchers did some real research to build these models, but later researchers have been blindly following the early ‘prophets’ (i.e., later research is fake).

Lines of code probably does have an impact on effort, but estimating lines of code is a fool’s errand, and plugging estimates into models built from actuals is just crazy.

April 17, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Big Picture: Truth & Trust Crises April 17, 2021 09:00 PM

A. Kircher, Ars magna lucis, Amsterdam, 1671

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 4 — The Big Picture
  • Section 2 — Truth & Trust Crises
Table Of Content
  • Individualism, Neoliberalism, And Relativism
  • Lack Of Trust — Doubt As The New Normal
  • The Collapse Of journalism
  • Mobs & Self-Justice

Nobody likes to be put in a box, nobody likes to be paralyzed. In a world where the individual is put first, we want to feel in control. Yet, we are lost in a whirlpool of chaotic and clashing ideologies, memeplexes and absolutist templates. Who are we, how should we define ourselves? This makes us distrust any form of authority and renounce classical forms of media. As individuals, we’re looking for authenticity, whatever form it can take even if disingenuous, so that we can find our new roots, our new base. Then, there’s nobody better suited to affirm justice than internet mobs.
It feels like the internet is the new far-west, a lawless land of incessant doubt, lacking meaning and trust, where gurus, inspirational influencers, and vigilantes reign and data, personal data, and metrics of reputation are the currency held by the banks of social media.
Or it may also feel like eunemics in action — the deliberate improvement of the meme pool — and people wanting to take part in the cultural meme wars and genocides.

A lot of this can be attributed to the wide spread of neoliberalism values and mindset, which as we’ve seen in the previous section are the most prevalent online. Particularly, we’re faced with individualism, relativism, and inequalities all justified by the protection of personal rights.

If you see a world full of individuals then you’ll want a morality that protects those individuals and their individual rights.

You’ll emphasize concerns about harm and fairness.

But if you live in a society in which people are more likely to see relationships, context, groups, and institutions. Then you won’t be so focus on satisfying individuals, you’ll have a more sociocentric morality.

Which means that you place the needs of groups and institutions first, often ahead of the needs of individuals.

If you do that then a morality based on harm and fairness won’t be sufficient. You’ll have additional concerns. And you’ll need additional virtues to bind people together.

The Righteous Mind

The individual becomes reason in and of itself, and not only a figurant in their own life. We are expected, in the neoliberal theory, to develop our own opinions, outlook, stances through a considerable amount of solitary reflection. However, most important issues, such as the definition of our identity, happen over dialogue. We can’t define ourselves or recognize ourselves, in a void.
We’ve discovered how this can be beneficial in part 2 section 2, as networked individuals connected by interests. Interests become part of our identity and how we generate meaning.

On the internet, however, this leads to struggles as we’re incessantly exposed to a wide range of ideas, some of which may be breaking our local taboos. The Overton window keeps moving, for the better or the worst.
Isolated societies, communities, and ways of living may be forcedly assailed by ideas outside their comfort zone. We’ve touched this topic in the cultural malaise section.

I went a bit over this in my article entitled cross culture frustration.

The new mentality appears different and frightening — different acting, different social dynamics, different manner of building the “self”, different definition of who we are within the society. In addition, we hate being wrong, and we like confirmation. We create a cocoon and hide ourselves in it repeating the hokum over and over again. And we are afraid of the unknown — afraid that the stable self we’ve built to be part of a society is not a self that could partake and feel comfortable in the activities of another society.

In this mindset, the individual always comes before the large collective forces and needs to be protected against them. Personal life choices can’t be criticised as this would be tantamount to an assault on personal dignity. Inherently, this carries the acceptance of a soft moral relativism, any opinion should be respected and nobody should impede on others.
This translates on the social plane into political acts for individual rights, recognition, and equality. Anything else is understood as inflicting damage and being a form of oppression. We are more interested in dignity, universals, innate attributes, and equality.

Additionally, this requires the breaking of hierarchies and a loss of a grand sense of meaning. The perception is now centered on the self to find these, rejecting, and even erasing from awareness, issues and concerns that transcend the self.
We aren’t recognized through honors, distinctions, or any rare achievements, but by our self-generated worth. We’ve replaced these forms of recognition, the need for belonging and authenticity for example, by other things.

Personal and intimate relationships are now the definition and emphasis of life. We now expect to find the greater meaning, which was previously found in institutions and traditional structures, in these relationships. The domestic becomes the loci of “the good life”.
This is the core of what social media are about. They are filling a cultural gap as we said before.

Self-fulfillment and worth is also found through the instrumental reasoning of neoliberalism — actions driven by profit, optimization, market, and metrics. Meaning lies in efficiency, success is found in numbers, tangible assets, products, and money. In a way, it has replaced grand narratives and sense with short-term narcissistic, selfish, pleasures and pains, with little regard for others or society as a whole.
Similarly, the intimate relationships are also seen as tools for self-development and so are also serving an end.
It permeates every aspect of life. Even simple things such as food have become an individual matter, for personal consumption, rather than a convivial mean of regrouping and conversations.

Old social orders are shattered and replaced with resources for projects, economic growth, and prestige in technological progress.

No empathy. No loyalty. No forgiveness. Thanks to the market, the old-fashioned virtues have been rendered obsolete.

Common goods and social objectives are de-emphasized. The protection of the individual is only an extension of the protection of their private economic rights. The new morals are the morals of protecting the individual and their economic rights, the sanctity of the individual and the pursuit of self-interest as the highest moral ideal.

However, this system generates a lot of inequalities, there’s a wide distribution of wealth and income. It is hard to keep competition open and fair — a competitive attitude that also fosters more individualism. Corporations are free to self-regulate.
Comparativeness and cooperation are both part of human nature, but the one that is valued and promoted by the culture will thrive.

On the internet we’re presented with the myth of individual geniuses working in their garages without any investment, self-made success stories. Yet, this is only a myth.

And because worth is found in such success, it is then displayed in clear light for everyone to admire. Social media make this even more apparent.
These inequalities will undeniably boil into social unrest. Individual persons will demand their individual rights and feel entitled to do so.

Overall, there’s a sentiment of being stranded on an island, lost, afraid, and missing something vital. Many societies embracing or exposed directly or indirectly through the internet (or other political pressures) to neoliberalism have a hard time coping with this. The competitive and instrumental mindset encourages a fear of the other. The moral relativism destroys grand narratives, old societies heroes, and replaces them with ambiguity and an identity crisis. The cultural malaise, the feeling of anomie, we’ve discussed earlier.
We’re left with defining ourselves through interests, intimate relationships, metrics, economic activities, and a moral that only has one thing to say: “everyone is on their own, let them be”.

More than anything, this damages and erodes our trust in any form of authority, this makes us doubt what we know and what others are telling us. We’ve seen how this is abused on social media in a previous section.

When reality doesn’t reflect our projections, it will backfire. Turning things upside down, changing love into hate, or any other ways of rebalancing our perceptions.
We project on people, ideologies, cultures, and politics, wanting to put our trust in something, to see them as extensions of us, to feel that our conception of reality is stable and secure.
When it doesn’t match the people revolt.

On social networks we are torn between different narratives that threaten each others. There’s no accountability or regulation for this user-generated content and it is often published under aliases that can’t be traced to any source.
Truth and trust are bent to their extreme. The digital revolution undermines mainly the control that governments used to have on them, replacing it with a technological alternative of wild-west, open, democratic, and sometimes manipulated discussions.
Infobesity and infoxication don’t help either, we’re inundated and can’t keep track of what is important, nor absorb it, nor understand it, nor know who said it. It makes it hard to effectively grasp issues and make decisions, as we’ve discussed in the paralysis section.
We’ve also previously seen how state entities are trying to get back the control by using speech itself as a censorial weapon.

The biggest consequence of this is that there has never been as much political discussion happening. It’s everywhere on the internet and it is what thrives.
However, the memetic material, the content, is always provocative, conflicting, shocking, and sensational because, as we keep repeating, this is exactly what thrives in the internet environment, namely: what touches our cultural sensitivities.
Consequentially, in relation to what we said before, this content will be about identity politics and identity in general, personal definitions. Discussions will rotate around recognitions of who some people are, the gaps they have in their cultures, how they patch them, the oppressions and damages they feel are being done to their previous or new-found identities.

Some discussions and content are more powerful and dangerous than others, activated ideologies that have real world consequences. This can lead to warranted or unwarranted protests, extreme ideologies, discriminations, assaults, suicides, acts of violence, and others.
In a previous section we’ve dabbled into how this can be used and initiated as part of PSYOPs.

This becomes alarming when doubt engenders the creation of an alternative reality that contests well-accepted scientific truth and facts, turning virtue into a vice. The lack of trust in authority being replaced by a trust in narratives that claim pseudo-authenticity and certainty. Sometimes coming from ill-informed people or from nations or entities that have an agenda — misinformation and disinformation.
This is what we’ve talked about when discussing memeoids, propaganda, and cults: a memetic equilibrium, a coherent world of cohesive meaning. The grand sense of meaning that was broken with neoliberalism but that couldn’t be recovered through anything else. The modern hermetism, or return of mysticism.
Doesn’t memetics itself reflect the concept of neoliberalism, all selfish memes fighting for their individual survival?

The word “truth” is often semantically associated with some religious or moral doctrines. This confusion also occurred in psychological literature.

In practice, this is embodied through conspiracy theories, built over time on the accumulation of confirmation biases and wishful thinking, and also through polarized and absolutist memetic wars that carry new narratives.

The side effect of these is the collapse of the classical form of journalism. Traditional media in its original shape is dead and along with it any form of “official narrative”.
There exists some media that are government owned, publicly funded, but the netizens do not trust them.

The online economy has ravage journalism. For example, in the USA, about a fifth of newspapers have closed and the survivors are shadows of what they were. The ones that are left rely on shock value to drive revenue from ads, the new business model, or have switched to subscriptions. Many geographical regions are now news deserts leaving people to rely on social media instead.
This results in the dramatic decline of professional reporting, yet a majority of people have not, and will not, notice any significant change.

The dominating ad networks, Google and Facebook, have their own alternative news channels and also decide on the internet dynamics: promoting or demoting a newspaper based on its popularity and how much revenue it attracts.

Citizen journalism is now king and the traditional media are following their footstep, redefining themselves as intermediates. Social media platforms have replaced the gatekeepers, as we’ve seen earlier, and the “democratic” curatorial systems decide which news are important and how they get framed.

While in the past the news were responsible for what they said, regulated, limited by their journalistic values, now it’s fully unregulated.

There are upsides and downsides to this new development. As we said, these are amateurs, people without journalism training, and so the quality of the reports and coverage can vary, is subjective, heavily opinionated, unregulated, and can be abused by malicious actors — such as spin doctors.
However, it can also promote human rights and democratic values to have the people join together and collaborate in this endeavor.

We’ve tackled before how the engines reinforce what people already believe, amplifying their narrative through their self-generated news.
It’s even more distressing when heads of nations publicly attack the press, calling them “enemy of the people”, inflaming opinions, polarization, and partisanships. Politicians have learned how to play the game on the internet.

Along with the self-generated meaning and self-generated news and facts, we see the rise of self-justice, private justice.
Ironically, this is in stark contrast with what neoliberalism is about, it is the large collective that fights individuals, the return of the battle of the collectivist narratives.
The polarization with its absolute us vs them mentality. A return of a sort of absolute moral rigidity with categorical imperatives.

Laws should be there to protect minorities, not lead to mob violence, and vigilantism, the kind of fast justice, with total impunity, and auto-defence that is justified by militias around the world — the wild-west and absence of authority that favors the creation of these dynamics.
This lack of trust in institutions and truth makes us want to create our own. We substituted the old gatekeepers, so we can also redefine the social contract.

Corporations and multinationals have been doing this for some time now, pushing their own justice, influencing legislations, spying, and thwarting competitors using any means. This has definitely been facilitated by social media.

On the individual side, it’s understandable, we’re presented a world, through the lenses of the news and social media, of constant chaos, clashes, and protests everywhere. It seems to be the only logical conclusion, when doubting all authorities, that we should fight for our own rights and that no one else will do it.
The offline consequences of this are apparent so let’s talk about the online ones.

The collectivists narrative, one that sees each person as a member of group and not an individual, a kind of “social tribalism”, is the way people bring back order from the chaos of getting exposed to different ideas — a pre-neoliberal view within a neoliberal mindset.
It is associated with a make-belief of simple dichotomies, groups at war, enemies, activism and counter-activism. Societies then become a struggle for power, a zero-sum war of ideologies. These ideologies are, within neoliberalism, related to identities.
Some of these are hijacked or pushed by political actors for gain. This is by various kind of parties that adhere to different political ideologies on the political spectrum.

This forcing of homogeneity, is the same force that drives indoctrination, cults, and memeoids. On themselves, these distorted worldviews and conflicts are wars of memeplexes, each side carrying their package of culture.

The combatants fight to annihilate the other memeplexe, to achieve eunemics — essentially trying to deliberately control the meme pool.
The war of memeoids is fought by quarantining other memes, restricting their speech/thoughts through censorship, and also done by executing attacks that ridicule memes to detach them from the enemy’s memeplex.
“Tyranny is the deliberate removal of nuance.” – Albert Maysles
Only one reality must survive!

When it comes to these attacks, they are filled with derogatory terms, loaded words that carry with them cultural ideas, politicized insults: essentially memes in the form of words. As we said, ridicule and satire are the best tools to fight narratives and drive discussions.
Creating a new language is also creating new ideas. Satiric words are thus both attacking memes toward the enemy memeplex, and new memes to be incorporated in the memeplex of the generator. A memetic warfare. Benjamin Lee Whorf said: “Language shapes the way we think, and determines what we can think about.”

A lot of the popular words are taken from the USA politics, so we’ll extract some examples from there, any reader should be familiar with the terms considering their invasiveness in the online sphere.

One of these words is SJW, “Social Justice Warrior”, a pejorative term used to describe someone who promotes “socially progressive views” for personal validation rather than conviction. This word carries a political meaning in the USA politics, is a satire thus a meme.
The term itself is used by the opposite ideologists to refer to the internet combatants of the other groups. These SJWs rely on victimhood talk, a twist on the oppressor narrative along with emotions, to silence/censor their opponents. Absurdingly, the groups using the words are themselves portraying the world in a similar way, the back to the collectivist narrative.

The personal validation is described as “virtue signalling”, a way to show your virtues and allegiance to a group. Virtue signalling is the common phenomenon we’ve seen happening when people share memes on social media: informing and showing others that they belong, that they know what is right, following what they think is prestigious, along with a sense of superiority.
It is most often used to point out “fake outrage”, “political correctness”, “microaggression”, “microinvalidations”, and “self-righteousness” (also meme terms), but it can also be applied more generally to anything used to get social karma/points.
Indeed, this is related to the neoliberalism narcissism coming from finding meaning through metrics. Obviously, and contrary to popular beliefs, this noisemaking is done by anybody on the political spectrum and not only by a single political camp. Only the association and its cultural reference that packages it in the name makes it a novel meme.
“In a democracy, recognition matters. Everyone wants to be seen as they are. If they are not, then it’s impossible for them to enjoy the experience of being full citizens.” — Melissa Harris-Perry

The term is also associated with slacktivism, hashtag activism, and pathological altruism. These are ways of supporting causes on social media while not involved in the on-the-ground activities and the messiness. These are terms used, and mostly prevalent, by Western netizens to describe how they are distanced from reality and act from the clean pristine coccoons of their sofa.
They are pejorative because they imply that these actions have little effects, might even be ineffective, are low-cost efforts, and can’t substitute more substantive actions. However, as we’ve seen over the past sections, we know that online political talks do indeed have an effect on mindsets and allow for more discussions around a topic — temporarily getting attention, selecting what is important.

One technique that is extensively used today by all groups, political parties, or association as a defense mechanism is a sort of deliberate social cooling of others, a self-surveillance that goes by the name of call-out culture and cancel culture.
The names are normally used by a specific political side in USA politics as a derogatory term describing another side, but the techniques themselves are applied by everyone anywhere in the world. It’s about using the online-outrage machine we talked about earlier to attempt to defame and vilify characters that go against the memeplex of the group.
This public lynching type of censorship should be a reminder of the propaganda we’ve seen earlier. Speech as a censorial weapon.
A perfect way to achieve eunemics through the suffocation of memeplexes.

This concludes our review of the truth and trust crises accelerated by the internet. In a first part we’ve seen how neoliberalism and individualism have shaken our notions of stability and broken our vision of authorities and hierarchies. The individual is favored before the collective, and they now find meaning through metrics, economic activities, and moral as a protection of new personal identities. Next, we’ve looked at how this unbalance can backfire if projections don’t match reality, especially when presented with so many worldviews on social networks.
Then we went over how this generates an enormous amount of political discussions, especially related to identity, and how this generates alternative universes of meanings that live on social media, some of it conspiratorial or based on novel narratives. A kind of self-generated meaning.
After that we’ve observed the collapse of traditional journalism and all that it implies, especially that now people can generate their own news, their own facts.
Finally, we’ve concluded with a glanced at a self-generated justice, a private justice created through the return of a collectivist narrative, fights between groups. These groups being memeoids, that fight eunemics wars on the internet trying to anhilate other memeplexes using some techniques we’ve encountered in previous sections such as speech as a censorial weapon.

Table Of Content

References








Attributions: A. Kircher, Ars magna lucis, Amsterdam, 1671

April 15, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Big Picture: The Formatted Individual April 15, 2021 09:00 PM

The Parmasayika grid is a fundamental religious diagram which divides up the Hindu Pantheon according to the measure of the purusha of the cosmic primal man

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 4 — The Big Picture
  • Section 1 — The Formatted Individual
Table Of Content
  • Paralysis
  • Normalcy and Innovation
  • The Neoliberal Mindset

In this part of the series, we’ll explore the bigger picture and the generic issues and “ill effects” on societies that are brought by the emergence of the internet or accelerated by it. We’ll begin with a look at three inter-related subjects: a general social paralysis, an apparent sentiment of homogeneity, and the relation with the widespread neoliberal mindset.

Paralysis is experienced at many levels and in different ways. The first of these are cognitives, which we’ve brushed in part 2 section 2.

One of the cognitive effect of the internet, proven by studies, is how much it affects our attention on the short term and long term. It is more divided and we’re less able to sustain concentration, it is more shallow and guided by attraction.
Research has shown the effect is clear on children, and long term studies have also shown that, contrary to popular belief, constant multi-tasking actually impedes the very act of task switching and does not improve it. This last one is due to how frequent exposure to this environment increases our susceptibility to distraction from irrelevant environmental stimuli.
Overall, multi-tasking in an internet environment has been associated with significantly poorer overall cognitive performance, it trains us to pay attention to what is flashier. We then feel frustrated and hindered when we have to perform tasks that require concentration. We feel obstructed and paralyzed, as if we’ve been robbed of our concentration.

Another effect we’ve looked at before is how the internet changed the way we retrieve information and value knowledge, acting as a transactive memory — an externalized memory.
The result is that we remember more where to find the information rather than the information itself. We know less about factual information because we’ve offloaded it to this new medium.
Hence, when we have decisions that rely on facts it makes us dependent on the internet. This can be both beneficial and harmful, as we said, giving us the opportunity to focus on conclusions, emergent hypothesis, overall aspect rather than facts. However, it also makes us focus more on opinions, trust the first results to our query (usually confirming our thoughts), ask fewer questions related to facts but questions related to individuals and society, and put an emphasis on fads and trends.
Our decisions are thus slower, having to consult online resources to make them, and hampered by what someone else has written about the subject we’re looking for.

We’ve previously also seen that social media had the same neurocognitive effects as real life social interactions but with an instant feedback through clear metrics. Time is now objective, not subjective, anything to turn us into machines. This has a direct impact on the concept of the self and self-esteem, making us pay attention to social judgement and social comparison. We’ve talked about influencers, hyper-successful individuals, and the unrealistic expectations these can create.
These can make us feel inadequate, especially for people that can’t manage properly their emotions.

Another aspect that gives us cold feet is the over-reliability of the medium. We’re deeply aware that nothing disappears on the internet, that any faux pas will be remembered.
Yet, we’ve also never been as uncertain about anything. We’re constantly the spectators of theatrical clashes of opinions and it makes us doubt our own.
Moreover, we’re constantly overthinking, overanalysing things, we are submerged in information and can’t take decisions fearing we’ll make the wrong ones. An analysis paralysis due to the infobesity.

Joining these ideas related to judgement makes us stay on the bench. These are compounded in phenomena like social cooling, the 1% rule, and lurking.

Social cooling is the self-censure that people exert on themselves because of social pressure. The tension between always feeling watched by surveillance algorithms, knowing our personal data is gathered against our will, understanding that our weaknesses and flaws could be exposed and used against us, and fearing how we’ll be scored based on the whims of the internet crowd. We’re intimately aware that our digital reputation is now serious business and can impact our real life.
For these reasons, people change their behavior and adapt to whatever makes them look better on the platforms, have better scores or avoid doing anything that could add a negative side to their reputation. We hesitate before making a move. All over, it makes an environment socially rigid with risk-averse people that conform.
A chilling effect on society and a powerful form of control.

This is in direct relation with the notion of lurkers, a silent majority of passive users that is said to constitute 99% of the internet, according to the 1% rule, and that doesn’t participate in the creative aspect.
The vast majority of people only consumes and doesn’t contribute any content. This participation inequality has been researched, verified, and shown to be consistent on different online forums and wiki pages such as Wikipedia and others.

There are multiple reasons why most stay on the periphery as read-only users. Social cooling plays a big part in it and the studies make it clear.
When asked why people only observed and did not participate, the common reply is that they were gathering cultural capital: the knowledge that enables an individual to interpret various cultural codes. We take the time to evaluate the implicit and explicit norms of the group, to learn if we can fit, to understand the preferred method of interaction without making mistakes and avoid being rejected, to see the topics of conversations that are allowed, and to learn the conventions of the community. This is commonly done by finding the typical figures representing the community, the models to adopt, templates that exhibit the kind of dialogues we can have.
People also lurk because they fear being redundant or that their contribution would not be worthy. By lurking they can glimpse at what has already been said and done.
We want to copy, belong, but also be unique, that’s something we’ve already seen in the biases section. Maybe it’s because our human experience is so similar and yet distinct.

Lastly, people stay afar because they feel that browsing is enough for them. That could be simply because they are looking to find particular content from a place and nothing else.

On the psychological side, lurking is associated with loneliness and apathy, a sense of not belonging. Lurkers are found to be less satisfied and experience more distractions, they are distant.
In the case of social networking, they experience less intimacy and feeling of personal well-being, and an overwhelming loneliness as they watch others participate in the community.
Social media does indeed feel cognitively similar to amplified real life interaction.
This is akin to a TV audience but with the sense that what we see is real and not a fabricated reality.

What makes someone de-lurk is situational and depends on both the individual’s personality, the environment, and the community. What remains is that only a handful of superusers generate the vast majority of content.

Research shows that these contributors are highly motivated, feel a sense of duty, attachment, and belonging to their online community. Other research show that active participants seemed to be more extroverted and open, felt they were more empowered and in control of their environment, had confidence in their ability to influence, had higher self-efficacy, and a greater need for gratification.
That is in sync with what we’ve seen regarding frustrated individuals that had their cultural malaise and gaps filled by the internet. Participants need a place to express themselves and that is relevant to them.

Statistics show that the people contributing the most are out of the ordinary, they exhibit extraordinary behavior. That means that when we consume any content on the internet, we’re mostly consuming content created by people who, for some reason, spend most of their time and energy creating this content. These people clearly differ from the general population in important ways. The kind of people that move the Overton window, which we talked about.
This is worth keeping in mind when on the internet.

Knapp, had been submitting an average of 385 edits per day since signing up in 2005 as of 2012. Assuming he doesn’t sleep or eat or anything else (currently my favored prediction), that’s still one edit every four minutes. He hasn’t slowed down either; he hit his one millionth edit after seven years of editing and is nearing his two millionth now at 13 years. This man has been editing a Wikipedia article every four minutes for 13 years. He is insane, and he has had a huge impact on what you and I read every day when we need more information about literally anything. And there are more like him; there is one user with 2.7 million edits and many others with more than one million.

These days the 1% rule has been more or less debunked as the barrier to engagement has decreased. However, it hasn’t decrease enough, the number of lurkers is still relatively high and social cooling adds to the equation.

This paradoxical balance between lurkers, conformity, social cooling, and out-of-the-norm influencers and contributors in different communities, makes us feel a sense of normalcy and lack of innovation.

Normalcy is the consequence of adherence and conformity. We lurk because we want to adhere to the norm, not stand out, to fit in. We’re also interested in participating because of the homophily phenomenon we’ve discussed before, “birds of feather flock together”.
This all keeps the status quo and homogeneity in the community.

This creates a situation in which we’re offered templatized, cookie-cutter identities which we can choose from, and indirectly coerced to fit into them by the persuasive technology. It’s easier to market to persons that fit into well-separated categories, to “buyer personas”. One could even call these memeplexes and memeoids, honed through eunemics.
This is all decided by fads, opinions, and interests, a sort of “choose your character/future” mode of identity building. With these templates there’s an emphasis on attitude rather than facts, truth, or moral. Sometime called “Hypernormalization”.
“How can you have a personality if you have no knowledge to base it upon, and if you merely have opinions that have been given to you through slogans or clichés.”

It is hard to get out of a template, these mass one-sided personalities, because of all that we’ve seen before especially the social judgement, social cooling, the action-attention-reward addictive mechanism, and others. We need privacy as a right to not fit in, a right not to be perfectly molded, well-behaved, docile, and conformed humans.
We are guided by biased algorithms that are mathwashed, believing they are impartial.

This has detrimental effects on kids that are in the process of finding themselves, in the process of building their personality. They feel enormous social pressure to be perfect. Self-inflicted injuries, such as cuttings that are serious enough to require treatment in an emergency room, in the USA, have increased dramatically in 10- to 14-year-old girls, up 19% per year since 2009.
Less physically harmful, is the effect on young boys that are looking for achievements and get addicted to “competency porn”, any type of media that fulfills the need for recognition, achievement, and control over the environment — Be it video games or animated movies. The epic tales and escapist fantasies of today’s world.

It is astonishing because different persons are part of different internet realms, with different templates, and are given skewed pictures of reality — it’s as if everyone lived in alternate universes that sometimes, but rarely, cross.

Indeed, even though the UN considers the right to internet access a human right, the effect of the globalisation of the internet are unclear.
Already we’ve seen filter bubbles, homophily, and confirmation bias that keep people within the same realms. Additionally, we have to add to this the privileges associated with the internet. Not everyone can access the internet, not everyone can use it powerfully, not everyone can participate in community content creation, and not everyone has the language skills to participate in these various communities.
The content is still subject to being part of a hierarchy, still based on the same social and international hierarchies that exist today. Minority views in the real world are still minority views online, the same people facing discriminations offline also tend to face them on the internet.
And yet, it also fosters the democratization of speech if used properly, but the content we are given somehow reinforces the status quo.

It’s a global and local presence, we could say a glocalization. Everyone that has internet uses it in their own way, and the way they use it affects their perception. Some use the internet for research, some for hobbies, some to socialize, some to learn, some as a tool for work, etc.. Internet usage is shaped by the myriads of ways that we access it for our needs and values.
Yet, this exact habit of usage is shaped by the local culture and thus also creates a divide as the internet is inherently based on connecting networks together.

For example, even though we can use the internet for different things, the language we speak will only give us a narrow view of the whole spectrum. If we search using a language that is more “popular”, let’s say English, we’re prone to be given the templatized identities related to the most vociferous online English speakers (USA).
Language then is part of the dynamic of who defines and who is defined.
There are other imbalances that exist based on geolocal preferences. For instance, speakers of a certain language will prefer domestic issues.

Moreover, advertisers, the main source of revenue online, will prioritize specific consumers that fit a template from a certain culture and geographical area, and not others. This will incentivize content creation for that specific audience, as this is where more money is made. This in turn will widen the cultural gap, now a digital advertising gap, thus a media content quantity and quality gap.

There are many factors that make the internet both global and local at the same time. Again, it depends on our usage of it but when we default to the average person, there’s no room around being presented only narrow views.
The question remains whether internet truly makes people conform, what’s the state of creativity and counterculture, and how much propaganda comes into play.

The literature shows that a small subset of the population, around 2 to 3%, have international ties and act as bridge figures between cultures. These persons show a higher creativity than their peers because of their exposure to different views.
From this we can imagine that it’s the same thing on the internet.

When using a service you become part of its ecosystem, there are many platforms and the big ones, the FAANG, have a tech hegemony. Internet in itself didn’t kill counterculture, it is just not present on these platforms as they encompass a sort of new overarching powerful entity, sometimes more organized than governments.

This means that counterculture today isn’t any more about a war of ideologies but about going against the trends of the internet. The counterculture of the past, which rotated around personal expressions and identity are now used by platforms to drive engagement and profit, it’s part of their lucrative business. Social media in particular have been catering to the “demands” of a generation by filling the gap that the culture couldn’t — self-fulfillment as a product, as we’ve seen earlier.
This means that these types of old countercultures are still part of the system, but by embodying them we’re not in charge. We pull ourselves together by gathering pieces of templatized subcultures. Then, this collection becomes our niche personal branding, our personal expressions and ideologies.

As we went over in part 2 section 2, we are now seeking individuation, we are networked individuals connected by interests. This is in stark contrast with counterculture which requires a group to thrive, an “us against the world” mindset. The internet does allow regrouping but only on a shallow level, particularly for collective dissent and unsatisfaction, turning the groups to absurdities and cult-like behavior.
Directly opposing the system will only make the system stronger.

True counter-culture can either be found by embracing the algorithms, understanding their inner-working and warping them to our needs, or by signing-off and rejecting the whole system for an alternative one. Both of these are difficult in the era of like-and-share.

So we are bound to use the internet, and the internet to project our anthropological nature, its current state and flaws, be it conformity, homogeneity, or others.
We’ll be given a reflection according to the function it has for us, our lifestyle, current society, culture, and time. For a big part of the world that means the internet is becoming inherently neoliberal, used for neoliberal ends, and encourages neoliberal behaviors.

This isn’t surprising considering the origin of the internet, who are the biggest internet companies, and who are the most vocal online. The internet embodies the “American mindset”, a mindset of personal brands and marketing, the self-made man.
A considerable amount of people are driven by this, with different subset cultural bubbles living outside the hegemony of these giants because of different environmental constraints.

The internet is the perfect tool for the workings and reinforcement of neoliberalism values. It is a perfect display of how this way of viewing the world, which was heavily questioned during the Cold War, has taken over.

It focuses on a certain flavor of democracy, individualism, and a laissez-faire absolutism of the marketplace. The belief and undeserved faith in the efficiency of markets. It centers around deregulation, free trade, privatization, and puts the individual at the center of everything. Any other attitude, and especially government intervention, is frowned upon.
We can clearly see how this has unfolded on the internet, with Facebook’s many scandals clarifying the limits of corporate self-regulation.

There’s an over-emphasis on economic efficiency and growth at all costs. Political theorist Wendy Brown has gone even further and asserted that the overriding objective of neoliberalism is “the economization of all features of life”.

The self becomes a product, a personified capital that needs to accumulate symbols of status, a storyline, and advertise itself. With regards to the internet, and because the money is where advertisers and the audience put their interests, a personal-brand will only be profitable if it fits the mold of the popular culture.
We have to partner with the brands and products that make us shine. We own things that reflect our personal meaning. We calculate the benefits, efficiency, and productivity of each action. We had a look at this in the new economies section, this is in relation to this. The commodification of personal data and attention for profit.

This unfettered exploitation puts gains before anything and it doesn’t only apply to individuals but especially to corporations. That is why marketing is prioritized on the internet, an optimization of products and surveillance. We de-emphasize public goods to turn our eyes towards the market idolization, a corporatocracy.

Everything has a monetary incentive rather than a social objective. Anything needs a metric, an indicator, to be tracked and quantifiable. This focus on economic efficiency can compromise other, perhaps more important, factors, or promote exploitation and social injustice. We see this on the personal level on social media through likes, shares, and subscribers.

It’s arguable if neoliberalism, the advent and rise of this way of life, are bi-products of the internet or if it’s the opposite. It’s arguable if individuation is the product of neoliberalisation or vice-versa, and if they are accelerated by the internet.
It’s probably a mix of everything, societal changes with the private enterprises reflecting the will of individuation of private individuals, which are also projected on their internet — the internet driven by the big players.

We’ll see later how these societal change and acceleration bring social unrest.

The individuals are at the heart, all the responsibility and weight are on them, that’s why they focus so much on identity politics and individual gains, while dismissing broader identities and class consciousness.
A mix of looking for authenticity, as it seems to be the gap in cultures that have embraced neoliberalism, and an overwhelming majority of narcissistic individuals.
A mix of losing political freedom because of apathy, and looking for self-fulfillment as withdrawal from moral concerns. Are lurkers neoliberal? Are counter-culture and creativity now anti-neoliberalism?

Globalisation doesn’t necessary mean homogenisation, but it’s undeniable that the neoliberal mindset has spread further than any other and can’t be dismissed. Its acceptance, normalization, the incentives, indirect influences on the internet and its dynamics are apparent.
Isn’t this the best propaganda, one we don’t even notice unless we’re someone in the middle, someone cross-cultural?

Nonetheless, what happens online is only a function of the offline status quo, the offline paralysis. The internet does not exist in a vacuum. It is made with the purpose of letting people connect with their own networks, assuming that users are individuals, and inherently individualising them.
The online world recreates and accelerates — there is nothing inherent within the technology that makes it neoliberal, it’s only a tool. Each can have their own different internet experience depending on how they use it, as we kept saying.
We’re both formatted and free individuals — freedom as a menu.

This concludes our review of how the internet has consequences in the acceleration of cultural evolution, formatting individuals, paralyzing them, imposing an overwhelming homogenisation while letting fringe people contribute, and fostering and spreading the neoliberal mindset.
At first, we’ve examined the cognitive effects: attention, memory, and social cognition. Our attention is divided, our memory externalized, and our social fear amplified. Then we’ve seen how the internet makes us constantly anxious of making indelible mistakes, embodied in the concept of social cooling and self-censorship.
Next, we’ve observed the link with the notion of lurkers, read-only users, the 1% rule, and the out-of-the-ordinary users that actually contribute the content. These contributors do so because the internet has filled something they were missing, a cultural gap.
After this we’ve questioned conformity and homogeneity, how the internet offers template we select from, and puts pressure on individuals to fit perfectly. Yet, we’ve also observed that these templates, these experiences, and the internet function as a whole, differ and are divided per culture, environment, language, and others — the glocalization of the internet.
Later, we’ve added to this the fact that the internet is driven by monetary incentives and that it will create a dynamic in which more content is targeted at people that fit certain templates and others will be enticed to fill the templates of other cultures, an indirect cultural assimilation.
Following this we’ve discussed what it would mean to stand out of these templates, of all these memeplexes, to be more creative or counter-cultural and we’ve concluded it means either embracing the tech or going against it. Finally, we’ve linked the topic with neoliberalisation and how it exemplifies the phenomenon we’re seeing, both by emphasizing the market, efficiency, and the individual.

Table Of Content

References








Attributions: Parmasayika grid

April 14, 2021

Gustaf Erikson (gerikson)

The Pacific War Trilogy by Ian W. Toll April 14, 2021 01:23 PM

  • Pacific Crucible: War at Sea in the Pacific, 1941–1942
  • The Conquering Tide: War in the Pacific Islands, 1942–1944
  • Twilight of the Gods: War in the Western Pacific, 1944–1945

An excellent and readable account of the (US) war in the Pacific against Japan in World War 2. Highly recommended.

Marc Brooker (mjb)

Redundant against what? April 14, 2021 12:00 AM

Redundant against what?

Threat modeling thinking to distributed systems.

There's basically one fundamental reason that distributed systems can achieve better availability than single-box systems: redundancy. The software, state, and other things needed to run a system are present in multiple places. When one of those places fails, the others can take over. This applies to replicated databases, load-balanced stateless systems, serverless systems, and almost all other common distributed patterns.

One problem with redundancy is that it adds complexity, which may reduce availability. Another problem, and the one that people tend to miss the most, is that redundancy isn't one thing. Like security, redundancy is a single word that we mean that our architectures and systems are resistant to different kinds of failures. That can mean infrastructure failures, where redundancy could mean multiple machines, multiple racks, multiple datacenters or even multiple continents. It can mean software failures, where common techniques like canary deployments help systems to be redundant when one software version failures. I can also mean logical failures, where we recognize that state can affect the performance or availability of our system, and we try ensure that the same state doesn't go to every host. Sometimes that state is configuration, sometimes it's stored data or requests and responses.

An Example

Unfortunately, when we talk about system designs, we tend to forget these multiple definitions of redundancy and instead just focus on infrastructure. To show why this matters, let's explore an example.

Event logs are rightfully a popular way to build large-scale systems. In these kinds of systems there's an ordered log which all changes (writes) flows through, and the changes are then applied to some systems that hang off the log. That could be read copies of the data, workflow systems taking action on the changes, and so on. In the simple version of this pattern one thing is true: every host in the log, and every consumer, sees the same changes in the same order.

Event bus architecture, with three replicas hanging off the bus

One advantage of this architecture is that it can offer a lot of redundancy against infrastructure failures. Common event log systems (like Kafka) can easily handle the failure of a single host. Surviving the failure of a single replica is also easy, because the architecture makes it very easy to keep multiple replicas in sync.

Event bus architecture, with three replicas hanging off the bus, with host failures

Now, consider the case where one of the events that comes down the log is a poison pill. This simply means that the consumers don't know how to process it. Maybe it says something that's illegal ("I can't decrement this unsigned 0!"), or doesn't make sense ("what's this data in column X? I've never heard of column X!"). Maybe it says something that only makes sense in a future, or past, version of the software. When faced with a poison pill, replicas have basically two options: ignore it, or stop.

Event bus architecture, with three replicas hanging off the bus, with logical failures

Ignoring it could lead to data loss, and stopping leads to writes being unavailable. Nobody wins. The problem here is a lack of redundancy: running the same (deterministic) software on the same state is going to have the same bad outcome every time.

More Generally

This problem doesn't only apply to event log architectures. Replicated state machines, famously, suffer from the same problem. So does primary/backup replication. It's not a problem with one architecture, but a problem with distributed systems designs in general. As you design systems, it's worth asking the question about what you're getting from your redundancy, and what failures it protects you against. In some sense, this is the same kind of thinking that security folks use when they do threat modeling:

Threat modeling answers questions like “Where am I most vulnerable to attack?”, “What are the most relevant threats?”, and “What do I need to do to safeguard against these threats?”.

A few years ago, I experimented with building a threat modeling framework for distributed system designs, called CALISTO, but I never found something I loved. I do love the way of thinking, though. "What failures am I vulnerable to?", "Which are the most relevant failures?", "What do I need to do to safeguard against those failures?"

If your answer to "What failures am I vulnerable to?" doesn't include software bugs, you're more optimistic than me.

April 13, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — Biases & Self: Cultural Ambiguities & Insecurities April 13, 2021 09:00 PM

The Sufi-infuenced teachings were aimed at destroying man's illusory sel-image, and revealing him as being guided by mechanical reflexes

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 3 — Biases & Self
  • Section 2 — Cultural Ambiguities & Insecurities
Table Of Content
  • Constrained By Platforms Or Platforms Constrained
  • Beliefs and Values
  • Repressed Cultural Weaknesses
  • Polarization As A Natural Phenomenon
  • Primal Needs
  • True Believers

To be a netizen means to be part of the online social sphere. There’s no way around it, to have a voice and participate people have to join a platform, which comes with its own limitations.
The rules of the platforms are the rules of the information society but the platforms adapt more to fit us than we adapt to them. Anything happening on them is directly because of real people interacting together. People that have their own hopes, emotions, values, prejudices, and beliefs. Consequently, through our own cultural differences, ambiguities, and insecurities, we are indirectly manipulating ourselves.

We look for confirmations of our own experiences, as we said in the previous section, confirmation bias. We want to share and search for things that relate to our local lives, to make us look smart, empathic, cast us in a positive light, or that is useful in our day-to-day lives.
On the internet this is proved through many experiments. Emily Falk’s Lab, Director of the University of Pennsylvania’s Communication Neuroscience Lab, has demonstrated how the expectation of social confirmation and reward influence the likelihood of someone sharing a meme with someone else. We share for social comparison, to look favorable to other group members, for “virtue signalling”.
Similarly, according to research, on average people visit a maximum of 30 websites before making their mind about something. The bulk of these websites are part of the top results of the search engines of which 70% will always support the view portrayed by how the query was formulated. This is what we refer to as an echo chamber and filter bubble, it echoes back what we think.
We seek information sources that support and reinforce existing attitudes or beliefs, as well as process them. “All else [being] equal, people seem to prefer not changing their opinions to changing them” (Lebo & Cassino, 2007, p. 722).

We increasingly, inadvertently through selective exposure by our own judgement, are localized, not only geographically but also through our language, ideologies, time, and cultures. As we said earlier, the internet is made of networked separated individuals, linked by their interests and similarities.
This isn’t only about filter bubbles created through algorithms, but it’s our natural tendency to pay attention to those who have a lot in common with us. We lack cognitive diversity.
We call this phenomenon homophily, “birds of feather flock together”. We’re really good at it, we have this inner urge to find people similar to us, and this is confirmed by over 100 studies.
This isn’t necessarily negative, this allows us to create communities based on niches, on interests, be more democratic, etc.. All things we’ve seen in part 2 section 2.

We all think that the way we live is the best way to live. We all think that we see the world as it is, that we’ve come to the best possible conclusion about our direction in life. However, that’s just our definition of normality. And this normality isn’t even ours as most of it happens without us being there. We often can’t justify why we do what we do or why we like what we like.

This is as much as about externalities as internalities. We associate ourselves with others like us, as a way of expressing our own individuality by proxy.
We said people spent more time viewing and sharing memes that confirmed their views with others, not only because it confirms them, but also because it’s a way of expressing one-self and indirectly promoting who we are. We went over this in the meme section, memes are a representation of our mental models, how we comprehend and make sense of the world.

Ideologies are never force-fed, we are deliberately looking for them. Nobody can be persuaded if they don’t want to be persuaded.
That is why when we discussed propaganda we said that the best messages are those that resonate with the audience, that aren’t perceived as propaganda. Not imposed, but emerging from themselves and expressing the concerns, tensions, aspirations, and hopes of the persons.

We live through our media, they define us, we identify with them. That is why the interactions on the internet are a reflection of our different cultures.
Everyone has an opinion, however, what is worrying is when they give rise to consequential beliefs. When they manifest in the real world.

On social media, these opinions are now all expressed and openly visible. The pristine walls protecting us have fallen and we are faced with a messy world full of ambiguities.

This is increasingly true when netizens use persona, and even more when they use hide under an anonymous pseudonym. People are then free to express themselves without restraints, breaking accepted cultural codes, expectations, unspoken tensions, and preconceived notions. They aren’t burdened by what their peers think of them.

However, we are all biased, as we saw in the previous section, and this confluence of voices brings uncertainty to our own. Doubt sets in when many claim the contrary of something widely accepted. It’s also hard for us to accept coincidence as an explanation, which gives rise to some shouting their conspiratorial thinking. Trust is easy to lose and hard to gain.
A trust crisis is setting in.

On social media everyone has an equal voice and different views. We said this was moving us towards a low-context society or informational society. We’ve also just seen the tendency to regroup in bubbles with people and content that is in agreement with our current views.
Yet, research tells us that social media users have a more diverse news diet than non-users. It shows how much we are bound to encounter opinions and information that might upset us because they go in the opposite direction of our preconceived notions. The bubbles are bursting against each others, and this implicitly creates hostility.

As humans, we tend to connect ideas in a coherent way, to make sense of the world around us based on our culture and routine. We have expectations of what is true, know historical facts set by our institutions, build a narrative from the events we are told, hold public opinions, have pictures of what is normal or not, assumptions about how structures fall into place, and which established powers we think is better.
Yet, again, reality isn’t sterile and spotless, these preconceptions are but a thin layer holding a story, and it only covers an infinitely small portion of the existence on this planet. Cultures often have unspoken tensions, anxieties, weaknesses — no story is perfect. This is felt even more heavily when they clash on the internet.
A truth crisis setting in.

These contradictions need to be accepted and that might generate cognitive dissonance in some people. Cognitive dissonance is a word used to describe the effects someone feels when holding contradictory beliefs, ideas, or values and it’s denoted by psychological stress. The human-psyche wants to be consistent, otherwise it creates discomfort. We feel an urge to resolve these contradictions.

Then it’s no wonder that some content go viral, it’s because they’re unleashing a repressed cultural agenda. Online discourse promotes outrage because we feel the need to defend or attack our positions so that our world view doesn’t shatter — to keep our own consistency or to find it again.
If we didn’t have issues in our societies, these extreme forms of discussions wouldn’t materialize. Thus, it is reductive to point the finger at just the social media platforms, that’s why we’re pointing it at ourselves here.

When facing an unclear situation, a tension, a question, or decision, we instinctively take stances. Naturally we either are with or against. We put ourselves in camps. At least, the most prominent and outspoken people take sides, so on the internet these are the ones whose voice break through the noise.
That person will be exposed to a disproportionately large amount of info similar to their view and a small amount of info against their view. Naturally, they’ll want to react the contrarian viewpoints to defend what they think is right.

After that dance, opinions will be polarized and the diversity of point of views and nuances will disappear — the system will have reached its equilibrium.
Absolutist thinking is in human nature when there is an absence of certainty. We want to grasp for reassurance.

The researcher Cass Sunstein has shown that this is even more apparent when putting people that have similar mindsets in the same room. They divide themselves according to more polarized version of the same mindset.
Initially the group shows a lot of internal disagreement over issues but then, again, fall unto an equilibrium. People held more-extreme positions after speaking with like-minded individuals and were more homogeneous, squelching diversity. Either an all or nothing, a black and white vision, highly moralistic, with expectations on us and others.
Similar studies show the same results and adds that people have a less negative perception of opposite views if they get exposed to them more often, instead of mainly the ones confirming their own.

We’ve seen that people self-sort and seek confirmating views online, this widens the rifts between different points. The internet makes it exceedingly easy for people to move into extreme absolute positions and versions of whatever is discussed.

The internet, and particularly social media, also fills the voids that our societies cannot. Gaps in our primal needs: the need for certainty, the need for identity, the need for belonging, the need for accomplishment, competency, and approval, or higher spiritual and moral callings. All things that are missing in contemporary civilizations according to Charles Taylor.

Our lack of certainty about fundamental assumptions and loss of control inclines us to the middle but opinions on extremes look steady and we slowly move towards them — a need for closure, narrowing our perception.
We generally adopt the first belief that fills the gap, which is given to us by the flashy promotional culture or fringe ones.
These fillings do not have to be understood but believable, to look consistent and absolutely certain. The quality of the ideas themselves play a minor role but the meaning and confidence that they bring matters.

The persons the most sensitive are the ones who are bored, feel powerless, have insecurities in their culture, feel uneasy about the ambiguity of the world, and are frustrated. A feeling of anomie.
They are driven primarily by hope for a coherent future, their gaps filled by the ideology, being consoled and rewarded for taking part in it.
They go under the name of fanatics, highly motivated individuals, defenders of culture, true believers, memeoids, or hyper partisans.

The transformation or indoctrination can happen slowly over time, through repetition and a slippery slope, as we said, reaching a target level of saturation creating a full universe of meanings. Then stuck in a feedback-loop.

Undeniably, micro-targeting can be used to find these individuals which needs aren’t filled by society, to then turn them into true believers. Monitoring posts can give us insights on whether someone feels stressed, defeated, overwhelmed, anxious, nervous, stupid, useless, and a failure.
It’s not surprising that this is going to be used because hyper partisans are the most motivated, they are driven by their ideology and occupy a bigger space in the information market. This is a sort of tyranny of the minority, which drags people that are undecided.

Sarah Golding, the president of the Institute of Practitioners in Advertising, says “It has essentially weaponized ad technology designed for consumer products and services. There is a danger that every single person can get their own concerns played back to them”.

Cognitive dissonance and uncertainty open a window of opportunity to hack the mind and bring these individuals in.
The memeoids are known to self-isolate into fringe communities. Ironically, statistically these communities, with their extreme viewpoints, are the most influential on the internet. This is all unsurprising because they offer certainty and answers for the lost netizens.

Additionally, these communities always have gatekeepers and look enticing. They isolate the members and shun anyone that has a different opinion. The stories are reinforced through time. This reminds us that certain communities on the internet are no different from cults, and of the reasons why people join them.

This concludes our review of cultural angst and how they are reflected on the internet. First of all, we’ve talked about how the internet has its limitations but is more shaped by us than we are shaped by it. We’ve then dabbled with the concepts of beliefs and values which always get confirmed through the results given by recommendation/curation engines and other algorithms. Next, we’ve seen how when cultures clash they make us feel discomfort and bring to the surface cultural anxieties and weaknesses, thus we react. After that, we’ve talked about how polarization is natural, how we instinctively take sides to become more homogeneous. On the internet this creates rifts between ideas, people want to take sides. Later, we’ve covered how the internet can fill needs that aren’t filled by societies, be it a need for closure, accomplishment, belonging, or approval. Finally, we’ve looked at hyper partisanship, how people get stuck in these communities, and why they are the most vociferous online.

Table Of Content

References








Attributions: Alexander de Salzmann, cover-design for the programme of the “Institute for the Harmonic Development of Man”, Tiflis, 1919

Pages From The Fire (kghose)

The hitch April 13, 2021 03:01 AM

I lay there staring up at the underbelly of the car, the muffler an inch beyond my nose. A thirty two pound steel hitch lay at my feet and I had no idea how I was going to drag it in that cramped space, over my chest, lift it up to the undercarriage and then… Read More The hitch

April 11, 2021

Ponylang (SeanTAllen)

Last Week in Pony - April 11, 2021 April 11, 2021 11:47 PM

New versions of the ponylang crypto, http, http_server, and net_ssl packages are available.

Derek Jones (derek-jones)

Pricing by quantity of source code April 11, 2021 09:45 PM

Software tool vendors have traditionally licensed their software on a per-seat basis, e.g., the cost increases with the number of concurrent users. Per-seat licensing works well when there is substantial user interaction, because the usage time is long enough for concurrent usage to build up. When a tool can be run non-interactively in the cloud, its use is effectively instantaneous. For instance, a tool that checks source code for suspicious constructs. Charging by lines of code processed is a pricing model used by some tool vendors.

Charging by lines of code processed creates an incentive to reduce the number of lines. This incentive was once very common, when screens supporting 24 lines of 80 characters were considered a luxury, or the BASIC interpreter limited programs to 1023 lines, or a hobby computer used a TV for its screen (a ‘tiny’ CRT screen, not a big flat one).

It’s easy enough to splice adjacent lines together, and halve the cost. Well, ease of splicing depends on programming language; various edge cases have to be handled (somebody is bound to write a tool that does a good job).

How does the tool vendor respond to a (potential) halving of their revenue?

Blindly splicing pairs of lines creates some easily detectable patterns in the generated source. In fact, some of these patterns are likely to be flagged as suspicious, e.g., if (x) a=1;b=2; (did the developer forget to bracket the two statements with { }).

The plot below shows the number of lines in gcc 2.95 containing a given number of characters (left, including indentation), and the same count after even-numbered lines (with leading whitespace removed) have been appended to odd-numbered lines (code+data, this version of gcc was using in my C book):

North Star Horizon with cover removed.

The obvious change is the introduction of a third straight’ish line segment (the increase in the offset of the sharp decline might be explained away as a consequence of developers using wider windows). By only slicing the ‘right’ pairs of lines together, the obvious patterns won’t be present.

Using lines of codes for pricing has the advantage of being easy to explain to management, the people who sign off the expense, who might not know much about source code. There are other metrics that are much harder for developers to game. Counting tokens is the obvious one, but has developer perception issues: Brackets, both round and curly. In the grand scheme of things, the use/non-use of brackets where they are optional has a minor impact on the token count, but brackets have an oversized presence in developer’s psyche.

Counting identifiers avoids the brackets issue, along with other developer perceptions associated with punctuation tokens, e.g., a null statement in an else arm.

If the amount charged is low enough, social pressure comes into play. Would you want to work for a company that penny pinches to save such a small amount of money?

As a former tool vendor, I’m strongly in favour of tool vendors making a healthy profit.

Creating an effective static analysis requires paying lots of attention to lots of details, which is very time-consuming. There are lots of not particularly good Open source tools out there; the implementers did all the interesting stuff, and then moved on. I know of several groups who got together to build tools for Java when it started to take-off in the mid-90s. When they went to market, they quickly found out that Java developers expected their tools to be free, and would not pay for claimed better versions. By making good enough Java tools freely available, Sun killed the commercial market for sales of Java tools (some companies used their own tools as a unique component of their consulting or service offerings).

Could vendors charge by the number of problems found in the code? This would create an incentive for them to report trivial issues, or be overly pessimistic about flagging issues that could occur (rather than will occur).

Why try selling a tool, why not offer a service selling issues found in code?

Back in the day a living could be made by offering a go-faster service, i.e., turn up at a company and reduce the usage cost of a company’s applications, or reducing the turn-around time (e.g., getting the daily management numbers to appear in less than 24-hours). This was back when mainframes ruled the computing world, and usage costs could be eye-watering.

Some companies offer bug-bounties to the first person reporting a serious vulnerability. These public offers are only viable when the source is publicly available.

There are companies who offer a code review service. Having people review code is very expensive; tools are good at finding certain kinds of problem, and investing in tools makes sense for companies looking to reduce review turn-around time, along with checking for more issues.

April 10, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — Biases & Self: Cognitive Biases April 10, 2021 09:00 PM

The art of memory is like an inner writing. Those who know the letters of the alphabet can write down what is dictated to them and read out what they have written

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 3 — Biases & Self
  • Section 1 — Cognitive Biases
Table Of Content
  • Biases Aren’t Long Term
  • Who
  • Physical
  • Biases Related To Visibility
  • Biases Related To Information Availability
  • Biases Related To Social Pressure

Humans are prone to manipulations by their own experiences and biases. Let’s review what are some of the cognitive biases and cultural weaknesses that make us fall for all sort of things on the internet.
We’ve already looked at coercion, deception, and other types of persuasion principles in a previous section. In the following two we’ll emphasize on ourselves instead of external factors.

There are many cognitive biases that have been documented over the years, this is nothing new. However, it’s interesting to see them under the light of this medium, how they come into play on the internet.
I’ll try to list the ones that are the most prominent in my opinion.

In general, biases can affect us when it comes to beliefs, decision-making, reasoning, economic decisions, and behavior. Yet, biases don’t have a significant influence on the long run, at least the ones that aren’t inherently connected to our cultures. We’ll dive into culture in the next section.
Cognitive biases are temporary tricks, temporary stimulations that hack our brain. When someone is aware that another person has used such bias against them, they know they have been cheated. Regardless, studies show that even when trying to influence subliminally, without the audience realising, these types of stimulations only have negligible effects.

In most cases, a bias isn’t something that someone creates but something we all naturally have and that is exposed. As such, we can all have them, different persons having different ones under different circumstances. Kids in particular are more vulnerable to them.

Before diving into cognitive biases, let’s stop to mention some physical reactions.
In part 2, section 2, we’ve mentioned social cognition and how communicating on the internet had the same bodily-effects and cognitive responses as other types of social interactions. This implies, as research shows, the same status-to-hormone correlation: stress with cortisol and social confirmation/success with oxytocin, endorphin, and dopamine.
Practically, in the social media environment, which works with interfaces showing quick metrics of social success or failure, that means people’s Action-Attention-Reward system will lead them towards addiction. Social media acting like slot-machines — classic conditioning — and keep us hooked.

We all react similarly to these hormones; biased to associate pleasant and positive feelings with what triggers some, and associate negative feelings with others. Hence, we are incentivized to repeat the same behaviors that once got us a dopamine hit — be it finding a niche of expertise we cater to, or posting and finding something shocking.

A market in which everybody has the right to speak comes with the biases of everyone in it.

Some of the biases we can have on social media are in relation to the control over visibility. Biases such as attentional bias, salience bias, framing effect, mere exposure effect, anchoring effect, and others.
Attentional bias is the tendency to pay attention to what is more recent in memory. Similarly, salience bias is about paying attention to what is more remarkable and emotionally striking. The anchor bias is a similar bias, fixating on a single part of a story. In the attention economy we definitely pay attention to what is more extraordinary and recent, and we’ll be shown these stories more often.
Mere exposure effect is about being influenced to like something because we are familiar with them, just by being exposed more frequently. Framing is similar, it’s about extracting different conclusions from the same story depending on how it’s presented. What we are exposed to on the internet is selected by the best buyers, and by our personal interests and stubbornness.

These are compounded into something called continued influence effect, which is a bias related to how hard it is to correct misinformation in our memory after having seen it the first time.
Moreover, our visibility is also affected by biases that make us seek certain things more than others. The automation bias, on the internet, is a tendency to rely too much on the algorithms for recommendations. We know these algorithms are not impartial and can be biased.
While the well-known confirmation bias, illusion of validity, optimism and pessimism biases, are all related to how much we chase, interpret, remember, and focus on information that is consistent and related to what we already believe. We like to stay in our bubble, we are our own filters of visibility with our biases. We are especially interested in reading and sharing content that connects to our own experiences and senses. These are biases that play a major role with what we’ll see in the next section, namely cultural insecurities.

Other biases are in relation to the over-availability of information and how it relates to our memory.
One such bias is the information bias, which is a tendency to look and search for information even though this information is meaningless to us, and cannot affect our actions.
The quest for information is also related to something we’ve seen in a previous section regarding the extension of our memory. The Google effect is a bias in which we are more likely to forget information if we know that it is readily available online.
Additionally, another popular bias under the name of Dunning-Kruger effect is the tendency of unskilled people to overestimate their own skills. With the whole internet acting as an external memory, it seems like more people are having this bias.
A bias that goes in the opposite direction is the imposter syndrome in which someone with enough experience feels they aren’t adequately qualified and fraudulent. With the whole internet pushing forward images of successful and perfect individuals, it seems like some people are more prone to this bias.

The over-availability of information itself can also lead to biases that makes it bypass certain of our defenses. As far as this is concerned, internet memes are an interesting case to examine.

The humor effect is a tendency to more easily remember things that are funny because they both capture our attention, stand out in a flood of information, and make us feel good. Humorous content also takes less time to process cognitively.
The picture superiority effect is the notion that pictures are remembered more easily and frequently than any other format. We’ve seen that one in the meme section and also said that pictures are processed more rapidly, 60 thousand times faster than text.
There are two ways to process information, through fast thinking or slow thinking, and memes harness speed to skip any form of slow thinking.

These effects, along with content that somewhat goes along with our preconceived views of the world, mean that visual internet memes will bypass criticism, go straight through a peripheral route, and lodge themselves in the brain, to be memorable.

Let’s now go briefly over some social biases, which will be helpful for us when tackling the cultural perspective in the next section.
Some of them are related to conformity and others to individuality in a social context. Online, as we now know, social interactions are mostly the same as in the real world, plus instant feedback, big exposure, and a sprinkle of anonymity and fabricated opinions.

When it comes to individuality, the biases of people on the internet are due to the constant exposure they have. For example, the spotlight effect is the tendency to overestimate the amount that others will pay attention to us.
Another bias, in relation, is the reactance: the urge to do the opposite of what others would want us to do, feeling a need to stand out and to assert freedom. It’s very common to see this contrarian behavior on social media.
The social comparison bias, yet again related, is about favoring candidates for a position that don’t compete with our own. We always compete on the internet, social media profiles are like public CVs and so this bias comes into play.

Other social biases tend to be linked to conformity such as the authority bias, the bandwagon effect, groupthink, social comparison bias, and in-group and out-group biases.
Authority bias is the tendency to trust and be more influenced by something or someone based on their origin and perception. We’ve seen earlier how state actors can abuse this to trade-up-the-chain to reach trusted mainstream media.
The bandwagon effect, groupthink, herd behavior, and in-group biases are all linked to preferential treatment to our own group, and keeping harmony, consensus, and conformity in it.
On the other side, there’s out-group homogeneity bias where we see members of an outside group as relatively similar. Confusingly, this bias also implies that the members of our own group are dissimilar.

Some people are more inclined to social biases than others. Some will recall images better than others. Distinctive people develop or react to biases in their own ways, be it because of their gender, sex, context, environment, geographical location, professional experiences, life experiences, education, etc..

This concludes our review of cognitive biases and how they get reflected on the internet. We’ve first seen some of the physical aspects of online addiction. Then we’ve covered three broad categories of biases: some related to the visibility, others to the information availability, and finally some related to social pressure. The biases related to visibility are about our fixation on what we see more frequently. The biases related to information availability are about what information we seek and how it affects our memory or confidence. Finally, the social biases are about how we conform to a group while still seeking individuality. On the internet these can be used to frame our views, bought or selected by algorithms, decide how information is accessible, and how social interactions get mapped unto social media with an instant feedback.

Table Of Content

References








Attributions: R. Fludd, Utriusque Cosmi, Tractatus primi, Oppenheim, 1620

Pepijn de Vos (pepijndevos)

SimServer: Tight Integration with Decoupled Simulators April 10, 2021 12:00 AM

I am working on Mosaic, a modern, open source schematic entry and simulation program for IC design. With a strong focus on simulation, I want to offer deep integration with the simulator, but also be able to run it on a beefy server and shield my program from simulator crashes. To this end, I have developed an RPC abstraction for interfacing with simulators remotely.

Here is a demo of a short Python script that uses Pandas, Matplotlib, and Cap’n Proto to run a CMOS netlist on Ngspice and Xyce and a behavioural Verilog equivalent on CXXRTL, allowing for easy verification of the transistor implementation.

You can see that the behavioural simulation is nearly instantaneously, while the spice results stream in much slower because they are doing a fully analog transistor simulation. You can see there is a bit of overshoot at the edges, and zooming in on that, you can see minor differences between the analog simulators because Xyce is using an adaptive timestep.

close up of overshoot

Now let’s take a step back and take a look at the design and implementation of this system. There are several reasons why I chose for a simulation server.

  • Ease of installation. Xyce is notoriously hard to install and only works on Linux as far as I know. An RPC protocol allows Xyce to run in a Docker container.
  • Performance. My laptop might not be the best place to run the simulation. An RPC protol allows the simulator to run on a beefy server, while running my user interface locally for a snappy experience.
  • Integration. Running a simulator in batch mode provides no indication of progress and requires setting up and parsing output files. An RPC protocol allows for direct, streaming access to simulation results.
  • Stability. It would not be the first time I’ve seen Ngspice segfault, and I’d hate for it to take the user interface along with it. An RPC protocol allows the same tight integration as its C API without linking the simulator into the GUI.

For the RPC library I settled on Cap’n Proto, but the next question is, what does the actual API look like? Ngspice has quite an extensive API, but the same can’t be said for Xyce and CXXRTL. So I could offer the lowest common denominator API of “load files, run, read results”, but one of my main goals was deep integration, so this is unsatisfactory. What I ended up doing is define small interfaces that expose a single functionality, and use multiple inheritance to assemble simulator implementations.

So I currently have 3 implementations of the run interface, and on top of that Ngspice implements the tran, op, and ac interfaces, with more to follow. I hope that in the future JuliaSpice will be a simulator that provides even deeper integration.

Please check out the code, and let me know your thoughts: github.com/NyanCAD/SimServer (How to expose simulator configuration and other functionality? Can we do remote cosimulation? Any other interesting usecases?)

Meanwhile, here is a demo of the example Python client running a transient and AC simulation on my VPS.

# on my VPS
docker pull pepijndevos/ngspicesimserver:latest
sudo docker run -d -p 5923:5923 pepijndevos/ngspicesimserver:latest
# in the examples folder
python ../client.py ngspice myvps:5923 rc.sp tran 1e-6 2e-3 0 ac 10 1 1e5

transient result AC result

April 09, 2021

Frederic Cambus (fcambus)

The state of toolchains in NetBSD April 09, 2021 10:42 PM

While FreeBSD and OpenBSD both switched to using LLVM/Clang as their base system compiler, NetBSD picked a different path and remained with GCC and binutils regardless of the license change to GPLv3. However, it doesn't mean that the NetBSD project endorses this license, and the NetBSD Foundation's has issued a statement about its position on the subject.

Realistically, NetBSD is more or less tied to GCC, as it supports more architectures than the other BSDs, some of which will likely never be supported in LLVM.

As of NetBSD 9.1, the latest released version, all supported platforms have recent versions of GCC (7.5.0) and binutils (2.31.1) in the base system. Newer (and older!) versions of GCC can be installed via Pkgsrc, and the following packages are available, going all the way back to GCC 3.3.6:

+---------+------------+-------------------+
| Package | Version    |      Release date |
+---------+------------+-------------------+
| gcc10   | GCC 10.2.0 |     July 23, 2020 |
| gcc9    | GCC  9.3.0 |    March 12, 2020 |
| gcc8    | GCC  8.4.0 |     March 4, 2020 |
| gcc7    | GCC  7.5.0 | November 14, 2019 |
| gcc6    | GCC  6.5.0 |  October 26, 2018 |
| gcc5    | GCC  5.5.0 |  October 10, 2017 |
| gcc49   | GCC  4.9.4 |    August 3, 2016 |
| gcc48   | GCC  4.8.5 |     June 23, 2015 |
| gcc3    | GCC  3.3.6 |       May 3, 2005 |
+---------+------------+-------------------+

The focus on GCC doesn't mean that the GNU and LLVM toolchains cannot coexist within NetBSD, and work has in fact been done during the last decade to make it happen.

Despite currently not being built by default in official NetBSD releases, LLVM has been imported in the NetBSD source tree in 2013. Daily images are built from NetBSD-current for selected platforms (at least amd64, i386 and evbarm) with the MKLLVM and HAVE_LLVM build options enabled, and contain LLVM and Clang.

Moreover, NetBSD has invested a lot of work on LLVM during the past few years, including funding some developer contracts for Kamil Rytarowski (kamil@) and Michał Górny (mgorny@), which allowed them to work on various parts of the LLVM toolchain to add and enhance support for sanitizers, and to improve LLDB support.

They both published several dozen articles on the NetBSD blog along the way, retracing their journey. Kamil's final report about upstreaming support to LLVM sanitizers summarizes the work accomplished. Thanks to this work, sanitizer support on NetBSD is mature and mostly on par with Linux. As a result, because LLVM is upstream for GCC sanitizers, they are also available in GCC on NetBSD. Similarly, Michał's final report on his LLDB work details the achievements on the debuggers front.

As always, work continues towards keeping the toolchains up to date, and upstreaming local changes whenever possible.

April 06, 2021

Bogdan Popa (bogdan)

Screencast: Writing a Resource Pool Library for Racket April 06, 2021 04:42 AM

After hacking on redis-lib for a bit on Sunday, I decided to write a general-purpose resource pooling library that I can re-use between it and http-easy and I recorded the process. You can check it out on YouTube: You can find the library on GitHub. One particularly interesting bit about the library, that I did not to record, is that the tests are all property-based. I might do another screencast at some point to talk about how they work and the bugs they found in my original implementation (from the video).

April 05, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Actors and Incentives: State Actors: PSYOP, Narrative Warfare, And Weaponized Tech April 05, 2021 09:00 PM

Access to the mountain of the philosophers is blocked by a wall of false, sophistical doctrines.

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 2 — The Actors and Incentives
  • Section 3 — State Actors: PSYOP, Narrative Warfare, And Weaponized Tech
Table Of Content
  • Nothing New — Censorship and Propaganda
  • Psychology as a Weapon
  • Information Society = Information War & Weaponized Tech
  • Internal: Population Control, Data and Surveillance
  • Computational Propaganda
  • External: Cyber Wars
  • Social Media As Battleground — State Sponsored Trolling
  • Memes as Vectors of Propaganda

Since ancient times, nations have tried to write history from their own point of view. As they say, history is written by the victors. Today, the speed of the internet allows rewriting the narrative in real-time, and state actors will certainly take advantage of this. Namely, there are two ways, or generations, of state information-control practices: an information scarcity approach, aka censorship, and an embracing of information approach, speech itself as a censorial weapon. Both require monitoring of the population to properly adapt the tactic in use.

State surveillance is nothing new either, but the era of information abundance gives the capability of amassing data on a colossal scale.

Historically, nations have used propaganda and surveillance campaigns internally, to control their population, and externally, to sway the opinion of other countries, direct revolts, and other favorable incentives.
Today, these dynamics have moved to the cyberspace. The internet has been part of all contemporary conflicts as a force acting on the public opinion. Ultimately, the entity that is able to control the media, the discussion environment, the social networking sites, is in control of the narrative.

Before diving into the new era war, let’s review how psychology can be used as a weapon.

Psychology has always been used for military gain but got more traction with the concept of total war. This type of warfare includes civilian-associated resources and infrastructure as legitimate targets, mobilizing all resources of society to fight the war, prioritizing it over other needs — an unrestricted war. Thus, having a way to sway civilians in this type of war is a must-have in the fighting arsenal.
Although alone it isn’t sufficient and needs to be complemented with simultaneous diplomatic, military, and political tactics. Psychological war takes into consideration the full environmental scope.

A psychological war consists of psychological operations, also known in the contracted form as PSYOPs, which consist of planning psychological activities designed to influence the attitudes and behavior for a political or military objective — to evoke a planned reaction from the people.
This is a vague definition that includes anything we’ve seen from the previous influence, propaganda, and persuasion section. These tactics, when used as weapons, will be backed up by important state incentives making them more powerful, and arguably more successful, than when they are employed by other entities.

In a total war the target can be individuals, civilians, organizations, groups, or governments. The audience needs to be well delineated and studied so that the message is appropriately formed, as we’ve discussed beforehand. There are four audiences to a message:

  • The ultimate one, the real target.
  • The intermediate one, which are likely to receive the message and could be part of the target audience.
  • The apparent one, an audience that seemingly appears to be the target but isn’t the real intended one.
  • The unintended one, which the planner didn’t intend to reach but still received the message.

The receptivity, as we’ve kept saying, depends on many cultural and environment factors. However, in a military setting and with government sponsorship, vulnerabilities can be artificially created through means such as kinetic (bombs and guns), non-lethal biological weapons affecting human psyche and mood, and physical threats. The full environmental scope comes into play.
These operations will employ the schemes we’ve seen such as anchors, hooks, and imageries. They will be delivered through different methods, and will have a clear aim. The objective is either a positive one — to reinforce the behaviour and feelings of friendliness —, a negative one — to destroy morale, weaken the adversary, create dissonance and disaffection —, or to gain support from the uncommitted and undecided.

When faced with a new weapon, states have to develop defensive systems. These include doing PSYOPs on their own population, censorship, and counterpropaganda. Although some countries have laws that forbid applying psychological operations on domestic grounds and on their own citizens.

Nonetheless, most countries have integrated in their military structure distinctive branches that are specialized in the type of operations related to the information and psychological sector.
Here is a generic definition of the role of such unit:

The integrated employment of the core capabilities of electronic warfare, computer network operations, psychological operations, military deception, and operations security, in concert with specified supporting and related capabilities, to influence, disrupt, corrupt or usurp adversarial human and automated decision-making while protecting our own.

When deployed these well-trained units can apply these tactics at multiple levels.

  • Strategic: applies to anything outside the military area, even in peacetime, to prepare the global information environment.
  • Operational: applies to a joint military operation to help achieve consistency between the national objective and the objective of the coalition who they are partnering with.
  • Tactical: applies to on the ground operations, especially when facing an opposing force, to support the actions of a tactical commander.

What’s important for these units is to be able to measure the effectiveness of their operations (MOE). In military, if it can’t be measured then it’s no good. The internet offers clear metrics and is the perfect ground for that.

The ethics of these methods is arguable. Some say that it’s an arms race and that they are forced to apply them, that the development is unavoidable. Others say they offer a way to reduce the “significant loss of life”. As with anything, there’s a lot of subjectivity when it comes to these.

In an information society, information is the center of the new-generation war and thus these operational units are put to the forefront.
Information superiority is a must to win. Everyone fights for their narrative.

This new generation war is often referred to as the 6th one, where the aim is to destroy economic potential, not take part in physical battlespace, and be the dominant information holder.
Like the information society, this type of war is network-centric, taking into consideration the inter-relations between bubbles and how affecting one could lead to a quick victory. This requires information superiority to be able to plan in advance, conceal, and prepare the multiple aspects that will jointly create a favorable ground for the goal of the operation — to easily make a nation ply to your intentions.
From information, moral, psychological, ideological, diplomatic, economic, and so on, these are all required. Practically, this means using mass media, religious organizations, cultural institutions, NGOs, scholars, etc.. In today’s world this translates into using the right influential methods, and influencers on social media.

The data that is available on the internet, and particularly social media, can be gathered, stored, and linked, to finally become an armor or weapon in the hands of a state entity. It is a tool that can be used both defensively and offensively.

This intelligence gathering often take the form of social media surveillance. The information can be used to identify possible bad actors or individuals vulnerable for a PSYOPs campaign.

On the defensive side, a prominent explicit example of intel gathering is how the USA is now asking VISA applicants to submit five years of social media handles for some selected platforms.
However, the intel gathering probably happens most of the time unbeknownst to the data subjects. This is what has been revealed by internal whistle-blowers, and by more explicit state requirements like ISP keeping internet connection logs for at least 2 months in some countries.

Additionally, we need to remember that social media platforms are “independent” businesses. As such, they are bound by the legalities of the geographic areas in which they want to operate. Consequentially, governments can pressure them by taking the legal route and force them to act a certain way to operate on their legislature. Or they can simply buy access to that data from data brokers.
For example, it’s not uncommon that the platforms will have to obey when receiving an order to censure certain messages. Sometimes notifying users, sometimes not notifying them, according to the transparency rules of the platforms and the legislation in which it is operating. This is the simplest method to get rid of dissidents and keep order in a country. However, for the citizen of information societies, censure isn’t looked at too well. The people are craving honesty and individuality, while rejecting authority or association to greater organizations like nations and their wealth, as we’ve seen in the previous section.

On the offensive side, micro-targeting — that is using a conjunction of very specific attributes such as political leaning and interests to target individuals — can be used to amplify a message and have measurable metrics when performing a psychological operation. We’ll come back to this topic in a bit.

No need to use forceful actions when speech itself can be used as a weapon. This is what computational propaganda is about, using social media political discussions to sway opinions. Especially that this is the go-to place to discuss these things. This is known as platform weaponization. It can be used internally, to thwart opposition, or externally, to influence the decision-making of other states.
The act itself could either be state-executed, by the related military branch, state-coordinated, state-incited, or state-leveraged or endorsed.

Computational propaganda relies on the anonymity provided by the platforms which favors black propaganda and gives more credibility, making it seem more genuine. This means that states can deny involvement while inciting or leveraging activities happening on social media for their gain.
Without this knowledge, it is hard to know whether an opponent is attacking or not and how to defend against such attacks.

The attacks can be conducted with bots, automated scripts to scrap and spread data, with trolls, online accounts that deliberately target individuals to trigger and harass them, PR, public relation through news outlets and others, and with memes, which we’ve covered previously.

Bots, and in particular social-bots, give the ability to create fake online voices, and so to fabricate a public opinion. They are one of the weapon of choice in the cyber warfare of who can shout their narrative the loudest. Online, popularity is measured with “likes”, “votes”, “friends”, “followers”, and others which can all be faked.

The advantage that algorithmic campaigns provide is the ability to manipulate what people see, to temporarily boost the visibility of a message, to bypass the social media “democratic” process.

Social bots are accounts that impersonate people of a certain demographic, mimicking social interactions on the platforms. These can be used to create a sense of consensus where there is none, or generate doubt on a subject that is supposed to be widely accepted in the target audience.
With the visibility also comes awareness. This can be used by political parties to obtain votes, and to look more popular than others, to look more seductive and attractive.

Another use for such bot is something called astroturfing, fighting for a “turf”. This is about creating the impression of grassroots movements in favor or against something, be it an event, person, or idea.

This visibility can be used to choke off debates, smothering adversaries by making it seem like issues are one-sided, controlling the bulk of the conversation. A sort of artificially created social censorship. Trolls can be used for such effect too as we’ll see.

Alternatively, algorithmic amplification can be used to muddy political issues, create chaos and instability in the discourse (local or another country), to generate uncertainty, distrust, and divisions.
This is absolutely destructive if combined with on-the-ground actions when discord and confusion reigns in the population, and devastating when combined with economic pressure.

Not all types of content prosper on social media, we’ve seen that before. We’ve learned that outrage and politics resonate well, the things that clash with our cultural insecurities. That is why the bot-led political campaigns tend to come from the most radical parties, it is the most appropriate medium for them.
Apart from these, any type of negative content spreads like wild-fire. This is useful to amplify a sentiment of dissatisfaction and disagreement.

In retrospect, social media certainly has some part to play in the process of radicalization, however none of the messages there would resonate if they didn’t reflect some issues that were already present in us. As usual, it’s an exaggeration of our own ambiguous cultural code.

This is exactly what some state entities use to attempt to influence elections in other countries. Like bots generating discord and chaos regarding a certain topic, micro-targeting can be used to personalize messages towards hyper-partisans. In turn, these selected partisans will be pushed into their narrative, radicalized, polarized, and moving the Overton window. Furthermore, this radicalization gives them a bigger exposure and sets the ground for instability in a population.
This can be an attractive outcome in coordination with other types of actions.

These hyper-partisans can be converted into trolls, online accounts that deliberately target and harass particular individuals. They can either be paid/sponsored real people or bots, either aware or unaware of the tacit agenda, with or without explicit instructions, and either real partisans or fabricated ones.
This is frequently used along with black PR that are campaigns to deliberately engage in disinformation and harassment against a perceived opponent. It is used to intimidate and silence individuals.

There are countless examples of this tactic being used by states, either deliberately launched by them, or the states harnessing an already ongoing attack. These include, but are not limited to, making death and rape threats, amplifying vitriolic attacks, making accusations of treason or collusion, disseminating libelous disinformation, spreading doctored images and memes, and sowing acrimonious sexism.
The target of these attacks are usually journalists, activists, human rights defenders, and vocal members of an opposite ideology. These types of attacks are used everywhere in the world, some for capitalistic gains, and others for state gains.

Yet another efficient way for states to impose their ideas on the minds is through the news networks, all the headlines. They can employ the typical colored language, the emotive stories, imageries, hooks, and anchors we’ve seen before to shape the narrative.
Promotional culture is also an important aspect of pushing a narrative through the headlines.

Aside from these, nations can rely on the visibility principle, once again, to promote the stories they want. This could be by voting on social media with bots, or by infiltrating the news network and trading up the chain. News websites can be constructed from scratch and filled with bogus content, to then rely on the spreading mechanism and citizen journalism to gain credibility and attention. It might finally emerge on the mainstream news media consumed by the desired target audience.
Moreover, trading up the chain can be achieved by using fringe social media platforms which are prone to influence other online ecosystems. This works very well when the message is based on a kernel of truth but with a spin. Even when more reputable outlets are only debunking them, it’s giving the content a position in mainstream news.

All these are extremely hard to defend against, a David vs Goliath affair, and so states prefer to defend through a good offense. Counterpropaganda, part of counter-insurgency (COIN), is hard to practice on social media. However, one particular vector of information that can be used as COIN are memes, the new favorite shell for political messages.

Governments are getting more and more interested in memes, especially as a useful method to compact narrative and culture in a transportable container.
All modern, arguably post-modern, PSYOPs involve the propagation of memes on social media. Meme wars are an inherent part of political life. We’ve amply discussed them in a previous section.
They are the embodiment of the competition over narrative, creating coherent constellations of meanings and stories. We know it’s easy to overlook them because they are a common way of expression and often use humor, but that makes them the perfect craft for information warfare.

What appears like a prank or trope, soon turns into an influential narrative campaign spread by bots and trolls. Remember that memes aren’t limited to their internet format but that they go beyond, that this is only the appropriate envelope they take to move on that medium.
As such, like all that we’ve seen in this section, they can be used defensively, offensively, and to sway people on the fence, to recruit them or drive them away. They are used both for local and international sentiments.

Many countries employ, or discuss employing, memetics in their military sector. It consists of having separate units in a specialized center for these type of operations, which would be a sub-branch of the usual information and psychological operations but tailored to social media.
These meme warfare centers, or referred to as meme farms when described by opponents, would include interdisciplinary experts such as cultural anthropologists, computer scientists, economists, and linguists.

There are many open discussions about which countries actually employ these units, knowing their associations would reduce their power, but we find implicit indicators in the wild. Most countries and international associations have them in sight, take them seriously, or are actually armed with them already. Nowadays, any side of a conflict will use them, that includes all NATO nations, Russia, Hong Kong, China, Armenia, Azerbaijan, Palestine, Israel, India, Pakistan, etc..

However, it is an unconventional mean and so it is tricky to be put in place, or to let the citizens know it is in place. It is hard to include in a military structure. Memes are abstract concepts and they might sound unconvincing and bring obstacles related to finance — a lack of investments —, culture — the mindset to grasp the new generation warfare —, legalities — to encompass the ethics around using them —, and bureaucracies — who should be in command.
Additionally, there needs to be scientific proofs of their efficiency to justify the investment. Efficiency should be tested and measurable, and this isn’t straight forward when a change in attitude doesn’t necessarily correlate with a change in behavior.

If a nation lacks the framework, mindset, resources, knowledge, and tools, they’ll leave the advantage to others that can embrace this new paradigm. Nonetheless, it’s apparent that this is a threat but it wasn’t until 2017 that the EU and NATO established together a center to counter “hybrid threats”. Though its purpose is more analytical than proactive, for them, at least in the public eye, the best form of defence is awareness.

Theoretically, a meme, like a virus, can be categorized in infect, inoculate, and treat. To transmit, prevent or minimize, and contain a message.
A quarantine doesn’t really work in the virtual world though and combining this with the neutralization of the infected memeoids by killing them in the real world might have the opposite effect. History has shown that it is likely it will instead validate the ideology of the meme.

Detection plays an important role in the defence mechanism, to then be able to launch a counter-message. In practice, it has been attempted for counter-radicalization, a form of counter-insurgency, with mixed effects.
An invisible enemy is and invincible enemy, that is why identifying, cataloguing, and tracing should be done beforehand. Which can be almost impossible when the opponent, be it foreign or local, hides within the civilian population, netizens. Thus, the focus on “hybrid treats” center.

In all cases, displacing and overwriting dangerous pathogenic memes is normally done by replacing them with more benign ones. These reactionary memes have different degrees of applicability, actionability, operationalization, and success.
The fruitful reactionary memes have one thing in common: they use humor, irony, and sarcasm to deligitimize the message of an opponent by ridiculing and mocking it. This is similar to WWII propaganda techniques used in cartoons.
“No one is above satire.”

This concludes our review of how state actors are employing the internet narrative as part of their overall information and psychological operations. We’ve first seen how this is nothing new, how psychology has been used as a weapon for quite a while now, both on the defensive and offensive. Then we’ve looked at how data is at the center of wars, now that we’ve moved to an information society. Next, we’ve seen how this data can be part of the surveillance campaign of states. Later we’ve examined computational propaganda, how algorithm dictates our world and how consequently they can be used by nations. Finally, we’ve dived into social media, bots, trolls, news control, and memes, all communication vectors that rely on gaining visibility and credibility. Speech as a method of censorship, speech as a method of propaganda, speech as a weapon.

Table Of Content

References








Attributions: Geheime Figuren der Rosenkreuzer, Altona, 1785

Ponylang (SeanTAllen)

Last Week in Pony - April 4, 2021 April 05, 2021 12:30 AM

Ponyc 0.39.1 has been released! The Roaring Pony group has made progress, including an addition to the standard library math package. Sean T. Allen is looking for assistance with the ponydoc project.

April 04, 2021

Derek Jones (derek-jones)

The first computer I owned April 04, 2021 09:06 PM

The first computer I owned was a North Star Horizon. I bought it in kit form, which meant bags of capacitors, resistors, transistors, chips, printed circuit boards, etc, along with the circuit diagrams for each board. These all had to be soldered in the right holes, the chips socketed (no surface mount soldering for such a low volume system), and wires connected. I was amazed when the system booted the first time I powered it up; debugging with the very basic equipment I had would have been a nightmare. The only missing component was the power supply transformer, and a trip to the London-based supplier sorted that out. I saved a months’ salary by building the kit (which cost me 4-months salary, and I was one of the highest paid people in my circle).

North Star Horizon with cover removed.

The few individuals who bought a computer in the late 1970s bought either a Horizon or a Commodore Pet (which was more expensive, but came with an integrated monitor and keyboard). Computer ownership really started to take off when the BBC micro came along at the end of 1981, and could be bought for less than a months’ salary (at least for a white-collar worker).

My Horizon contained a Z80A clocking at 4MHz, 32K of RAM, and two 5 1/4-inch floppy drives (each holding 360K; the Wikipedia article says the drives held 90K, mine {according to the labels on the floppies, MD 525-10} are 40-track, 10-sector, double density). I later bought another 32K of memory; the system ROM was at 56K, and contained 4K of code, various tricks allowed the 4K above 60K to be used (the consistent quality of the soldering on one of the boards below identifies the non-hand built board).

North Star Horizon underside of boards.

The OS that came with the system was CP/M, renamed to CP/M-80 when the Intel 8086 came along, and will be familiar to anybody used to working with early versions of MS-DOS.

As a fan of Pascal, my development environment of choice was UCSD Pascal. The C compiler of choice was BDS C.

Horizon owners are total computer people :-) An emulator, running under Linux and capable of running Horizon disk images, is available for those wanting a taste of being a Horizon owner. I didn’t see any mention of audio emulation in the documentation; clicks and whirls from the floppy drive were a good way of monitoring compile progress without needing to look at the screen (not content with using our Horizon’s at home, another Horizon owner and I implemented a Horizon emulator in Fortran, running on the University’s Prime computers). I wonder how many Nobel-prize winners did their calculations on a Horizon?

The Horizon spec needs to be appreciated in the context of its time. When I worked in application support at the University of Surrey, users had a default file allocation of around 100K’ish (memory is foggy). So being able to store stuff on a 360K floppy, which could be purchased in boxes of 10, was a big deal. The mainframe/minicomputers of the day were available with single-digit megabytes, but many previous generation systems had under 100K of RAM. There were lots of programs out there still running in 64K. In terms of cpu power, nearly all existing systems were multi-user, and a less powerful, single-user, cpu beats sharing a more powerful cpu with 10-100 people.

In terms of sheer weight, visual appearance and electrical clout, the Horizon power supply far exceeds those seen in today’s computers, which look tame by comparison (two of those capacitors are 4-inches tall):

North Star Horizon power supply.

My Horizon has been sitting in the garage for 32-years, and tucked away in unused rooms for years before that. The main problem with finding out whether it still works is finding a device to connect to the 25-pin serial port. I have an old PC with a 9-pin serial port, but I have spent enough of my life fiddling around with serial-port cables and Kermit to be content trying a simpler approach. I connect the power supply and switched it on. There was a loud crack and a flash on the disk-controller board; probably a tantalum capacitor giving up the ghost (easy enough to replace). The primary floppy drive did spin up and shutdown after some seconds (as expected), but the internal floppy engagement arm (probably not its real name) does not swing free when I open the bay door (so I cannot insert a floppy).

I am hoping to find a home for it in a computer museum, and have emailed the two closest museums. If these museums are not interested, the first person to knock on my door can take it away, along with manuals and floppies.

Bogdan Popa (bogdan)

Screencast: Building a Redis Session Store for Koyo April 04, 2021 10:37 AM

I decided to write a library for storing koyo sessions in Redis today and I recorded the process. If that sounds appealing, you can check it out on YouTube:

April 02, 2021

Carlos Fenollosa (carlesfe)

Fed up with the Mac, I spent six months with a Linux laptop. The grass is not greener on the other side April 02, 2021 09:11 AM

This article is part of a series:

  1. Seven years later, I bought a new Macbook. For the first time, I don't love it
  2. How I moved my setup from a Mac to a Linux laptop
  3. This article
  4. (TBD) My review of the M1 Macbook Air

Due to very bad decisions by Apple's product marketing teams, Mac hardware and software had been in steady decline since 2016.

Therefore, there has been a trickle of articles on the Geekosphere about people switching from Macs to Linux or Windows.

This is the contrarian view. Don't do it.

The TL;DR is right there in the title: migrating to Linux is fine, but don't expect a better experience than the Mac.

My experience with the Dell XPS 13" Developer Edition was positive in general, including a self-hosted Cloud setup, but not good enough to convince me to stay with it.

We will cover:

  1. A comparison of generic productivity software: email, calendar, image manipulation, etc.
  2. Available power tools to customize your keyboard, trackpad, and more.
  3. A quick jab at decades-old issues which still haven't been solved.
  4. Misc stuff that Linux does better than the Mac.
~~~~~

I feel like I need to clarify that this is an article aimed at Mac users who are considering a migration to Linux in hope of a more polished system. As usual, personal experiences and requirements are subjective. I know that Ubuntu ≠ Gnome ≠ Linux. I also know that I'm not entitled to anything, everybody is welcome to send patches. Just let me say that if you try to cherry-pick any single issue, you're missing the forest for the trees.

~~~~~

Linux productivity software is fine, but there are rough edges for the power user

The typical disclaimer when recommending Linux to a Mac/Windows user is that some proprietary software may not be available, like MS Office, Photoshop, games, etc.

Nobody says, "the main problem you will find with Linux is that email and calendar clients fall apart when you scratch under the surface."

It is truly ironic because I ran MS Office with Wine and it worked well but I was unhappy with my email workflow.

Yes, the apps I missed the most from the Mac were Mail.app, Calendar.app, and Preview.app.

I am an extreme power user, to the point that many of the keys on my keyboard don't do what the keycap says. I want my apps to let me do easy things fast while allowing me to do complex tasks with a bit of extra work.

I send and receive maybe 100 emails per day. Most of them are HTML, with attachments, video conference invitations, and such. I don't live in a vacuum. I can't ask my clients to send me plaintext email only. I need to send long emails with pictures, I want my zoom invites to appear automatically in my calendar.

For some reason Mail.app gets a lot of criticism, but it does almost everything well. It has conversation view, search is fast and helpful, multiple accounts are combined seamlessly including autodetection of the "From" field based on the recipient, and smart folders (search folders) are updated when you need them.

On Linux, the only email client with a native "conversation view" is Geary, which is in early development and still very buggy. Evolution is fine and well-integrated with the rest of the desktop apps, but the lack of conversation view was a deal-breaker for me. Thunderbird is an excellent email client, but conversation view is provided by a plugin that is also buggy. Other options like Claws, Sylpheed, Kmail, and terminal clients are more limited in terms of features and don't work for me.

I ended up using Thunderbird, but I felt like I was doing my email with handcuffs. Suffice to say, I had both Thunderbird and Gmail open and used either one depending on the task I needed to complete.

The situation of calendar and contacts clients is similar, with the same contenders. I also ended up using Thunderbird along with Google Calendar.

About PDF and basic image management, anybody who has used Preview.app will realize that it's much more than just a viewer. There is simply no replacement on Linux. You'll need to open either the Gimp or Xournal for any basic editing. I am an advanced Gimp user, but for most operations, Preview.app is faster and more convenient.

Desktop notifications are something we don't think a lot about, but a bad system can be very annoying. Gnome has a system-wide framework, which is not well thought in terms of dealing with the actual notifications.

Most apps have their own notifications system which runs in parallel, especially Thunderbird and Evolution. You end up with different types of notifications on different parts of the screen, and a non-consistent UI to deal with them.

Finally, on the Mac, you can find an ecosystem of alternative paid PIM apps, like Spark, Fantastical, Things, and others. There is no equivalent ecosystem on Linux, probably because they would be difficult to monetize.

Power tools are more limited and more difficult to use

The previous section could be summarized as "Linux PIM software is fine in general, but gets in the way of power users."

That is counterintuitive, right? Linux is a much nerdier OS than the Mac and everything is customizable.

But when you jump from theory to practice, at some point you just want a tool to help you set up your config, without the need to edit your trackpad driver source file.

Any advanced Mac user knows about Karabiner, BetterTouchTool, Choosy, Alfred, Automator, and more.

With Linux, you can achieve almost the same feature set, but it is harder and more limited.

For example. To customize your keyboard, you will need a combination of xdotool, xbindkeys, xcape, xmodmap and setxkbmap to capture some event and then run a shell script. There is a Gnome Shell plugin that allows you to tweak your keyboard, but it's nowhere near Karabiner.

If you want to achieve some specific action you need to read four or five manpages, search online, and figure out how you are going to put the pieces together. That made me appreciate Karabiner and BTT much more.

Furthermore, I couldn't find a real alternative to BTT to customize trackpad multi-touch gestures. I tried a few approaches with libinput-gestures but none worked.

In the end, I was able to replicate most of my macOS power tools setup via input hooks and shell scripts, but it took much longer than it should have. I found it surprising that, given the number of nerds using Linux every day, there are no specific tools equivalent to those mentioned above.

"I Can't believe we're still protesting this crap"

Please allow me to make a bit of fun of issues that existed back in 1999 when I started using Linux and still exist today.

  • Screen tearing with the intel driver. Come on. This was solved on xorg and now with Wayland it's back. I fiddled multiple times with the settings but couldn't fix it. Even with OpenBSD it was easier to fix. The default settings should be better. I don't care if the video driver will use an extra buffer or whatever.
  • Resolving new hosts is slow, with a delay of about 2-3 seconds. I tried to disable IPv6 and other tricks from Stackoverflow threads, but none solved the issue completely. Again, an issue with the default settings. macOS does some DNS magic or something and the network feels much faster.
  • Resuming after suspend seems to work at first. As soon as you start to trust it and not save your work before closing the lid, it betrays you and you lose your work. Later, you upgrade the kernel and it works all the time for weeks until you upgrade the kernel again and it goes back to working 80% of the time. What a mess.

We've come a long way with Linux on the desktop but I think it's funny that some things never change.

Linux also hides some gems

I want to end this review on a positive note.

During those six months, I also took notes on apps and workflows that are still better on Linux.

  • Tracker/search is better and faster than Spotlight. It's a shame that not all apps take advantage of it, especially Thunderbird.
  • Firefox is amazing. On the Mac, Safari is a better choice, but I was very happy using Firefox full-time on Linux. I am going to miss some great plugins, like Multi-account containers, Instagram-guest, Reddit Enhancement Suite, and of course NoScript and uBlock Origin
  • Nautilus is better than the Finder. It's not even close.
  • The Gnome Shell Extension Gallery has many hidden gems, like Nothing to say which mutes the microphone system-wide with a shortcut, the Emoji selector, Caffeine to keep your computer awake, a Clipboard manager, and Unite to tweak basic UI settings. I am now using macOS equivalents to those, and I discovered their existence thanks to the Linux counterparts.
  • Insync for Linux is better than the official Google Drive File Stream for the Mac. In fact, I am now using the Mac version of Insync.
  • Gimp and Inkscape are excellent apps, and it's a pity that the macOS ports are mediocre. I'd rather use them than Pixelmator/Affinity Designer. Hopefully, someday either GTK or these apps will get decent macOS versions.
  • apt-get was a revolution when it was released in 1998 and it is still the best way to manage software today. brew is a mediocre replacement.
  • I paid for Crossover which allowed me to use MS Office and other Windows apps I needed. Kudos to the Wine developers for almost 30 years of continuous effort.
  • Xournal is an obscure app that allows you to annotate PDF documents as well as draw with a Wacom tablet. I used it constantly as a whiteboard for online presentations. The macOS port is very buggy, unfortunately, so I use OneNote which is not that good.

Hopefully, the success of paid tools like Insync or Crossover can encourage the developer ecosystem to continue developing quality apps, even if they are non-free, or supported by donations.

What's next?

On November 10th Apple showed us the future of the Mac and released again laptops worth buying. So I bought the 2020 M1 Macbook Air. You will read a review of it soon.

The hardware is much better than the Dell's and, I guess, every other PC laptop. The software ecosystem is a big improvement over my Linux setup, and Big Sur course corrects the absolute mess that Catalina was. With every passing year, the iCloud offering keeps getting better, especially if you have other Apple devices.

I am somewhat sad that I couldn't join the Linux Resistance. After all, I've been an annoying proselytizer heavy free software advocate in the past, and I still am, though I nowadays admit there are many nuances.

The experience of using Linux as a daily driver has been very positive for me, but I do need my productivity. I can work much faster with macOS and iCloud than I was with Linux and my self-hosted cloud setup.

If there ever was a period where the Mac experience was worse than Linux, it is now over. The Mac ecosystem wins again. Don't switch to Linux expecting it to have fewer papercuts than the Mac. It's quite the opposite.

There is definitely grass on the other side of the fence, but it is not greener.

Tags: apple, linux

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

April 01, 2021

Gergely Nagy (algernon)

The Logical Next Step April 01, 2021 12:30 PM

It's been a while I posted here, even though there were a lot of things happening around me. Alas, some of those led me onto a path that I am now ready to talk about: it's been brewing for a good while now, and I'm excited to announce it today!

I've spent the past two years working on Kaleidoscope and Chrysalis, and enjoyed every moment of it, keyboards, ergonomic keyboards, are my passion. Today, I'm taking that a step further, and am joining a major keyboard manufacturer to work on their input devices from now on. Not only will I be working with an incredible team developing the next generation of smart, cloud-connected input devices, but we will be bringing my own devices to the market aswell!

I was originally approached by a different team from the company, due to having a background in IoT and security, but the talks quickly turned to ergonomics and input devices (my greatest fault is that I can't stop talking about things I'm passionate about), and after a bit of back and forth, we ended up deciding that we can combine all our experiences, all our knowledge, and build products where all of them are put to good use. This is an incredible opportunity to put everything I've learned so far while working on Kaleidoscope, Chrysalis, and the many different projects of the past, towards the goal of building products that will revolutionize what we think about input devices.

I can't wait to share the crazy new ideas we've come up with during the past year of negotiation!

Gustaf Erikson (gerikson)

March April 01, 2021 07:35 AM

Derek Jones (derek-jones)

Linux has a sleeper agent working as a core developer April 01, 2021 12:36 AM

The latest news from Wikileaks, that GCHQ, the UK’s signal intelligence agency, has a sleeper agent working as a trusted member on the Linux kernel core development team should not come as a surprise to anybody.

The Linux kernel is embedded as a core component inside many critical systems; the kind of systems that intelligence agencies and other organizations would like full access.

The open nature of Linux kernel development makes it very difficult to surreptitiously introduce a hidden vulnerability. A friendly gatekeeper on the core developer team is needed.

In the Open source world, trust is built up through years of dedicated work. Funding the right developer to spend many years doing solid work on the Linux kernel is a worthwhile investment. Such a person eventually reaches a position where the updates they claim to have scrutinized are accepted into the codebase without a second look.

The need for the agent to maintain plausible deniability requires an arm’s length approach, and the GCHQ team made a wise choice in targeting device drivers as cost-effective propagators of hidden weaknesses.

Writing a device driver requires the kinds of specific know-how that is not widely available. A device driver written by somebody new to the kernel world is not suspicious. The sleeper agent has deniability in that they did not write the code, they simply ‘failed’ to spot a well hidden vulnerability.

Lack of know-how means that the software for a new device is often created by cutting-and-pasting code from an existing driver for a similar chip set, i.e., once a vulnerability has been inserted it is likely to propagate.

Perhaps it’s my lack of knowledge of clandestine control of third-party computers, but the leak reveals the GCHQ team having an obsession with state machines controlled by pseudo random inputs.

With their background in code breaking I appreciate that GCHQ have lots of expertise to throw at doing clever things with pseudo random numbers (other than introducing subtle flaws in public key encryption).

What about the possibility of introducing non-random patterns in randomised storage layout algorithms (he says waving his clueless arms around)?

Which of the core developers is most likely to be the sleeper agent? His codename, Basil Brush, suggests somebody from the boomer generation, or perhaps reflects some personal characteristic; it might also be intended to distract.

What steps need to be taken to prevent more sleeper agents joining the Linux kernel development team?

Requiring developers to provide a record of their financial history (say, 10-years worth), before being accepted as a core developer, will rule out many capable people. Also, this approach does not filter out ideologically motivated developers.

The world may have to accept that intelligence agencies are the future of major funding for widely used Open source projects.

March 31, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Actors and Incentives: Internet, In Between The Corporate, Private, And Public Spheres March 31, 2021 09:00 PM

For all are Men in Eternity, Rivers, Mountains, Cities, Villages. In your imagination, of which this World of Mortality is but a shadow. For it includes all things, and everything that exists, only exists through it

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 2 — The Actors and Incentives
  • Section 2 — Internet, In Between The Corporate, Private, And Public Spheres
Table Of Content
  • Availability of Information
  • Networks And Cultures
  • Individuation
  • Self-Generated Societies
  • Making Up Your Own Mind + Learning & Creativity
  • Citizen Journalist
  • Cognitive effects
  • Private Life Becoming Public Life
  • Passion Economy
  • Extreme Noeliberalisation

The internet is a new resource and when introduced in our social structures it has fueled the construction of utilities around it. Like any tool it has no effects on its own but only through its usages. In particular, it has altered our capacity of communication making it interactive, multimodal, asynchronous or synchronous, global or local, many-to-many, one-to-many, one-to-object, object-to-object.
In this section we’ll go over some of the things the internet allows to do through the platforms and services hosted on it. We can’t go over everything because that would be equivalent to describing what modern life is about. Instead, we’ll paint a rough picture of the internet in between the corporate, private, and public sphere.

We can describe ourselves as an information society, an internet-everywhere society. Calling ourselves digital natives, digital citizens, digital cosmopolitans, and netizens.
Everything that can be digitized gets digitized. According to a study published by Martin Hilbert in Science in 2011, 95% of all information in existence on the planet is already digitized and accessible through the internet.
This has shrunk the globe, brought knowledge to everyone, altered the way we communicate and connect, changed the way we make money, and changed the way we think.

The internet reflects us, it can convey the best and worst of humankind. It is an open-ended system, like our societies, with the inter-relations mapped in the digital space, with all the same logics that exist between cultures, organizations, and technologies.
Similarly, the internet is networked and so are our societies: constructed around personal and organizational networks, without clear boundaries. Thus, the same paradigms and dynamics that apply in our real world are echoed online. Studying the internet is studying anthropology.

Previously we’ve seen how this bridging of cultures online makes us shift from high-context cultures to low-context ones, or informational ones. Because of this, we are forced to learn to communicate in a different way, with an emphasis on collaboration, learning to negotiate, to accept disagreement as normal.
Paradoxically, this has made us more socially conscious, and social in general, on the internet than in the physical world. There is less distance, it feels like the globe has shrunk, and that has intensified social interactions, for better or worse.
Some studies have observed that overall, there is a correlation between sociability and internet usage. This applies to civic engagements and other types of relationships. This is a new kind of sociability, one that is driven by permanent connectivity, especially on social media.

When bridging cultures a common language has to be chosen, and on the internet it is often English that dominates. It composes around 30% of the internet according to estimates, but isn’t the native tongue of most users. This can limit the spread of information from one cluster to another, create boundaries for communities, an inequality of information, an inequality of representation, and shape the digital experience and views of people on the internet. The language, and indirectly culture, barriers can direct the flow of information, tending in the direction where it is sometimes more domestic. We’ll get back to this in the consequences part of the series.

The apparent reduction in distance also creates networking opportunities. In the past interactions were limited to the personal networks; people being favored for whether they went to a prestigious school, university, or worked at a well-known company. These days we can be exposed to a diverse range of networks that are much more accessible, increasing the employment prospect.
This applies to creativity too, being exposed to so many diverse ideas, different ways to see a problem. Creativity is an import-export business. Taking things that are ordinary from multiple places and putting them in totally different settings. What is called cognitive diversity.
Additionally, widespread connectivity brings the possibility of working from almost anywhere in the world, at any time of the day, if the job doesn’t require to be on-place — that is if it is an information related job. This unlocks tremendous freedom and autonomy.

The concept of autonomy is an important one, as it relates to another one: individuation. When moving from high-context society to low-context information society, the process of individuation takes place. Some ironically calls it the “ME-centered” society, or the “selfie generation”. This isn’t something new, it’s been happening for a couple of years, but it is more widely spread and the internet is a catalyst.
Concretely, this translates into the decline of the definition of the self based on organizations and organizational roles, such as work, nation, and family. The individual is now in the center and doesn’t find meaning in these classical system, sometimes leading to a cross culture frustration.

Nevertheless, this is not the end of the concept of community but the end of having it based on places of interactions. Instead, it has shifted toward the self-authorship of the community, one based on personal ties, social relationships, individual interests, values, and projects. As a matter of fact, the individual is not isolated, nor less sociable, but redefined as a networked individual with a quest for like-minded people.
The information society is a network of self-authored individuals.

The transformation and evolution in culture bring with them a reorganization of economic, communication, and political activities. According to some research, it is correlated with a sense of empowerment, increased feelings of security, personal freedom, and influence — all in relation with overall happiness and well-being.

Whole communities join together and participate in their own identity development, as a choice. In this model, people replace blind reverence for authority with authenticity.
In the past information were shaped and based on hype, now people crave what is genuine and transparent. This is a change in perception with a focus on raw emotions and connections.

This authenticity is an essential aspect for identity development, it is how we identify ourselves and create our stories. Our narratives are based on our interactions in this participatory network. We are the creators, refining our persona with the help of the individuals that surround us, to whom we tell our story.
McAdams (1993) asserts, “Once an individual realizes that he or she is responsible for defining the self, the issue of self definition remains a preoccupation through most of the adult years”.

Being the subject, the individual, in the process of creation is what is meant by autonomy. This is accompanied by an urge to not submit to institutions, especially for minorities. This switch in mindset is highly correlated with social autonomy, independent-minded citizen able to participate in self-generated networks.
This autonomy materializes itself in society through entrepreneurship, creative works, proactive consumers, self-informed critical thinkers, pedagogical platforms that allow self-learning, e-governments, grassroots movements, etc..
The paper entitled “The Social Media Mindset” lists six major types of autonomy:

  • Professional development
  • Communicative autonomy
  • Entrepreneurship
  • Autonomy of the body
  • Sociopolitical participation
  • Personal, individual autonomy

More than anything, to be autonomous means mastering our tools, grasping how to make algorithms work for us. They are ubiquitous, we are surrounded by them, the internet is mediated by them. Hence, we need to muster their strength.
To be internet literate implies understanding the technology that drives it. The customization and recommendation engines can be used to facilitate our daily routine, letting the users create their own personal optimized environments.

Social networks are used to create our own constructed world, which includes everything from e-commerce, education, entertainment, to sociopolitical activism. The users group together according to their interests and tailor it to their needs. A self-constructed society with people yearning for honesty, genuineness, transparency, common interests, and connections.
Many of the real world has migrated to the internet, from web marketers, work organizations, service agencies, governments, and civil societies.
Technology, being material, is based on products made by people based on their ideas, values, and knowledge. We adapt the technology to fit us, rather than adopting it. It is built into our society and thus it has to follow its constraints. Obviously, social media platforms are trying to monetize the process, turning it into a business.

Collectively, this has brought us to an era of creativity. Where everything is a remix and shared. We aren’t bound by the mainstream media gatekeepers like TV, radio, newspapers, and academia. Information is now freely available on the internet, accessible by anyone, from the worst atrocities to the most awe-inspiring instances of human kindness.
In the internet society we are forced to broaden our collective perspective, a perspective that is more global.

The informational freedom carries with it transformations in the sociopolitical realm. There’s a widespread availability of tools for creating and distributing content across the globe. Most often, there is no need for training nor expertise to participate, everyone is empowered to mass media — A direct access to million of other people in an expanded community.
We bypass traditional media, letting us challenge the assertions of powerful corporations, government officials, news reports, and television advertisements. We do it through blogs, Twitter, Facebook, and others.

Everyone is an author and can express their thoughts to the world and this undoubtedly means that some people are going to spread disinformation, misinformation, and other types of erroneous information.
Our new distrust in authority and increasing doubts, can be abused through our cognitive biases. The attention economy, with the never ending competition between all actors, is filled with manipulation.
This is something the netizens are slowly learning. We’ll come back to biases later.

The constant connectivity has given rise to a more democratic and participatory form of journalism: citizen journalists. The audience employs the press tools to inform one another. The public now plays the role of collecting, reporting, analyzing, and disseminating news and information.

There are upsides and downsides to this new development. As we said, these are amateurs, people without journalism training, and so the quality of the reports and coverage can vary, is subjective, heavily opinionated, unregulated, and can be abused by malicious actors — such as spin doctors.
However, it can also promote human rights and democratic values to have the people join together and collaborate in this endeavor.

Social network websites, accessible to everyone, have provided an alternative for activists to report on their activities. Social media also allow real time participation and messages can be sent quickly in reaction to events.
This is particularly helpful during an earthquake, street movements, and protests.

Lots of things are now shaped and decided on the internet, it is tightly linked to political changes in the process. Citizen journalists now surround all the protests and movements these days. These movements, rejecting authority, are often without leaders, more spontaneous, and full of emotional outbursts.

The media of choice to express feelings about a situation are internet memes. They act like the modern era version of caricatures found in traditional newspapers.

Some memes have been put forward in democratic campaigns such as sharing messages about climate change. Though arguably, the research hasn’t found their impact in altering mindsets that impressive. In that study only 5% of participants changed their minds.
Memes, like caricatures, can also act as a power and counterpower, a battle of communication. Humor has always been a tool for people to express feelings and thoughts that were repressed, to fight against oppression.
This is the case in countries where people have to bypass censorship and rhetoric by reusing the semantic meaning of the slogans, ideas, and terminology. They reuse the language, twist it, satire it, to reinterpret it in their own way. They use the official language in reverse by applying it with humor.

Still, we have to remember that citizen reporting is full of biases, doubts, lies, polarization, conspiracy theories, and subjectivities. These can also obviously be manipulated by other actors, as we said.

Similar to citizen journalists we are seeing the rise of citizen science, which we won’t dive into here.

Let’s now jump a bit into some of the effects that being constantly online can have on our cognitive abilities. In a place of never-ending multitasking, attention economy, infobesity, and new social concepts.
This will be reviewed again in the consequences part of the series.

Our attention is now always divided between multiple media sources at the expense of sustained concentration. Everything tries to attract our attention in this hyperlinked addictive environment and we got used to it.
Education providers can already see the effects this has on children attention span: their attention is more shallow.
Studies show a significant effect on cognitive performance, where even short term engagement reduces attention scope for a sustained duration after coming offline.

Memory-wise, the vast array of information has changed the way we retrieve, store, and even value knowledge. The internet is now acting like a transactive memory, externalizing our thoughts — a helper, a memo, a memento.
We have most of the factual information in existence at our fingertips, introducing the possibility that the internet could ultimately negate or replace certain parts of our memory system, especially the semantic memory — memory of facts.
Multiple research show that people searching online were more likely to remember where they found the facts rather than the facts themselves. We are more reliant on the internet for information retrieval.
I’ve dived into this topic previously here and here.
At the group level this can be efficient but at the individual level it can hinder how we recall the actual information. The internet sometimes even bypasses the ability to retain where we found the information when it gives us the opportunity to search for it. We don’t have to remember the exact information anymore, nor where its located, just that it can be searched and that it’ll be available.
This cognitive offloading using an external long term storage can be beneficial to focus, not on facts, but on aspects that are not retrievable like meta information, conclusions, and emergent hypothesis. It frees some space to think about other things than plain facts.
Though other research show that analytical thinkers are less likely to rely on such means in their day-to-day than other people.

Cognitively, the internet also creates new ways to learn. Any new skills and information are available in this space. People now have more breadth of knowledge instead of depth of knowledge.

When it comes to social cognition, there are a lot of debates regarding whether the internet provokes the same effects as real interactions would. So far, the studies and evidences show that it does indeed reflect real things like self-concepts and self-esteem. We clearly see this through the neurocognitive responses to online social interactions.
These online social relationships, like any relationships, are connected to feelings of happiness, and mental and physical well-being. Thus being accepted or rejected online feels the same way in the brain, an interpretation, which we’d like to remember, in the offline world is often ambiguous and left to self-interpretation.
However, on social media, unlike the real world, we are presented with clear metrics which give us an indication of whether we fit in or not. They take the form of “followers”, “shares”, “likes”, “friends”. The potential can be painful, or addictive because of the immediacy. There is a direct feedback on our self-esteem.
Additionally, the public life on these social media also means that upward social comparisons are omnipresent. People take part in artificial environments manufactured to picture hyper-successful persona. Obviously, this is what the platforms capitalize on. You can go back to the previous section to get an idea.

On social media we aren’t anonymous but often create persona, a self-presentation of a real person. One that we mold to be perfect. Everyone has the ability to become an orator, a super-star in front of a gigantic crowd. An online life is like being followed by a million paparazzi.

We know that the medium is quick, indelible, and that a misstep can make us victim of the internet outrage machine. Be it positive, or not, because of private justice or vigilantism, or any other reasons. We are intimately aware of how extreme views proliferate and how ideas are hard to separate — always coming as blocks, hooks, and anchors.

With social media our personal life and work life are intertwined, they come as a package that can’t be separated. Our views in one field can propagate and affect other aspects of our lives.
Personal life is now public life, and not only for celebrities.

This is compounded with the feeling of always being scored and tracked. According to the Pew Research Center, 72% of people feel that almost all of what they do online is being tracked by advertisers, technology firms or other companies.

Having private personal details online and our digital reputation can have repercussions in the real world affecting our relationships, job opportunities, bank loans, visa applications, and more.
It is hard to erase something once it is on the internet, and so we apply self-surveillance between ourselves. Indeed, we know our weaknesses are mapped and become increasingly transparent and this leads to self-censorship, conformity, risk-aversion, and social rigidity.
This is a phenomenon called “social cooling”. “You feel you are being watched, you change your behavior.” Isn’t this a form of censorship? Privacy is the right to be imperfect and not judged. We’ll come back to this in the last section of the article when discussing some of the solutions.

I was told that I couldn’t be trusted since people can’t check online what I’m doing when I’m not around.

On the good side, the ability to easily get exposure, the appetite for authenticity, and the itch for autonomy have given birth to the passion economy.
The passion economy consists of taking the cliché of “following your passion” to the extreme. While it didn’t work properly in the past, it is now increasingly easier to achieve, either through an online patronage system or simply because of the new exposure.
The individual is empowered and liberated from previous notions of fixed career path.

The passion economy is centered around purposeful creators that are motivated and driven by their passion, displaying their expertise and experiences with the world.

The audience is as passionate and niche as the creator, influenced and transported through the craft. This is in stark contrast with the apathetic consumers of previous years that made price-focused decision about trinkets of mass productions.
People want to be involved in the making process, they want the whole experience, to engage with the services and ideas they like, to associate with their stories, to build a relationship with the authors. They want to find individuality in their products.

Along with an always public life comes the idolization of hyper-successful individuals. There is a rise in unrealistic expectations of oneself.
Individuation puts the weight on people to find who they are, and it is easy to go back to role models for escapism. These people might in turn use their influence to create a monetizable audience.
The gold rush for scores and fame, the new cult of personality, the new promotional culture going viral.
This is the rise of the “influencers” who’ll sell their authenticity for paid product placements and advertisements.

Lastly, these coalesce — individuation, influencers, passion economy, private life becoming public life, and social media — into neoliberalization.
We’ll see more of this later, as it also depends on culture and how people perceive and use the internet in different ways.

This is the ailment of modern society in general, and the internet makes society more “modern”. A shift in motivations to be goal-oriented and efficient, instead of being based on the more traditional values.
The individuals, with the help of the internet, are now accustomed to marketing themselves as products. This is the modern rule of the game: corporate life merging with private life.

Many products are sold to fill this “need”. I’ll digress, as I have discussed this in other articles, and here. We’ll come back to this later.

This concludes our review of the person’s life on the internet in between the corporate, private, and public spheres. We started by seeing how the availability of information is now everywhere and shrinks distances. Then we’ve taken a look at the relationship between networks in societies being mapped online and the effects on cultures. Next we’ve discussed the switch to the information society which creates a need for individuation, networks of people grouping based on their interests. Then we’ve seen how this allows self-generated societies and making our own mind about things. Later we’ve discussed the empowerment of the average person through citizen journalism. After that we’ve glanced at some of the cognitive effects in the realm of attention, memory, and social cognition. Following this we’ve concluded with three subjects: the private life becoming public, the passion economy, and a new extreme form of neoliberalisation that is normalized.

Table Of Content

References








Attributions: W. Blake, The Sun at its Eastern Gate, c 1815

March 29, 2021

Ponylang (SeanTAllen)

Last Week in Pony - March 29, 2021 March 29, 2021 04:39 PM

The ‘Force declaration of FFI functions’ RFC has entered final comment period. OpenSSL and LibreSSL builders have been updated. There are also some releases from last week.

Kevin Burke (kb)

Preferred Stock and Common Stock Aren’t The Same March 29, 2021 04:00 PM

When you get an offer from a tech company it will usually be some combination of cash and stock. Small companies give out stock because it's hard to compete, cash-wise, with bigger companies, and a grant of stock or options offers the chance at a large payday down the road.

Valuing the cash part of an offer is pretty simple. How should you value the stock? Well, one answer is whatever someone's willing to pay you for it.

To that end recruiters will sometimes give you information like, "our last round closed at $4.80 per share, so, if you get X shares per year, your stock compensation is worth $4.80 * X." Sometimes recruiters will send a fancy tables showing the value if the company doubles or 10X's its value.

This is a classic bait and switch. When a company raises a round from, say, Sequoia, Sequoia wires money to the company and gets preferred shares. When you are granted stock options as an employee, you are getting common shares. The differences will vary from company to company, but commonly:

  • Holders of preferred shares get paid out before common shareholders (ie you). Bankruptcy is not intuitive. If you get in a traffic accident, the insurers will usually say something like, the other party is X% at fault, and you were Y% at fault, so this is what you owe. Bankruptcy is not like traffic court. All the rights holders ahead of you will get paid out 100% before you see a cent. If the company is sold or liquidated, the preferred shareholders will likely be paid in full before any holder of common stock sees a dollar. Because of this, preferred shares are more valuable.

  • Preferred shares have different voting rights than common shares. A preferred share might get five or ten (or zero) votes, to one for a common share.

  • Preferred shares may have other downside protection in the event an IPO or sale does not reach a target price.

So preferred shares are worth more than common shares. It is a mistake to look at the preferred share price and multiply by the number of shares and use that as the current value of your common shares. Your shares are worth much less than that.

One reason this happens is that preferred shares are easier to value, because there are regular funding rounds, insider sales. Common stock doesn't trade hands that often before an IPO because stock sales often require board approval. But that doesn't excuse anyone from giving you common shares and pretending they are worth as much as preferred shares.

The recruiters and VC's next trick is to pretend that you shouldn't be concerned about the difference between common and preferred stock, because in an IPO preferred stock is usually converted to common stock. That's true. But this is sort of like saying a home without fire insurance and a home with fire insurance are worth the same as long as they don't burn down. If you have common stock, you don't have fire insurance. And a startup is significantly more likely to fail than a home is to burn down.

If anyone tries to convince you that the difference doesn't matter, ask them if they'd like to swap their preferred shares one-for-one with your common shares.

If you are being recruited and someone tries this on you, please point out the difference and explain that you don't appreciate being condescended to. You should also think less of the company. Every one of your coworkers went through the same process of being lied to about their potential share value.

If you are an employer and you want to do this honestly, quote the most recent share price you have, and then explain that that's the preferred share price, but you are not giving out preferred shares. Explain that recruits should value their shares lower than the preferred share price you quoted - exactly how much is difficult to say, but the preferred share number should be an upper bound on that value. If your common stock is traded, or any of your shareholders are forced to mark their shares to market (Fidelity holds them in a mutual fund, for example), you should disclose that.

(You could also let your employees sell their equity more often, maybe to insiders.)

March 28, 2021

Derek Jones (derek-jones)

The aura of software quality March 28, 2021 09:45 PM

Bad money drives out good money, is a financial adage. The corresponding research adage might be “research hyperbole incentivizes more hyperbole”.

Software quality appears to be the most commonly studied problem in software engineering. The reason for this is that use of the term software quality imbues what is said with an aura of relevance; all that is needed is a willingness to assert that some measured attribute is a metric for software quality.

Using the term “software quality” to appear relevant is not limited to researchers; consultants, tool vendors and marketers are equally willing to attach “software quality” to whatever they are selling.

When reading a research paper, I usually hit the delete button as soon as the authors start talking about software quality. I get very irritated when what looks like an interesting paper starts spewing “software quality” nonsense.

The paper: A Family of Experiments on Test-Driven Development commits the ‘crime’ of framing what looks like an interesting experiment in terms of software quality. Because it looked interesting, and the data was available, I endured 12 pages of software quality marketing nonsense to find out how the authors had defined this term (the percentage of tests passed), and get to the point where I could start learning about the experiments.

While the experiments were interesting, a multi-site effort and just the kind of thing others should be doing, the results were hardly earth-shattering (the experimental setup was dictated by the practicalities of obtaining the data). I understand why the authors felt the need for some hyperbole (but 12-pages). I hope they continue with this work (with less hyperbole).

Anybody skimming the software engineering research literature will be dazed by the number and range of factors appearing to play a major role in software quality. Once they realize that “software quality” is actually a meaningless marketing term, they are back to knowing nothing. Every paper has to be read to figure out what definition is being used for “software quality”; reading a paper’s abstract does not provide the needed information. This is a nightmare for anybody seeking some understanding of what is known about software engineering.

When writing my evidence-based software engineering book I was very careful to stay away from the term “software quality” (one paper on perceptions of software product quality is discussed, and there are around 35 occurrences of the word “quality”).

People in industry are very interested in software quality, and sometimes they have the confusing experience of talking to me about it. My first response, on being asked about software quality, is to ask what the questioner means by software quality. After letting them fumble around for 10 seconds or so, trying to articulate an answer, I offer several possibilities (which they are often not happy with). Then I explain how “software quality” is a meaningless marketing term. This leaves them confused and unhappy. People have a yearning for software quality which makes them easy prey for the snake-oil salesmen.

Gustaf Erikson (gerikson)

The Eighteenth Brumaire of Louis Bonaparte by Karl Marx March 28, 2021 10:37 AM

Available online here.

Come for the class analysis, stay for the bon mots.

It’s probably fitting that the only way obscure French politicians are remembered today is through their skewering in this piece.

Hegel remarks somewhere that all great world-historic facts and personages appear, so to speak, twice. He forgot to add: the first time as tragedy, the second time as farce. Caussidière for Danton, Louis Blanc for Robespierre, the Montagne of 1848 to 1851 for the Montagne of 1793 to 1795, the nephew for the uncle. And the same caricature occurs in the circumstances of the second edition of the Eighteenth Brumaire.

The period that we have before us comprises the most motley mixture of crying contradictions: constitutionalists who conspire openly against the constitution; revolutionists who are confessedly constitutional; a National Assembly that wants to be omnipotent and always remains parliamentary; a Montagne that finds its vocation in patience and counters its present defeats by prophesying future victories; royalists who form the patres conscripti of the republic and are forced by the situation to keep the hostile royal houses they adhere to abroad, and the republic, which they hate, in France; an executive power that finds its strength in its very weakness and its respectability in the contempt that it calls forth; a republic that is nothing but the combined infamy of two monarchies, the Restoration and the July Monarchy, with an imperial label – alliances whose first proviso is separation; struggles whose first law is indecision; wild, inane agitation in the name of tranquillity, most solemn preaching of tranquillity in the name of revolution – passions without truth, truths without passion; heroes without heroic deeds, history without events; development, whose sole driving force seems to be the calendar, wearying with constant repetition of the same tensions and relaxations; antagonisms that periodically seem to work themselves up to a climax only to lose their sharpness and fall away without being able to resolve themselves; pretentiously paraded exertions and philistine terror at the danger of the world’s coming to an end, and at the same time the pettiest intrigues and court comedies played by the world redeemers, who in their laisser aller remind us less of the Day of Judgment than of the times of the Fronde – the official collective genius of France brought to naught by the artful stupidity of a single individual; the collective will of the nation, as often as it speaks through universal suffrage, seeking its appropriate expression through the inveterate enemies of the interests of the masses, until at length it finds it in the self-will of a filibuster. If any section of history has been painted gray on gray, it is this. Men and events appear as reverse Schlemihls, as shadows that have lost their bodies. The revolution itself paralyzes its own bearers and endows only its adversaries with passionate forcefulness. When the “red specter,” continually conjured up and exercised by the counterrevolutionaries finally appears, it appears not with the Phrygian cap of anarchy on its head, but in the uniform of order, in red breeches.

The coup d’etat was ever the fixed idea of Bonaparte. With this idea he had again set foot on French soil. He was so obsessed by it that he continually betrayed it and blurted it out. He was so weak that, just as continually, he gave it up again.

The army itself is no longer the flower of the peasant youth; it is the swamp flower of the peasant lumpen proletariat. It consists largely of replacements, of substitutes, just as the second Bonaparte is himself only a replacement, the substitute for Napoleon. It now performs its deeds of valor by hounding the peasants in masses like chamois, by doing gendarme duty; and if the natural contradictions of his system chase the Chief of the Society of December 10 across the French border, his army, after some acts of brigandage, will reap, not laurels, but thrashings.

Confessions of a Long-Distance Sailor by Paul Lutus March 28, 2021 08:17 AM

A self-published book available online recounting the author’s solo round the world sail.

A worthy entry in the long roster of such accounts.

Libra Shrugged: How Facebook’s dream of controlling the world’s money crashed and burned by David Gerard March 28, 2021 08:09 AM

A short account of how Bitcoiners tried to create a Facebook currency and how the rest of the world reacted.

March 27, 2021

Pierre Chapuis (catwell)

Booting GDM on my XPS with kernel 5.11 March 27, 2021 08:15 PM

When I updated my Linux kernel to 5.11 I had the bad surprise to end up with a blinking underscore on reboot. It had been many years since an update had broken my system like that. I fixed it rather easily by booting in rescue mode and downgrading the kernel. I had no time to investigate so I just added linux to IgnorePkg at the time, But I don't use Arch to run old kernels so today I took the time to fix it "properly".

To do so, I reproduced the issue, then downgraded again and looked at the logs with journalctl -b --boot=-1. It quickly let me understand that it was GDM that was failing due to something wrong with graphics initialization.

To keep things short, let me skip to the conclusion: if you run into this issue on an old-ish Dell XPS with an Intel Iris Plus 640 graphics card like mine with GDM, Arch and Wayland (or something similar), try enabling early KMS by adding i915 to the MODULES array in mkinitcpio.conf and rebuilding the initramfs, that fixed it for me.

Andreas Zwinkau (qznc)

How you can handle The Diamond with CMake March 27, 2021 12:00 AM

CMake requires old-school include-guards and prefix at scale

Read full article!

March 25, 2021

Marc Brooker (mjb)

What You Can Learn From Old Hard Drive Adverts March 25, 2021 12:00 AM

What You Can Learn From Old Hard Drive Adverts

The single most important trend in systems.

Adverts for old computer hardware, especially hard drives, are a fun staple of computer forums and the nerdier side of the internet1. For example, a couple days ago, Glenn Lockwood tweeted out this old ad:

Apparently from the early '80s, these drives offered seek times of 70ms, access speeds of about 900kB/s, and capacities up to 10MB. Laughable, right? But these same ads hide a really important trend that's informed system design more than any other. To understand what's going on, let's compare this creaky old 10MB drive to a modern competitor. Most consumers don't buy magnetic drives anymore, so we'll throw in an SSD for good measure.

XCOMP 10MB    Modern HDD   Change Modern SSD  Change
Capacity 10MB 18TiB 1.8 million times   2 TiB 200,000x
Latency 70ms 5ms 14x 50μs 1400x
Throughput 900kB/s 220MB/s 250x 3000MB/s 3300x
IOPS/GiB (QD1) 1400 0.01 0.00007x 10 0.007x

Or there abouts2. Starting with the magnetic disk, we've made HUGE gains in storage size, big gains in throughput, modest gains in latency, and a seen a massive drop in random IO per unit of storage. What may be surprising to you is that SSDs, despite being much faster in every department, have seen pretty much the same overall trend.

This is not, by any stretch, a new observation. 15 years ago the great Jim Gray said "Disk is Tape". David Patterson (you know, Turing award winner, RISC co-inventor, etc) wrote a great paper back in 2004 titled Latency Lags Bandwidth that made the same observation. He wrote:

I am struck by a consistent theme across many technologies: bandwidth improves much more quickly than latency.

and

In the time that bandwidth doubles, latency improves by no more than a factor of 1.2 to 1.4.

That may not sound like a huge amount, but remember that we're talking about exponential growth here, and exponential growth is a wicked thing that breaks our minds. Multiplying Patterson's trend out, by the time bandwidth improves 1000x, latency improves only 6-30x. That's about what we're seeing on the table above: a 250x improvement in bandwidth, and a 14x improvement in latency. Latency lags bandwidth. Bandwidth lags capacity.

One way to look at this is how long it would take to read the whole drive with a serial stream of 4kB random reads. The 1980s drive would take about 3 minutes. The SSD would take around 8 hours. The modern hard drive would take about 10 months. It's not a surprise to anybody that small random IOs are slow, but maybe not how slow. It's a problem that's getting exponentially worse.

So what?

Every stateful system we build brings with it some tradeoff between latency, bandwidth, and storage costs. For example, RAID5-style 4+1 erasure coding allows a system to survive the loss of one disk. 2-replication can do the same thing, with 1.6x the storage cost and 2/5ths the IOPS cost. Log-structured databases, filesytems and file formats all make bets about storage cost, bandwidth cost, and random access cost. The changing ratio between the hardware capabilities require that systems are re-designed over time to meet the capabilities of new hardware: yesterday's software and approaches just aren't efficient on today's systems.

The other important thing is parallelism. I pulled a bit of a slight-of-hand up there by using QD1. That's a queue depth of one. Send an IO, wait for it to complete, send the next one. Real storage devices can do better when you give them multiple IOs at a time. Hard drives do better with scheduling trickery, handling "nearby" IOs first. Operating systems have done IO scheduling for this purpose forever, and for the last couple decades drives have been smart enough to do it themselves. SSDs, on the other hand, have real internal parallelism because they aren't constrained by the bounds of physical heads. Offering lots of IOs to an SSD at once can improve performance by as much as 50x. Back in the 80's, IO parallelism didn't matter. It's a huge deal now.

There are two conclusions here for the working systems designer. First, pay attention to hardware trends. Stay curious, and update your internal constants from time to time. Exponential growth may mean that your mental model of hardware performance is completely wrong, even if it's only a couple years out of date. Second, system designs rot. The real-world tradeoffs change, for this reasons as well as many others. The data structures and storage strategies in your favorite textbook likely haven't stood the test of time. The POSIX IO API definitely hasn't.

Footnotes

  1. See, for example, this Reddit thread, unraid forums, this site and so on. They're everywhere.
  2. I extracted these numbers from my head, but I think they're more-or-less representative of modern mainstream NVMe and enterprise magnetic drives.

March 24, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Actors And Incentives: New Economies March 24, 2021 10:00 PM

The spectacular widespread magnetic cures of doctor Franz Anton Mesmer. Work based on the magnetic bipolarity of the human body and the influence of an all pervasive vital fluid.

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 2 — The Actors and Incentives
  • Section 1 — New Economies: Information Economy, Attention Economy, Data Economy, Surveillance Economy
Table Of Content
  • Data And Attention As New Assets
  • How Much Value
  • The Actors (Social Media, Ads Networks)
  • What They’d Do To Get It
  • What They’d Do To Protect It

The artifacts we’ve previously seen have given rise to new types of economies, tools, and arsenals that can be used by different actors for different incentives.
Let’s start by focusing on the economic aspects by looking at actors such as social media platforms, newspapers, and advertisement companies.

Life in an information society is about moving data, the new raw material, around our manufacturing pipelines. This data is then consumed by either paying with attention or money.
Data and attention are the two assets of the digital economies that emerged.

Whoever holds the means of productions, by centralizing and monopolizing them, has a tremendous advantage in any economy. In a data economy these are the data brokers who figuratively data mine the raw material, store it, and keep it well guarded — as is required in any intangible economy to succeed (See Capitalism Without Capital: The Rise of the Intangible Economy).

The other new asset is attention. In an environment where it is at the same time so hard to reach an audience and, once reaching a threshold, so easy to spread virally, having the skills to capture attention is valuable.
This is due to multiple factors such as the rise of social media, the average users becoming generator of content, and the ever-growing infobesity.

Davenport & Beck (2001) define attention as “focused mental engagement on a particular item of information. Items come into our awareness, we attend to a particular item, and then we decide whether to act.
Its scarcity turns it into a prized resource, constructing an attention economy. Attention is exchanged as a commodity, either by giving it or taking it — an intangible good on one side, money and physical goods on the other.
There is a fight for this resource, the corporate entities need it because the digital economy is in sync with advertising. The drivers and generators of money are buyers of attention and data, namely marketers and advertisers. Just like traditional media advertising, the online world follows a model that uses a linear process called AIDA: Attention, Interest, Desire, and Action.
Hence, social media platforms, online newspapers, and others make money by selling ad spaces, selling targeting information using the data gathered, and giving access to or selling the data itself.

Marketers have two choices when it comes to social media: either passive marketing — gathering data from their consumers posts and public information — or active — engaging with users, creating polls, paying influencers to advertise a product, etc..
Both parts are facilitated by platforms that allow tracking and analyzing the results of their campaigns. Advertisement is only a sub-section of marketing.

As for online newspapers, and many others, they have chosen to either sell ad space, thus attention, or switched to a subscription model in which the customer pays to get access.
Some argue that the centralization of ad services forces companies to offer paid services, leading to an overall more expensive online experience. There are big questions regarding the value of the content we consume online and how much should be paid for it. The concept of micro-transactions and micro-payments comes to mind, we’ll come back to them in the solutions part.

The internet conundrum is that users have developed a habit of expecting online services to be free. So, to make the services offered profitable, the owners have to extract valuable things from their users, namely their attention and generated data.

The digital culture expert Kevin Kelly describes the expected attributes of this economic space as:

  • Immediacy - priority access, immediate delivery
  • Personalization - tailored just for you
  • Interpretation - support and guidance
  • Authenticity - how can you be sure it is the real thing?
  • Accessibility - wherever, whenever
  • Embodiment - books, live music
  • Patronage - “paying simply because it feels good”,
  • Findability - “When there are millions of books, millions of songs, millions of films, millions of applications, millions of everything requesting our attention — and most of it free — being found is valuable.”

In direct relation with the data economy, a sub-type of economy has appeared that specializes in a particularly profitable data type: personal data. The surveillance economy, as it came to be called, centers itself around making profit from gathering personal information, commodifying it.
This data is obviously more valuable and can be sold for reasons other than precise targeted marketing, such as creating a voters profiles database to be sold to state actors. This was the case of Cambridge Analytica, who used Facebook as a tool to gather information. Allowing them to join up to 7 thousand data points on 240M people. We’ll see more of that later.

This grants companies access to information that can be used to tailor their products to their customers’ needs, and customize their marketing campaigns to specific categories of people, to what we call “buyer personas”. This is good for passive marketing.
For example, a job ad can be sent precisely to men between 25 and 36 that are single and have interests in cars.
When so many data points are involved in identifying an audience, this is referred to as microtargeting.

In the past advertising was used with the goal to inform consumer choices, today it is predominantly used to persuade them, going back to our section on persuasion and propaganda. It steers consumers toward the brands which invest more in advertising, rather than those that are sustainable and responsible but less competitive. Thus, driving global consumption artificially.

Active marketing, engaging with users, works well in a surveillance economy too. Companies can perform direct continual experiments on their customers without them being aware of it nor giving consent.
An A/B or Wizard of Oz testing phase that would’ve taken months to perform can be done in a matter of hours on social media, an almost instantaneous feedback.

In the past marketing used to be more of a one-to-many type of communication, with the internet it now appears more intimate, a one-to-one customized experience with the brand. The company can build a relationship with their customers for long-term success.
This relationship can then work in their favor through the viral nature of social media. The consumers who find what they are looking for will share their positive experience, while those who didn’t will also communicate it. The opinions will proliferate and increase in number.

From the data gathering side of the equation, big tech companies are then justified in amassing data at all costs, tracking and analyzing. This affects privacy and control over our lives, we’ll see some of the deleterious consequences later. The data includes location history, preferences, age, name, and other things we’ve seen in the previous data and metadata section.

These three or four types of economies, digital, data, attention, and surveillance, clash together. The first one generates a mountain of advertisements and content that floods people, making them more picky about what they pay attention to. Accordingly, the value of attention rises, it being more scarce, creating additional incentives to invest into specific targeted advertisement, hence more surveillance.
A true arms race driven by a contamination of the supply that is often filled with irrelevant, redundant, unsolicited, low-value information. An adverse effect of this information revolution.

One of the many consequences, which we’ll see later, is something called continuous partial attention, a phenomenon where someone’s attention is constantly divided. It can lead to increase stress and an inability to reflect or make thoughtful decisions, sometimes related to analysis paralysis.
This consequence is convenient to some actors, especially in relation to what we’ve seen in the propaganda and persuasion section.

There’s a lot of money involved and many reasons to secure a position in these new economies. As with any new commodity that spawns, is lucrative, and fast growing, it attracts regulators and policymakers that want to restrain its control and flow.

Every year, companies worldwide spend more than 1.3 trillion dollars on commercial communication and 600 billion dollars on advertising. The titans of the internet, Alphabet, Amazon, Apple, Facebook and Microsoft look unstoppable. They are part of the most valuable listed firms in the world. Their profits keep surging: they collectively racked up over 25 billion dollars in net profit in the first quarter of 2017.

This is compounded with the feedback effect that more data encourages more data. This confers an enormous advantage to already large incumbents, the ad technology suppliers that have amassed a wall of well guarded data.
In this economy it is quantity over quality.

The giants span different fields, Google can see what people search for, Facebook what they share, Amazon what they buy. There’s no room for small players such as online newspapers, or any other SMEs, they can’t compete. This means the biggest players have a strong grip on the supply chain, both the data, attention, and the ad tech.

Size isn’t the issue here but the danger it can have on the consumers in these economies. It impacts everyone along the supply chain, from publishers, to advertisers buying ads, and the consumers.
This can create conflicts of interests, giving a self-preferenciation to specific suppliers of ad technology. These later can then control the price of access to it, its quality of service, and the transparency going along with it.
With such control, these companies can thwart competitors, either by hoarding more data and making it impossible for others to rival them, or by acquiring or destroying contenders.
For instance, many think that Facebook’s acquisition of WhatsApp for $22 billion falls in that category, namely creating a barrier of entry and stifling competition.

Unquestionably, these data brokers are nothing without their network of buyers, the advertising companies and other entities interested in getting access to mined data and attention. While there are many small players, the bigger ones will often sign partnership deals with the data brokers, so-called third-party partnership (or sub-contractors), to exchange and interoperate data laterally.
There are countless examples of these generally non-transparent deals, at least non-transparent with regard to the data subjects which have their information exchanged without their consent.

For example, AT&T may use third-party data to categorize subscribers without using their real names to show them ads. In practice, that means AT&T can find out which subscribers have which health issues — or which ones bought over-the-counter drugs to treat it. The companies extracted this kind of information from the databases accessed through third parties which contained subscribers’ spending.

An investigation by the New York Times from 2018 detailed how Facebook had made special agreements to share/give access to personal user information and private chats to 150 tech firms, including Amazon, Apple, Microsoft, Netflix, Spotify, and Yandex. The list also included online retailers, car-makers, banks, phone manufacturers, and media organizations.

Facebook distinguished between two types of relationships: integration and instant personalization partnerships.
The first one is about offering Facebook’s features outside its own app or website, the users having to sign in to their account to allow it.
The second one, “instant personalization”, is about allowing other apps to interact with Facebook private messages to customize them.
Facebook noted that it ended “nearly all” of the integration partnership and most of its personalization partnerships.

Facebook is a gargantuan player, but only one among many like Google and Twitter. They all have such deals and act as data brokers.
The most interesting type of partnership relying on these gigantic databases of personal information are the ones with state actors. For now, let’s only note that the USA government requires would-be immigrants and visa applicants to submit five years of social media handles for specific platforms identified by them, mainly Facebook. We’ll see how lucrative social media are for states later.

With lots of money and people involved, many aggressive tactics are used to get the attention and personal data of people, any legal means are allowed. That is why they rely on persuasive technology, a term used to refer to any technology designed to change attitudes or behaviors through persuasion and social influence. In our case, they are used to persuade to act based on ads, pay attention, and give personal data.

The use of persuasive design is custom-fit to the receiver, for example it can be based on age-appropriate desires.
The preliminary information can be gathered from all the sources we mentioned in our data and metadata section such as smart utilities, and our daily ubiquitous online activities. This is all possible because the user inadvertently agreed to a hidden privacy policy, in most cases without reading it. In so far as this is the argument used by these platforms to allow this, while making it as cumbersome as possible to understand what the policies imply.

Then, once the target is set, the software platform algorithms will explicitly or implicitly attempt to win the attention of the users through interface and psychological tricks based on the interests and personalizations from the data extracted earlier.
This includes mundane things such as catchy click-bait titles of news articles, and visual tricks such as using shocking pictures, impactful fonts, and attractive colors. The usual in marketing and other types of media (TV, newspapers, radio).
The new nefarious techniques rely on dark patterns, user interfaces that are designed to mislead and direct users into acting a certain way. This includes hiding parts of the interface that are supposed to be there for legal reasons, encouraging users to sign up or buy services or items they didn’t intend to, and more

Additionally, the platforms will create an environment that nurtures attention, shaping it using the data acquired. This ranges from benign things such as showing users what they are looking for, to turning the digital environment into a casino-like environment.
Social media are especially known to employ this stratagem, relying on the craving for social feedback that people want and mixing it with a reward system, be it “like”, “views”, or “shares”. Overall, this is a vicious cycle made up of a classical conditioning scheme that creates a drug-like addiction for the users participating on social media.

These techniques work well on adults but even better on kids. Perniciously, the different platforms involved have used them to target younger and younger people. The typical age when kids get their first smartphone has fallen to 10yo.
There’s more profit to be made out of a blank mind than an already filled one. This manipulation for profit is a reminder of our section on propaganda and how having full control of an environment allows for indoctrination.

The children are then caught in a virtual environment replacing their real-world needs and basic human drives: to be social and obtain goals.
To reiterate clichés, for teenage girls it could be wanting to be socially successful, and for teenage boys to want to earn competences. Meanwhile, they can easily fall victim to the internet’s attention and surveillance economies — an endless pursuit of stimulating virtual points which they believe will make them happy and successful.
I digress, we’ll have a full part with consequences.

Strikingly, the tracking and fight for attention starts earlier than childhood, the digitalization of our lives starts before birth. This is why data on pregnant women is the most valuable. We are the very first generation to be datafied before birth, this is a historical transformation.

On social networks, people most of the time want to entertain themselves, and entertainment is what makes them stay. One type of very popular entertainment is political infotainment, a sort of caricature of politics. A large portion of end users visit social media only for politics, and so the platforms will be incentivized in promoting this type of content. They give what people are looking for.

Many of these digital spaces rely on something called a “feed” or “home” page that will be generated and presented by default. This page will be optimized and curated by algorithms for the goal of the platforms. It is highly individualized, the news and information are selected by engagement value, intentionally designed to isolate end users from information that will make them leave, and creating a self-reinforcing feedback loop of confirmation, thus satisfaction.

From recommendation engines, search results, trends, autocomplete forms, it all happens through the mediation of algorithms. Whatever content is able to satisfy the criteria of the algorithms will be promoted, be it through clickbait titles, paying for fake views, fake reviews of products, bots interaction, and outrage-related and emotion inducing content.
The objective is to predict what we are more likely to interact with: click, view, engage.

The algorithms themselves are impartial, they do not know what the actual content is, and cannot discern between what is true and what isn’t. They are simply good at what they were programmed to do.
The algorithms are trade secrets of the platforms, opaque tools that influences millions of people each day. The data brokers and networks use this as a justification of non-accountability for the consequences, something we call mathwashing. Hiding behind the algorithm and pleading free speech.

Algorithm amplification is the phenomenon we are experiencing, the automatic radicalization of views and disinformation as an instrument for more views and spiraling out of control. It is a business choice made in pursuit of profit.
Social media will reward, through the point systems, extreme stances on subjects, these serve as the lubricant for these businesses, making them more profitable and influential. Studies have shown how users feel satisfied when finding a niche to cater to that generates a lot of these points (likes/shares).

This is an effect that is comparable with TV long-term media effects theory. It states that the more an individual watches television, the more they believe social reality matches what they see on it.

These have ramification in the real world, the consequences can be dire, as we’ll see later. For example, hate speech amplification can thrive and fuel genocide. Internet platforms play a dominant role in our conversation, extremism cultivated online has real world ramifications.

Evidently, they are now under public scrutiny and lawmakers want to intervene but social network, ad tech service providers, and others in the loop will fight to defend their castle of data.

The most common argument is that users have consented to what is happening by agreeing to the privacy policy and EULA they were presented with when signing up to the service. It is ok because it is the inevitable conclusion of our online activity. In other words, asking companies whose business models revolve around exploiting personal data for consumer-influence techniques to explain their privacy policies is like asking someone to shoot themselves in the foot.

The companies are now tweaking their tactics, using a patch-like approach rather than tackling the real issue, which is the core of their business. We’ll see in the last part some real solutions.

For example, AT&T recently said they would stop sharing users’ location details with data brokers, Facebook said it stopped allowing advertisers to use sensitive categories for targeting, Google said it’s now allowing users to download masses of their data.
Recently Google is also going to remove third-party cookies, to then allow third-parties to have access to the data only through them and getting information about groups and not individuals. They also get information directly from other sources such as their analytics and AMP service. A sort of differential privacy. Arguably, these give Google even more power over this type of data and drives away competitors.

In some countries, like the USA, the companies hide behind laws related to free speech to justify their policies. They are claiming that they do not want to be arbiters of truth.
That helps in taking the blame away from them, leaving them unaccountable as no USA critics want any platform to act as a censor.

As a last resort, like in any industry big enough, these companies don’t shy away from using their money to convince the lawmakers anywhere in the world.
Between 2005 and 2018, the big five tech in the USA have spent more than half a billion dollars lobbying the USA Congress, and donating to parties. Literally purchasing power.
Moreover, other databrokers also spend as much, companies such as CoreLogic spent as much as $215,000 on lobbying in 2020; and Acxiom, which spent $360,000 on lobbying in 2020 for issues related to data security and privacy.

Amazon, Apple, Facebook, Google and Microsoft have collectively thrown $582 million at legislators in an attempt to influence the lawmaking process and make sure their voices (and interests) are taken into account.

All of the firms have taken a position on privacy policy, with the issue coming up 3,240 times in all reports files—by far the most talked about topic. Last year alone, $66.4 million was spent on privacy issues by the tech giants. That includes $16.6 million from Google, $15.1 million from Amazon and $13 million from Facebook.

Whatever legality they deal with, like anything legal, depends on the country where the law is applied. Thus, any solutions will be localized, as we’ll see later.

This concludes our review of the emergence of new economies in the digital space with regards to social networks and ad tech providers. We’ve first seen the two new assets: data and attention. Then we’ve looked at their worth and which actors had incentives to capture them. Next, we’ve seen how a few are owning the pipeline in this new infrastructure and how they have agreements with third parties. After that we’ve dived into some of the manipulation tactics used to gain such assets. Finally, we’ve listed a few of the defenses some of these companies have used against policymakers.

Table Of Content

References








Attributions: E. Sibley, A Key to Magic & the Occult Sciences, c. 1800

March 21, 2021

Ponylang (SeanTAllen)

Last Week in Pony - March 21, 2021 March 21, 2021 11:19 PM

Version 0.6.3 of ponylang/ponyup has been released. The first recording is available for implementing RoaringBitmap in Pony.

Derek Jones (derek-jones)

Software engineering research problems having worthwhile benefits March 21, 2021 11:06 PM

Which software engineering research problems are likely to yield good-enough solutions that provide worthwhile benefits to professional software developers?

I can think of two (hopefully there are more):

  • what is the lifecycle of software? For instance, the expected time-span of the active use of its various components, and the evolution of its dependency ecosystem,
  • a model of the main processes involved in a software development project.

Solving problems requires data, and I think it is practical to collect the data needed to solve these two problems; here is some: application lifetime data, and detailed project data (a lot more is needed).

Once a good-enough solution is available, its practical application needs to provide a worthwhile benefit to the customer (when I was in the optimizing compiler business, I found that many customers were not interested in more compact code unless the executable was at least a 10% smaller; this was the era of computer memory often measured in kilobytes).

Investment decisions require information about what is likely to happen in the future, and an understanding of common software lifecycles is needed. The fact that most source code has a brief existence (a few years) and is rarely modified by somebody other than the original author, has obvious implications for investment decisions intended to reduce future maintenance costs.

Running a software development project requires an understanding of the processes involved. This knowledge is currently acquired by working on projects managed by people who have successfully done it before. A good-enough model is not going to replace the need for previous experience, some amount of experience is always going to be needed, but it will provide an effective way of understanding what is going on. There are probably lots of different good-enough ways of running a project, and I’m not expecting there to be a one-true-way of optimally running a project.

Perhaps the defining characteristic of the solution to both of these problems is lots of replication data.

Applications are developed in many ecosystems, and there is likely to be variations between the lifecycles that occur in different ecosystems. Researchers tend to focus on Github because it is easily accessible, which is no good when replications from many ecosystems are needed (an analysis of Github source lifetime has been done).

Projects come in various shapes and sizes, and a good-enough model needs to handle all the combinations that regularly occur. Project level data is not really present on Github, so researchers need to get out from behind their computers and visit real companies.

Given the payback time-frame for software engineering research, there are problems which are not cost-effective to attempt to answer. Suggestions for other software engineering problems likely to be worthwhile trying to solve welcome.

Gokberk Yaltirakli (gkbrk)

Constructing a sponge function from MD5 March 21, 2021 09:00 PM

.iocell { display: flex; justify-content: space-between; align-items: center; } @media (max-width: 540px) { .iocell { flex-direction: column; } .iocell * { width: 100%; } }

While doing some research, I came across the term “Sponge function”. After playing around with them and implementing one inside my kernel, I decided to write this blog post about how to write a simplified version. To keep low-level cryptographic code to a minimum, we will be relying on the hash function MD5. Buckle in, this is going to be a long one.

This article will start from the simple concept of an MD5 hash, and incrementally build on it until we implement a lot of common functionality that seems like black boxes. Every step should be small enough to be digested individually, while still contributing to the overall understanding of the topic. If anything is unclear, feel free to discuss it in the comments. The post is designed so that you can pause and play around with the concepts yourself any time you want, or speed through it if that is what you prefer.

Since we are going to base our project on the MD5 hash function, let’s set aside a small section to go through what it is. We will treat MD5 as a black box and ignore any complicated details for the sake of brevity.

MD5 - Background

MD5 is a cryptographic hash function that maps an arbitrary amount of data into 16 bytes (or 128 bits). In its heyday, MD5 was the go-to choice for hashing passwords, checking files for corruption, and tagging data against tampering. Those days are long gone. It has been considered broken for some time, and it is not recommended using it for anything security-related. But it is well-known and has been implemented for practically any computing device ever created. Luckily for us, Python comes bundled with a collection of hash functions in the hashlib module. Let’s quickly see how it works.

In [3]:
md5(b"Test").hex()
md5(b"Test 123").hex()
Out [3]:
'0cbc6611f5540bd0809a388dc95a615b'
Out [3]:
'f3957228139a2686632e206478ad1c9e'

As we can see, inputs of different lengths map to fixed-size outputs, and small changes in the input lead to completely different output values. Basically what we would expect from a hash function. In this article, we will be (ab)using the MD5 hash function to create a sponge function. But before we can do that, we need to know a sponge function is.

Sponge functions

A sponge function is a cryptographic function that can “absorb” any number of bits and “squeeze” any number of bits, like a sponge. This is different from what we observed about MD5; while MD5 will only produce fixed-size outputs of 16 bytes, a sponge function can output 1 byte, 26 bytes, 5000 bytes or any number that you like. This sounds fun and could be useful for a lot of different tasks, so let’s do some unholy programming and turn MD5 into a sponge function.

Sponge functions are fascinating. You can use a sponge function as a hash function, random number generator, Message Authentication Code or for data encryption. It would be apt to describe one as a cryptographic Swiss army knife.

Theory

In order to create a sponge function, we need an internal state (which is just a buffer), and function to pseudorandomly transform one state into another. We will take advantage of the two properties of the MD5 hash, our state buffer will be the 16 bytes of MD5 output and our transform function will be MD5 itself.

Sponge functions keep most of their internal state hidden. Both the bits absorbed and the bits squeezed only touch a small portion of it, so the output never reveals the full state of the function.

  • The first step is to initialize the state, either to zero or to a sensible default value.
  • For each byte of the input
    • The first byte of the state is XOR’ed with the input byte.
    • State is replaced by MD5(State).

This process absorbs all the input data into the state. After we have absorbed our input, we can squeeze out as many output bytes as we want by following a very similar process. For each byte that we want to produce:

  • Output the first byte of the state.
  • Transform the state using MD5.

Warning! You probably don’t want to use this for anything too sensitive. This is a proof-of-concept implementation using the broken MD5 function as a base. At least pick something better like ChaCha20 or SHA-512. In general, we want a large state and a transform function that mixes the state really well.

Implementation

Now that we have briefly gone over the theory, let’s get to the implementation. We will take this step by step and implement each operation we mentioned above. Our first step is the transform function.

Transform function

According to the theory we outlined above, we need a transform function that will take our state and pseudorandomly map it to another state. In our case, the MD5 hash function will be doing the heavy lifting for us. And by heavy lifting I mean MD5 will be doing pretty much the whole job.

We can transform the current state by passing it through the MD5 function. Here’s a small demonstration.

In [5]:
# Initial state
md5(b"").hex()

# Transform once
md5(md5(b"")).hex()

# Transform again
md5(md5(md5(b""))).hex()

# And so on...
Out [5]:
'd41d8cd98f00b204e9800998ecf8427e'
Out [5]:
'59adb24ef3cdbe0297f05b395827453f'
Out [5]:
'8b8154f03b75f58a6c702235bf643629'

Looks like everything is working. Let’s encapsulate this in a method of the Sponge class. Every time we absorb or squeeze a byte, we will mix the state using this method.

In [6]:
class Sponge(Sponge):
    def transform(self):
        self.state = md5(self.state)

Initialization

As we mentioned before, the state needs to be initialized before we start absorbing and squeezing any bits. Since we are using MD5, we want our state to be 16 bytes. Fortunately, MD5 makes sure that no matter the value we provide, the state will end up being 16 bytes. So we can pick any value, including an empty string. Let’s go with that option.

In [7]:
class Sponge(Sponge):
    def __init__(self):
        self.state = b""
        self.transform()

Let’s see if everything is working. After creating a Sponge instance, we should be greeted with the MD5 of the empty string, d41d8cd98f00b204e9800998ecf8427e.

In [8]:
s = Sponge()
s.state.hex()
Out [8]:
'd41d8cd98f00b204e9800998ecf8427e'

Absorbing a byte

Using the logic from the theory section, we can write the code for absorbing a single byte easily. We will replace the first byte of the state with input XOR first byte, and then transform the state.

In [9]:
class Sponge(Sponge):
    def absorb_byte(self, byte):
        self.state[0] = byte ^ self.state[0]
        self.transform()

We can quickly test that this results in different states after we absorb different data. Let’s try to absorb [1,2] and [2,1] and observe the difference in the states.

In [10]:
s = Sponge()
s.absorb_byte(1)
s.absorb_byte(2)

s.state.hex()
Out [10]:
'29a3a137fccfa18e5cfb5054b13aa412'
In [11]:
s = Sponge()
s.absorb_byte(3)
s.absorb_byte(4)

s.state.hex()
Out [11]:
'0291c72acd7e7da67bedcb15aa4733c6'

Absorbing a buffer

Generalizing this to buffers of arbitrary sizes is trivial. Just iterate over a buffer and absorb the bytes one-by-one. This is a useful abstraction because in real code we will commonly work with buffers instead of individual bytes.

In [12]:
class Sponge(Sponge):
    def absorb(self, buffer):
        for byte in buffer:
            self.absorb_byte(byte)

Here’s a quick sanity check: Our state should be different from the empty state after absorbing bytes. Let’s quickly verify this before moving on.

In [13]:
s = Sponge()
s.absorb(b"Test")
s.state.hex()
Out [13]:
'28a7cbf238c85bad13cc0fc4933a68ae'

Squeezing a byte

Since we don’t need to do any input-mixing, our squeeze logic will be simpler than our absorb logic. Following the theory part, we will output the first byte and transform the state again in order to produce one byte.

In [14]:
class Sponge(Sponge):
    def squeeze_byte(self):
        byte = self.state[0]
        self.transform()
        return byte

Let’s produce some bytes and see if it’s working.

In [15]:
s = Sponge()
s.absorb(b"Test")
[s.squeeze_byte() for _ in range(5)]
Out [15]:
[40, 243, 39, 189, 220]

Squeezing a buffer

Going from extracting single bytes to exctracting buffers is not too difficult. We can use a list comprehension to write this in a concise way.

In [16]:
class Sponge(Sponge):
    def squeeze(self, size):
        buf = [self.squeeze_byte() for _ in range(size)]
        return bytes(buf)
In [17]:
s = Sponge()
s.absorb(b"Test")
s.squeeze(5).hex()
Out [17]:
'28f327bddc'

It might seem like a very small amount of code, but this is all we need. It might be useful to add some convenience funtions later, but for 99% of the use cases these methods will be sufficient. Now we can start playing around with our sponge function.

Use cases

In the beginning, we mentioned that sponge functions have a wide range of cryptographic use cases. In this section I will implement them in simple ways to provide some examples on how useful sponge functions can be.

Hash function

Hashing is the easiest thing to implement with a sponge function. In fact, we already saw a demonstration of this when testing the squeeze function above. To clarify; we can produce a hash by absorbing all the input and squeezing a fixed number of bytes.

In [18]:
def sponge_hash(data):
    s = Sponge()
    s.absorb(data)
    return s.squeeze(10).hex()

sponge_hash(b"123")
sponge_hash(b"Test 123")
sponge_hash(b"Test 113")
Out [18]:
'91e292b50acc3c838a0a'
Out [18]:
'b7a2027b77e56ca5d11f'
Out [18]:
'62eb28a8017c976f7ccc'

This seems to fit our criteria of a hash function; inputs of different sizes map to fixed-size outputs, and small changes in the output result in completely different hashes. You can substitute 10 with any other length in order to change the output size of your hash. In general, longer hashes are less likely to have collisions but take up more space. You can play around and pick the sweet spot for your use case.

Random number generator

Random number generation is also something that can be done with sponge functions. The basic idea is to absorb the RNG seed and then squeeze out bytes for as many random numbers as you need. In the following example, I am using a fixed seed in order to generate 10 unsigned 16-bit integers.

In [19]:
import struct

s = Sponge()
s.absorb(b"Seeding the RNG")

def rng():
    buf = s.squeeze(2)
    return struct.unpack('H', buf)[0]

[rng() for _ in range(10)]
Out [19]:
[29342, 19407, 47040, 9984, 55893, 40500, 56312, 36293, 58610, 10880]

If we use the same seed, we will always get the same output. This might sound counterintuitive for the goal of generating “random” numbers, but it is commonly required to be able to replicate random results. If this is not something you need, you can seed from an actually random source or something that changes regularly like the current time. This depends on what qualities you expect from your random numbers. Below is a demonstration of how to read a random seed from /dev/urandom.

In [20]:
s = Sponge()

with open("/dev/urandom", "rb") as urandom:
    s.absorb(urandom.read(64))

[rng() for _ in range(10)]
Out [20]:
[56437, 39690, 47308, 16515, 29378, 11318, 32523, 18419, 47972, 4874]

Idea! You can absorb values while generating them as well, this allows you to periodically reseed your RNG using external sources.

Message Authentication Code

We can use a sponge function in order to produce a mechanism that can produce and verify signatures using a secret key. This is a very common technique, especially in mobile and web applications where it is used to store the session on the client without letting them tamper with it. If you want to read more about this use-case, check out JSON Web Tokens.

In order to produce a signature; we will absorb the data, and the secret key. After this, we can squeeze any number of bits that can be used as the signature.

In [21]:
def sign(data, key):
    s = Sponge()
    s.absorb(data)
    s.absorb(key)
    return s.squeeze(5)

data = b"Hello world!"
key  = b"password123"
signature = sign(data, key)

signature.hex()
Out [21]:
'480e4c2b9d'

Verification of a signature can be done by generating the signature yourself and comparing the signature you received with the signature you generated. If they match up, the data and the signature have not been tampered with.

In [22]:
def verify(data, sig, key):
    correct = sign(data, key)
    return sig == correct

verify(data, signature, key)
Out [22]:
True

As expected, the signature can be verified successfully. Let’s try to modify the data a little and switch two characters around.

In [23]:
data = b"Hello wordl!"

verify(data, signature, key)
Out [23]:
False

Similarly, we can have the correct data and tamper with the signature instead. The verification fails, showing that both the signature and the data are protected against corruption and tampering.

In [24]:
data = b"Hello world!"
signature = bytes.fromhex("481e4c2b9d")

verify(data, signature, key)
Out [24]:
False

Stream cipher

A stream cipher allows us to encrypt and decrypt a stream of bytes using a single secret key. This can be used to make sure only you, or anyone you entrust with the secret key can decrypt the data.

In [25]:
def stream_cipher(data, key):
    s = Sponge()
    s.absorb(key)
    
    output = bytearray(data)
    
    for i in range(len(data)):
        key = s.squeeze_byte()
        output[i] ^= key
        
    return output

data = b"Hello, world!"
encrypted = stream_cipher(data, b"password123")

encrypted.hex()
Out [25]:
'b571d4065c54547bdf1a002d8e'

Decoding a stream cipher is very simple, in fact it takes no code at all. Simply encrypting the already encrypted value with the correct key will end up decrypt your data. Let’s try to decode our data with the correct and incorrect passwords.

In [26]:
stream_cipher(encrypted, b"password123")
stream_cipher(encrypted, b"password132")
Out [26]:
bytearray(b'Hello, world!')
Out [26]:
bytearray(b'\x12\x88\x98?\x9aESh\x9a\x96\x9d\x17\x1d')

Idea! You can combine the Message Authentication Code and the Stream Cipher in order make a chunk of data that is encrypted and resitant against tampering. This is called Authenticated encryption commonly done in real protocols. Try to implement the AE and AEAD variants.

Warning! It is recommended to also include an IV / nonce with your key in order to make sure the same plaintext encrypts to different ciphertexts.

Time-based one-time password

You might have noticed that a lot of services these days ask you for one-time tokens when trying to authenticate. These tokens are usually displayed as 6 digits and expire in ~30 seconds. Using a sponge function, we can implement our own version pretty easily. Here’s how these one-time tokens work.

  1. The server and the client have a pre-agreed secret key.
  2. When authenticating, the server asks the client to produce a token.
  3. The client absorbs the current time and the secret key in order to produce a token, and sends it to the server.
  4. The server independently produces a token using the same key and following the same rules.
  5. If the tokens match up, the client is granted access.
In [27]:
import time

key = b"Secret key 123"

def get_otp(key, period=10):
    t = time.time()
    value = int(t / period)
    time_left = period - (t % period)
    
    s = Sponge()
    s.absorb(key)
    s.absorb(str(value).encode('ascii'))
    
    otp = [s.squeeze(1).hex() for _ in range(3)]
    otp = ' '.join(otp)
    
    return otp, int(time_left)

otp, time_left = get_otp(key)

f"OTP is '{otp}'."
f"Valid for {time_left} more seconds."
Out [27]:
"OTP is '7c 0b c8'."
Out [27]:
'Valid for 7 more seconds.'

If the code is still valid, meaning that time_left is still not zero, the OTP will be considered valid.

In [28]:
otp == get_otp(key)[0]
Out [28]:
True

If we wait until the timer runs out, our OTP will no longer validate.

In [29]:
time.sleep(time_left + 1)

otp == get_otp(key)[0]
Out [29]:
False

Idea! It is recommended to also accept codes that should have been generated before or after the current time in order to account for clock skew. After all, the current time is the input that determines what the code will be, so authentication won’t be possible if the clocks don’t match up.

Block cipher

Stream ciphers that use cryptographic hashes have a risk of running into cycles. This is when calling the transform function on the state will eventually go back to a previous one. In order to mitigate this, we can use a block cipher instead.

The difference of a block cipher is; instead of constructing the sponge once and squeezing bytes from it for the whole stream, we instead absorb a counter into the sponge along with the key and nonce in order to generate a fixed block of bytes. This is where we get the name “block cipher”.

In [30]:
BLOCKSIZE = 10

def get_block(key, counter):
    s = Sponge()
    s.absorb(key)
    s.absorb(str(counter).encode("ascii"))
    return bytearray(s.squeeze(BLOCKSIZE))

def block_encrypt(data, key):
    size = len(data)
    result = b""
    
    counter = 0
    while data:
        # Chop off BLOCKSIZE bytes from the data
        data_block = data[:BLOCKSIZE]
        data = data[BLOCKSIZE:]
        
        # Generate a block cipher block
        block = get_block(key, counter)
        
        for i, byte in enumerate(data_block):
            block[i] ^= byte
        
        result += block
        counter += 1
        
    return result[:size]

data = b"Hello, world! Don't forget to stay hydrated."
encrypted = block_encrypt(data, b"test")
encrypted.hex()
Out [30]:
'eec587d16686e81d26ed800677e609a6d2fed11b7a27bbb233370cdba1d941cdc01d42c4c3e7ee90a09333c1'

As we did with the stream cipher, let’s try to decrypt our data with correct and incorrect keys.

In [31]:
block_encrypt(encrypted, b"test")
block_encrypt(encrypted, b"TEST")
Out [31]:
b"Hello, world! Don't forget to stay hydrated."
Out [31]:
b'\xd1%\x17\xd9\xe0\x1bh\xaf~2\xc0\x9f\x8da\xb2\xe4\xa4\x05\x99\xc4\x82\xf7\x02\x0c\xed+\xa1\xf4\xefa?\x82l9Q\x05=B>p%\x9e\xa0q'

Closing words

If you made it this far, thank you for reading my article. I’d appreciate any emails or comments. You now have an understanding of how to implement some commonly used cryptographic technologies from scratch, please let me know what projects you end up doing with sponge functions.

March 20, 2021

Gonçalo Valério (dethos)

And… the blog is back March 20, 2021 10:30 PM

You might have noticed that the website has been unavailable during the last week (or a bit longer than that), well, the reason is quite simple:

OVH Strasbourg datacenter burning OVH Strasbourg datacenter burning (10-03-2021)

It took sometime but the blog was finally put online again, new content should be flowing in soon.

And kids, don’t forget about the backups, because the good old Murphy’s law never disappoints:

Anything that can go wrong will go wrong

Wikipedia

March 19, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Artifacts And Spaces: Memes & Cults March 19, 2021 10:00 PM

Metamorphosis from a frog's head to Apollo. Repelled bestiality

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 1 — The Artifacts and Spaces
  • Section 4 — Memes & Cults
Table Of Content
  • Defining Memes
  • Internet Memes As A new Mode Of Communication
  • Memeplex — The Meme Ecosystems
  • Effective Memes
  • Memeoids
  • Relation To Cults

In this last section of our first part about artifacts and places, we’ll explore memes, internet memes, and cults, their definitions, how they have transformed and crisscrossed, their relationship, their role as communication vectors, and the extreme forms that they can take.

Meme is a word that comes up in many conversations. It can refer to two related definitions, the classic one and the internet-related one.

Let’s start with the classic definition.
The Oxford English Dictionary defines a meme as “an element of culture that may be considered to be passed on by non-genetic means, especially imitation”.
Keith Henson defines memes as “replicating information patterns: ways to do things, learned elements of culture, beliefs or ideas.”

A meme is a unit of culture, it is copied from other individuals, and propagates by being actively shared. It encompasses constituents of culture within itself. A mix of narrative, ideas, behaviors, and tropes.
The meme information has not only to propagate but also to have an impact and persist.

The memotype, the actual information-content of a meme, its semantic core, can have direct or indirect meaning. It can use figurative language that resonates within a culture such as commonly recurring images, rhetorics, motifs, and clichés.
In its physical form, its appearance, a meme can be contained in any media shell: verbal, visual, textual, auditory, gustatory, sensory, or anything the senses can allow.

These definitions are enticing but don’t include the essential novelty that the concept of meme has brought: a parallel with genes. The name comes from a mix of gene and mimetes, from the Greek mimetes, the imitator or pretender. It was initially supposed to be called Mimeme but got abbreviated to meme.
Thus, memes also add the notion that the transmitted information is granular and can be studied from an evolutionary standpoint. Memes are transported in their media package, the copies are subject to variation, and they compete for space in our memories and for the chance to be copied again. Only successful variants can survive.

Susan Blackmore re-stated the definition of memes as: “whatever is copied from one person to another person, whether habits, skills, songs, stories, or any other kind of information. Memes, like genes, are replicators in the sense as defined by Dawkins.”

Memetics is the study of the analogy between the elements moving in an “information society” and Darwinian evolution. This is done in multiple ways, one of them is to apply models to visualize the transfer of information, to see the criteria that makes them stick, to analyze what makes them successful and reproduce. We’ve done a great overview in the previous sections on big data and models of information flow.

There are two movements that argue different definitions of what memes are.

  1. A unit of cultural transmission (that can be copied, located in the brain). That can be described as a memory item, a portion of neurally-stored information that is instantiated/caused by interacting with another person’s nervous system. (I know this is heavy).
  2. Observable cultural artifacts and behavior.

Some people think memes have a physical nature in the brain and others think that they only exist within society and not as observable entities in the brain. Essentially, one group defines memes as internal and another defines them as external to oneself.
The first group argues that in the future we could obtain technologies that would allow us to find such observable meme.
Due to these separate definitions, it’s better to differentiate and study memes using two words.

  • i-meme, for the internal cognitive phenomena like neuronal behavior, studied via fMRI, neuroimaging techniques, genetic profiling, hormonal tests, and neurochemical reactions.
  • e-meme, for the external phenomena like culture and behavior studied by simulated propagation on social networks, game theory, attention and awareness studies, motivations, emotions, etc..

Like any scientific endeavor, it’s important that things be quantifiable and well-defined, otherwise it’s a show-stopper. Research on memes always precisely do that, for example let’s quote a paper: “A meme is information (using Shannon’s definition as that which reduces uncertainty) transmitted by one or more primary sources to recipients who, as secondary sources, retransmit the information to at least an order of magnitude more recipients than primary sources, where propagation or the information persist at least ten hours and the information has observable impact in addition to its transmission.”
Most research focus on e-memes.

It’s also good to be aware that there are skeptics about the terminology used when comparing memes to genes as they don’t equate to one another. The critics call memetics a pseudoscience because memes do not follow the same selection criteria as genes. Namely, the definition of a meme is still vague, there is no scientific demonstration of the replication happening, the encoding of a meme isn’t known, and the mechanisms of replication aren’t stable.
Memes can thus be placed in between the science fiction ideas of brainwashing, viral marketing, and Darwinian evolution.
Regardless, playing with the notion that cultural information moves like genes is interesting in and of itself as a tool for thoughts.

In common parlance when someone says meme they can mean “internet meme,” usually a photograph with a clever caption that is shared around the Web. Often created anonymously, remixed endlessly, and shared constantly, the most viral memes seem to materialize out of nowhere.
The term has seen a rebirth with the internet, even the Merriam-Webster notes that a meme is now popularly defined as “an amusing or interesting item or genre of items that is spread widely online especially through social media.”

Limor Shifman characterizes these internet memes as:

  • A group of digital items sharing common characteristics of content, form, and/or stance, which
  • were created with awareness of each other, and
  • were circulated, imitated, and/or transformed via the Internet by many users.

The author of the original meme idea, Dawkins, noted in 2013 that it “has itself mutated and evolved in a new direction. An internet meme is a hijacking of the original idea. Instead of mutating by random chance, before spreading by a form of Darwinian selection, internet memes are altered deliberately by human creativity. In the hijacked version, mutations are designed — not random — with the full knowledge of the person doing the mutating”

However, even though internet memes are associated with creative medium like funny image macros, this perception is limiting. Internet memes are “culturally resonant items easily shared and spread online”, and as with anything on this communication channel they have to follow its rules to succeed.
As a consequence, the sharing and virality aspects are self-explanatory, these criteria are required of social media messages for them to pierce through bubbles. In that case, it isn’t surprising that the concept of memes, when applied to the internet, will have to be deliberately designed using attention grabbers such as visual content. In that sense, internet memes are only instances of memes and not a brand-new definition.

However, unlike generic memes, the appearance of internet memes is limited to what can be transported on the internet. This can be textual, audio, or visual.

  • Verbal media memes: slogans, words, text, keywords, hashtags, etc..
  • Audio media memes: songs, melodies, etc..
  • Visual media memes: images, posters, “Photoshoped” pictures, caricatures, etc..
  • Hybrid/audiovisual media memes: video, vines, etc..

Internet memes are culturally relevant, broadly resonant, organically developed, and voluntarily spread.
On the internet they have become a new shared cultural language, a way to convey meaning. It’s a new type of vehicle for communication, packing ideas into transportable containers that can evoke meaning, emotions, memories and more.
The sentence “a picture is worth a thousand words” applies well to internet memes.

Memes are convenient and accessible mental model creators. They offer a quick way to construct a mental representation of a situation, event, or object and allow to process, organize, comprehend, explain, judge, and formulate predictions and inferences about these former.
They provide a narrative in a condensed package, and so are tools for analysis. They are media rhetoric, deconstructing complex ideas using compressed symbolic images/text/audio.
As you can see, it makes memes excellent persuasion and propaganda anchors and hooks, you can revisit the section on propaganda and persuasion for more info.

Even if some memes weren’t created with a purpose, they will sooner or later be appropriated by political or business elites for leverage. Many memes, because of their symbiosis with social media, have to frequently show strong emotions that is with or against ideologies, show humor, parody, be snarky, silly, witty, angry, and most predominantly be poignant.
As was said in the social media section, it is our own confused cultural codes that are abused to make messages spread. That is why successful memes take stances, either as meme-aggressor — discrediting an idea, political program, politics, or media person or authority — or as meme-protector — a message to keep the status quo, enhance a person’s profile or authority, or protect a state ideology.
Memes frame the attention on a single aspect of an issue and cannot confer enough nuances due to their condensed form. They are filtered views of the world.

However, in the eye of the average person, internet memes are looked at as entertainment content, more precisely infotainment — like anything that is on social media. It is easy to overlook their power as they do not appear to have substantive content and can take the form of hastily constructed cartoons. Nevertheless, after the 2016 USA elections, the media outlets of the world started crowning memes as “the new king of political communications”.

Unlike TV, printed media, and most other forms of communication, internet memes are built for speedy consumption on smartphones and are prioritized because of the visual aspect of social networking. They provide a quick emotional hit in comparison with long-winded articles, and thus an instant payoff.
Studies have shown that people perceive internet memes via a peripheral route in their brain, according to Richard Petty’s elaboration likelihood model of persuasion (ELM). This means that they rely on a general impression, the mood, and early part of the message contained in the meme and not on any significant cognitive efforts.
This goes together with the notion that internet memes are perceived as infotainment.
This peripheral route isn’t effective when it comes to long-term changes in attitudes but can have some influence on the short-term. However, as you can imagine, this can be countered with repetition, more units of the same message.

More than anything, memes are a way for individuals to express themselves, to show who they are and what they believe in. Consuming and sharing memes is an act of identification.
It has been found that there is a close relation between belonging to a sub-culture or having affinity to a group and the propagation of related internet memes. They are used as a form of commentary, or even rebellion against or for official discourse.
People might even seek out memes as a type of emotional release.

This creates a constellation of memes, a meme pool. In it, there are organizations of large groups that are copied and passed together, co-adapted for their survival. We call them meme complexes, or memeplexes.
The memes are aware of each others, like cross-breeding, sometimes cross-referencing the format, appearance, message, or semantic core of other memes.
This symbiosis reinforces individual memes.

The memeplex, like an ecosystem, balances itself with different memes, creating a memetic equilibrium. Similar to how propaganda sells a coherent world of cohesive meaning. Some people like to talk of memes as the new urban legends medium, the modern hermetism or mysticism, others as an ecological forest.
Experts in the domain of semiotic are getting interested in memes for this reason. Semiotic being the study of signs and symbols with an emphasis on their interpretation, context, and meaning. It covers many of the things we’ve seen and we’ll see, such as: ideologies, persistent beliefs, commonly held views, cultural anxieties, dominant mindsets, received wisdoms, expectations, preconceived notions, assumptions, widespread preoccupations, historical facts, principles, ideals, supposed eternal truths, unspoken tensions, shared hopes, master narratives, public opinions, prevalent attitudes, considerations of what is normal, outlook, established power structures, and more.

Through different means, such as the gathering of big data, it is possible to track the propagation of the memes across multiple communities and measure their influence, discover who influences others the most and which variants are best.

A research analyzing memes generation on online platforms has showed that fringe communities such as 4chan and reddit r/The_Donald (defunct) were big generators of memes, which then were influential on more frequented social networks such as Twitter. Their messages were mainly political ones, imbued with emotional and hateful content.
Nothing surprising here either.

Even though we can track the spread of memes, the real origin of them is often unknown. Some have come to call this phenomenon crowd-sourced democratic expression, and others an intellectual apocalypse. That reminds us of what social media are about.
This means that there is no accountability for the creation of the message. It makes it harder for political groups to blame or retaliate against the memotype, or even criticise the creator, as they could with mainstream media reporters or journalists.
Indirectly, that’s also an advantage for the propagandist, they can use memes as tools of black propaganda, hiding their identity.

We’ve seen that emotions and shock-value make a meme spread. What other criteria make a meme successful?

Let’s state something from the get go: Memes do not have to be true to replicate. Memes aren’t about their creation but about their spreading. The constant production of new content leads to a fight for the survival of the fittest.

Like genes, meme survival is measured along 3 axis fecundity, fidelity, and longevity. They also spread like viruses. There are studies quantifying the analogy with evolutionary pressure, showing the selection process and the movement between different communities, moving from fringe ones to more mainstream ones. Prosperous memes have to bridge cultures to live on. Being restricted to silos is a disadvantage and can lead to extinction.
Other studies show how the SIR (Susceptible - Infected - Resistant) model of disease applies perfectly to memes. Susceptible people being the ones who hadn’t seen the meme, infected people those who were actively interested in its content and to spread it further, and the recovered were those who had seen the meme and lost interest.
These research and models confirm that memes can be studied like infections. Consequently, they can be treated or have their propagation hindered by quarantine for example.

This is a topic that the author of the concept of memes, Richard Dawkins, directly tackles in the essay entitled “Viruses of the Mind”. Though the essay dives into the parallel with religious doctrines, it still puts down criteria that viruses have to fulfill to survive:

  • Spread silently, be difficult to detect (by its host)
  • Not spread if the virus is already present in the host (double infection)
  • Not kill its host, or infer too much damage, at least not rapidly enough to stop the spread

The essay then extracts and applies this to memes with two conditions:

  • A readiness to replicate information accurately, perhaps with some mistakes that are subsequently reproduced accurately.
  • A readiness to obey instructions encoded in the information, so replicated.

When it comes to the shell of the message, the visual nature of internet memes adds to their potential persuasive power on social media, making them more attractive. They are comparable to editorial cartoons in newspapers.
An image has the advantage of stickiness and brevity, communicating concepts quickly. This is something that has been studied. Our brain interprets images faster than text, thus reducing the likelihood of thinking deeply about the content, as we said earlier. 70% of our neurons are in our eyes, and we process images 60 thousand times faster than any other media type.
Rational discussion is less effective than a visual campaign. Images are more emotionally evocative, and emotional cues have a preferential path in the brain. We do not even need to be aware that these emotions are elicited, they could be subliminal.
As far as metrics go, research in the field of marketing clearly show that visual content is more shared than anything else, help retain information, and make following instructions easier.

When it comes to the memotype, the semantic core of the message, we’ve already mentioned that successful memes are reflections of us. It is worth reiterating.
A meme is an expression of our culture, our beliefs, behaviors, and values. The ones that go viral are unleashing our repressed cultural agenda, they don’t use novelty but use a potential that was already there. They exploit our latent, yet intrinsic, gaps in our cultural code, using our cultural vulnerabilities, the ones we aren’t immune to. Therefore, memes don’t compete for dominance by appealing to our intellect, our compassion, or anything related, but they compete on triggering our automatic impulses.
Logic and truth have nothing to do with memes, but they rely on our flight-or-flight reactions, the fastest reactions. From that perspective, memes aren’t really pro-social but anti-social.
Like viruses, memes reveal less about themselves than about their hosts. We wouldn’t be sick if we had an immune system capable of recognizing the shell and then neutralizing the code it contains.

Thus, memetic material that is provocative, sensational, making news headlines, and shocking will be more successful. Studies show that people are more willing to share memes that evoke stronger disgusts (Heath et al.).
Another study conducted in 2017 related to an analysis of the keywords used in memes shows that the most popular memes were politically related. From USA MAGA movement, to the USA libertarian, to the president of the USA at the time (This study was USA-centric).
That means the content often acts like a propaganda anchor or hook, and is receptive to the current paradigms and belief systems.

Then this all manifests through real consequences such as media confusions, lack of trusts, protests on the streets. It’s all viral memetics in action. We’ll dive into consequences in another part of this series.

Another criteria that helps survival is whether a meme can create a symbiotic relationship with other memes, what we referred to as memeplex. The meme could already fit properly with others, cooperating, or it could develop progressively a real or artificial compatibility.
Ganging up helps the meme flourish, and the longer they are attached together, the more they’ll merge into a single package, a co-evolution.
Additionally, memes can have defensive systems, reject other memes that would hurt it. Consequently, creating an intolerance within the meme complex.

The content and the medium aren’t sufficient alone, the context is extremely important too. We need to take into account the external factors such as:

  • The timing to coincide with major events
  • The architecture and dynamics of the network
  • The media literacy and facility of the target population

Let’s now review some of the results of research related to meme success.

The University of Memphis research shows that for an image macro to be successful the predictor variables were:

  • A shorter memes (fewer than four words)
  • The use of concrete terms (more memorable than abstract language)
  • Avoiding swear words as they hamper virality (negatively correlated)

A research on military memetics by Dr.Robert Finkelstein digs into the measurement of the fitness of memes. It calculates the memetic fitness unit as the number of person-hours transformed. The most fit memes correlated with:

  • Propagation (.428): number of recipients, type of recipients, dispersion of recipients
  • Persistence (.284): duration of transmission, duration of memory
  • Entropy (.087): small, medium, large (At this time, we consider smaller memes to be more important than larger memes. Larger memes may all be memeplexes (e.g., the typical human brain cannot hold more than 10 “chunks” of information in short-term memory))
  • Impact (.20): Individual consequence, societal consequence

That means, according to this research, that the most important factor to fitness are propagation and persistence, and impact of a meme, but mostly propagation.

The research also summarizes some criteria that should be applied to the information content of the meme, such as:

  • accuracy
  • relevance
  • timeliness
  • usability
  • completeness
  • brevity
  • security

The computer scientists at the Hebrew University tried to use an algorithm to predict the spread of certain hashtags on Twitter using a dataset of more than 400 million Tweets. Their findings were that successful memes factors were based on:

  • The meme’s content
  • The meme’s context
  • The social graph

Another paper on the military application of memes proposes the following:

  • Meme conciseness (1-10 words)
  • Choosing information which propagates, has impact and persists (Info-PIP), and be able to monitor them via metrics

Some papers even go as far as to say that it’s not really about the meme criteria but the network criteria. That’s what the Indiana University’s Center for Complex Networks and Systems Research found: memes that went viral were no different from those that didn’t, the success was due to the structure of the social network.

These research all go in a direction that echoes what we said earlier: memes are a reflection of us. These memes do not necessarily have to be malicious, they could have beneficial purposes. Today memes have surpassed genes as the dominant driver in human behavior and we have to understand their importance. For that we have to develop a healthy cultural immune response to hostile ones. We should learn to quarantine, distance ourselves, and apply immune attacks to neutralize them when necessary.
These are requirements in an information society.

Two concepts come to mind in relation to our society: memocides — the deliberate eradication of a memeplex from a population, by either killing the host or by censorship — and eunemics — inspired by eugenics but applied to memes, deliberately improving the quality of the meme pool by selective breeding. Eunemics is done through memetic engineering, using engineering principles to mold them.
This means memes can also be informational weapons, propaganda tools.

When meme and memeplex are brought to their extreme, a person is overtaken by them, letting go of their own survival instinct and favoring the meme instead. In the symbiotic relationship, the meme has won and consumed its host. The people affected by this phenomenon are called memeoid.
Examples of this include kamikazes, suicide bombers, cult members, and others. Keeping in mind that the previous criteria for reproduction still applies, namely that the meme should keep their host alive long enough to spread properly.

This is a trading between genetic reproductivity and memetic reproductivity, which might not coincide. Promoting the memeplex might not be correlated with promoting a gene pools. Replacing the purpose of one’s existence with the spread of the meme.

Richard Dawkins cites the following when it comes to memeplex consuming individuals, it becomes a sort of narrative.

  • Impelled by deep inner conviction that something is true/right/virtuous, against all reasons
  • Make positive virtue out of the strong stance on the meme
  • Conviction that mystery is a good thing
  • intolerant behaviour towards perceived rivals of the meme
  • If follow a different meme from parent it comes from a charismatic individual
  • The internal sensations of the meme holder are similar to ones associated with sexual love

Strong memeplex are thus similar to cults, people being mind controlled by ideologies and its members turning into puppets of the memes.
Singer’s six conditions of mind control reverberate this idea:

  • Keep the person unaware of what is going on, and what changes in themselves/their behavior is happening.
  • Control the person’s social and/or physical environment, especially the person’s time. Remind them of the narrative all the time.
  • Systematically create a sense of powerlessness in the person. Keep them away from their former social support group or identity.
  • Manipulate a system of rewards, punishments, and experiences to inhibit the former identity. Accomplished through various methods of trance induction, speaking patterns, guided imagery, etc..
  • Manipulate a system of rewards, punishments, and experiences to promote the group’s ideology and behaviors. Compliance are rewarded while questioning or doubts are met with redressed and rejection.
  • Put forth a closed system of logic and an authoritarian structure that doesn’t permit feedback and refuses to be modified except by leadership.

These are all things that we’ve seen in the mix of social media space and the usage of memes in their memeplex.
This is especially true of the attention-reward system of social media. As social primates, attention is a measure of status. When we get attention our brain releases chemicals such as dopamine and endorphins, creating an addiction (AAR: Action - Attention - Reward). This system can be shortcut with memes. The memeplex, and the ones carrying it, requires intense social interactions and behaviors such as sharing the meme or reenacting its content.

The members then create their own isolated bubbles of coherent meaning. Some of them opposed, clashing, and creating the rise of many of the things we’ve seen so far.

Clearly, as with propaganda, children are more vulnerable to subversion and are easy pray for memeplex. Children are pre-programmed to absorb useful information at thigh rate. It is harder as adults to remove the pernicious and damaging information that have been introduced early.

Deprogramming studies, used to get people out of cults, can be applied to memeoids too. They rely on invoking new capture-bonding social reorientation mechanisms.

Similar to our conclusion on propaganda, we have to say that some memeplex can be considered pathogenic but others can be beneficial or relatively harmless. The judgement is subjective.
In a strict sense, everyone, including you and I, are memeoids, carriers of memeplexes. We are all hosts, one way or another, but not everyone reaches a degenerative state, even though we have the potential.

A skill that should be developed is metamemetic thinking: being able to recognize and acknowledge the content of memeplexes, the illogicality in them, and anything related to their attributes.
The metamemetic can also be used to understand the people associated with them. Coming back to our data and metadata section, memes are information and thus can be used to characterize, analyze, and classify people.

This concludes our review of what memes are and their relation with the internet. We’ve first seen their definitions, one is based on a comparison with genes applied to cultural information, and the other to internet image macros and tropes. We’ve then made the parallel to how these two aren’t dissimilar and how internet memes are only instances of memes adapted to social media. We’ve taken a look at how people perceive these memes, how they are used as a way to communicate and express one’s identity. Then we’ve dived into the memeplex, how memes associate themselves to create a world, to better survive. Next, we’ve inspected what makes some memeplex successful by summarizing some research. Finally, we’ve investigated the close relation between the extreme form that memeplex can take, turning people into memoids, and cults.

Table Of Content

References








Attributions: J.C. Lavater, Physiognomik, Vienna, 1829

March 17, 2021

Gustaf Erikson (gerikson)

One year since WFH started March 17, 2021 05:55 AM

One year since WHO declared a pandemic March 17, 2021 05:54 AM

March 15, 2021

Ponylang (SeanTAllen)

Last Week in Pony - March 14, 2021 March 15, 2021 01:58 AM

We’re migrating the Pony Tutorial and Pony Patterns to MkDocs-Material. There’s a new tool for interactively exploring the Pony AST from a debugger. ponylang/http has been updated.

March 14, 2021

Derek Jones (derek-jones)

The impact of believability on reasoning performance March 14, 2021 10:38 PM

What are the processes involved in reasoning? While philosophers have been thinking about this question for several thousand years, psychologists have been running human reasoning experiments for less than a hundred years (things took off in the late 1960s with the Wason selection task).

Reasoning is a crucial ability for software developers, and I thought that there would be lots to learn from the cognitive psychologists research into reasoning. After buying all the books, and reading lots of papers, I realised that the subject was mostly convoluted rabbit holes individually constructed by tiny groups of researchers. The field of decision-making is where those psychologists interested in reasoning, and a connection to reality, hang-out.

Is there anything that can be learned from research into human reasoning (other than that different people appear to use different techniques, and some problems are more likely to involve particular techniques)?

A consistent result from experiments involving syllogistic reasoning is that subjects are more likely to agree that a conclusion they find believable follows from the premise (and are more likely to disagree with a conclusion they find unbelievable). The following is perhaps the most famous syllogism (the first two lines are known as the premise, and the last line is the conclusion):

    All men are mortal.
    Socrates is a man.
    Therefore, Socrates is mortal. 

Would anybody other than a classically trained scholar consider that a form of logic invented by Aristotle provides a reasonable basis for evaluating reasoning performance?

Given the importance of reasoning ability in software development, there ought to be some selection pressure on those who regularly write software, e.g., software developers ought to give a higher percentage of correct answers to reasoning problems than the general population. If the selection pressure for reasoning ability is not that great, at least software developers have had a lot more experience solving this kind of problem, and practice should improve performance.

The subjects in most psychology experiments are psychology undergraduates studying in the department of the researcher running the experiment, i.e., not the general population. Psychology is a numerate discipline, or at least the components I have read up on have a numeric orientation, and I have met a fair few psychology researchers who are decent programmers. Psychology undergraduates must have an above general-population performance on syllogism problems, but better than professional developers? I don’t think so, but then I may be biased.

A study by Winiger, Singmann, and Kellen asked subjects to specify whether the conclusion of a syllogism was valid/invalid/don’t know. The syllogisms used were some combination of valid/invalid and believable/unbelievable; examples below:

        Believable                  Unbelievable
Valid
        No oaks are jubs.           No trees are punds.
        Some trees are jubs.        Some Oaks are punds.
        Therefore, some trees       Therefore, some oaks
                   are not oaks.               are not trees.
Invalid
        No tree are brops.          No oaks are foins.
        Some oaks are brops.        Some trees are foins.
        Therefore, some trees       Therefore, some oaks
                   are not oaks.               are not trees.

The experiment was run using an online crowdsource site, and 354 data sets were obtained.

The plot below shows the impact of conclusion believability (red)/unbelievability (blue/green) on subject performance, when deciding whether a syllogism was valid (left) or invalid (right), (code+data):

Benchmark runtime at various array sizes, for each algorithm using a 32-bit datatype.

The believability of the conclusion biases the responses away/towards the correct answer (the error bars are tiny, and have not been plotted). Building a regression model puts numbers to the difference, and information on the kind of premise can also be included in the model.

Do professional developers exhibit such a large response bias (I would expect their average performance to be better)?

People tend to write fewer negative tests, than positive tests. Is this behavior related to the believability that certain negative events can occur?

Believability is an underappreciated coding issue.

Hopefully people will start doing experiments to investigate this issue :-)

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Artifacts And Spaces: Data & Metadata: Personal Data & Models March 14, 2021 10:00 PM

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 1 — The Artifacts and Spaces
  • Section 3 — Data & Metadata: Personal Data & Models
Table Of Content
  • What Is Personal Data
  • How Is It Extracted, What’s Its Value
  • Information Flow Analysis
  • The Power of Models

The internet is littered with nuggets of information, some made of gold and some worthless. From this mine we can extract meta-information, inferences that can lead to more lucrative valleys.

Two of the most talked about are personal data and models, also known as statistical trends visualizations, big data analysis, and predictive studies. Let’s take a look at what is personal data, where and why people share them, why they’re valuable, and how they can be used with different types of models.
In another part of the series we’ll focus on the actors and consequences.

Personal data is any information that can be used to directly or indirectly identify a person, an attribute currently attached to them, a behavior, a preference or interest, personal history, anything.
These include things such as name, phone number, address, email, schoolbook, credit scores, geolocation, travel logs, interests, information about past purchases, health records, insurance records, online behavior, etc..
We call the person to which the personal data belongs a data subject.

Information gathering is omnipresent, we are deliberately, or not, leaving trails in the digital space. On a daily basis, in modern society, through our regular actions, it is inevitable that our activities will generate data that is collected, which may or may not require our permission or knowledge.
As we’ll see later, the online services and their sub-contractors benefit from this exchange process, relying on the brokerage of big data about their consumers.

The digital world is interwoven with our physical world, we are incentivized to enter information to access services and utilities. Everything is being datafied, anything that can be turned digital will be, including us from birth till death. Are we an information society?
Data collection has been normalized, and we will see the general effects of this in another section. This is justified by the advantages this can convey such as having an optimized service.

There are five places where data gathering happens: casual online browsing and searches, social media, third party marketing, smart widgets, and traditional data repositories.

The biggest collectors are internet-based navigation sites like Google, Yahoo, and Bing. Apart from the data source coming from their search engine services, these companies have a broader business model where they offer tools such as emails, document editors, file storage facilities, and others in which they gather information about users.
These are used for targeted ads. For example, Gmail, a Google owned email service, will direct their marketing campaigns based on the type of mails a user has exchanged.

Likewise, massive amount of data is collected via social networking websites and mobile applications like Facebook, Twitter, LinkedIn, etc..
Users on these platforms either voluntarily communicate information with a public audience by messaging them, or indirectly share their behavioral patterns and preferences by filling a persona — The persona having meta-information stored and used by the platforms and advertising partners.
The data subjects might be lead to believe they own the data they generate through their activities on the platforms, however depending on the legal policy of the services and the legislation in which they reside, it is often not the case.

Third-party marketing providers are good source of consumer data. This includes companies such as KBM, Acxiom, and Equifax which have built consumer databases with information including wage data, occupations, past purchases, transactions, etc.. It is frequent that these third-parties partner or sell their information database to other entities.
In this category we also find credit card companies that might share processing data.

Yet another source of data are all the connected widgets. That include devices such as smart watches, smart doorbells, facial recognition cameras, fingerprint scanners, toys, and others. These could connect to smartphone applications, or directly to the internet, and the data might be forwarded to a third-party by the parent company.

Finally, the classical places where we find data collections are the traditional repositories. These are the ones that have been used since forever by insurance companies and that contain information such as credit scores, vehicle registration records, medical records, and other official and non-official knowledge.

Each piece of information in itself might not be valuable but the aggregation and processing makes it so. It’s quantity over quality, and the more data the more the approximate improvement.

This data confers a huge advantage to companies that use it compared to competitors that don’t. Like a GPS using multiple points to find a spot, the data allows for finely grained personalization of services and data-driven decisions. Companies can understand customers needs and wants, and have better marketing, products, and services.
Some argue that the companies might even know more about you than you know yourself.

Most companies and industries already used similar knowledge base in the past for marketing, as a result they only supplement their existing internal source with the external ones they can buy.
This gives rise to a business model, a new type of economy, in which personal data is commodified, gathered at all costs to be resold later to these companies. We’ll get back to this in the next part.
This type of business is lucrative, there is a lot of money involved. Simply taking a look at names such as Alphabet, Amazon, Apple, Facebook, and Microsoft that have insane profits racking up over $25B each year should give an idea. Or even looking at commercial consumer database owners such as Acxiom that makes sales of $1.13 billions and have customers such as big banks, investment services, department stores, and automakers.

This data is pieced together, shared, aggregated, and monetized, fueling a $227 billion-a-year industry. This occurs every day, as people go about their daily lives, often without their knowledge or permission.

There is so much money involved because the return-on-investment for firms that have embraced it shows that it works. In an ever-moving world, it is always a leverage to know about your consumer target and predict trends.
The data isn’t available to everyone though, and the companies owning them aren’t keen on letting go of their business. Moreover, there are moral, ethical, and legal regulatory concerns that are starting to develop around the topic.

Practically, this gathered data needs to be processed to be used by companies or other entities. That amounts to collecting, structuring, organizing, storing, sharing, and modeling the information.

The digitalization allows not only to deduce direct information from the data (such as personal interests in a topic), but also indirect ones like the flow of information. For example, it can allow knowing which persons are more prone to interact together. How data moves is information in itself.

This can help to know via clear metrics if a message was received by the community or group it was intended to target. The message could then be honed or sent through different more adapted means for the audience. Essentially letting the message sender influence a group without interacting directly with them, while still receiving feedback response.

Seeing how a message propagates can also be used to identify which hubs or communities are more influential than others, if information is more likely to originates from certain bubbles.
We can examine what are the criteria of these messages originating from the influential communities. For example, some studies have noticed that more fringe and extreme communities can influence other ecosystems on the internet.
The flow of ideas between these interconnected networks is likely why the mainstream media now gets some of their news from social media.

Indirectly, the collection of information and the visualization of silos can be used to categorized people belonging to them. We could attribute certain characteristics and qualities to persons frequenting a group.

Epidemiological model such as the SIR spread model can be used to see the propagation of messages through time and space. Similarly, percolation models can be used to see which variety of information can reach a tipping point and spread virally: its qualitative attributes or the ones of the network it’s spreading on.

Another information we can get is whether applying different reproductive strategies for messages work better. Answering questions such as: Does quality or quantity matters? Which works best an R vs K selection strategies? What makes a message good, is it the high-volume, frequent repetition or high quality and low-volume?
Some studies have shown that on the internet it’s a game of volume and brute force and not specifically intent and design to be able to reach a large audience.

Theoretically, all this data can be used to see trends, the spirit of the time, the zeitgeist. Having insights into the architecture and dynamics of the networks and the information that lives on them open many possibilities. The related field of big data, consisting of extracting value out of a humongous amount of data, is booming.

Unfortunately, as with anything on the internet, a considerable quantity of the data generated doesn’t come from real humans but from algorithms. Studies approximate that 40% of the internet consists of non-human users. Consequently, the visualization of flow, the insights, the models, and the refining of messages are partly based on information given by bots.
Thus, a company’s decisions based on such data might be indirectly based on the will of algorithms, hence leading them to deceitful conclusions.

Furthermore, the knowledge and visualization of the flow of information can be abused by actors wanting to spread a narrative.
For instance, they might rely on side channels such as citizen journalists and fake news portals to then have their message be regurgitated by mainstream media. Effectively, they would have traded up the chain by using their insight of the connection between different networks.

Lastly, there are still questions regarding personal data and data gathering in general. Who is responsible when there is a breach? What type of processing is allowed on the data? Is the data subject allowed to update or correct their data? Who is the actual owner of the data? Can the data be transferred freely without authorization from the data subject?
These are all important questions we’ll tackle in the last part of this series when we’ll discuss the solutions to the issues brought by the internet.

This concludes our review of what personal data is and how data on the internet is collected to be used in models. We’ve first seen what personal data consists of. Then we’ve looked at all the places personal data is generated and collected. We’ve seen the normalization of this process in societies. Later, we’ve examined how this data is useful for marketers, and how it creates a lucrative business to gather the data to resell it to them. Next we’ve pondered about different types of models and ways to use the data apart from direct marketing. And lastly, we’ve considered how this model could lead to misleading insights if it is based on algorithms and not human behaviors.

Table Of Content

References








Attributions: W. Blake, Death Door, from: Gates of Paradise 1793

Gokberk Yaltirakli (gkbrk)

Status update, March 2021 March 14, 2021 09:00 PM

It’s been a while since the last Status Update on November 2020 so I thought it would be good to have a catch-up post. Be prepared, this is going to be a long one. After all, I haven’t done one of these in 120 days.

Development environment

There has been one small change to my development environment. I’ve gotten a CLion license. I’m planning to play around with it for a while to see if it improves my C++ workflow.

DSP / Radio

There has been some good progress on signal processing and radio-related projects since the last status update.

Frequency-hopping spread spectrum (FHSS)

I made a modulator for a frequency-hopping signal, and wrote receiver plugins for SDR Sharp and GQRX. The modulator changes the frequency of transmission based on the hash of the current time, and hops frequencies 10 times a second. It is able to be received and demodulated without problems.

It is possible to share the same frequencies between multiple transmitters, as their collisions are not very common. This modulation is good at avoiding narrowband noise, but it provides no processing gain to the signal as there is no correlation done. All I’ve done is transmit a narrowband FM signal with a hopping frequency.

APT image encoder fixes

There has been some improvements to my APT image modulator apt-encoder. Previously, the images created by GIMP needed to be manually edited to get rid of comments. Now the program knows how to ignore comments in image files.

Spectrogram drawing

I wrote a modulator that can transmit image files using the HackRF One. It can draw simple pictures on the spectrogram. The height and width of the transmitted image, which translate to the transmission speed and the bandwidth, can be configured.

I have written an article about it here.

Signal correlation

I started learning how to do signal correlation in order to get processing gain from spread-spectrum signals. I haven’t written a full modulator/demodulator yet, but I have tested locally with Python and things are looking promising.

Kernel / OSDev

There has been some kernel development in February. Below are the main points. As always, the kernel source code can be found in gkbrk/kernel.

Threading improvements

From the first days of multi-processing in the kernel, it was not possible to pass any data when spawning threads. You could pass a function pointer and communicate with the thread using shared memory. Thanks to some changes in the Task class, it is now possible to push and pop arbitrary objects to the stack of the process.

The first thing I did after implementing this is to make the music player fork to another process so it can play in the background while you are dropped back in the command prompt.

Improved PS/2 driver

One of the first drivers we implemented was a polling keyboard reader. We now replaced this with a proper PS/2 driver that handles keyboard and mouse events. Yes, we also support PS/2 mouse now.

A driver for the “mouse integration” that is common in VM emulation software was also implemented, so we can seamlessly use the mouse in the kernel now.

RNG and entropy pool

The only method of generating random numbers in the kernel was a simple LCG PRNG. This made the random number quality bad, and (most importantly) not suitable for anything that needs to be secure.

We now have a system that can collect and extract entropy from kernel events and CPU interrupts. This means anything that depends on external input, such as keyboard and mouse events, and hard drive access patterns, now contribute to a “system entropy pool”.

For now, the only output is the occasional line on the debug console with a random byte. In the future, I am planning to add a sponge function that will collect entropy and produce arbitrary amounts of randomness.

Build system

We were using Ninja as the build system for some time, but recently I switched the ./configure script to emit Makefile syntax again. This was mostly done to integrate better with tooling and to get rid of a build dependency.

We also hash the output file names and put them all in a flat build directory. This makes it easier to clean the project, and allows us to reuse file names without having a nested build folder.

When using the ./mach command to build, all the subcommands still work the same so there are no changes.

Blog comments

I used to have a comment system that used CGI scripts to collect, process, filter and display comments. It required JavaScript for fetching and displaying the comments. At some point, I migrated servers and never bothered to set up comments again.

I decided to fix that, so now we have comments again. And guess what, they even work without JavaScript now. I am using Paul Graham’s spam filtering algorithm, which seems to be doing a great job so far.

Android application template

I wrote a small template for an Android application that displays a web page in a webview while exposing some useful APIs over the JavaScript bridge. It can be compiled and packaged entirely from the command-line, avoiding the Android Studio bloat.

Small utilities

bencode2json

I wrote a small utility called bencode2json to convert bencoded data to JSON. This is useful to me because I end up using bencode a lot when prototyping network protocols, but I don’t feel like rewriting jq for bencode.

Since I wanted to use this utility for external data from the network, I wanted to make sure it wouldn’t be able to do anything bad to my system. To this end, I learned about how to use seccomp to sandbox a Linux process. For better security and for a small challenge, I used the strict mode which only lets the process call read, write, exit and sigreturn. This pretty much eliminates almost all attack vectors.

LZ77 Compression / Decompression utility

In order to understand compression algorithms better, I decided to write a worse version of a common compression algorithm called LZ77.

The version I wrote keeps a 255-byte window in order to use only a single byte for the length values. Due to the very small window size, it is only able to achieve a 90% compression ratio on plaintext.

I am planning to play around with different algorithms and larger window sizes in the future.

Container runtime script

In order have more reproducible build without messing around with build environments and configurations, I wrote a shell script that uses systemd-nspawn to start and interact with temporary Linux containers. I decided to call this script runcontainer.sh, you can find it on my github.

Caius Durling (caius)

Fix Edgerouter DHCP ? entries March 14, 2021 03:55 PM

Occasionally I end up with devices on the local network that don’t emit their hostname over DHCP, so when listing the current leases on the EdgeRouter’s cli, they just appear as “?".1 These usually just irritate me, but occasionally when I’m looking for a machine on the network it means I can’t find it and end up poking the different “?” IPs using nmap or ssh to discover which machines they are.

$ show dhcp leases
IP address Hardware Address Lease expiration Pool Client Name
---------- ---------------- ---------------- ---- -----------
10.0.0.34 a8:1d:16:74:xx:yy 2021/03/14 16:38:27 trusted ?
10.0.0.40 b8:27:eb:c5:xx:yy 2021/03/14 16:36:13 trusted picontrol1
10.0.0.46 a8:1d:16:75:xx:yy 2021/03/14 16:37:48 trusted ?
10.0.0.93 14:f6:d8:53:xx:yy 2021/03/14 16:40:29 trusted ?

The EdgeRouter lets me assign static entries in the DHCP subnet, which solves the problem of knowing which hostnames they are, but also pins those devices to (effectively) static IPs within the subnet which leads to me having to know which IPs are free when I assign them, etc. Avoiding that is why I have DHCP on the local network.2

Provided the EdgeRouter is configured to use dnsmasq to provide DHCP services3, you can lean on the dhcp-host option in the dnsmasq configuration to assign a hostname based on MAC address, without prescribing a specific IP address for the machine. This solves the issue of “?” devices showing up in show dhcp leases, whilst also allowing dynamic IP assignment.

You’ll need to know the MAC address in question, and pick a hostname to be assigned to the machine. You’ll then want to inject these through dnsmasq’s configuration file, which set service dns forwarding options xxx nicely injects into on the EdgeRouter.

$ configure
set service dns forwarding options "dhcp-host=14:f6:d8:53:xx:yy,cb1"
set service dns forwarding options "dhcp-host=a8:1d:16:75:xx:yy,cb3"
set service dns forwarding options "dhcp-host=a8:1d:16:74:xx:yy,cb2"

Then follow the usual compare, commit, verify your DNS/DHCP still works, save dance to apply & persist the changes.

Now when you login to the router and list the current DHCP leases, you’ll see the hostnames available - and you can now lookup the machines in local DNS via their hostname too. 🎉

$ show dhcp leases
IP address Hardware Address Lease expiration Pool Client Name
---------- ---------------- ---------------- ---- -----------
10.0.0.34 a8:1d:16:74:xx:yy 2021/03/14 16:38:27 trusted cb2
10.0.0.40 b8:27:eb:c5:xx:yy 2021/03/14 16:36:13 trusted picontrol1
10.0.0.46 a8:1d:16:75:xx:yy 2021/03/14 16:37:48 trusted cb3
10.0.0.93 14:f6:d8:53:xx:yy 2021/03/14 16:40:29 trusted cb1

  1. On my network currently these are Chromebooks, and Sonos speakers. I’ve also observed native SmartOS Zones behaving like this previously (I think they might have fixed this now.) I believe the device fails to send the current hostname (option 12) in either the DHCPDISCOVER or DHCPREQUEST packets. ↩︎

  2. Also, if I assign static host mappings to a device they vanish entirely from show dhcp leases, which stops me being lazy and checking one place to figure out where a device is. ↩︎

  3. To find out if you’re using dnsmasq for DHCP, check show service dhcp-server use-dnsmasq returns “enable” ↩︎

Andrew Owen (yumaikas)

Learning without Burnout March 14, 2021 12:45 AM

This article is a response to the excellent question by @Nellies.

This lifelong learning thing, is there any talk on how to deal with the infinite amount of stuff there is to learn?

I do ok, but sometimes the todo list does get cumbersome.

Is that part of the whole burnout in tech? No worries, I have broad shoulders, just curious.

Initial caveats

Most of my examples are going to biased towards full stack or frontend web development, since that’s where I have the most familiarity. This is also written to people trying to break into software development, or just want to understand what it’s like to manage this aspect of a software development career. I also gloss over a lot of things, especially cultural factors that make software development harder for those who can’t pass as white males.

Types of knowledge in software development

As mentioned by Nellie, one of the aspects of software development is that software developers are expected to constantly be learning new things. Or, at least, that’s the impression that happens from the outside. There are some mitigating factors, however.

The first is that once you learn the fundamentals for a particular technology stack, moving between similar flavors of that stack is made much easier. Once you understand the fundamentals of HTTP, HTML, CSS and JS, picking up Rails, Django or ASP.NET becomes much easier.

The second is varying types of knowledge have different shelf-lives. The cool new shiny JS framework being promoted by its creator? That could be relevant for only 6 months, or it could be the next React. It can be hard to tell up front, especially when you’re new. Thankfully, most jobs also lag the hotness by a certain amount, not everything asks for the hottest new tech.

But, if you’re learning HTTP status codes, or headers and auth works? That knowledge stays relevant a lot longer, and is part of all web apps. As a bonus, learning more fundamental knowledge gives you an edge when it comes to debugging problems, which is one thing that can help you stand out in your early years. Also, different types of knowledge have different acquisition costs. Basic JS or HTML is relatively easy compared to writing a full stack web app, which is easier than building a reasonably full scripting language, which, in turn, is easier than building a production grade data store like Postgresql or MongoDB.

Another factor is that it’s possible to specialize inside a given stack. So, for instance, with Rails, if you hit an area where your co-workers don’t seem to know a lot, that’s a good place to start a deep dive into how it works. Because then you’ll be able to build a reputation as the person who can figure out tricky things. You don’t have to know the whole stack at that same level of detail. If you’re known as “The ActiveRecord Bender” or “The Database Index Guru” or “The CSS Animations Witch”, people will give you more credit when you ask for help in their area of expertise. This also synergizes with learning fundamentals, and is where fundamentals can help you out.

Tradeoffs of what to learn

There are some trade-offs in play here. The first is that most foundational knowledge doesn’t show up as keywords on a resume, and most companies are poor at assessing it (to my knowledge). Either they assume that you have it if you have relevant keywords, or that you can pick up (or have) foundational as you go, or they pattern match you during the interview against what they expect people to know for the job. And nobody has time to learn everything all at once, no, not even that super impressive open source developer or Indie game development.

Learning on the Job

So, don’t learn everything all at once. You will have to look a lot of things up during your day to day work, even after a decade in the field. But, especially in your early years, do make time to learn things. Jobs for junior candidates will expect you spend a lot of time learning things. If they don’t they shouldn’t have hired you for that level. Any programming job involves learning the companies code base for at least the first 2-6 months you’re on a project, unless they follow a very conventional structure. So take advantage of the junior position and learn your job’s tech stack as effectively as you can.

Be sure to take deep dives into the tech involved at your first position, trying to understand one layer deeper than the one you’re working at. This doesn’t have to happen every ticket, but it should be happening at least once every paycheck or two. Often the lower layers are simpler than you might think, or make understanding the higher levels a lot easier. Understanding the lower levels also gives you a much better nose for when code is fluff vs substance. This guides your debugging, grows your fundamentals, and speeds up future work.

Initial learning time

When you are in your early years, you will need to front-load learning the basics. Unless you have some other skill set that can get you into the tech industry (there are a few, Quality Assurance/Testing comes to mind, as does being an engineer in a more continuous field, like aerospace, materials design, or the like), this is going to have to come out of your own time. Bootcamps are good for getting resume keywords under your belt, though they typically don’t spend much time on fundamentals. Colleges focus a lot more on fundamentals, but might use tech 5-20 year out of date. Either one will require you to figure out how to fill the gaps left by it.

For me, I had the luck to be able to carve out school time in high school (I was home schooled) on Fridays for morning each week to work through Head First C#. At the same time, I had an aspirational project to build a Scientific Calculator GUI app, which was solidly outside what I could pull off with what I started out knowing. For the first year or two, especially before you get a job, I’d recommend setting aside a time box in a similar fashion. If life doesn’t afford that timebox, then you’ll have to be a lot more proactive about making time.

Also, don’t try to code in a void, sit down with a game plan of what you’re going to try to do or learn. If you get to your time, and you don’t have that, take a walk and think about it, or do a chore with the plan in the back of your head. Often problems are solved away from the screen, or in conversations with ducks or teddy bears. Tunnel vision is a dangerous trap in software development

Maintaining a learning pace over time

Once you have your first job (or two), and have established your base knowledge, then the question comes up of how to maintain the learning required to keep up with tech. And this ultimately depends on what you plan do to with your career. If software is a stepping stone that takes you on to the next thing, then you can afford to burn a lot more of your motivation and time, and focus where the market is hot, and try to get 3-5 jobs out of that, setting yourself up as much for the next step as you can.

If you’re wanting to stay in software for the long haul, then I’d recommend a different approach from chasing the hype. Try to find the fun in software development. And, as you learn how to find the fun in building and learning things, be sure to pace yourself. I’ve had 6-week stretches of working hard at learning a new language, ecosystem, or side-project, followed by months of focusing on other things.

You don’t have to learn at 100% every week once you feel comfortable in your ability to build projects of a small-medium size. For web development, this would probably be about the point you’re able to write your own blogging software, for whatever functionality you’re looking for. You should be learning new things on a semi-regular basis. But keep it sustainable. One thing that really helps here is to have a process for documenting what you’ve learned, in some fashion. I’ve biased towards publishing code in the past, because that’s what’s been easiest for me.

Don’t use side projects as your only tool for learning software. I used to think this was the best way to learn, because it was how I had learned a lot of things. It has disadvantages, however. For one, side projects that are actually useful are more time-consuming than something you can throw away, and the kernel of what you’re learning might be discoverable in the shorter exercise. Noah Gibbs has a lot more to say here, and I owe a lot to his writing and companionship for changing how I approach learning in the last year.

But, whether through side projects, blogging, or building a portfolio, I do think that finding a way to publish your learnings is valuable. The nice thing about publishing and shipping smaller things on the side is that you do end up building a signal to employers that you have “passion”. This also allows you to ratchet your perceived learnings more sustainably than always trying to be at 100% learning mode.

High Output times

The main time I think that it’s worth going a highly regular output (like, once a week or more often) on the side is if you are aiming to become a proper software development celebrity. This is not for everyone, and likely doesn’t pay as well as you’d think, unless you spend a lot of time figuring out how to get it to do so, and you’re magically successful with it. There’s also a whole host of Not Writing Code that goes into making money off of it.

Doing this for a season, akin to an art challenge, is a great way to learn a lot in a short time, if you have the time and energy. If you’re looking to make a lot of money on the side, however, freelancing is likely a better bet, once you have the skillset to pull that off. As much as selling your hours for money isn’t the path to higher wealth bands, it’s a much less risky tradeoff compared to product development or trying to build a personal brand.

Final thoughts

Try to find the fun, as well as a sustainable long-term motivation. Look for trade-offs rather than dogma. Use binary search debugging. Listen to your body, and be honest with yourself.

March 12, 2021

Jeremy Morgan (JeremyMorgan)

Should Developers Join Clubhouse? March 12, 2021 12:18 AM

Clubhouse has been all the rage lately online, and people are flocking to it. If you’re a developer or tech person, is it worth signing up for? A total waste of time? Let’s take a look at it. Already on Clubhouse? Join me tomorrow for a chat! My Experience With Clubhouse I was hesitant to join Clubhouse. The premise seems silly. My first response: “So we’re going to volunteer to go on conference calls now?

March 11, 2021

Gustaf Erikson (gerikson)

11,000 dead in Sweden March 11, 2021 02:11 PM

Simple chart showing the number of days between every 1 000 deaths, and the average deaths per day.

Update Monday, 2021-03-01: revised dates based on official stats from FHM.

DateDeaths DaysDeaths/day
2020-03-11 1 0 0.0
2020-04-09 1 000 29 34.4
2020-04-19 2 000 10 100.0
2020-05-02 3 000 13 76.9
2020-05-18 4 000 16 62.5
2020-06-11 5 000 24 41.7
2020-11-01 6 000 143 7.0
2020-11-27 7 000 26 38.5
2020-12-12 8 000 15 66.7
2020-12-23 9 000 11 90.9
2021-01-02 10 000 10 100.0
2021-01-14 11 000 12 83.3
2021-01-27 12 000 13 76.9
2021-03-01 13 000 33 30.3

13,000 dead in Sweden March 11, 2021 02:09 PM

March 09, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Artifacts And Spaces: Social Media And The Democratization Of Speech March 09, 2021 10:00 PM

It is clad in seven robes or skins (tunicas), to which the seven planetary spheres correspond in the macrocosm. The Cabalists connected this ten part structure of the eye with the Sephiroth. The blind spot was a term applied to the high Sephira

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 1 — The Artifacts and Spaces
  • Section 2 — Social Media And the Democratization Of Speech
Table Of Content
  • Efficiency
  • Democratization of Speech and Removal of Gatekeepers
  • Intersection of Social and Media Space
  • Infotainment And Overton Window
  • Anonymity, Automation, And The Role Of Platforms
  • It’s About Us

With the advent of smartphones, social media platforms are on the rise. Let’s step back and try to understand what social media are, not by citing events and instances happening on the different platforms, nor the countless consequences but by describing the characteristics of this new informational channel. We’ll go over the rest later but let’s take some distance for now.

A media is an outlet or tool used to deliver information. Before social media, this term most often referred to mass media communication such as newspaper, cinema, advertising, political speech, printing press, etc..
In contrast, social media are platforms hosted on the internet that facilitate the creation, engagement, and sharing of information between the average person and virtual communities. It relies on simulating aspects of social interactions.
Social media is the mix of media space and social space.

As a media, it stands out for its efficiency. This is characterized by numerous people on the platforms, high spreading speed, massive amount of information available, and the long-lasting lifetime of this information.

According to a Facebook’s statistics, there are 2.8 billion monthly active users on the platform. Pew research also shows that, in 2019 in the USA, Facebook was used by 69% of adults, while YouTube was used by 73% of them. Additionally, Worldwide statistics of the same year (2019) show that people spend on average 2 and a half hours a day on social media. These massive penetration rates are clear indicators of how social media have taken part in the lives of many.
Many of the platforms rely on a network effect to attract and create social hubs — The more the people on a platform, the more others will want to join it. This is the inherent social aspect, that’s why we also call them “social networks”.
Consequently, any information circulating on the popular social media could benefit from exposure that no other medium has.

The exposure is complemented with an instantaneous spread, a fast pass-it on effect. The engagement of users allowing the fast sharing of information. Quickly creating an amplification of the message, letting it travel through social circles that would’ve never been reached before — a wider transmission that breaks distances.

Another fitness criteria is how enduring and suitable the storage media is. The internet has a long-lasting memory, by the time the message is out it will have been copied and stored in multiple places. It will be almost impossible to erase.

These three criteria: exposure, speed, and memory, break previous media standards. However, to exploit this new medium successfully the information sent through it needs to adapt symbiotically to the ecosystem and niche in which it exists. If it is hostile, the information won’t be transmitted.
For example, the message will need to take in consideration the augmented Chinese Whisper effect taking place due to the speed and social aspects.

Another component that needs to be considered in the social media space, is the infobesity, also known as the infoxication phenomenon. The information is long-lasting, as we said, and accumulates over time. The users of the social media are overloaded with them.
The senders of mass messages will have to keep this in mind and craft their messages appropriately to penetrate social media. Indeed, the people on the platforms will never be able to consume all that is presented to them, and have to be picky with what they choose to process.

These are all indicators of the scale that the internet platforms, particularly social platforms, have and their potential use.
Most importantly, they have dethroned traditional mainstream media gatekeepers and allowed the common people to reach a wide audience, bypassing filters and safeguards.

In the past, there were two main ways to get information: through peers or through traditional media.
With peers, we’d learn about the world events through their stories, rumors, and anecdotes.
Traditional media, as the only other source, were subject to different types of manipulation. To protect the public and stand as gatekeepers of truth they erected journalism standards, free press, fact checking, and other types of regulation present in multiple countries.
At least that’s the theory. It remains ambiguous if traditional media are still influenced, controlled, and driven by ideological stances or corporate incentives.

Nowadays, we are presented with an alternative: the ordinary person as a media producer and filter. On social media, we aren’t subject to the standards and safeguards that mainstream media have.
People democratically filter and promote the news, ideas, and information that they consider important. The majority decides what is relevant and what isn’t. The phenomenon is similar to conversations with friends being converted from one-to-one into one-to-many, an amplification of decentralized voices. These voices are driven by an appeal to emotions and popularity, and not by reason, deliberations, qualifications, backgrounds, incentives, history, or others.
These are the messages that pierce through.

It is now frequent that mainstream media accentuate the legitimacy and credibility of social media by regurgitating their news from the different platforms. This can be interpreted as a confirmation and encouragement for participating on them.

The generation and distribution of media has been made effortless. Anyone can easily take part in it. This is the will of the actors owning the platforms, which we’ll see in another section.
The availability of digital equipment makes it a breeze to edit, write, capture, record, and film content.
The public is socially rewarded for their engagement, be it sharing, commenting, producing, or distributing. The usual social rules are in places such as wishing for fame or conforming and belonging to a community. We’ll also discuss in depth the psychological aspects in another section.
In sum, the public discourse on social media is a new place where the creation of narratives about events happens, and where they are discovered.

The digital platforms are also social places used for cultural, personal, and intimate relationships. This combines senses of trust and familiarity along with individualization and personality.

People maintain profiles on social media, facades of their personality that they mold with the aim of displaying it to an audience. The platforms normally allow for the customization and individualization of what can be interacted with, is recommended, and the style and types of messages received. It goes both ways: the person forms the virtual persona they want, and the digital platform assists in surrounding that persona with an environment that will foster it within boundaries.
Practically, that means the social media user will have personalized news and media, and interact with others, according to how they portray themselves.

Furthermore, the individualization, along with the fact that other users on the platform might already have social bonds with the person outside the digital space, induce feelings of trust and familiarity. However, this is complemented with an inability to establish new organic social bonds through social media — we aren’t able to connect deeply. We feel closer, yet further.
Familiarity is also felt through the repetition of the customized messages targeted for the particular persona.

As with any type of social place where communication happens, conversations will touch topics that conflict across cultures.
Just like mainstream media newspapers could be leaning toward different points of view, now everyone on social media reiterate their ideologies. An outer expression and clash of the inner confused and unaddressed stances of our societies and cultures.

Unquestionably, the most favored topic, the one driving the most engagement, is politics. Social media are the hubs for political discussions. According to Pew Research Center, 39% of adults in the USA used social media solely for political purposes. Interacting with media that has political meaning, which translates into “liking”, sharing, reposting, discussing, creating political posts.

Interestingly, social media are positioned between socialization, entertainment, and information consumption. For a vast majority of users political interaction has turned into infotainment, information sharing as entertainment. A caricaturization of politics, reducing political acts to quick and easy bona fide participation on social platforms — What is called hashtag activism or slacktivism.

Along with the overload of information and the democratic filter, users will indirectly encounter all types of conflicting ideas while browsing to entertain themselves. These having spread exactly because they interpolated the engagement of so many others.
The types of information that usually stand out are the ones that challenge our ideologies and culture, as we said. These types of messages can pierce through.

Some theory implies that because divergent views exists at the same time on social media, this latter is akin to low-context societies.
High-context mode are the communications that happen with an inner-group, including the subtle codes, traditions, and meanings. While low-context mode are the communications that are straight forward, reducing ambiguity, flat, and global.

This can be what some call an information society, a society in which creating, manipulating, and sharing of information is central. The people being part of this society would then be digital citizens.

We are having an incredibly difficult time adapting to this new medium, now that everyone has an equal voice.
We still refuge ourselves in different cocoons that collide in parody-like confrontations with one another. Again, a reflection of our own cultural weaknesses, our cultural insecurities.

These bubbles then can only be pierced through by more extreme content and ideas that will travel further. That means we are moving the Overton Window — the ideas tolerated in public discourse. Fringe ideas on social media are the ones that are heard the most and might be accepted as normal. The medium having voted democratically that this was what the majority voted as relevant.

Obviously, the votes are not all done by real human beings, the platforms have their share of responsibilities and reasons to promote certain types of discourse that drive engagement, as we’ll see soon.
This reminds us of the nudge theory and persuasive tech we’ve learned about in the previous section and how it could only be achieved when an actor has full control of the environment; This is the case with most social media. They can, and do, deliberately create digital environments that users feel fulfill their basic human drives.

Hiding the actors affect the messages greatly, this is done in two ways: anonymity and automation.
The use of persona that aren’t attached to real individuals let people express themselves without restraints. This attracts bitter views and clashes, as people can now break the accepted cultural codes.
These anonymized profiles can then be automated by algorithms, what we call bots, and exchange with other social media users to promote a message. Consequentially, cheating the democratization of speech and artificially creating a public opinion by giving importance to selected ideas.
We referred to these autonomous agents as computational propaganda in the previous section. Statistics show that around 40% of the traffic on social media is composed of these autonomous agents. We’ll explore later the incentives of the actors.

Let’s finally mention that there are different types of social media platforms. Some have specific rules, some are only used to share a single type of media, some are centralized, some are decentralized, some are anonymous, some require identification, some are more fringe than others, etc..
Social media have all the essential components we’ve described, but they are mainly guided by the platforms themselves. They are new instruments, new tools in our hands.

That is why some persons start to see social media as public utilities, even though they are far from it. A public utility being “an infrastructural necessity for the general public where the supply conditions are such that the public may not be provided with a reasonable service at reasonable prices because of monopoly in the area.” For that, a governing body will need to be erected and have monopoly over the social media in place and justify them being a necessity.

Overall, the medium reveals more about the people than about itself. It is how we use it that matters and the confusion we feel could be a manifestation of our own confusion. Or is it?

This concludes our review of what social media are as new communication channels. We’ve first seen its efficiency characteristics: huge reach, high speed, reliability. Then we’ve contemplated how it has broken the walls of the mainstream media gatekeepers and putting the keys of the castle in the hands of the common person. Later we’ve seen how it intertwines a social space with a media space, including al our cultural issues and how we want constant entertainment. Next, we’ve described how hard it is for the information to pierce through and how that increases extremeness in the messages to actually reach us. And finally, we’ve discussed anonymity and automation as ways to cheat the pseudo-democratic process that is supposed to happen on social media to elect what is important and what isn’t.

Table Of Content

References








Attributions: Gregor Reisch, Pretiosa Margarita, Freiburg, 1503; Basle, 1508

March 08, 2021

eta (eta)

Why asynchronous Rust doesn't work March 08, 2021 12:00 AM

In 2017, I said that “asynchronous Rust programming is a disaster and a mess”. In 2021 a lot more of the Rust ecosystem has become asynchronous – such that it might be appropriate to just say that Rust programming is now a disaster and a mess. As someone who used to really love Rust, this makes me quite sad.

I’ve had a think about this, and I’m going to attempt to explain how we got here. Many people have explained the problems with asynchronous programming – the famous what colour is your function essay, for example.1 However, I think there are a number of things specific to the design of Rust that make asynchronous Rust particularly messy, on top of the problems inherent to doing any sort of asynchronous programming.

In particular, I actually think the design of Rust is almost fundamentally incompatible with a lot of asynchronous paradigms. It’s not that the people designing async were incompetent or bad at their jobs – they actually did a surprisingly good job given the circumstances!2 I just don’t think it was ever going to work out cleanly – and to see why, you’re going to have to read a somewhat long blog post!

A study in async

I’d like to make a simple function that does some work in the background, and lets us know when it’s done by running another function with the results of said background work.

use std::thread;

/// Does some strenuous work "asynchronously", and calls `func` with the
/// result of the work when done.
fn do_work_and_then(func: fn(i32)) {
    thread::spawn(move || {
        // Figuring out the meaning of life...
        thread::sleep_ms(1000); // gee, this takes time to do...
        // ah, that's it!
        let result: i32 = 42;
        // let's call the `func` and tell it the good news...
        func(result)
    });
}

There’s this idea called “first-class functions” which says you can pass around functions as if they were objects. This would be great to have in Rust, right?

See that func: fn(i32)? fn(i32) is the type of a function that takes in one singular i32 and returns nothing. Thanks to first-class functions, I can pass a function to do_work_and_then specifying what should happen next after I’m done with my work – like this:

fn main() {
    do_work_and_then(|meaning_of_life| {
        println!("oh man, I found it: {}", meaning_of_life);
    });
    // do other stuff
    thread::sleep_ms(2000);
}

Because do_work_and_then is asynchronous, it returns immediately and does its thing in the background, so the control flow of main isn’t disrupted. I could do some other form of work, which would be nice (but here I just wait for 2 seconds, because there’s nothing better to do). Meanwhile, when we do figure out the meaning of life, it gets printed out. Indeed, if you run this program, you get:

oh man, I found it: 42

This is really exciting; we could build whole web servers and network stuff and whatever out of this! Let’s try a more advanced example: I have a database I want to store the meaning of life in when I find it, and then I can run a web server in the foreground that enables people to get it once I’m done (and returns some error if I’m not done yet).

struct Database {
    data: Vec<i32>
}
impl Database {
    fn store(&mut self, data: i32) {
        self.data.push(data);
    }
}

fn main() {
    let mut db = Database { data: vec![] };
    do_work_and_then(|meaning_of_life| {
        println!("oh man, I found it: {}", meaning_of_life);
        db.store(meaning_of_life);
    });
    // I'd read from `db` here if I really were making a web server.
    // But that's beside the point, so I'm not going to.
    // (also `db` would have to be wrapped in an `Arc<Mutex<T>>`)
    thread::sleep_ms(2000);
}

Let’s run this…oh.

error[E0308]: mismatched types
  --> src/main.rs:27:22
   |
27 |       do_work_and_then(|meaning_of_life| {
   |  ______________________^
28 | |         println!("oh man, I found it: {}", meaning_of_life);
29 | |         db.store(meaning_of_life);
30 | |     });
   | |_____^ expected fn pointer, found closure
   |
   = note: expected fn pointer `fn(i32)`
                 found closure `[closure@src/main.rs:27:22: 30:6]`

I see.

Hang on a minute…

So, this is actually quite complicated. Before, the function we passed to do_work_and_then was pure: it didn’t have any associated data, so you could just pass it around as a function pointer (fn(i32)) and all was grand. However, this new function in that last example is a closure: a function object with a bit of data (a &mut Database) tacked onto it.

Closures are kind of magic. We can’t actually name their type – as seen above, the Rust compiler called it a [closure@src/main.rs:27:22: 30:6], but we can’t actually write that in valid Rust code. If you were to write it out explicitly, a closure would look something like this:

struct Closure<'a> {
    data: &'a mut Database,
    func: fn(i32, &mut Database)
}

impl<'a> Closure<'a> {
    fn call(&mut self, arg: i32) {
        (self.func)(arg, self.data)
    }
}

There are a number of things to unpack here.


An aside on naming types

Being able to name types in Rust is quite important. With a regular old type, like u8, life is easy. I can write a function fn add_one(in: u8) -> u8 that takes one and returns one without any hassle.

If you can’t actually name a type, working with it becomes somewhat cumbersome. What you end up having to do instead is refer to it using generics – for example, closures’ types can’t be named directly, but since they implement one of the Fn family of traits, I can write functions like:

fn closure_accepting_function<F>(func: F)
where
    F: Fn(i32), // <-- look!
{
    /* do stuff */
}

If I want to store them in a struct or something, I’ll also need to do this dance with the where clause every time they’re used. This is annoying and makes things harder for me, but it’s still vaguely workable. For now.

msql-srv example code

[image: from the msql-srv crate, showing an example of many where clauses as a result of using closures]


An aside on ‘radioactive’ types

The way Rust is designed tends to encourage certain patterns while discouraging others. Because of ownership and lifetimes, having pieces of data that hold references to other pieces of data becomes a bit of a problem. If my type has a & or a &mut reference to something, Rust makes me ensure that:

  • the something in question outlives my type; you can’t go and drop the thing I’m referring to if I still have a reference to it, otherwise my reference will become invalid
  • the something in question doesn’t move while I have the reference
  • my reference to the something doesn’t conflict with other references to the something (e.g. I can’t have my & reference if something else has a &mut reference)

So types with references in them are almost ‘radioactive’; you can keep them around for a bit (e.g. inside one particular function), but attempting to make them long-lived is usually a bit of an issue (requiring advanced tricks such as the Pin<T> type which didn’t even exist until a few Rust versions ago). Generally Rust doesn’t really like it when you use radioactive types for too long – they make the borrow checker uneasy, because you’re borrowing something for an extended period of time.

borrowing semantics visualization

[image: from the RustViz paper, showing borrowing semantics]


Closures can be pretty radioactive. Look at the thing we just wrote out: it has a &'a mut Database reference in it! That means while we’re passing our Closure object around, we have to be mindful of the three rules (outlives, doesn’t move, no conflicting) – which makes things pretty hard. I can’t just hand off the Closure to another function (for example, the do_work_and_then function), because then I have to make all of those rules work, and that’s not necessarily easy all the time.

(Not all closures are radioactive: if you make them move closures, they’ll take everything by value instead, and create closure objects that own data instead of having radioactive references to data. Slightly more of a pain to deal with, but you lose the blue radiation glow the things give out when you look at them.)

Also, remember what I said about being able to name types? We’re not actually dealing with a nice, written-out Closure object here; we’re dealing with something the compiler generated for us that we can’t name, which is annoying. I also lied when I said that it was as simple as making all of your functions take F, where F: Fn(i32) or something – there are actually three different Fn-style traits, Fn, FnMut, and FnOnce. Do you know the difference between them?

So. A closure is this magical, un-nameable type that the compiler makes for us whenever we use || {...} syntax, which implements one of three traits (and it’s not immediately obvious which), and it also might be radioactive. Try and use one of these, and the Rust compiler is probably going to be watching you very carefully.


The thing I really want to try and get across here is that Rust is not a language where first-class functions are ergonomic. It’s a lot easier to make some data (a struct) with some functions attached (methods) than it is to make some functions with some data attached (closures).

Trying to use ordinary structs is downright easy:

  • they’re explicitly written out by the programmer with no funky business
  • you choose what traits and methods to implement on them and how to set them out / implement them
  • the struct can actually be referred to by other parts of the code by its type

Trying to use closures is hard:

  • the compiler does some magic to make a closure type for you
  • it implements some obscure Fn trait (and it’s not immediately obvious which)
  • it might be radioactive (or force you to use move and maybe insert a bunch of clone() calls)3
  • you can’t actually name their type anywhere or do things like return them from a function

Importantly, the restrictions applied to using closures infect types that contain them – if you’re writing a type that contains a closure, you’ll have to make it generic over some Fn-trait-implementing type parameter, and it’s going to be impossible for people to name your type as a result.

(Other languages, like Haskell, flip this upside down: functions are everywhere, you can pass them around with reckless abandon, etc. Of course, these other languages usually have garbage collection to make it all work…)


Bearing this in mind, it is really quite hard to make a lot of asynchronous paradigms (like async/await) work well in Rust. As the what colour is your function post says, async/await (as well as things like promises, callbacks, and futures) are really a big abstraction over continuation-passing style – an idea closely related to the Scheme programming language. Basically, the idea is you take your normal, garden-variety function and smear it out into a bunch of closures. (Well, not quite. You can read the blue links for more; I’m not going to explain CPS here for the sake of brevity.)

Hopefully by now you can see that making a bunch of closures is really not going to be a good idea (!)

*wibbly wobbly scene transition*

And then fast forward a few years and you have an entire language ecosystem built on top of the idea of making these Future objects that actually have a load of closures inside4, and all of the problems listed above (hard to name, can contain references which make them radioactive, usually require using a generic where clause, etc) apply to them because of how “infectious” closures are.

The language people have actually been hard at work to solve some (some!) of these problems by introducing features like impl Trait and async fn that make dealing with these not immediately totally terrible, but trying to use other language features (like traits) soon makes it clear that the problems aren’t really gone; just hidden.

Oh, and all the problems from what colour is your function are still there too, by the way – on top of the Rust-specific ones.

Beginner (and experienced) Rust programmers look at the state of the world as it is and try and build things on top of these shaky abstractions, and end up running into obscure compiler errors, and using hacks like the async_trait crate to glue things together, and end up with projects that depend on like 3 different versions of tokio and futures (perhaps some async-std in there if you’re feeling spicy) because people have differing opinions on how to try and avoid the fundamentally unavoidable problems, and it’s all a bit frustrating, and ultimately, all a bit sad.


Did it really have to end this way? Was spinning up a bunch of OS threads not an acceptable solution for the majority of situations? Could we have explored solutions more like Go, where a language-provided runtime makes blocking more of an acceptable thing to do?

Maybe we could just have kept Rust as it was circa 2016, and let the crazy non-blocking folks5 write hand-crafted epoll() loops like they do in C++. I honestly don’t know, and think it’s a difficult problem to solve.

But as far as my money goes, I’m finding it difficult to justify starting new projects in Rust when the ecosystem is like this. And, as I said at the start, that makes me kinda sad, because I do actually like Rust.


(Common Lisp is pretty nice, though. We have crazy macros and parentheses and a language ecosystem that is older than I am and isn’t showing any signs of changing…)


  1. This is really recommended reading if you aren’t already familiar with it (as you’ll soon see) 

  2. Seriously – when I put out the last blog post, the actual async core team members commented saying how much they appreciated the feedback, and then they actually went and made futures 1.0 better as a result. Kudos! 

  3. You might be thinking “well, why don’t you just only use move closures then?” – but that’s beside the point; it’s often a lot harder to do so, because now you might have to wrap your data in an Arc or something, by which point the ergonomic gains of using the closure are outweighed by the borrow checker-induced pain. 

  4. You can actually manually implement Future on a regular old struct. If you do this, things suddenly become a lot simpler, but also you can’t easily perform more async operations inside that struct’s methods. 

  5. (sorry, I mean, the esteemed companies that deign to use Rust for their low-latency production services) 

March 07, 2021

Ponylang (SeanTAllen)

Last Week in Pony - March 7, 2021 March 07, 2021 11:49 PM

We have new releases of Ponyc and Corral.

Derek Jones (derek-jones)

Code bureaucracy can reduce the demand for cognitive resources March 07, 2021 09:46 PM

A few weeks ago I discussed why I thought that research code was likely to remain a tangled mess of spaghetti code.

Everybody’s writing, independent of work-place, starts out as a tangled mess of spaghetti code; some people learn to write code in a less cognitively demanding style, and others stick with stream-of-conscious writing.

Why is writing a tangled mess of spaghetti code (sometimes) not cost-effective, and what are the benefits in making a personal investment in learning to write code in another style?

Perhaps the defining characteristic of a tangled mess of spaghetti code is that everything appears to depend on everything else, consequently: working out the impact of a change to some sequence of code requires an understanding of all the other code (to find out what really does depend on what).

When first starting to learn to program, the people who can hold the necessary information on increasing amounts of code in their head are the ones who manage to create running (of sorts) programs; they have the ‘knack’.

The limiting factor for an individual’s software development is the amount of code they can fit in their head, while going about their daily activities. The metric ‘code that can be fitted in a person’s head’ is an easy concept to grasp, but its definition in terms of the cognitive capacity to store, combine and analyse information in long term memory and the episodic memory of earlier work is difficult to pin down. The reason people live a monks existence when single-handedly writing 30-100 KLOC spaghetti programs (the C preprocessor Richard Stallman wrote for gcc is a good example), is that they have to shut out all other calls on their cognitive resources.

Given time, and the opportunity for some trial and error, a newbie programmer who does not shut their non-coding life down can create, say, a 1,000+ LOC program. Things work well enough, what is the problem?

The problems start when the author stops working on the code for long enough for them to forget important dependencies; making changes to the code now causes things to mysteriously stop working. Our not so newbie programmer now has to go through the frustrating and ego-denting experience of reacquainting themselves with how the code fits together.

There are ways of organizing code such that less cognitive resources are needed to work on it, compared to a tangled mess of spaghetti code. Every professional developer has a view on how best to organize code, what they all have in common is a lack of evidence for their performance relative to other possibilities.

Code bureaucracy does not sound like something that anybody would want to add to their program, but it succinctly describes the underlying principle of all the effective organizational techniques for code.

Bureaucracy compartmentalizes code and arranges the compartments into some form of hierarchy. The hoped-for benefit of this bureaucracy is a reduction in the cognitive resources needed to work on the code. Compartmentalization can significantly reduce the amount of a program’s code that a developer needs to keep in their head, when working on some functionality. It is possible for code to be compartmentalized in a way that requires even more cognitive resources to implement some functionality than without the bureaucracy. Figuring out the appropriate bureaucracy is a skill that comes with practice and knowledge of the application domain.

Once a newbie programmer is up and running (i.e., creating programs that work well enough), they often view the code bureaucracy approach as something that does not apply to them (and if they rarely write code, it might not apply to them). Stream of conscious coding works for them, why change?

I have seen people switch to using code bureaucracy for two reasons:

  • peer pressure. They join a group of developers who develop using some form of code bureaucracy, and their boss tells them that this is the way they have to work. In this case there is the added benefit of being able to discuss things with others,
  • multiple experiences of the costs of failure. The costs may come from the failure to scale a program beyond some amount of code, or having to keep investing in learning how previously written programs work.

Code bureaucracy has many layers. At the bottom there is splitting code up into functions/methods, then at the next layer related functions are collected together into files/classes, then the layers become less generally agreed upon (different directories are often involved).

One of the benefits of bureaucracy, from the management perspective, is interchangeability of people. Why would somebody make an investment in code bureaucracy if they were not the one likely to reap the benefit?

A claimed benefit of code bureaucracy is ease of wholesale replacement of one compartment by a new one. My experience, along with the little data I have seen, suggests that major replacement is rare, i.e., this is not a commonly accrued benefit.

Another claimed benefit of code bureaucracy is that it makes programs easier to test. What does ‘easier to test’ mean? I have seen reliable programs built from spaghetti code, and unreliable programs packed with code bureaucracy. A more accurate claim is that it can be unexpectedly costly to test programs built from spaghetti code after they have been changed (because of the greater likelihood of the changes having unexpected consequences). A surprising number of programs built from spaghetti code continue to be used in unmodified form for years, because nobody dare risk the cost of checking that they continue to work as expected after a modification

March 06, 2021

eta (eta)

Getting PIV-based SSH working on a YubiKey March 06, 2021 12:00 AM

I bought a YubiKey 5C Nano recently. These devices are great – I’ve built a lot of my (metaphorical) empire on top of them, seeing as they’re capable of acting as an SSH agent (store your SSH keys on them, securely!), an OpenPGP smartcard (do encryption and decryption on the key!), FIDO U2F ‘security keys’ (use them as a 2-factor authentication method!), and probably more.

Getting the thing to work as an SSH agent was, however, not the easiest thing I’ve ever done. There are multiple options here – you can use the OpenPGP applet and then configure GnuPG to work as an SSH agent, but that’s a brittle solution in my experience (gpg-agent is quite flaky, and often requires restarting when it forgets about the YubiKey). Instead, I wanted to see whether I could use the YubiKey’s PIV (Personal Identity Verification) applet to get this working.

Procedure

For the SSH agent part, we’re going to use Filippo Valsorda’s yubikey-agent, so you’ll want to have that installed. In my testing, yubikey-agent’s built-in yubikey-agent -setup command errored out, so we’ll configure the PIV applet of the YubiKey manually. (This’ll also leave the door open for you to do other things with the PIV applet later if you like.)

You’ll also need Yubico’s own yubikey-manager (the ykman cli tool) installed.

Okay, here goes:

  1. Ensure pcscd (the PC/SC smartcard daemon) is installed and running. This might already have been done for you by your Linux distribution but, if not:
    • $ sudo systemctl enable --now pcscd
    • (This is safe to run if it’s already been set up; the command will just do nothing).
  2. This exercise assumes a fresh YubiKey (i.e. one where you haven’t touched the PIV applet yet). If that’s not the case, and you want to erase all of the PIV data and start afresh (losing all data encrypted with the PIV keys!), use the ykman piv reset command.
  3. First, change the PIN from the default one (123456).
    • $ ykman piv change-pin
    • As the command’s help says: The PIN must be between 6 and 8 characters long, and supports any type of alphanumeric characters. For cross-platform compatibility, numeric digits are recommended.
  4. Then, change the PUK (‘personal unblocking key’). This is used to reset the PIN if you ever forget it.
    • $ ykman piv change-puk
    • The PUK has the same entry requirements as the PIN (i.e. also 6-8 ASCII characters).
    • It might be prudent to generate a random PUK and keep it safe (e.g. by writing it down and locking the paper away). If you lose both the PIN and PUK, you will need to reset the PIV applet, losing all data encrypted with the PIV keys (and SSH access to hosts you don’t otherwise have access to).
  5. The YubiKey PIV applet by default has a well-known management key used to make changes to the PIV keys (etc.). It’s best practice to change this to something else. We’ll use the option to generate a random one, store it in the YubiKey, and secure it with the PIN.
    • $ ykman piv change-management-key -pt
    • -p: Store new management key on your YubiKey, protected by PIN. A random key will be used if no key is provided.
    • -t: Require touch on YubiKey when prompted for management key.
    • If you’re more paranoid than me, you can use the -g option instead, which will generate a key for you to note down and give back later. However, the extent to which you can cause damage with a management key is limited (you can delete and regenerate keys, but not decrypt data or anything), so this is arguably not worth the hassle – especially since forgetting the management key means you’d have to reset the PIV applet to make changes.
  6. Generate a public/private keypair on the key, using the 9a (“PIV Authentication”) key slot. We’ll use ECC (elliptic-curve cryptography) because it’s fast, secure, and has vastly smaller key sizes, and configure the key to always require touches and require the PIN on first use. You might want to use different settings here.
    • $ ykman piv generate-key -a ECCP256 --touch-policy ALWAYS --pin-policy ONCE 9a ./yubikey-public.pem
    • Note: Some outdated SSH servers/implementations only support RSA keys. If this applies to you, leave off the -a option to use RSA 2048 instead.
    • See ykman piv generate-key -h for a full description of all available options.
  7. Use the newly generated key to make a self-signed PKCS#11 certificate to act as our SSH identity.
    • $ ykman piv generate-certificate -s 'my-yubikey-ssh' -d 365 9a ./yubikey-public.pem
    • Modify the -s parameter to include a human-readable description of the key or the machine the key is installed in.
    • Modify the -d parameter to set how many days the key will be valid for. The value above specifices 1 year.
    • You can get rid of yubikey-public.pem after this step.
  8. Enable yubikey-agent.
    • $ systemctl --user enable --now yubikey-agent
    • Note: You might need to kill any running instances of gpg-agent if you had that running (and it decided to try and use your YubiKey), and potentially restart pcscd after doing so.
    • Note: Check that yubikey-agent started properly with systemctl --user status yubikey-agent.
      • On my machine, it failed to start because of namespacing issues.
      • If this happens, edit the unit file with systemctl --user edit --full yubikey-agent, and remove all lines under [Service] apart from ExecStart and ExecReload.
  9. Update SSH_AUTH_SOCK to point to the running instance of yubikey-agent.
    • $ export SSH_AUTH_SOCK=/run/user/1000/yubikey-agent/yubikey-agent.sock
    • The above command assumes bash. Other shells may vary.
    • If your user ID isn’t 1000, you’ll need to change the above command (you can find the right path by ps aux | grep yubikey-agent).
    • Note: You’ll want to put the above command in .bashrc or similar.
  10. Get the key fingerprint for your newly generated SSH identity.
    • $ ssh-add -L
  11. Copy the key fingerprint to your remote host(s), and put it in ~/.ssh/authorized_keys.
  12. You should now be able to SSH using the YubiKey!
    • $ ssh <host>
    • The first time you try this, it should pop up a window asking for the PIN, after which you’ll need to touch the flashing YubiKey.
    • Subsequent attempts will only require a touch.
    • If it didn’t work, check the notes under step 8. In particular, using OpenPGP and SSH simultaneously with the YubiKey usually requires you to kill either pcscd or gpg-agent, depending on what you want to do.
  13. Ensure you have a backup plan in place for when this YubiKey fails, or is lost.
    • The key we generated is local to the YubiKey, and cannot be exported. It can very easily be wiped by anyone running ykman piv reset (which requires no authentication!), or you could lose the key.
    • Make sure you have another way of getting into all hosts you use this YubiKey with (encrypted regular SSH key, out-of-band console, etc.).
  14. You’re done!

Hopefully that worked. If it didn’t, well, there’s a comments thing below if you can be bothered, I guess?

March 05, 2021

Sevan Janiyan (sevan)

LFS, round #5 March 05, 2021 05:35 PM

Up to this point I’ve been working with a chroot to build OS images from loop back mounted flat file which is then converted to the vmdk format for testing with virtualbox. I created packages for bpftrace and BCC, BCC was fairly trivial and the availability of a single archive which includes submodules as bcc-src-with-submodule.tar.gz …

March 04, 2021

Frederic Cambus (fcambus)

OpenBSD/loongson on the Lemote Fuloong March 04, 2021 11:10 PM

In my article about running OpenBSD/loongson on the Lemote Yeeloong back in 2016, I mentioned looking for a Fuloong. All hope seemed lost until the Summer of 2017, when a fellow OpenBSD developer was contacted by a generous user (Thanks again, Lars!) offering to donate two Lemote Fuloong machines, and I was lucky enough to get one of those units.

This machine uses the same CPU as the Yeeloong, a Loongson 2F which is a single-core MIPS-III 64-bit processor running at 800/900 MHz.

As hinted in my previous article, contrarily to the Yeeloong, the Fuloong is less strict with the type of RAM it accepts, and my device is happily running with a Kingston 2GB DDR2 SO-DIMM module (ASU256X64D2S800C6), replacing the original 512MB module.

Here is the result of a quick md5 -t benchmark:

MD5 time trial.  Processing 10000 10000-byte blocks...
Digest = 52e5f9c9e6f656f3e1800dfa5579d089
Time   = 1.726563 seconds
Speed  = 57918535.263411 bytes/second

For the record, LibreSSL speed benchmark results are available here.

System message buffer (dmesg output):

Copyright (c) 1982, 1986, 1989, 1991, 1993
	The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2021 OpenBSD. All rights reserved.  https://www.OpenBSD.org

OpenBSD 6.9-beta (GENERIC) #74: Fri Feb 26 08:02:25 MST 2021
    deraadt@loongson.openbsd.org:/usr/src/sys/arch/loongson/compile/GENERIC
real mem = 2147483648 (2048MB)
avail mem = 2116452352 (2018MB)
random: boothowto does not indicate good seed
mainbus0 at root: Lemote Fuloong
cpu0 at mainbus0: STC Loongson2F CPU 797 MHz, STC Loongson2F FPU
cpu0: cache L1-I 64KB D 64KB 4 way, L2 512KB 4 way
bonito0 at mainbus0: memory and PCI-X controller, rev 1
pci0 at bonito0 bus 0
re0 at pci0 dev 6 function 0 "Realtek 8169" rev 0x10: RTL8169/8110SCd (0x1800), irq 4, address 00:23:9e:00:0f:71
rgephy0 at re0 phy 7: RTL8169S/8110S/8211 PHY, rev. 2
sisfb0 at pci0 dev 8 function 0 "SiS 315 Pro VGA" rev 0x00: 640x400, 8bpp
wsdisplay0 at sisfb0 mux 1: console (std, vt100 emulation)
glxpcib0 at pci0 dev 14 function 0 "AMD CS5536 ISA" rev 0x03: rev 3, 32-bit 3579545Hz timer, watchdog, gpio, i2c
isa0 at glxpcib0
com0 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com1 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
mcclock0 at isa0 port 0x70/2: mc146818 or compatible
gpio1 at glxpcib0: 32 pins
iic at glxpcib0 not configured
glxclk0 at glxpcib0: clock, prof
pciide0 at pci0 dev 14 function 2 "AMD CS5536 IDE" rev 0x01: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <WDC WD1600BEVS-00VAT0>
wd0: 16-sector PIO, LBA48, 152627MB, 312581808 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
auglx0 at pci0 dev 14 function 3 "AMD CS5536 Audio" rev 0x01: isa irq 9, CS5536 AC97
ac97: codec id 0x414c4760 (Avance Logic ALC655 rev 0)
audio0 at auglx0
ohci0 at pci0 dev 14 function 4 "AMD CS5536 USB" rev 0x02: isa irq 11, version 1.0, legacy support
ehci0 at pci0 dev 14 function 5 "AMD CS5536 USB" rev 0x02: isa irq 11
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00 addr 1
apm0 at mainbus0
vscsi0 at root
scsibus0 at vscsi0: 256 targets
softraid0 at root
scsibus1 at softraid0: 256 targets
pmon bootpath: bootduid=53153d3cd8ddc482
root on wd0a (53153d3cd8ddc482.a) swap on wd0b dump on wd0b

PCI device data:

# pcidump
Domain /dev/pci0:
 0:6:0: Realtek 8169
 0:8:0: SiS 315 Pro VGA
 0:14:0: AMD CS5536 ISA
 0:14:2: AMD CS5536 IDE
 0:14:3: AMD CS5536 Audio
 0:14:4: AMD CS5536 USB
 0:14:5: AMD CS5536 USB

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — The Artifacts And Spaces: Defining and Understanding Propaganda, Influence, And Persuasion March 04, 2021 10:00 PM

Descartes compared the creation of pictures of memory in the brain with the traces left by needles in fabric

  • Internet: Medium For Communication, Medium For Narrative Control
  • Part 1 — The Artifacts and Spaces
  • Section 1 — Defining and Understanding Propaganda, Influence, And Persuasion
Table Of Content
  • Communication
  • Influence
  • Persuasion
  • Propaganda
  • Types Of Propaganda
  • Administering
  • Reinforcement, Anchor, Hook, Imagery
  • Ethics And Issues Of Identification

Our journey begins with three terms: propaganda, influence, and persuasion. To most, they sound like synonyms but each convey a different concept. Like any form of communication the internet can be used as a channel for them. What does communication consist of?

The very broad domain of communication studies — with multiple sub-disciplines such as journalism, film critic, public relation, and political science — often use a simple reductionist model called the Shannon–Weaver model of communication.

Shannon-Weaver's Model Of Communication Diagram

The model has 7 different concepts: Information source, Encoder, Channel, Decoder, Destination, and Noise. In our case, the internet is used as the pipe that conducts the signal, the transmitter and receiver are the chosen form of media (video, text, audio), and the information source and destination are the people and algorithms at both ends.
This model is not complete as it does not take into consideration the personalities of the persons involved, nor context, history, time, or any other clues that make communication complex. Moreover, it is limited to one-to-one scenarios and doesn’t cover one-to-many.
Regardless of these flaws, the model gives a generic picture of what communication is: It is about sharing meaning — a message — through the exchange of information via symbols that have perceived meanings on both ends.
The Shannon-Weaver model is a great starting point to see what needs to be done so that the conduct becomes more reliable, reducing uncertainty. That is, we need to keep in mind the qualities of context, sender, intent, message, channel, audience, and response. Concretely, the more the exchange the less the ambiguity, both parties converging towards a common point, common interest, or focus.

All this to say that the internet is a medium for transporting information. Just like the printing press lead to novelties, change in thoughts, and brought doctrines to the public like never before, the internet has its own characteristics but can only be defined by what we use it for.
For now, let’s pause and go back to what propaganda, influence, and persuasion are.

Influence is a generic word to convey that someone’s decisions have been affected by something. Influence is passive, the person makes the decision themselves unconsciously. It can happen with or without any direct communication (verbal or nonverbal). It is a spontaneous mix of inner motivations and environment. Consequently, it is rarely, if ever, controlled by another entity’s will unless this entity has full control over the environment.
The nudge theory argues that an entity can influence people through positive or negative reinforcement, indirect suggestions. However, the long term effects of these suggestions are highly contested as they get mixed with other environmental factors like peer pressure and social norms.
An example of influence is product placement, where a product is desired based on its indirect association with a portrayed lifestyle that the viewer would, in theory, like to obtain.

Persuasion, on the other hand, is based on a two-way interaction. It is often used in negotiation when one party or many parties want to take over the other, imposing their point. The difference with negotiation is that the core of the message never changes, it is not about resolving a common conflict but gradually convincing someone.
Persuasion is thus transactional and voluntary. The party that wants their message to come across has to bridge through the individual psychological process, to exchange ideas freely and openly so that the other will be convinced. In persuasion there is a back and forth between persuader and persuadee, the persuader adapting, shaping, reinforcing, changing the message at each iteration to make it more appealing depending on the persuadee’s responses.
If the persuasion succeeds, it should result in a reaction such as “I never saw it that way before”.

Persuasion can be verbal or nonverbal. It can either look like a consensual dialogue to reach an understanding, or look like strategic information or awareness campaigns that appear one-way but are still using a feedback mechanism to hone the message. This former is called Organized Persuasive Communication (OPC), and refers to all organized persuasion activities (advertising, marketing, public relations, organizational communication), but the term can also refer to more manipulative means.
In both these cases there is a free flow of information and the persuadee is fully aware of what is happening.

People are reluctant to change and so to convince them the persuader has to shape the message in a relatable way to something the persuadee already believes, or use lies and tactics morphing the facts.
This means persuasion can take a manipulative form, which may or may not be consensual. For example when someone is persuaded under false pretences, incentivized via promises, provided extra benefits, or coerced through threats or actual infliction of costs (including withdrawal of benefits).

Propaganda goes in that manipulative direction. It is exclusively a one-to-many massive persuasive strategy that is deliberately trying to shape perceptions, cognitions, and the direct behavior to achieve the desired intent of the propagandist.

Historically, propaganda didn’t have the same meaning it has today. It meant “to propagate” or “to disseminate”, but the term has changed over time as it got attached with the dissemination of information by governments for particular purposes.
Purpose is what differentiate propaganda. It is always meant to be targeted at a specific group, category of people, or society with the goal to push them in the desired direction. This objective is often linked to institutional ideologies, that is why we call propaganda activated ideology.
This means it is inherently advantageous to the propagandists, and their ideological fellows, but not necessarily the recipients. In some cases, it is arguable, and depends on the viewer, if the message is propaganda or education.

Propaganda doesn’t shy away from using whatever means available to inculcate the message. It isn’t based on mutual understanding, omits or hides information, uses deception, uses manipulation, frames issues, coerces or deceptively coerces, distorts, lies, overemphasizes, and misdirects. Anything goes, as long as the goal is reached.
The message is crafted in a deliberate way to support a viewpoint, a narrative, a goal of an ideology.
This is why propagandist consultants are often called “spin doctors” or “spinmeisters”, reinterpreting the events under a particular perspective.

The corollary of propaganda is censorship. The entity that wants to control the perception of its ideology silences any message that doesn’t fit. It achieves the same purpose by, not filing people’s mind with approved information, but preventing them from being confronted with opposing points of view.

That is why propagandists require control of information flow so that they can manufacture it. This allows them to select the current agenda — telling the public what is important and what isn’t. Or to frame problems, influencing how the public pictures a situation, creating the images and opinions that go along the ideological goal. The propagandists try to control the media as a source of information distribution and to present distorted information from what appears to be credible sources.
Indeed, it is also imperative that the message should be packaged in ways that conceal its manipulative persuasive purpose, as we’ll see later.

Another interesting typical aspect of propaganda as communication is its heavy use of symbols and association. That is in direct relation to its institutional and ideological root. For instance: justice, liberty, devotion to a country, etc..
Symbolism is also heavily employed in advertising propaganda to sell products as ideologies and ways of living.

Wherever you look and ask, all domains of research and work that touch propaganda dismiss their association with it. Public relation dismisses it as consensual and not manipulative, and the same in the advertising industry. The studies of propaganda are filled with euphemism, diverting attention, using the techniques they are themselves describing.

The historian Zbyněk Zeman categorizes propaganda as either white, grey or black, but there are other categorizations too.

White propaganda is when the message is sent from an official source and the intent is more or less clear. For example, a message on TV by the president of a country.

Black propaganda is when the message conceals the source and intent by making it seem as if it emerged from the same people, group, or organization it is targeting. Obscuring the identity of the originator is done to make the message more credible, giving the impression it comes from the people it’s trying to discredit or manipulate. It is also used for diplomatic reasons to hide government involvement.
For the propagandist, this is the type that is safer and has greater persuasiveness.

Grey propaganda lives in between, it is a message coming from an ambiguous or non-disclosed source, and the intent isn’t known. This could be a rumor or a message based on unnamed insider’s info.

Counter-propaganda is the reaction to propaganda. It differs from it in how it is defensive, based on facts and truth, clear (understood by everyone), calling out logical fallacies. It is most often administered as a white propaganda.
Propaganda can only be countered effectively if the counter-propaganda acts quickly to overcome its damage. Propaganda uses an anchor, as we’ll see, and it needs to be dismantled as soon as possible. That means the older the propaganda the more it will be deeply rooted in people’s mind.

Self-propaganda is a type of propaganda where someone deliberately tries to convince themselves of something, even if irrational. This is akin to a Stockholm syndrome in some cases.

Sub-propaganda refers to anything that isn’t direct propaganda but to ways to keep the door open for future opportunities of propaganda.
Within this category we find promotional culture that is used in marketing and other domains. It consists of promoting commodities, celebrities, politics, civil society, social cultural traditions, economic ways, etc.. It is the salesmanship of a culture and thus cobbled with direct and indirect marketing means.

Computational propaganda is, according to Phil Howard, “the assemblage of social media platforms, autonomous agents, and big data tasked with the manipulation of public opinion.”
Computational propaganda can be directed by agendas or not. It could passively happen because of the incentive of the algorithms on the platforms — usually social media and thus views and clicks as we’ll see in other sections.
Propaganda, like any other type of communication, depends heavily on the environment and context in which it is moving.

Another related concept is persuasive technology, whenever a digital platform has full control of the environment of a user and attempts to change their attitudes through social influence and persuasion techniques. This consists in creating an environment that guides the person to perform certain actions and not others, inciting them through different means like social feedback, extrinsic or intrinsic motivations, and positive and negative reinforcement..

When it comes to administering persuasion and propaganda, the process is intuitive: Choose the target group and message to transmit. Choose the media form, the media shell, depending on how easily it can spread. Pack the message material in the shell, and send it towards either the mass/group, or towards specific individuals that could spread it further. If an opinion leader spread it then their credibility, expertise, trustworthiness, and attractiveness with respect to the target group should be analyzed to match the message.
Then ensues a series of iterations to polish the message through reinforcement, repetition, targeting opinion leaders through indoctrination, and the addition of new psychological techniques, manipulation of the message, emotions, and logical fallacies appropriate to the target audience.

If the propagandists’ or persuaders’ goal go to the contrary of the habits of the audience, it will be difficult to achieve and will require awareness and information related to behavioral change and predictors of behavior of the target audience.
That is, the message, to have its effect, will need to be adapted and employ tactics to resonate with the audience.

The message shouldn’t be imposed, instead the recipient should feel it flow effortlessly within themselves. Any propaganda shouldn’t look like propaganda to the target audience. “Propaganda is most effective when it is least noticeable”. The change will need to be self-imposed. This is similar to the nudge theory of reinforcement, it should appear as influence and not propaganda.
For that, the message should seem to give expression to the recipient’s own concerns, tensions, aspirations, and hopes. It must identify with them, this is what we call the anchor.

This anchor activates the ideology of the propagandist by tying down new attitudes or behaviors along with existing ones. It is the starting point for change because it represents something already widely accepted by potential persuadees — it is the path of least resistance.

These anchors can be categorized as either belief, value, attitude, behavior, and group norms.
An anchor of belief is one of the best anchor, and the stronger the belief of a receiver, the more likely they will attach to the propaganda goal.
A value is a kind of long-term belief that is not likely to change. It is hard to use as anchor because these are unwavering and so might conflict with the propagandist goal.
An attitude is the readiness to respond to an idea, object, or course of action. It is a predisposition that already resides within the receiver. The propagandist can activate the predisposition to further achieve the goal.
A behavior or behavioral pattern is a clear indication of how receivers are currently acting and often predicts future behaviors.
Group norms are beliefs, values, attitudes, and behaviors derived from the membership of the receiver in a group. The propagandist can use this tendency to conformity to make the message organically move laterally within the group.

A concept similar to the anchor is called the hook. It differs from the anchor in that it is based on a unique event or idea that is constantly repeated to amplify its effect.
The hook is generally a story which central element makes it newsworthy and evokes strong emotional responses, making it stick to memory. It could be fabricated, misleading and creating doubts, or real, but most importantly it should spread quickly and inundate all conversations on a topic. The hook is then attached with whatever piece of information it was packed with.

Hooks and anchors regularly use typical imagery to reach the ideological goal. For example, portraying the enemy as a baby killer.
The compaction of big ideas into small packages that can be distributed effortlessly is a must. This can be done through emotional wording, hook-like stories, symbolic images, and impactful videos. This is a form of portable philosophy created through repetition, emotions, and stories that lead to new meanings.
Notably, when using symbols, words, and images, that already live in the collective imagination, they will bring with them a baggage of feelings, concepts, and ideas. Think of all the recently rehashed tropes in media like movies, songs, images, and news. The use of strong wording will color the message, creating an association in the collective memory: genocide, patriotism, invader, victim, democracy, anarchy, enemy, confrontation, massacre, coup, radicalization, etc..

To spread further and faster, the anchors, hooks, and imageries must all interpolate and abuse our own confused cultural code. They work by exploiting issues we haven’t adequately addressed as a society. That is why people show strong responses to messages that are linked to injustice, outrage, partisanship à la we-vs-them, and controversies. We are more likely to react to things we don’t agree with.
From economic inequalities, race, nationalism, gender roles, sexual norms, etc.. It doesn’t matter which side of the issue people are on, the message will be spread and will provoke a reaction — if this is the goal of the propagandist. It could also be fabricated controversies, seeding doubt on purpose, making it seem like we’re still on the fence, ignorant (the domain of agnotology). The propagandist expertly triggers hope, satisfaction, pride, enthusiasm, but also fear, anxiety, anger, outrage, and disgust.

Let’s take a look at how Robert Cialdini details 6 principles of persuasion:

  • Reciprocity: People feel the need to give back to someone who provided a product, service, or information.
  • Scarcity: People want items that they believe are in short supply.
  • Authority: People are swayed by a credible expert on a particular topic.
  • Consistency: People strive to be consistent in their beliefs and behaviors.
  • Likability: People are influenced by those who are similar, complimentary, and cooperative.
  • Consensus: People tend to make choices that seem popular among others.

These go hand in hand with what we described so far and also reminds us of the conveyor of the packed message. We should pay attention to the credibility of the source, if the source is revealed.
Yet, a trick in propaganda is to keep confirming what you said, to never argue and assume the truth. Our mind is wired to accept and believe what others say as true, especially if not contradicting with any prior belief or value.

The rule of thumb is that the message shouldn’t create any cognitive dissonance but instead build a world that feels real to the recipient. To create a cohesive cosmology of meanings, to sell a coherent memorable story.
However, propaganda still denies the distance between the source and audience because it puts the ideology first.

We’ll dive in the psychological aspects in another part. For now, this should have created a strong foundation to understand influence, persuasion, and propaganda.

The exact identification of what can and should be considered propaganda is debatable and extremely hard to do. Some researchers have attempted to create a framework to discern manipulative persuasion from consensual informational and dialogical ones, but this is hardly applicable to the real world. And, as we said, propaganda could be considered education in the eye of some people.

When it comes to ethics, the subject of children indoctrination comes to mind. Children are the most vulnerable targets of propaganda as they haven’t built strong defenses against them. They are the least prepared for critical reasoning and contextual comprehension and thus are impressionable. Their mental world is in the process of being built and is easier to mold. There is no strong forces to oppose the propagandist goal.

On that note, it is subjective whether certain types of persuasions and propagandas are ethical or not. Propagandists are driven by ideologies, and consequently strongly believe that all means are necessary. As with anything related to ethics and moral, it differs based on personal judgement of the actions. For some people, it is ethically wrong to employ any subterfuge or tactics to reach a goal. For others, it is not about the tactics but whether the goal itself is right or wrong.
There’s a lot to be said on the black and white division of ethics, but we’ll leave it at that because the world is inherently nebulous.

This concludes our review of propaganda, influence, and persuasion. We’ve taken a look at the basic idea of communication, the packing of a message. We’ve then defined influence, persuasion, and propaganda, with propaganda being a type of persuasion that is one-to-many, manipulative, and driven by ideology. We’ve seen different types of propaganda and their definitions such as black, white, grey, self-, and others. Then we’ve seen how this package can be administered effectively through the person or channel carrying it and the actual shell of the message. This message will be successful if it doesn’t conflict with the receiver, if it is reinforced, uses an anchor, a hook, or strong imagery. The message will spread further and faster if it triggers strong emotional responses, especially something people feel they have to stand against. Finally, we’ve glanced at the issue of identification and the ethics concerning propaganda and the most vulnerable.

Table Of Content

References








Attributions: René Descartes, Traité de l’homme

March 03, 2021

Jeremy Morgan (JeremyMorgan)

What Do You Do for a Living Anyway? March 03, 2021 05:36 PM

I get asked this question a lot. My friends and family barely understand what I do, and my title Technical Marketing Manager doesn’t help at all. So I’ll try to explain it. TL;DR: My job is awesome. I build cool stuff, talk with amazing people, and have fun. If you want to join me, click here and let’s talk In 2002, I went from being a Network Administrator/IT person to a developer.

Robin Schroer (sulami)

The Shape of Tests March 03, 2021 12:00 AM

Many tests for an operation iterate over a mapping of different inputs to expected outcomes. By looking at the tests for a single operation as the same test with different inputs and output expectations, we can start to question how we should model those tests.

Tests as Matrices

By simply enumerating every possible combination of input values, we can construct a matrix with as many dimensions as inputs. We can then define the expected result for each set of inputs, and write a generalised test function:

∀ input∈{(a, b, …, n) | a∈A, b∈B, …, n∈N} f(input)

The number of possible test cases is thus:

|inputs| = |A × B × ⋯ × N|

As soon as our operation accepts an input that has more than a few possible values, that is any kind of number, string, or complex data structure, enumerating every possible input combination becomes impractical. Instead we can resort to groupings of values via properties.

This is a test matrix for division which uses properties instead of values, with the rows being dividends, and the columns divisors:

÷ Positive Zero Negative
Positive Positive undefined Negative
Zero Zero undefined Zero
Negative Negative undefined Positive

Matrices like this are necessarily exhaustive, and force us to think about the result for every possible combination of the input values we have included.

This is an implementation of the same property matrix in Clojure:

(ns division-matrix-test
  (:require [clojure.test :refer [deftest is testing]]
            [clojure.spec.alpha :as s]
            [clojure.spec.gen.alpha :as gen]))

(defn safe-divide
  "2-arity `/`, but returns `nil` on division by zero."
  [dividend divisor]
  (try
    (/ dividend divisor)
    (catch ArithmeticException _
      nil)))

(defmacro test-matrix
  "Generates tests for a two-dimensional test matrix."
  [test-fn matrix]
  (let [columns (rest (first matrix))
        rows (map first (rest matrix))
        combinations (for [[row idy] (map #(vector %1 %2) rows (range))
                           [col idx] (map #(vector %1 %2) columns (range))]
                       [row col (-> matrix
                                    (nth (inc idy))
                                    (nth (inc idx)))])]
    `(doseq [combination# [~@combinations]]
       (apply ~test-fn combination#))))

(deftest safe-division-test
  (let [gen-input
        (fn [kind]
          (case kind
            :pos (gen/generate (s/gen pos-int?))
            :neg (gen/generate (s/gen neg-int?))
            :zero 0))]

    (test-matrix

     (fn [x y result-pred]
       (let [dividend (gen-input x)
             divisor (gen-input y)]
         (is (result-pred (safe-divide dividend divisor))
             (format "Failed with: %s / %s" dividend divisor))))

     [[nil    :pos   :zero  :neg ]
      [:pos   pos?   nil?   neg? ]
      [:zero  zero?  nil?   zero?]
      [:neg   neg?   nil?   pos? ]])))

In this case we are testing a safe variant of the division function /, which returns nil if the divisor is zero. This simplifies the testing process, because we do not have to include any exception catching logic in our test function, or invent a notation to mean this set of inputs should result in a thrown exception.

It is worth noting that such a direct interpretation of a matrix is only possible in a language as malleable as Clojure. In other languages, we might have to resort to enumerating a set of (dividend divisor result) tuples, losing the guarantee of covering all possible combinations.

But even in Clojure, more than two dimensions in this matrix will quickly become unwieldy and hard to follow, and a tuple-based approach would scale better to larger numbers of input parameters.

Tests as Trees

Another way we could structure our tests is as a tree. A tree does not have to be exhaustive the same way a matrix has to be. We can omit certain combinations of inputs by pruning their branches. In this way we are implying that if a single input has a given value, it defines the result regardless of the other inputs’ values.

In the division example all branches with a divisor of zero could be collapsed into a single case, as the dividend does not matter in this case. This only works if the first level of branching describes the divisor, and the dividends are on the second level.

(ns division-tree-test
  (:require [clojure.test :refer [are deftest testing]]
            [clojure.spec.alpha :as s]
            [clojure.spec.gen.alpha :as gen]))

(deftest safe-division-test

  (testing "with a positive divisor"
    (let [divisor (gen/generate (s/gen pos-int?))]

      (testing "and a positive dividend"
        (let [dividend (gen/generate (s/gen pos-int?))]
          (is (pos? (safe-divide dividend divisor)))))

      (testing "and a zero dividend"
        (let [dividend 0]
          (is (zero? (safe-divide dividend divisor)))))

      (testing "and a negative dividend"
        (let [dividend (gen/generate (s/gen neg-int?))]
          (is (neg? (safe-divide dividend divisor)))))))

  (testing "with a divisor of zero"
    (let [dividend (gen/generate (s/gen int?))]
      (is (nil? (safe-divide dividend 0)))))

  (testing "with a negative divisor"
    (let [divisor (gen/generate (s/gen neg-int?))]

      (testing "and a positive dividend"
        (let [dividend (gen/generate (s/gen pos-int?))]
          (is (neg? (safe-divide dividend divisor)))))

      (testing "and a zero dividend"
        (let [dividend 0]
          (is (zero? (safe-divide dividend divisor)))))

      (testing "and a negative dividend"
        (let [dividend (gen/generate (s/gen neg-int?))]
          (is (pos? (safe-divide dividend divisor))))))))

This might look more verbose, but in exchange we get a unique label for every tree branch, which can improve readability. The nesting also naturally lends itself to lexical scoping, so we only have the values in scope which apply on a given branch.

A key advantage of the tree structure is flexibility. If one of the branches requires special code, we can confine it to that branch, avoiding complicating the remaining branches more than necessary.

Trees also scale better with larger numbers of inputs or options for inputs. A tree might grow overly wide or deep, but we can split it if that becomes a problem.

There is a downside to omitting branches though. If we change our safe-divide function to return different results depending on the dividend when the divisor is zero, our tests might still pass, depending on the specific inputs used, but we will lose test coverage for certain code paths. We have chosen to not test certain input combinations, and we need to be aware of this omission when we are changing the code under test.

Tests as Definitions

Considering the formula describing the generalised test function above, we could also consider translating this directly into code. This can work, but only if we can test results without re-implementing large parts of the code under test, otherwise we are overly coupling the tests to the code. In the division case, we can decide the sign of the result based on the signs of the inputs.

(ns division-spec-test
  (:require [clojure.test :refer [deftest is]]
            [clojure.spec.alpha :as s]
            [clojure.spec.test.alpha :as stest]))

(defn- check-safe-divide-result [{{:keys [dividend divisor]} :args
                                  ret :ret}]
  (cond
    (zero? divisor) (nil? ret)

    (zero? dividend) (zero? ret)

    (or (and (pos? dividend) (pos? divisor))
        (and (neg? dividend) (neg? divisor)))
    (pos? ret)

    :else (neg? ret)))

(s/fdef safe-divide
  :args (s/cat :dividend number?
               :divisor number?)
  :ret (s/nilable number?)
  :fn check-safe-divide-result)

(deftest safe-divide-spec-test
  (let [check-result (stest/check `safe-divide)]
    (is (not check-result)
        (format "Failed with: %s"
                (-> check-result
                    first
                    stest/abbrev-result
                    :failure
                    ::stest/val)))))

This solution is specific to Clojure, though many other languages have property based testing tools that work similarly.

By adding a spec to our function, we can run a large number of different inputs against our function, and assert a property about the result based on the inputs. It will even shrink the inputs to find the simplest set of inputs to trigger a spec failure.

This means we do not have a programmer writing a matrix or a tree by hand anymore, which has some advantages. The main one being that a programmer might not consider all possible inputs.

Fail in safe-divide-spec-test
Failed with: {:args {:dividend ##NaN, :divisor 0}, :ret ##NaN}

Fail in safe-divide-spec-test
Failed with: {:args {:dividend 1, :divisor ##Inf}, :ret 0.0}

Fail in safe-divide-spec-test
Failed with: {:args {:dividend 6.812735744013041E-108, :divisor 2.7578261315509936E216}, :ret 0.0}

Conclusion

The optimal shape of a test depends mainly on the structure of the inputs to the operation we are testing, as well as its nature.

For pure functions which we expect to use widely and change rarely, property-based testing can be desirable to avoid unintended consequences. There is also a certain speed requirement for test shrinking to work effectively.

Operations with a small number of possible inputs can also be tested via test matrices, which have fewer limitations, but do not guarantee correctness, as only the programmer can assert the completeness of the matrix. They are easy to extend with additional values for parameters, but harder to extend with additional values. Their declarative nature can be useful for documentation purposes.

At the other end of the spectrum, tree-shaped tests are the most flexible, and scale best for larger operations with many inputs. If different branches require fundamentally different setup, test trees can isolate that complexity to where it is required. They also require the most care to keep tidy, and have a tendency to sprawl if unsupervised.

March 02, 2021

Gokberk Yaltirakli (gkbrk)

Emacs idle config March 02, 2021 09:00 PM

Due to some bad computer habits, I end up opening and closing editor instances a lot instead of starting my editor and using it for the duration of the project. In particularly bad cases, I end up closing my editor for every code change to run a compiler or check some files in the same terminal.

The proper solution to this problem is to get rid of bad habits and start using multiple windows like a modern human in 2021. But the easier solution is to make my editor startup time faster in order to mitigate some of the pain in using it.

This is not the fault of Emacs of course, as Emacs is expected to be running for at least the whole programming session. This doesn’t change the fact that I need to speed it up to make my life better though.

Profiling

The first step to making a program faster is to run a profiler to see which parts are slowing it down. Since running emacs -Q (which makes Emacs start without loading the user config) is instant, this means my customization is causing Emacs to start up slower.

I used two methods for profiling. The first method is to run ProfileDotEmacs, which is a small Emacs Lisp script that prepares a report on how long each section of your .emacs file takes to execute. If all your configurations are done in your .emacs file, in a mostly top-level fashion, ProfileDotEmacs will probably be sufficient to know what is causing the slowdown.

The second method, which actually provided more useful information for me, is to wrap your config in profiler statements. This can be done like this.

(profiler-start 'cpu)

;; Your existing config

(profiler-end)

Now you will be able to run M-x profiler-report whenever you want in order to a nested profiler report that you can interactively drill down to.

Speeding everything up

The title is a little dishonest, as we won’t actually be improving the run-time of any code. What we will be doing is to take the blocking code that runs on startup, and make it run once the editor is idle for a few seconds. The most important thing is that the editor starts up quickly, displays the contents of the file instantly and allows me to move around. Less crucial settings and comfort plugins can be loaded once I stop pressing the keys to think for a few seconds.

The way to schedule a command to run when the editor is idle is to call run-with-idle-timer. You can choose how many idle seconds you want to wait before executing your callback and what arguments to pass to you callback. I ended up writing these two helper functions for loading a file after the editor becomes idle.

(defun leo/load-file (file)
  "Load an elisp file from .emacs.d"
  (load (concat "~/.emacs.d/" file)))

(defun leo/load-idle (sec file)
  "Load a file after N idle seconds"
  (run-with-idle-timer sec nil 'leo/load-file file))

Delaying everything by one second was enough for my needs, but you can choose a tiered approach where the essentials are loaded immediately and heavier, less-important plugins get loaded over time.

March 01, 2021

Gustaf Erikson (gerikson)

On the dates March 01, 2021 06:13 PM

When I started recording the dates when Sweden’s death toll from COVID-19 exceeded round thousands, I did not foresee the project continuing into the next year. But here we are.

I used to set the dates when I noticed Swedish media report them, but I’ve now gone to FHM’s stats page and got them from there.

This has led to some reshuffling - especially on Jan 6 2021 which now has its own tally.

This table has also been updated.

12,000 dead in Sweden March 01, 2021 12:28 PM

Mark J. Nelson (mjn)

Converting arbitrary base^exponent to scientific notation March 01, 2021 12:00 PM

I had the following question: Given a number in $base^{exponent}$ format, like $3^{500}$ or $7^{300}$, is there a way to convert it to scientific notation without explicitly constructing the (possibly very large) number as an intermediate step? We will assume here that both $base$ and $exponent$ are positive. For example, $7^{300}$ should be converted to $3.384\ldots \cdot 10^{253}$, also written 3.384...e+253.

This can be useful when looking at exponential growth with different bases. I happened to come across this particular question, right now, with game tree branching factors. If one game has an average branching factor of 3 and lasts 500 turns on average, is that more or fewer total states than one with an average branching factor of 7, which lasts 300 turns on average? We can eyeball the answer easily by converting to scientific notation, with its more familiar base 10: $3^{500} \approx 3.636 \cdot 10^{238}$, while $7^{300} \approx 3.384 \cdot 10^{253}$, so the latter is larger by 15 orders of magnitude.

Beyond games, exponential growth is recently in the news in epidemiology of course. Given an effective reproduction number $R$ and current cases $i$, there will be an expected $i \cdot R^t$ cases after $t$ additional time units pass. But how to easily compare different $R$ and $t$ combinations? Converting to scientific notation is one way.

I spent more time googling unsuccessfully for this answer than it ultimately took to just derive it from equations solvable using high school mathematics. Maybe I am bad at Google. But nonetheless, I thought I'd summarize the result.

There are two steps. First, change bases from $base$ to $10$. The resulting exponent is likely not to be an integer. Secondly, take this $10^x$ and convert it to scientific notation, meaning an expression of the form $m \cdot 10^e$, for some mantissa $m$, $0 m 10$ and integral exponent $e$.

Step 1. Changing bases

We want to turn $base^{exponent}$ into $10^x$. That means solving for $x$ in the equation $10^x = base^{exponent}$. For example, $10^x = 7^{300}$.

First, take the base-10 logarithm of both sides: $$\log_{10} 10^x = log_{10} base^{exponent}$$ Applying the definition of a logarithm as the inverse of exponentiation, this is equivalent to: $$x = \log_{10} base^{exponent}$$ This gives us a closed form for $x$, but still requires evaluating $base^{exponent}$, which is what we were hoping to avoid. But we can apply the logarithmic identity that $\log a^b = b \cdot \log a$, resulting in: $$x = exponent \cdot \log_{10} base$$

Now we have something that can be computed efficiently, at least in approximate terms. For example, we can convert $7^{300}$ to $10^x$ by computing $x = 300 \cdot \log_{10} 7$, yielding $10^{253.529\ldots}$.

This is almost scientific notation, except that scientific notation has integral exponents, not exponents like $253.529\ldots$.

Step 2. Round down the exponent and attach a mantissa

We now have a $10^x$ with probably non-integral $x$, such as the running example of $7^{300} \approx 10^{253.529}$.

The exponent for the purposes of scientific notation will simply be this $x$ rounded down to the nearest integer, denoted $\lfloor x \rfloor$. That will naturally result in a smaller number. So to approximate the original number, we need to multiply it by a mantissa $m$. This is the basic idea of scientific notation: a number between 0 and 10 times an integral power of 10. Put in the form of an equation, we want to solve for $m$ in: $$m \cdot 10^{\lfloor x \rfloor} = 10^x$$ This can be solved with high school mathematics actually a little more elementary than Step 1, though for some reason it took me longer to recall that fact: $$m = \frac{10^x}{10^{\lfloor x \rfloor}}$$ $$m = 10^{x - \lfloor x \rfloor}$$ Now we again have a closed-form equation that can be computed reasonably efficiently with floating-point arithmetic (not requiring a gigantic intermediate result).

To return again to our running example of $7^{300}$. In step 1 we determined this was equal to $10^{253.529\ldots}$. Now we turn it into scientific notation of the form $m \cdot 10^e$, where:

  • $e = \lfloor 253.529\ldots \rfloor = 253$
  • $m = 10^{253.529\ldots - e} = 10^{0.529\ldots} \approx 3.384$
Yielding $7^{300} \approx 3.384 \cdot 10^{253}$ or 3.384e253, without explicitly constructing the large intermediate number.

* * *

Translating the above to Python:
  import math

  base = 7
  exponent = 300

  raw_exponent = exponent * math.log10(base)

  sci_notation_exponent = math.floor(raw_exponent)
  sci_notation_mantissa = 10 ** (raw_exponent - sci_notation_exponent)

Gustaf Erikson (gerikson)

The Anarchy: The East India Company, Corporate Violence, and the Pillage of an Empire by William Dalrymple March 01, 2021 09:36 AM

A good history of the EIC. Dalrymple gives equal space to the “opposing” viewpoints, sidestepping the historiographical triumphalism of most English-language histories.

February March 01, 2021 09:01 AM

Strandvägen

Last image(?) from this area of Stockholm from me, as we’ve moved our office.

Feb 2020 | Feb 2019 | Feb 2018 | Feb 2017 | Feb 2016 | Feb 2015 | Feb 2014 | Feb 2013 | Feb 2012 | Feb 2011

February 28, 2021

Derek Jones (derek-jones)

Fitting discontinuous data from disparate sources February 28, 2021 11:16 PM

Sorting and searching are probably the most widely performed operations in computing; they are extensively covered in volume 3 of The Art of Computer Programming. Algorithm performance is influence by the characteristics of the processor on which it runs, and the size of the processor cache(s) has a significant impact on performance.

A study by Khuong and Morin investigated the performance of various search algorithms on 46 different processors. Khuong The two authors kindly sent me a copy of the raw data; the study webpage includes lots of plots.

The performance comparison involved 46 processors (mostly Intel x86 compatible cpus, plus a few ARM cpus) times 3 array datatypes times 81 array sizes times 28 search algorithms. First a 32/64/128-bit array of unsigned integers containing N elements was initialized with known values. The benchmark iterated 2-million times around randomly selecting one of the known values, and then searching for it using the algorithm under test. The time taken to iterate 2-million times was recorded. This was repeated for the 81 values of N, up to 63,095,734, on each of the 46 processors.

The plot below shows the results of running each algorithm benchmarked (colored lines) on an Intel Atom D2700 @ 2.13GHz, for 32-bit array elements; the kink in the lines occur roughly at the point where the size of the array exceeds the cache size (all code+data):

Benchmark runtime at various array sizes, for each algorithm using a 32-bit datatype.

What is the most effective way of analyzing the measurements to produce consistent results?

One approach is to build two regression models, one for the measurements before the cache ‘kink’ and one for the measurements after this kink. By adding in a dummy variable at the kink-point, it is possible to merge these two models into one model. The problem with this approach is that the kink-point has to be chosen in advance. The plot shows that the performance kink occurs before the array size exceeds the cache size; other variables are using up some of the cache storage.

This approach requires fitting 46*3=138 models (I think the algorithm used can be integrated into the model).

If data from lots of processors is to be fitted, or the three datatypes handled, an automatic way of picking where the first regression model should end, and where the second regression model should start is needed.

Regression discontinuity design looks like it might be applicable; treating the point where the array size exceeds the cache size as the discontinuity. Traditionally discontinuity designs assume a sharp discontinuity, which is not the case for these benchmarks (R’s rdd package worked for one algorithm, one datatype running on one processor); the more recent continuity-based approach supports a transition interval before/after the discontinuity. The R package rdrobust supports a continued-based approach, but seems to expect the discontinuity to be a change of intercept, rather than a change of slope (or rather, I could not figure out how to get it to model a just change of slope; suggestions welcome).

Another approach is to use segmented regression, i.e., one of more distinct lines. The package segmented supports fitting this kind of model, and does estimate what they call the breakpoint (the user has to provide a first estimate).

I managed to fit a segmented model that included all the algorithms for 32-bit data, running on one processor (code+data). Looking at the fitted model I am not hopeful that adding data from more than one processor would produce something that contained useful information. I suspect that there are enough irregular behaviors in the benchmark runs to throw off fitting quality.

I’m always asking for more data, and now I have more data than I know how to analyze in a way that does not require me to build 100+ models :-(

Suggestions welcome.

February 27, 2021

Patrick Louis (venam)

Internet: Medium For Communication, Medium For Narrative Control — Introduction February 27, 2021 10:00 PM

Astronomicum Cæsareum,

  • Internet: Medium For Communication, Medium For Narrative Control
  • Introduction

To no one’s surprise, the internet has permeated all aspects of our lives. All other means of communication have dwindled in comparison, even though the technological behemoth is relatively young (around 50 years old as of 2021).
Worldwide, according to statistics from 2019, people spent an average of 2 and a half hours a day on social media. The top place goes to The Philippines with 3h53min per day.

This raises an iceberg of loaded questions.
At the top: How controlling is the internet today?
Or, asking in reverse: How does internet, as a new vector of communication, let different actors control us? How is the internet changing the way communication works and, indirectly, how we think?

These are broad questions and there are plenty of sub-questions underneath. Indeed, we keep hearing in the news about social media, extremism on the rise, and a salad of convoluted keywords thrown in articles trying to rationalize all this.
Is there really an information war?
What are the elements of it?
Who are the actors?
What’s the relation with social media?
Is it related to all the conspiracy theories we keep hearing about?
Is journalism dying?
What is the attention and data economy?
Are we all part of a giant hypnosis experiment?
More importantly, can we, and should we, do something about this?

Like many of you, I’ve asked myself these questions countless times, got buried in a mountain of headlines and news articles selling shock-value pseudo-deciphering. I temporarily felt clear-headed but quickly got back to a muddy state of comprehension.
Thus, I’ve set myself to consume all the literature I could find, peel it, parse it, organize it, categorize it, hone it, filter it, etc.. So that I could, at least partly, remove the haze surrounding the topic.

This series of articles is going to be my attempt at answering how the internet changes communication regarding narrative control. My own mini internet studies.

Here is the rough plan of our exploration.

Table Of Content

Get ready because this is going to be a long ride!








Attributions:

  • Peter Apian, Astronomicum Caesareum (Ingoldstadt, 1540)

February 26, 2021

Gokberk Yaltirakli (gkbrk)

Giving search engines a fair access to data February 26, 2021 09:00 PM

Search engines are difficult to create. They are even harder to improve to a point where you get good-enough results to keep regular users. This is why it’s so rare to see decent search engines that aren’t front-ends to the Bing or Google APIs.

This doesn’t mean there are none though. There are a small number of search engines with their own crawlers and search logic. More and more of them appear over time, but most of them cannot improve to the point of catching on. This is because of a common resource they lack: Data.

I am not talking about the slimy, personal kind of data that Google and friends like so much. What are people searching for right now? How many of those do I have good results for? How do people form their queries? Those are all difficult to answer and improve if people aren’t using you search engine. But no one will use your search engine unless you improve those. Great then, we are in a chicken-and-egg situation with no escape in sight.

The data problem

Before tackling the problem, let’s explore what the problem is in the first place. The first problem is the number of humans testing the result quality. In almost all cases, the creator(s) will be testing the results. Friends and family will try it a few times before going back to their default search engine. Social media and Hacker News will provide a swarm of clicks that only last for a few hours. This is not data, at least not enough data.

The second problem is a little trickier. Most people from our already small set of users will not provide data that is too valuable. Let’s break down our users into two segments, the creators and the people testing it out.

The creators are programmers who research very specific pieces of information all day. While this makes them very good at using search engines, it makes them very bad at testing the results. A programmer knows the exact query that will bring them results before typing it. This query is usually so good that even a bad algorithm will find the results they are looking for.

The people testing it out have a different problem. When put on the spot for testing a search engine, it is not easy to come up with queries for difficult questions. But those are the exact situations that need the help of a good search engine. You will only see these queries once people see you as reliable and pick you as their default engine.

The current situation

We can separate the current search ecosystem into three distinct groups.

Google gets almost all the search traffic. They have more than enough data, both personal and aggregated, to serve all their needs. Their monopoly on search, and their hostility for the open web makes this undesirable. A good solution will decrease the amount of data and they get, or give more data to their competitors.

DuckDuckGo and other API front-ends get a small chunk of search traffic. They are a good compromise between keeping the web open, and having a good search experience as a user. Most of these engines stay as API wrappers forever, so the data they get doesn’t improve them much.

Independent search engines have to make do with scraps. This makes it hard for them to become popular or earn money to support themselves.

How to improve the situation

In this post; I will propose different ways to improve this situation. Each have different trade-offs in user convenience and their usefulness to search engines. The best option would be to use and promote independant search engines. But for a lot of people, it is hard to commit to a sub-par experience even if it is the better long-term option. One can look at how people handle environmental issues to see a prime example of this effect.

Feeding data to search engines

With the first option, you keep using your favourite search engine.

An automated process will send a sample of the queries to different search engines. This way, the engines can get their hands on organic usage data before finding full-time users.

This approach makes automated requests without user interaction. Depending on their systems, this might mess up their data collection or make it difficult to serve real users. To be considerate to the service operators, we should make our requests with a User-Agent header that explains what is happening. This header will allow them to log our requests, handle them in a cheaper way, and to filter them out of the data for their real users.

Redirecting to different search engines

Another approach is to have each search go to a random search engine. Compared to the previous approach, this one is more beneficial to search engines and more incovenient for the user. The user won’t be able to reproduce searches as the same query will end up going to different search providers. Similarly, a smaller search engine might give unsatisfactory results to the user, forcing them to perform the same query multiple times.

This approach can be combined with the previous one as well. By putting a few “good” engines on the random redirect list and feeding data automatically to the rest of them, the downsides could be improved.

Embedding the results of multiple engines

There are already meta-search engines, like Searx, that satisfy some of these requirements. The problem with them though is, each data source they add clutters the main results and slows down search. I think if Searx adds the option of sending data to small search engines in the background without slowing down the main UI, it will be a really good solution to this.

One could use iframes to do this as well, but browsers not being “User Agents” any more, they allow the websites to control their embeddability.

Centralized vs. shared

Another trade-off to consider is where the automated query submission should happen. If you choose a centralized approach, you end up trusting a third-party with your search queries. If you instead choose to handle this yourself without a centralized third-party, you are now sending all your queries to all the other engines in an identifiable way.

There are a few ways to work around this. One of them is to have small public instances like the Fediverse. Everyone would pick who to trust with their queries, and even on small instances the queries would be mixed enough to protect identities. Another approach would be to keep the queries saved locally, and submit them using random proxies.

Implementation

If there are solutions satifying this need in the future, I am planning to implement this. I just wanted to write this and put it on the internet in case other people are planning similar things. I already have the random search engine redirect working, but in my opinion the most important piece is the automatic data feeding.

The way I will most likely implement this is either a web endpoint that can be added to browsers as a search engine, which can be hosted locally or on a server, or a browser extension.

Gonçalo Valério (dethos)

Django Friday Tips: Subresource Integrity February 26, 2021 06:26 PM

As you might have guessed from the title, today’s tip is about how to add “Subresource integrity” (SRI) checks to your website’s static assets.

First lets see what SRI is. According to the Mozilla’s Developers Network:

Subresource Integrity (SRI) is a security feature that enables browsers to verify that resources they fetch (for example, from a CDN) are delivered without unexpected manipulation. It works by allowing you to provide a cryptographic hash that a fetched resource must match.

Source: MDN

So basically, if you don’t serve all your static assets and rely on any sort of external provider, you can force the browser to check that the delivered contents are exactly the ones you expect.

To trigger that behavior you just need to add the hash of the content to the integrity attribute of the <script> and/or <link> elements in question.

Something like this:

<script src="https://cdn.jsdelivr.net/npm/vue@2.6.12/dist/vue.min.js" integrity="sha256-KSlsysqp7TXtFo/FHjb1T9b425x3hrvzjMWaJyKbpcI=" crossorigin="anonymous"></script>

Using SRI in a Django project

This is all very nice but adding this info manually isn’t that fun or even practical, when your resources might change frequently or are built dynamically on each deployment.

To help with this task I recently found a little tool called django-sri that automates these steps for you (and is compatible with whitenoise if you happen to use it).

After the install, you just need to replace the {% static ... %} tags in your templates with the new one provided by this package ({% sri_static .. %}) and the integrity attribute will be automatically added.

February 22, 2021

Ponylang (SeanTAllen)

Last Week in Pony - February 22, 2021 February 22, 2021 03:11 PM

Ponycheck has become an official Ponylang project. @ergl has opened a new RFC related to FFI declarations. We also have notes from Sean T. Allen and Theo Butler on how to start contributing to Pony.

Marc Brooker (mjb)

Incident Response Isn't Enough February 22, 2021 12:00 AM

Incident Response Isn't Enough

Single points of failure become invisible.

Postmortems, COEs, incident reports. Whatever your organization calls them, when done right they are a popular and effective way of formalizing the process of digging into system failures, and driving change. The success of this approach has lead some to believe that postmortems are the best, or even only, way to improve the long-term availability of systems. Unfortunately, that isn't true. A good availability program requires deep insight into the design of the system.

To understand why, let's build a house, then a small community.

A house, with four things it needs to be a working home

Our house has four walls, a roof, and a few things it needs to be a habitable home. We've got a well for water, a field of corn for food, a wood pile for heat, and a septic tank. If any one of these things is not working, let's say that the house is unavailable. Our goal is to build many houses, and make sure they are unavailable for as little of the time as possible.

When we want to build a second house, we're faced with a choice. The simple approach is just to stamp out a second copy of the entire house, with it's own field, wood, well, and tank. That approach is great: the failure of the two houses is completely independent, and availability is very easy to reason about.

Two houses, with full redundancy

As we scale this approach up, however, we're met with the economic pressure to share components. This makes a lot of sense: wells are expensive to drill, and don't break down often, so sharing one between many houses could save the home owners a lot of money. Not only does sharing a well reduce construction costs, but thanks to the averaging effect of adding the demand of multiple houses together, reduces the peak-to-average ratio of water demand. That improves ongoing economics, too.

Five houses, sharing a well

In exchange for the improved economics, we've bought ourselves a potential problem. The failure of the well will cause all the houses in our community to become unavailable. The well has high blast radius. Mitigating that is well-trodden technical ground, but there's a second-order organizational and cultural effect worth paying attention to.

Every week, our community's maintenance folks get together and talk about problems that occurred during the week. Dead corn, full tanks, empty woodpiles, etc. They're great people with good intentions, so for each of these issues they carefully draw up plans to prevent recurrence of the issue, and invest the right amount in following up on those issues. They invest in the most urgent issues, and talk a lot about the most common issues. The community grows, and the number of issues grows. The system of reacting to them scales nicely.

Everything is great until the well breaks. The community is without water, and everybody is mad at the maintenance staff. They'd hardly done any maintenance on the well all year! It wasn't being improved! They spent all their attention elsewhere! Why?

The problem here is simple. With 100 houses in the community, there were 100 fields, 100 tanks, 100 piles, and one well. The well was only responsible for 1 in every 301 issues, just 0.33%. So, naturally, the frequency-based maintenance plan spent just 0.33% of the maintenance effort on it. Over time, with so little maintenance, it got a little creaky, but was still only a tiny part of the overall set of problems.

Plot showing how the percentage of action items related to the well drops with scale

This is one major problem with driving any availability program only from postmortems. It feels like a data-driven approach, but tends to be biased in exactly the ways we don't want a data-driven approach to be biased. As a start, the frequency measurement needs to be weighted based on impact. That doesn't solve the problem. The people making decisions are human, and humans are bad at making decisions. One way we're bad at decisions is called the Availability Heuristic: We tend to place more importance on things we can remember easily. Like those empty wood piles we talk about every week, and not the well issue from two years ago. Fixing this requires that an availability program takes risk into account, not only in how we measure, but also in how often we talk about issues.

It's very easy to forget about your single point of failure. After all, there's just one.

February 21, 2021

Derek Jones (derek-jones)

Research software code is likely to remain a tangled mess February 21, 2021 11:32 PM

Research software (i.e., software written to support research in engineering or the sciences) is usually a tangled mess of spaghetti code that only the author knows how to use. Very occasionally I encounter well organized research software that can be used without having an email conversation with the author (who has invariably spent years iterating through many versions).

Spaghetti code is not unique to academia, there is plenty to be found in industry.

Structural differences between academia and industry make it likely that research software will always be a tangled mess, only usable by the person who wrote it. These structural differences include:

  • writing software is a low status academic activity; it is a low status activity in some companies, but those involved don’t commonly have other higher status tasks available to work on. Why would a researcher want to invest in becoming proficient in a low status activity? Why would the principal investigator spend lots of their grant money hiring a proficient developer to work on a low status activity?

    I think the lack of status is rooted in researchers’ lack of appreciation of the effort and skill needed to become a proficient developer of software. Software differs from that other essential tool, mathematics, in that most researchers have spent many years studying mathematics and understand that effort/skill is needed to be able to use it.

    Academic performance is often measured using citations, and there is a growing move towards citing software,

  • many of those writing software know very little about how to do it, and don’t have daily contact with people who do. Recent graduates are the pool from which many new researchers are drawn. People in industry are intimately familiar with the software development skills of recent graduates, i.e., the majority are essentially beginners; most developers in industry were once recent graduates, and the stream of new employees reminds them of the skill level of such people. Academics see a constant stream of people new to software development, this group forms the norm they have to work within, and many don’t appreciate the skill gulf that exists between a recent graduate and an experienced software developer,
  • paid a lot less. The handful of very competent software developers I know working in engineering/scientific research are doing it for their love of the engineering/scientific field in which they are active. Take this love away, and they will find that not only does industry pay better, but it also provides lots of interesting projects for them to work on (academics often have the idea that all work in industry is dull).

    I have met people who have taken jobs writing research software to learn about software development, to make themselves more employable outside academia.

Does it matter that the source code of research software is a tangled mess?

The author of a published paper is supposed to provide enough information to enable their work to be reproduced. It is very unlikely that I would be able to reproduce the results in a chemistry or genetics paper, because I don’t know enough about the subject, i.e., I am not skilled in the art. Given a tangled mess of source code, I think I could reproduce the results in the associated paper (assuming the author was shipping the code associated with the paper; I have encountered cases where this was not true). If the code failed to build correctly, I could figure out (eventually) what needed to be fixed. I think people have an unrealistic expectation that research code should just build out of the box. It takes a lot of work by a skilled person to create to build portable software that just builds.

Is it really cost-effective to insist on even a medium-degree of buildability for research software?

I suspect that the lifetime of source code used in research is just as short and lonely as it is in other domains. One study of 214 packages associated with papers published between 2001-2015 found that 73% had not been updated since publication.

I would argue that a more useful investment would be in testing that the software behaves as expected. Many researchers I have spoken to have not appreciated the importance of testing. A common misconception is that because the mathematics is correct, the software must be correct (completely ignoring the possibility of silly coding mistakes, which everybody makes). Commercial software has the benefit of user feedback, for detecting some incorrect failures. Research software may only ever have one user.

Research software engineer is the fancy title now being applied to people who write the software used in research. Originally this struck me as an example of what companies do when they cannot pay people more, they give them a fancy title. Recently the Society of Research Software Engineering was setup. This society could certainly help with training, but I don’t see it making much difference with regard status and salary.

Carlos Fenollosa (carlesfe)

Whatever Clubhouse is, they are onto something February 21, 2021 11:35 AM

I've been following Clubhouse for a few weeks. As a podcaster, it piqued my interest. So it's like podcasts, but live?

The official slogan is drop-in audio chat. But that's not good. It only makes sense once you've used the app, and it doesn't describe the whole thing.

For me, the perfect definition is: it's Twitch for audio. But then, you need to know what Twitch is.

Yesterday I received an invitation and finally got to try it first hand. And I think that Clubhouse is onto something.

Radio vs Podcasts

Everybody knows radio. Even during this Internet revolution, it still has survived. Why? Because it's convenient. You tune in to some station and listen to music or people talking. It requires zero effort.

Radio has two problems: the fact that it's live, and the selection of topics.

Nowadays it's easy to download aired shows, so if you really like some program but you missed it when it was live, just go to their website and download the mp3 file.

However, the selection of topics still is an issue. Due to the fact that a station is a business, and that its model is airing ads, it requires volume. Therefore most radio stations produce mainstram content.

With the coming of the internet, a few nerds started using a new technology called Podcasts. You could record any audio content with a 1€ microphone and publish it on the internet.

Even though podcasts are naturally asynchronous, many shows air live too. Some listeners can listen to the stream, but most of them just download the audio file later.

Publicly searchable podcast directories aggregate both amateur and professional audios. Thanks to that, we have reached this point where anybody in the world has access to an ocean of audio content about any topic, either mainstream or niche.

Enter Clubhouse

What Twitch did to Youtube, Clubhouse has done to podcasts. For the sake of this explanation, let's ignore that podcasts are an open ecosystem and Youtube is proprietary.

Youtube is a video discovery platform. It has some tools to livestream, but it's not their main focus. Twitch has a much better product (and ToS) for livestreamers and their audience.

Want to watch somebody playing Minecraft? Open Twitch, search for Minecraft, and boom! hundreds of streams right there. Join one, chat with the community, and if you're lucky the streamer may shout out to you.

You can't do that with podcasts.

First of all, there can be some interactivity by combining an Icecast stream with an IRC channel, but it is not a good system.

Second, live podcasts are not aggregated anywhere. It is just impossible to search for "strategies to control your stress during covid-19" and find live shows.

So, if only as a directory of live audio content, Clubhouse has future.

But it is not only that. The product is very well thought and lets the audience participate, with audio.

A naive approach would have been to include a text chat on top of the audio stream. That would replicate the current solution on an integrated app. Okay, not bad.

However, the Clubhouse team spent some time thinking about the use case for audio streaming, which is not the same as for video streaming, nor public chat rooms.

Most of us listen to audio while we are doing other tasks and most of the times our hands are busy. This is why people jokingly call it the Airpods social network. You can participate while being away from a phone or computer.

In Clubhouse, you can tap a button to "raise your hand", and the moderators may "unmute" you. Then you can talk to the rest of the audience. Of course, not all show formats allow for that, but the option is there.

Being able to talk to your idols or even talk to the community of fans is very powerful. My first experience with Clubhouse was moving. I was listening to a concert and after the show all the listeners gathered up to talk about their experience and to have a chat with the band. Everybody agreed that with Clubhouse you can feel that there's people at the other end. Not only the speakers, but also the audience.

You don't get that with podcasts, even with live ones with a chat room.

A new category

Clubhouse has definitely invented a new category which combines the best of radio and the best of podcasts.

The product implements a selection of novel features which, when brought together, create an exciting and very addictive experience:

  • Directory of live audio streams ("rooms") about any imaginable topic
  • You can quickly drop in any room, listen for a few minutes, and jump to another one
  • The audience can participate via audio, which creates a great sense of community
  • Basic tools to follow people and interests, and get notified when they live stream
  • Of course, streamers may record the audio and publish it afterwards, so it's trivial to use Clubhouse in combination with the current podcasting ecosystem.

If you're in the podcasting community you should try to find an invitation. It is the real deal.

Tags: internet, podcasting

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

Pepijn de Vos (pepijndevos)

Switching Continuously Variable Transmission February 21, 2021 12:00 AM

What if you took a boost converter and converted it to the rotational mechanical domain? Switching CVT!

At the University of Twente, they teach Bond Graphs, a modelling system for multi-domain systems that is just perfect for this job. Unlike domaing-specific systems or block diagrams, Bond Graphs model a system as abstract connections of power. Power is what you get when you multiply an effort with a flow. The two examples we’re interested is voltage × current and force × velocity, or to be exact, angular momentum × angular velocity.

Here is a schematic of a boost converter (source). It goes from a high voltage (effort, force) to a low voltage, but from a high current (flow, velocity) to a low current. It works by charging the inductor by shorting it to ground, and then discharging it via the diode into the capacitor.

boost converter

The classic example of model equivalence is that an electrical inductor-capacitor-resistor system behaves equivalent to a mechanical mass-spring-damper system. In the rotational domain, the equivalent of a switch is a clutch, and the equivalent of a diode is a ratchet. So we have all we need to convert the system! Step one is making the bond graph from the electrical system.

boost converter bond graph

Quick Bond Graph primer if you’re too lazy to read the Wikipedia page. Se is a source of effort. R, I, and C are generalized resistance, inertance, and compliance. mR is a modulated resistance I used for the switch/clutch. D is a diode/ratchet that I just made up. 0 junctions are sum of flows, equal effort. 1 junctions are sum of effort, equal flow. An ideal electrical net has equal voltage (effort), and a sum of currents, but a mechanical joint has an equal velocity (flow), but a sum of forces. With that in mind, we can convert the bond graph to the mechanical system.

mechanical boost converter

I’m not sure if those are even remotely sane mechanical symbols, so I added labels just in case. The motor spins up a flywheel, and then when the clutch engages it winds up the spring. Then when the clutch is disengaged, the ratched keeps the spring wound up, driving the output while the motor can once more spin up the flywheel.

It works exactly analogous to the boost converter, and also suffers from the same problems. Most ciritically, switching/clutching losses. I imagine applying PWM to your clutch will at best wear it down quickly, and maybe just make it go up in smoke. Like with a MOSFET, the transition period where there is a nonzero effort and flow on the clutch, there is power loss and heat.

Anyway, I decided to build it in LEGO to see if it’d work. I used a high-speed ungeared motor that can drive absolutely no load at all, and connected it with a 1:1 gear ratio to the wheels with only a flywheel, clutch, ratchet, and spring inbetween. This proves that there is actually power conversion going on!

lego mechanical boost converter

If you get rich making cars with this CVT system, please spare me a few coins. If you burn out your clutch… I told you so ;)

February 18, 2021

Pete Corey (petecorey)

Genuary 2021 February 18, 2021 12:00 AM

I didn’t participate in Genuary this year, but I was inspired by a few of the submissions I saw to hack together my own sketches. Here’s what I came up with.

I was originally inspired by this Reddit post on expanding circles, so I recreated it and added some extra layers of depth. My kingdom for a plotter and a mechanical pencil:

From there, I thickened the stroke width of each circle, and colored each pixel based on the number of circle intersections there (using 2D SDF to check for intersections, and the cubehelix algorithm for coloring). There’s some really cool kaleidoscope vibes in some of these variations:

The SDF technique caught my imagination, so I spent some more time playing with using SDF and cubehelix to render simple circles:

This post inspired me to play with turmites a bit. The &lbrace&lbrace&lbrace1, 2, 1}, &lbrace1, 8, 1}}, &lbrace&lbrace1, 2, 1}, &lbrace0, 2, 0}}} turmite is especially cool. Coloring it based on the number of visits to each cell, and removing the “state lines” shows some interesting structures:

While I didn’t officially participate, I had a lot of fun with Genuary this year.

February 15, 2021

Ponylang (SeanTAllen)

Last Week in Pony - February 14, 2021 February 15, 2021 02:47 AM

The supported version of FreeBSD is moving from 12.1 to 12.2. The Apple M1 support team has an initial report. The documentation site website, main.actor, is being shut down. The default branch renaming is underway. Interested in contributing to Corral or other Pony tools?

Andreas Zwinkau (qznc)

Software Architecture: How You Make Them Care February 15, 2021 12:00 AM

Software architects must code, talk business, and tell stories.

Read full article!

February 14, 2021

Derek Jones (derek-jones)

Performance impact of comments on tasks taking a few minutes February 14, 2021 11:23 PM

How cost-effective is an investment in commenting code?

Answering this question requires knowing the time needed to write the comment and the time they save for later readers of the code.

A recent study investigated the impact of comments in small programming tasks on developer performance, and Sebastian Nielebock, the first author, kindly sent me a copy of the data.

How might the performance impact of comments be measured?

The obvious answer is to ask subjects to solve a coding problem, with half the subjects working with code containing comments and the other half the same code without the comments. This study used three kinds of commenting: No comments, Implementation comments and Documentation comments; the source was in Java.

Were the comments in the experiment useful, in the sense of providing information that was likely to save readers some time? A preliminary run was used to check that the comments provided some benefit.

The experiment was designed to be short enough that most subjects could complete it in less than an hour (average time to complete all tasks was 31 minutes). My own experience with running experiments is that it is possible to get professional developers to donate an hour of their time.

What is a realistic and experimentally useful amount of work to ask developers to in an hour?

The authors asked subjects to complete 9-tasks; three each of applying the code (i.e., use the code’s API), fix a bug in the code, and extend the code. Would a longer version of one of each, rather than a shorter three of each been better? I think the only way to find out is to try it both ways (I hope the authors plan to do another version).

What were the results (code+data)?

Regular readers will know, from other posts discussing experiments, that the biggest factor is likely to be subject (professional developers+students) differences, and this is true here.

Based on a fitted regression model, Documentation comments slowed performance on a task by 30 seconds, compared to No comments and Implementation comments (which both had the same performance impact). Given that average task completion time was 205 seconds, this is a 15% slowdown for Documentation comments.

This study set out to measure the performance impact of comments on small programming tasks. The answer, at least for tasks designed to take a few minutes, is that No comments, or if comments are required, then write Implementation comments.

This experiment measured the performance impact of comments on developers who did not write the code containing them. These developers have to first read and understand the comments (which takes time). However, the evidence suggests that code is mostly modified by the developer who wrote it (just reading the code does not leave a record that can be analysed). In this case, the reading a comment (that the developer previously wrote) can trigger existing memories, i.e., it has a greater information content for the original author.

Will comments have a bigger impact when read by the person who wrote them (and the code), or on tasks taking more than a few minutes? I await the results of more experiments…

Update: I have updated the script based on feedback about the data from Sebastian Nielebock.

Noon van der Silk (silky)

2020 Books February 14, 2021 12:00 AM

Posted on February 14, 2021 by Noon van der Silk
div.book { display: flex; flex-direction: row; padding: 10px; } div.book img { margin-right: 10px; width: 150px; } div.alt { background: #eaeaea; }

Continuing the tradition started in 2018, continued in 2019 over on the Braneshop blog, I was reminded I haven’t posted the books I read in 2020 yet.

So, here we are:

How Long ’til Black Future Month? by Jemisin, N.K.

As a big fan of N.K. Jemisin I just wanted to read more of what she’d written. This is good for that; it’s a really nice dip-in to the worlds she has created and leaves you wanting more!

Gravity’s Century: From Einstein’s Eclipse to Images of Black Holes by Cowen, Ron

Pretty good. This was a nice history of black holes and an introduction to some of the pressing issues presently.

The Cloudspotter’s Guide by Pretor-Pinney, Gavin

I picked this up after starting (but not finishing) The Wavewatcher’s Companion, which I found to be hilarious. This book was not as funny, but was still pretty good. Gavin has a very nice way of appreciating the world.

Altruism: The Power of Compassion to Change Yourself and the World by Ricard, Matthieu

I really quite enjoyed this book, and infact wrote a longer review over on the Between Books website.

Girl, Woman, Other by Evaristo, Bernardine

A nice collection of stories featuring different women and how their lives intersect. I think it’s quite nice to get so many stories from the everyday lives of women; especially for me, for whom it’s a bit unknown territory.

The White Album by Didion, Joan

The first Joan Didion book I’ve read. She’s very famous of course; and I don’t know. It was good to read; but was it objectionably good? Maybe. It’s at least nice to know a bit about her life and how she writes. I’ll probably read more.

Mullumbimby by Lucashenko, Melissa

I quite enjoyed this story. It’s also the first time I’ve read a modern fictional Australian indigenous-centered story. Will be on the look out for more books of this kind! Came up as part of the OC House bookclub, I think.

Down Girl: The Logic of Misogyny by Manne, Kate

This book is amazing. I found it to be written in a bit of an academic-y style; so it can be a bit dense, but it does an amazing job of framing what misogyny is and how it is present in all aspects of society.

Unfree Speech: The Threat to Global Democracy and Why We Must Act, Now by Wong, Joshua

This is a nice short history of free speech and how China is taking over Hong Kong. It was interesting to read about how they attempted to make some large changes to their democracy through grass-roots organisation. In some ways quite inspiring, but also a reality check that it isn’t easy to defeat power. Worth a read.

A Month in Siena by Matar, Hisham

This was just a nice simple book about someone getting amongst life in a new town. In many ways I think this kind of experience is how many people wish their holidays to go: meeting some nice stranger, integrating into the community, and just appreciating the joy of life.

The Weil Conjectures: On Math and the Pursuit of the Unknown by Olsson, Karen

I can’t remember much about this book at this point; I think I enjoyed it.

The Worst Journey in the World by Cherry-Garrard, Apsley

I read the next two books back-to-back. This one was strange. It’s written from the English perspective; it’s quite racist at times. I wouldn’t recommend reading it, unless you really want to compare perspectives on the journeys of Scott and Amundsen.

The Last Place on Earth: Scott and Amundsen’s Race to the South Pole by Huntford, Roland

This was a really interesting read. It was so fascinating to compare how Amundsen approached the journey compared to Scott. According to this book, Scott was woefully underprepared and arrogant, and Amundsen spent many many years training and learning the right skills from different indigenous groups in order to survive.

If you want to learn about the race to the south pole; this book is definitely better than the above.

Food Rules: An Eater’s Manual by Pollan, Michael

More of a blog post than a book; but still, nice to keep next to you somewhere when you want to remember if it’s okay to eat chocolate everyday (probably not) or have a wine with dinner (probably fine).

Men Explain Things to Me by Solnit, Rebecca

Hilarious. This is a collection of stories; some funnier than others, but overall great.

The Shadow of the Sun by Kapuściński, Ryszard

I enjoyed this as a nice way to get a bit of, albeit an outsiders, insight into how different people live in some of the poorer parts of Africa.

Capital and Ideology by Piketty, Thomas

I really enjoyed this book, and wrote about it more over on Between Books - Capital and Ideology

Capital in the Twenty-First Century by Piketty, Thomas

This one I also enjoyed; I don’t think you really need to read it first before Ideology, but I opted to do so. Reading this really inspired a love and interest of economics, and Piketty seems to do an amazing job of exploring these topics in an approachable way.

Affluence Without Abundance: The Disappearing World of the Bushmen by Suzman, James

This book wasn’t bad; it’s an exploration of life with a marginalised tribe referred to as the Bushmen. It doesn’t totally take the expected view that the simple life is better; but it does discuss how politics and the community encroaches on this tribe, and how they live life.

A Woman in the Polar Night by Ritter, Christiane

I think this is a classic book; it’s a little strange, but it does show one persons journey into the Arctic and how she learned to love it. It didn’t exactly inspire me to go and life there, but I do admire her approach!

Touching the Void: The True Story of One Man’s Miraculous Survival by Simpson, Joe

This is amazing. I read this assuming a particular fact about the story (I won’t spoil it by telling you), but it turned out that my mind was blown by what actually happened. I’ll probably read it again!

Into Thin Air: A Personal Account of the Mount Everest Disaster by Krakauer, Jon

I bought this I think after the one above; desperate for more books about climbing. This is also an amazing story; I think if anything it made me feel certain that I’ll never attempt to climb Everest.

The Case of the Honjin Murder by Yokomizo, Seishi

I just happened across this one in the bookshop and thought I’d give it a go. I can’t say it was the best; it was quite sexist. The story was probably pretty good, if you could ignore that; but I couldn’t.

Tall Man: The Death of Doomadgee by Hooper, Chloe

This was a good read but super aggravating. It’s really unsettling to learn how terrible some of the policing is; and the subsequent investigations that yielded no useful outcome. Eye-opening for me in terms of how bad racism is in Australia.

Mountains of the Mind: A History of a Fascination by Macfarlane, Robert

Pretty good. I went through a phase of reading about mountain climbing. This was a nice little overview of how people justify climbing. Good for an introduction to other things to read.

The City We Became (Great Cities, #1) by Jemisin, N.K.

For how much I loved her other books; I have to say I found this one a bit disappointing. I think by normal sci-fi standards, it’s certainly excellent, and I’ll definitely read the rest of the series; but if you’re looking for something as amazing as her other books, you might not find it here.

Annapurna: A Woman’s Place by Blum, Arlene

I really enjoyed this one. It was interesting to compare this to other climbing books written by men, which almost never feature much uncertainty or collaborative leadership.

Underland: A Deep Time Journey by Macfarlane, Robert

This is just super cool. We are taken on a journey through lots of different underground worlds. Certainly makes you want to do this kind of exploring.

The Eastern Curlew by Saddler, Harry

I read this after meeting the author at a party! I felt so cool; I’d never met an author in real life before. Inspired by that meeting, I picked it up. It’s really a nice story about following the migratory path of a bird, and thinking about how their ecosystem is being impacted. Would definitely recommend!

Her Body and Other Parties by Machado, Carmen Maria

Not bad. Probably not my favourite style of book, but if you like weird kind of magical fiction with a message, probably quite good.

My Sister, the Serial Killer by Braithwaite, Oyinkan

I quite enjoyed this one; it was a very quick read; so I’m looking forward to more by this author!

In Praise of Shadows by Tanizaki, Jun’ichirō

Pretty good. Very very short; but a thoughtful analysis of how light impacts space. I think it’s a classic of the architecture world.

Barbarian Days: A Surfing Life by Finnegan, William

I really really enjoyed this one. Very nicely written, you feel like you’re living the life alongside the author. It’s nice to read about someone who follows their passion so directly.

Why I’m No Longer Talking to White People About Race by Eddo-Lodge, Reni

I picked this up as soon as I arrived in the UK, to gain an understanding for how people here think about race issues. Pretty good reading.

Kon-Tiki by Heyerdahl, Thor

This was a funny one. A classic kind of adventure story, from real life, I have to say I enjoyed it.

Mad, Bad & Dangerous to Know: The Autobiography by Fiennes, Ranulph

I found this one also kind of funny. I suppose this guy is very famous in the UK, but I’d not heard of him really. It’s funny to read about how he thinks of endurance, and his claim that “anyone” can be like him; in terms of running 7 marathons at the age of 70 across 7 countries; or something along those lines. Didn’t exactly encourage me to do the same, but did give me some food for thought about willpower and energy.

See What You Made Me Do: Power, Control and Domestic Violence by Hill, Jess

This is a very hard book to read, emotionally. It contains some exceptionally difficult stories. Definitely recommended reading, but if you want a more academic treatment see the earlier book by Kate Manne.

Becoming Bodhisattvas by Chödrön, Pema

Easily the best book I read last year. Of course it’s a buddhist view on how to live life; but I found it very practical and thoughtful. I’ve read it again during some difficult times, and found it very uplifting.

February 12, 2021

Jan van den Berg (j11g)

Thoughts on Clubhouse February 12, 2021 09:44 AM

You can’t

  • Listen on demand
  • Restart/replay a conversation
  • Record a conversation or audio snippets
  • Trace back when a conversation started
  • See who is talking, you can only hear them
  • Send (text) messages to other Clubhouse users
  • Share pictures, videos, gifs or audio files
  • Use it on Android

You can

  • Listen to audio conversations as they happen
  • See avatars of who is in the conversation
  • Start a group audio conversation with people you know
  • See Clubhouse member profiles with who they follow and who follow them
  • Minimize the app and do other things while listening
  • Receive conversation notices, as they happen, based on people you follow*

What you get

  • Raw, unscripted watercooler conversations

So there are a lot of things Clubhouse doesn’t do and only a few things it does do. But the almost archaic possibilities of the Clubhouse app are a large part of the appeal. The constraints punctuate the experience.

Developers, developers, developers

Ephemeral group conversations are of course as old as humans. And we didn’t even have to wait for the internet and smartphones for this human need to be implemented in technology. Theoretically the Clubhouse experience was already possible — pre-internet — with the plain old telephony system and it is also basically what happens on the amateur radio network frequencies (this is still a thing).

Remember this bad boy?

Which is why it is remarkable that it took to 2020 for such an app to exist on a smartphone. Was the idea maybe too simple? No. Clubhouse may be a primal experience but it is also a logical iteration from text messages, Instagram Stories, Snapchat and TikTok. Clubhouse adds something new to this line of — increasingly real-time — social interactions, by taking away a lot of barriers. And by being the only one that is actually real-time.

The Clubhouse experience is the lowest bar to participation of any social app out there. You don’t have to leave the house, sit at a desk, straighten your hair, you don’t even have to be able to type. It is just you talking or listening to people.

And Clubhouse strips down the human need for sharing without showing your face (Zoom), or having to be overly creative (TikTok). Remember that Instagram and Snapchat filters are not only successful because they are fun, they also obfuscate what you don’t want to be seen. Clubhouse doesn’t have this problem.

Realness

This all boils down to the lowest denominator of participation of any social app out there and the result is a very real experience. So real that it hardly makes sense for people to get ‘verified’ (blue checkmarks) you know right away if the person talking is who they say they are. I was listening to a room with Steve Ballmer, and trust me, that was Steve Ballmer.

It’s the algorithm

So Clubhouse offers one of the oldest human experiences of just people talking. But here is the really clever part and why we did need the internet and smartphones.
*The Clubhouse algorithm sends a push notification when conversations you might be interested in are happening. This is probably also the only reason Clubhouse uses profiles and following/followers lists. Because your interests are, of course, based on people you follow. And this — social graph — is exactly what the internet and smartphones bring to the table that the telephone system and amateur radio can’t.

So now what?

A lot of Clubhouse conversations are about Clubhouse. Also a lot of people on Clubhouse are talking about social media and social influencing in general. It feels very meta. But I guess that is what typically happens when these things start.

Clubhouse is the hottest new app at the moment, either because of their clever iOS only, FOMO inducing invite only approach, or because of the tech VC entourage that pushes the interest for the app, or maybe because that the pandemic has emphasized the need for humans to connect. It’s probably a little bit of all of the above. But you also know because of the app’s succes either one of these two things will happen: 1. Facebook will buy Clubhouse or 2. Facebook will clone Clubhouse. We’ll see.

I see different paths forward for Clubhouse and I am curious to see how it will pan out. And the app right now is very bare, which is also the appeal. So it’ll be interesting to see how and whether they will pivot, maybe they will start adding features, maybe they will introduce recording conversations (podcasts)? And they of course will have to find ways to monetize. And they will have to do so all while the question looms: will it stay fun or is it just the newness that is appealing?

The post Thoughts on Clubhouse appeared first on Jan van den Berg.

Pete Corey (petecorey)

The Long and Short Case for Elixir February 12, 2021 12:00 AM

I recently landed a new contract with a client gracious enough to hear my thoughts about why they should make radical changes to their stack and choose Elixir to build their greenfield data pipeline.

As part of my pitch, I put together two documents:

The “long case” is a detailed, low level document that outlines why Elixir might be a better choice than Node.js for building the server in question. The “short case” is a distilled and higher level version of the longer document, intended for the project owner.

May these documents help you as much as they’ve helped me. If you have any suggestions for improving them, please let me know!

February 11, 2021

Carlos Fenollosa (carlesfe)

How I moved my setup from a Mac to a Linux laptop February 11, 2021 08:38 AM

This article is part of a series:

  1. Seven years later, I bought a new Macbook. For the first time, I don't love it
  2. This article
  3. Fed up with the Mac, I spent six months with a Linux laptop. The grass is not greener on the other side
  4. (TBD) My review of the M1 Macbook Air

Returning the Macbook Pro

In the previous installment of this series I explained how I had been disenchanted with recent Macs.

The fact is that shortly after writing that review I returned the 2020 Macbook Pro.

I tried to love it, honest to God, but it was mediocre.

Let me share with you the list of issues I found while using it for two weeks:

  • USB2 keyboard and mouse randomly disconnecting. I tried 3 different USB/Thunderbolt adapters, including Apple's, to no avail.
  • Abnormal rattling sound when fans start spinning, as if they were not properly seated.
  • Not detecting the internal keyboard sometimes after resuming sleep.
  • Only 5 hours of battery life when streaming video with the screen off, connected to a TV.
  • Battery drained from 70% to 15% after sleeping overnight.
  • The touchbar. I spent time trying to make it useful with BetterTouchTool. Then I discovered that the keypresses don't register correctly most of the times, and I had to tap twice or three times, which defeats the purpose of having a dedicated shortcut.
  • Trackpad registers too many spurious taps and the cursor jumps around the screen while typing.
  • Airpods Pro have ~400ms of audio lag when watching videos. Solved with a reboot but it appears again after a few minutes.

The fact that it had issues with the internal keyboard and the USB subsystem made me think that the unit may have a faulty logic board. I discovered later, through a Reddit thread, that the USB2 issue was not specific to my unit, but it didn't matter much.

I was feeling really ripped off with a machine I had spent 2.000€ on and which, speed aside, was worse than my 2013 Air in every way I cared.

In the end, my gut told my brain, "Stop rationalizing it, this laptop makes you unhappy", and I came to terms with it.

Now what?

I had been dreading this moment since 2016, when I realized that Apple didn't care about my demographic anymore.

Migrating platforms is a big hassle, so after I made the decision to return the Macbook Pro, I thought carefully what the next steps would be.

In order to transition from the Mac to Linux I had to prepare a plan for for new hardware, new software, and a new cloud ecosystem.

At that point there were strong rumors about ARM Macs. I thought I'd use Linux for an indeterminate amount of time, until Apple hopefully released a good Mac again. Which may have been "never", though I was optimistic.

I have used Linux extensively in my life, since 1999, but mostly as a developer. Nowadays my requirements are more "mainstream" and I need apps like Microsoft Office and Adobe Reader to do my job. This is an important point to make. This computer is my main work tool, and it needs to accommodate my own needs as well as my job's. I spent some time making sure that I could run all the software I needed. As a last resort, there is always the option of using a VM.

As a final step, I had to move all the iCloud stuff out of there, because it is not interoperable with standard clients. I decided I would self-host it and see how difficult it is to leave the walled garden.

Therefore, I needed to fulfil the following requirements:

  1. Good laptop hardware with Linux support
  2. Ability to run work-related software
  3. Self-hosted (or almost) cloud services

1. Choosing a new laptop: The 2018 Dell XPS 13"

Before buying—and potentially returning—a new machine I drove to the office and grabbed two old laptops to see if they would fit: a Thinkpad 420 and a 2018 Dell XPS 13".

I decided to test drive the five of them: the 2020 MBP, my 2013 MBA, the Thinkpad, the Dell, and my tinkering laptop, a Thinkpad x230 with OpenBSD.

I then spent a couple days trying to make some sense of the situation. You can see them running a group video chat and some benchmarks.

Fortunately, a clear winner emerged: the 2018 Dell XPS with Ubuntu-Dell installed.

How good is the 2018 XPS? Excellent. 9.5/10 would recommend. I got in love with that machine. Very good hardware, with just a few minor issues.

Pros:

  • Good screen quality
  • Small bezels. It makes a difference and I still miss them today.
  • Light, nice in my lap. The Macbook Pros have air vents that "cut" into your legs when you're wearing shorts.
  • All I/O worked fine. I used the official Dell Thunderbolt Dock.
  • Excellent keyboard. I liked the pgup/pgdn keys on the arrow cluster and welcomed back the function keys.
  • Good battery life (6h of streaming video) even though the laptop had been used daily for almost 3 years.

Cons:

  • The speakers are of laughable quality. Not just bad, but why-would-Dell-do-that bad. Extremely quiet and terrible quality.
  • The webcam is on a bad location. Not really a big deal, but I'm glad they fixed it in recent revisions.
  • The trackpad is kinda like the 2013 Air's, but a bit worse.
  • Coil whine. I tried to be positive and used it as an "activity indicator" like in the old days of spinning hard drives, but I'd rather not have it.

That really is it. The Dell XPS is probably the best go-to PC laptop. Excellent set of compromises, good price, good support. If you want to use Linux on a laptop, you can't go wrong with the XPS.

2. Doing my job with Linux

I knew beforehand that hardware support was not going to be an issue. Linux drivers in general are pretty good nowadays, and that XPS specifically was designed to work well with Linux, that is why we bought it for the office.

On first boot everything works. Ubuntu is pretty good. Gnome 3 tries to be like a Mac, which I liked, and the basic software is fine for most of the people.

I then installed my work software. Most of it is either standard (email, calendar...) or multi-platform via Electron or webapps. For Windows-specific software I purchased a license of Crossover and also installed a Windows 10 VM on Virtualbox. It was not super convenient and sometimes things crashed with Crossover, but I could manage.

Overall, the desktop environment and base apps are not as polished as macOS, which I will discuss later, but it worked.

I am happy to realize that I can continue recommending Linux to "regular people" who want a computer that just works, doesn't get viruses, and is very low maintenance.

3. My self-hosted cloud setup

This is a topic that is on everybody's mind right now. We went from the original, decentralized internet, to a network centralized in a few vendors like Facebook, Google, Cloudflare and Amazon, and I think that is a bad idea.

The walled garden is comfortable, but what happens when you want to make the switch? How easy it really is to migrate your online infrastructure to another vendor?

Well, I was going to discover that soon. I like the iCloud ecosystem, and in general am fine with Apple's privacy policies, but I just couldn't continue using it. Apart from email, all other services (pictures, calendars, files, notes, etc.) cannot be used in Linux, and the browser client is extremely bad.

I am a geek, and have been a sysadmin since college, so I took it as a personal challenge to create my own personal cloud infrastructure.

First I tried Nextcloud. It mostly works and I recommend it in general, but the server components are too heavy and the file syncing is slow and unreliable.

I decided to self-host every individual piece of the puzzle:

  • My mail has been managed by postfix/dovecot for a few years now. I don't recommend anybody self-hosting email due to deliverability issues, but I'm that stubborn.
  • I set up radicale for contacts, calendars and tasks. It had issues connecting to some clients due to, I believe, SSL negotiation. If I had to set it up again I'd try another alternative.
  • All my files got synced over my laptops and the server thanks to Syncthing. I can't stress enough how great of a software is Syncthing. Really, if you're looking for an alternative to Dropbox, try it out. It will amaze you.
  • Syncthing does not expose files publicly, so I span up the Apache Webdav server to share files.
  • Joplin is a good alternative to take rich text notes and sync them over the internet. The clients are not very polished, but it works.
  • For passwords I've been using Lastpass for some time.
  • I kept using iCloud for pictures, because it's the best solution if you have an iPhone. It is fine because I don't need to work with pictures on my daily workflow.

It took some time of researching and deploying all the pieces, and I'm quite happy with the result. It feels really great to manage your online infrastructure, even though it requires technical knowledge and regular maintenance to keep everything up to date.

So how did it all work out?

Well, I have been repeating this term all over the article: it works. I could do my job, and it was a very gratifying learning experience. Overall, I do encourage geeks to spin up their own cloud infra and work with Linux or BSD boxes. I do have some self-hosted cloud services and I also keep a laptop with OpenBSD which I use regularly.

It is possible to get out of the walled garden. Of course, it's not within reach of the general public yet, even though Nextcloud is very close, and some third party vendors are starting to offer an integrated cloud experience outside the world of the Big Cloud.

But I'm writing this in the past tense because I went back to the Mac. Unfortunately, after six months of using this setup full-time I started noticing very rough edges, which I will explain on the next article.

Stay tuned!

Tags: apple, linux, hardware

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

Maxwell Bernstein (tekknolagi)

Small objects and pointer tagging February 11, 2021 12:00 AM

Welcome back to the third post in what is quickly turning into a series on runtime optimization. The last two posts were about inline caching and quickening, two techniques for speeding up the interpreter loop.

In this post, we will instead look at a different part of the runtime: the object model.

The problem

Right now, we represent objects as tagged unions.

typedef enum {
  kInt,
  kStr,
} ObjectType;

typedef struct {
  ObjectType type;
  union {
    const char *str_value;
    word int_value;
  };
} Object;

This C struct contains two components: a tag for the type, and space for either an integer or a C string pointer. In order to create a new Object, we allocate space for it on the C heap and pass a pointer around. For example, here is the current constructor for integers:

Object* new_int(word value) {
  Object* result = malloc(sizeof *result);
  CHECK(result != NULL && "could not allocate object");
  *result = (Object){.type = kInt, .int_value = value};
  return result;
}

I don’t know about you, but to me this seems a little wasteful. Allocating every single new integer object on the heap leaves a lot to be desired. malloc is slow and while memory is cheaper these days than it used to be, it still is not free. We’ll need to do something about this.

In addition, even if we somehow reduced allocation to the bare minimum, reading from and writing to memory is still slow. If we can avoid that, we could greatly speed up our runtime.

This post will focus on a particular strategy for optimizing operations on integer objects, but the ideas also apply broadly to other small objects. See Exploring further for more food for thought.

The solution space

There are a number of strategies to mitigate this gratuitous allocation, most commonly:

  1. Interning a small set of canonical integer objects. CPython does this for the integers between -5 and 256.
  2. Interning all integers by looking them up in a specialized hash table before allocating. This is also called hash consing.
  3. Have a statically typed low-level language where the compiler can know ahead of time what type every variable is and how much space it requires. C and Rust compilers, for example, can do this.

The first two approaches don’t reduce the memory traffic, but the third approach does. Our runtime has no such type guarantees and no compiler to speak of, so that’s a no-go, and I think we can do better than the first two strategies. We’ll just need to get a little clever and re-use some old knowledge from the 80s.

What’s in a pointer?

Before we get clever, we should take a step back and think about the Object pointers we pass around. The C standard guarantees that malloc will return an aligned pointer. On 32-bit systems, this means that the result will be 4-byte aligned, and on 64-bit systems, it will be 8-byte aligned. This post will only focus on 64-bit systems, so for our purposes all malloced objects will be 8-byte aligned.

Being 8-byte aligned means that all pointers are multiples of 8. This means that if you look at the pointer representation in binary, they look like:

High                                                           Low
0bxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx000

See that? The three lowest bits are zero. Since we’re guaranteed the pointers will always be given to us with the three zero bits, we can use those bits to store some extra information. Lisp and Smalltalk runtimes have been doing this for at least 30 years.

On some hardware, there are also bits unused in the high part of the address. We will only use the lower part of the address, though, because the high bits are reserved for future use.

The scheme

To start, we will tag all pointers to heap-allocated objects with a lower bit of 11. This means that now all real heap pointers will end in 001 instead of 000. We will then assume that any pointer with a lowest bit of 0 is actually an integer. This leaves us 63 bits of integer space. This is one less bit than we had before, which we will talk about more in Exploring further.

We are doing this because the assumption behind this pointer tagging is that integer objects are both small and common. Adding and subtracting them should be very cheap. And it’s not so bad if all operations on pointers have to remove the low 1 bit, either. x86-64 addressing modes make it easy to fold that into normal struct member reads and writes2.

And guess what? The best part is, since we were smart and used helper functions to allocate, type check, read from, and write to the objects, we don’t even need to touch the interpreter core or library functions. We only need to touch the functions that work directly on objects. Let’s take a look.

New object representation

Okay, we should also look at at the struct definition. I think we should first make Object opaque. We don’t want anyone trying to dereference a tagged pointer!

struct Object;
typedef struct Object Object;

Now we’ll need to represent the rest of the heap-allocated objects in some other type. I think HeapObject is a reasonable and descriptive name. We can keep using the tagged union approach from earlier.

typedef struct {
  ObjectType type;
  union {
    const char* str_value;
  };
} HeapObject;

Right now we only have strings but I imagine it would be useful to add some more types later on.

Now, it’s important to keep the invariant that whenever we have a HeapObject*, it is a valid pointer. This means that we should always untag before casting from Object* to HeapObject*. This will help both keep our interfaces clean and avoid bugs. You’ll see what I mean in a little bit.

Helper functions

Now that we have our object representation down, we can take a look at the helper functions. Let’s start with the easiest three, object_is_int, object_as_int, and new_int:

enum {
  kIntegerTag = 0x0,      // 0b0
  kIntegerTagMask = 0x1,  // 0b1
  kIntegerShift = 1,
};

bool object_is_int(Object* obj) {
  return ((uword)obj & kIntegerTagMask) == kIntegerTag;
}

word object_as_int(Object* obj) {
  CHECK(object_is_int(obj));
  return (word)obj >> kIntegerShift;
}

Object* new_int(word value) {
  CHECK(value < INTEGER_MAX && "too big");
  CHECK(value > INTEGER_MIN && "too small");
  return (Object*)((uword)value << kIntegerShift);
}

We decided that integer tags are one bit wide, zero, and the lowest bit. This function puts that in code. If you are unfamiliar with bit manipulation, check out the Wikipedia article on bit masks. The constants INTEGER_MAX and INTEGER_MIN refer to the maximum and minimum values that will fit in the 63 bits of space we have. Right now there are some CHECKs that will abort the program if the integer does not fit in 63 bits. Implementing a fallback to heap-allocated 64-bit integers or even big integers is a potential extension (see Exploring further).

The test for heap objects is similar to the test for ints. We use the same tag width (one bit) but we expect the bit to be 1, not 0.

enum {
  // ...
  kHeapObjectTag = 0x1,      // 0b01
  kHeapObjectTagMask = 0x1,  // 0b01
};

bool object_is_heap_object(Object* obj) {
  return ((uword)obj & kHeapObjectTagMask) == kHeapObjectTag;
}

Any pointer that passes object_is_heap_object should be dereferenceable once unmasked.

Speaking of unmasking, we also have a function to do that. And we also have a function that can cast the other way, too.

HeapObject* object_address(Object* obj) {
  CHECK(object_is_heap_object(obj));
  return (HeapObject*)((uword)obj & ~kHeapObjectTagMask);
}

Object* object_from_address(HeapObject* obj) {
  return (Object*)((uword)obj | kHeapObjectTag);
}

The function object_address will be the only function that returns a HeapObject*. It checks that the object passed in is actually a heap object before casting and untagging. This should be safe enough.

Alright, so we can make integers and cast between Object* and HeapObject*. We still need to think about object_type and the string functions. Fortunately, they can mostly be implemented in terms of the functions we implemented above!

Let’s take a look at object_type. For non-heap objects, it has to do some special casing. Otherwise we can just pull out the type field from the HeapObject*.

ObjectType object_type(Object* obj) {
  if (object_is_int(obj)) {
    return kInt;
  }
  return object_address(obj)->type;
}

And now for the string functions. These are similar enough to their previous implementations, with some small adaptations for the new object model.

bool object_is_str(Object* obj) { return object_type(obj) == kStr; }

const char* object_as_str(Object* obj) {
  CHECK(object_is_str(obj));
  return object_address(obj)->str_value;
}

Object* new_str(const char* value) {
  HeapObject* result = malloc(sizeof *result);
  CHECK(result != NULL && "could not allocate object");
  *result = (HeapObject){.type = kStr, .str_value = value};
  return object_from_address(result);
}

That’s that for the helper functions. It’s a good thing we implemented int_add, str_print, etc in terms of our helpers. None of those have to change a bit.

Performance analysis

I am not going to run a tiny snippet of bytecode in a loop and call it faster than the previous interpreter. See Performance analysis from the first post for an explanation.

I am, however, going to walk through some of the generated code for functions we care about.

We said that integer operations were important, so let’s take a look at what kind of code the compiler generates for int_add. For a refresher, let’s look at the definition for int_add (which, mind you, has not changed since the last post):

Object* int_add(Object* left, Object* right) {
  CHECK(object_is_int(left));
  CHECK(object_is_int(right));
  return new_int(object_as_int(left) + object_as_int(right));
}

Previously this would read from memory in object_as_int, call malloc in new_int, and then write to memory. That’s a whole lot of overhead and function calls. Even if malloc were free, memory traffic would still take quite a bit of time.

Now let’s take a look at the code generated by a C compiler. To get this code, I pasted interpreter.c into The Compiler Explorer. You could also run objdump -S ./interpreter or gdb -batch -ex "disassemble/rs int_add" ./interpreter. Or even run GDB and poke at the code manually. Anwyay, here’s the generated code:

int_add:                                # @int_add
        and     rdi, -2
        lea     rax, [rdi + rsi]
        and     rax, -2
        ret

How about that, eh? What was previously a monster of a function is now four whole instructions3 and no memory operations. Put that in your pipeline and smoke it.

This is the kind of benefit we can reap from having small objects inside pointers.

Thanks for reading! Make sure to check out the repo and poke at the code.

Exploring further

In this post, we made an executive decision to shrink the available integer range by one bit. We didn’t add a fallback to heap-allocated 64-bit numbers. This is an interesting extension to consider if you occasionally need some big numbers. Or maybe, if you need really big numbers, you could also add a fallback to heap allocated bignums! If you don’t care at all, it’s not unreasonable to decide to make your integer operations cut off at 63 bits.

This post spent a decent chunk of time fitting integers into pointers. I chose to write about integers because it was probably the simplest way to demonstrate pointer tagging and immediate objects. However, your application may very well not be integer heavy. It’s entirely possible that in a typical workload, the majority of objects floating around your runtime are small strings! Or maybe floats, or maybe something else entirely. The point is, you need to measure and figure out what you need before implementing something. Consider implementing small strings or immediate booleans as a fun exercise. You will have to think some more about your object tagging system!

Pointer tagging is not the only way to compress values into pointer-sized objects. For runtimes whose primary numeric type is a double, it may make sense to implement NaN boxing. This is what VMs like SpiderMonkey4 and LuaJIT do.

Remember my suggestion about the template interpreter from the quickening post? Well, that idea is even more attractive now. You, the runtime writer, get to write a lot less assembly. And your computer gets to run a lot less code.

This post puts two distinct concepts together: small objects and pointer tagging. Maybe you don’t really need small objects, though, and want to store other information in the tag bits of your pointer. What other kinds of information could you store in there that is relevant to your workload? Perhaps you can tag pointers to all prime integers. Or maybe you want to tag different heap-allocated objects with their type tags. Either way, the two techniques are independent and can be used separately.


  1. In my blog post about the Ghuloum compiler, I used the bit patterns from the original Ghuloum paper to tag integers, characters, and different types of heap-allocated objects. Feel free to skim that if you want a taste for different pointer tagging schemes. 

  2. (Adapted from my Reddit comment)

    Say you have a C struct:

    struct foo {
      int bar;
    };
    

    and a heap-allocated instance of that struct:

    struct foo *instance = malloc(...);
    

    Reading an attribute of that struct in C looks like:

    instance->bar;
    

    and gets compiled down to something like the following pseudo-assembly (which assumes the instance pointer is stored in a register):

    mov rax, [instance+offsetof(foo, bar)]
    

    This is read as:

    1. take pointer from whatever register instance is in
    2. add the offset for the bar field to the pointer
    3. dereference that resulting pointer
    4. store that in rax

    And if you tag your pointers with, say, 0x1, you want to remove the 0x1. Your C code will look like:

    instance->bar & ~0x1;
    

    or maybe:

    instance->bar - 1;
    

    Which compiles down to:

    mov rax, [instance+offsetof(foo, bar)-1]
    

    and since both offsetof(foo, bar) and 1 are compile-time constants, that can get folded into the same mov instruction. 

  3. And guess what? This is just what the C compiler can generate from a C description of our object model. I have not figured out how to add the right compiler hints, but another correct implementation of int_add is just two instructions:

    int_add:                                # @int_add
        lea     rax, [rdi + rsi]
        ret
    

    I’m not sure what’s going on with the code generation that prevents this optimization, but we really should be able to add two integers without doing any fiddling with tags. In an assembly/template interpreter world, this kind of fine-grained control becomes much easier. 

  4. This is interesting because V8 and Dart, other JS(-like) VMs use pointer tagging. Seach “Smi” (or perhaps “Smi integer”) if you want to learn more. 

February 09, 2021

asrpo (asrp)

Adding a minimal foreign function interface to your language February 09, 2021 08:21 PM

A foreign function interface (FFI) is a way for one language to call functions from another language.

The most common FFIs are C FFIs where the target language is C.

I just pushed a new version of Flpc where I added a very limited FFI for calling Python. Highlights of some other changes are:

February 08, 2021

Gonçalo Valério (dethos)

Documentation done right February 08, 2021 11:03 PM

One critical piece of the software development process that often gets neglected by companies and also by many open-source projects is explaining how it works and how it can be used to solve the problem in question.

Documentation is often lacking and people have an hard time figuring out how they can use or contribute to a particular piece of software. I think most developers and users have faced this situation at least once.

Looking at it from the other side, it isn’t always easy to pick and share the right information so others can hit the ground running. The fact that not everybody is starting from the same point and have the same goal, makes the job a bit harder.

One approach to solve this problem that I like is Divio’s documentation system.

Divio's documentation system explained. Showing the 4 quadrants and their relations.The components of Divio’s documentation system.

It splits the problems in 4 areas targeting different stages and needs of the person reading the documentation. Django uses this system and is frequently praised for having great documentation.

From a user point of view it looks solid. You should take a look and apply it on your packages/projects, I surely will.

Ponylang (SeanTAllen)

Last Week in Pony - February 7, 2021 February 08, 2021 12:36 AM

@ergl, also know as Borja o’Cook on the Ponylang Zulip, has become a Pony committer. Also, we have started the process of renaming the default branches on our repos from master to main.

February 07, 2021

Carlos Fenollosa (carlesfe)

Linux vs GNU/Linux February 07, 2021 09:41 PM

Why I do call the system Linux

I personally use the term Linux for simplicity. I know that it's the kernel name, but that's how naming works, sometimes it's convenient to use a synecdoche.

Linux is useful to identify the platform as a whole and it is recognizable to the general public. No-one is using it as demeaning, and when technical people need to be more specific they can include any complements to that term.

Why I don't call it GNU/Linux anymore

Some time ago I was an advocate for the term GNU/Linux, following the rationale of the FSF.

I still do recognize the importance of the FSF and the GNU project into making Linux what it is today. However, I think that nowadays this controversy does more harm than good to the FSF, and I encourage people to be careful when discussing it.

The FSF arguments boil down to:

  1. Many years ago, a big percentage of the codebase of GNU/Linux systems was sourced from the GNU project.
  2. The GNU project is not just a series of tools, but rather an integrated system, which was missing only the kernel. It existed before Linux.

The thing is, with every passing year Linux systems have less and less GNU components. Some vendors are even preferring software with alternative licenses such as BSD or Apache.

Should we then, using the same arguments, advocate for a name which reflects the more recent critical components, as GNU/FreeDesktop/Apache/OSI/BSD/.../Linux?

I am not trying to ridicule the FSF argument. To the contrary, my point is that, while they have been a very important contributor to the project, those specific arguments carry less weight as the project progresses. Therefore, even if this discussion could have been productive in 2001, nowadays it is either moot, or worse, plays against the GNU project's interests.

I am sure that the FSF will continue calling the system GNU/Linux and I believe they are entitled to it. But I don't think anybody should continue to proselytize about this anymore. And I also don't think that calling the system Linux in 2021 is neither morally nor technically wrong.

Tags: linux

&via=cfenollosa">&via=cfenollosa">Comments? Tweet  

Derek Jones (derek-jones)

Widely used programming languages: past, present, and future February 07, 2021 09:09 PM

Programming languages are like pop groups in that they have followers, fans and supporters; new ones are constantly being created and some eventually become widely popular, while those that were once popular slowly fade away or mutate into something else.

Creating a language is a relatively popular activity. Science fiction and fantasy authors have been doing it since before computers existed, e.g., the Elf language Quenya devised by Tolkien, and in the computer age Star Trek’s Klingon. Some very good how-to books have been written on the subject.

As soon as computers became available, people started inventing programming languages.

What have been the major factors influencing the growth to widespread use of a new programming languages (I’m ignoring languages that become widespread within application niches)?

Cobol and Fortran became widely used because there was widespread implementation support for them across computer manufacturers, and they did not have to compete with any existing widely used languages. Various niches had one or more languages that were widely used in that niche, e.g., Algol 60 in academia.

To become widely used during the mainframe/minicomputer age, a new language first had to be ported to the major computers of the day, whose products sometimes supported multiple, incompatible operating systems. No new languages became widely used, in the sense of across computer vendors. Some new languages were widely used by developers, because they were available on IBM computers; for several decades a large percentage of developers used IBM computers. Based on job adverts, RPG was widely used, but PL/1 not so. The use of RPG declined with the decline of IBM.

The introduction of microcomputers (originally 8-bit, then 16, then 32, and finally 64-bit) opened up an opportunity for new languages to become widely used in that niche (which would eventually grow to be the primary computing platform of its day). This opportunity occurred because compiler vendors for the major languages of the day did not want to cannibalize their existing market (i.e., selling compilers for a lot more than the price of a microcomputer) by selling a much lower priced product on microcomputers.

BASIC became available on practically all microcomputers, or rather some dialect of BASIC that was incompatible with all the other dialects. The availability of BASIC on a vendor’s computer promoted sales of the hardware, and it was not worthwhile for the major vendors to create a version of BASIC that reduced portability costs; the profit was in games.

The dominance of the Microsoft/Intel partnership removed the high cost of porting to lots of platforms (by driving them out of business), but created a major new obstacle to the wide adoption of new languages: Developer choice. There had always been lots of new languages floating around, but people only got to see the subset that were available on the particular hardware they targeted. Once the cpu/OS (essentially) became a monoculture most new languages had to compete for developer attention in one ecosystem.

Pascal was in widespread use for a few years on micros (in the form of Turbo Pascal) and university computers (the source of Wirth’s ETH compiler was freely available for porting), but eventually C won developer mindshare and became the most widely used language. In the early 1990s C++ compiler sales took off, but many developers were writing C with a few C++ constructs scattered about the code (e.g., use of new, rather than malloc/free).

Next, the Internet took off, and opened up an opportunity for new languages to become dominant. This opportunity occurred because Internet related software was being made freely available, and established compiler vendors were not interested in making their products freely available.

There were people willing to invest in creating a good-enough implementation of the language they had invented, and giving it away for free. Luck, plus being in the right place at the right time resulted in PHP and Javascript becoming widely used. Network effects prevent any other language becoming widely used. Compatible dialects of PHP and Javascript may migrate widespread usage to quite different languages over time, e.g., Facebook’s Hack.

Java rode to popularity on the coat-tails of the Internet, and when it looked like security issues would reduce it to niche status, it became the vendor supported language for one of the major smart-phone OSs.

Next, smart-phones took off, but the availability of Open Source compilers closed the opportunity window for new languages to become dominant through lack of interest from existing compiler vendors. Smart-phone vendors wanted to quickly attract developers, which meant throwing their weight behind a language that many developers were already familiar with; Apple went with Objective-C (which evolved to Swift), Google with Java (which evolved to Kotlin, because of the Oracle lawsuit).

Where does Python fit in this grand scheme? I don’t yet have an answer, or is my world-view wrong to treat Python usage as being as widespread as C/C++/Java?

New programming languages continue to be implemented; I don’t see this ever stopping. Most don’t attract more users than their implementer, but a few become fashionable amongst the young, who are always looking to attach themselves to something new and shiny.

Will a new programming language ever again become widely used?

Like human languages, programming languages experience strong networking effects. Widely used languages continue to be widely used because many companies depend on code written in it, and many developers who can use it can obtain jobs; what company wants to risk using a new language only to find they cannot hire staff who know it, and there are not many people willing to invest in becoming fluent in a language with no immediate job prospects.

Today’s widely used programmings languages succeeded in a niche that eventually grew larger than all the other computing ecosystems. The Internet and smart-phones are used by everybody on the planet, there are no bigger ecosystems to provide new languages with a possible route to widespread use. To be widely used a language first has to become fashionable, but from now on, new programming languages that don’t evolve from (i.e., be compatible with) current widely used languages are very unlikely to migrate from fashionable to widely used.

It has always been possible for a proficient developer to dedicate a year+ of effort to create a new language implementation. Adding the polish need to make it production ready used to take much longer, but these days tool chains such as LLVM supply a lot of the heavy lifting. The problem for almost all language creators/implementers is community building; they are terrible at dealing with other developers.

It’s no surprise that nearly all the new languages that become fashionable originate with language creators who work for a company that happens to feel a need for a new language. Examples include:

  • Go created by Google for internal use, and attracted an outside fan base. Company languages are not new, with IBM’s PL/1 being the poster child (or is there a more modern poster child). At the moment Go is a trendy language, and this feeds a supply of young developers willing to invest in learning it. Once the trendiness wears off, Google will start to have problems recruiting developers, the reason: Being labelled as a Go developer limits job prospects when few other companies use the language. Talk to a manager who has tried to recruit developers to work on applications written in Fortran, Pascal and other once-widely used languages (and even wannabe widely used languages, such as Ada),
  • Rust a vanity project from Mozilla, which they have now abandoned cast adrift. Did Rust become fashionable because it arrived at the right time to become the not-Google language? I await a PhD thesis on the topic of the rise and fall of Rust,
  • Microsoft’s C# ceased being trendy some years ago. These days I don’t have much contact with developers working in the Microsoft ecosystem, so I don’t know anything about the state of the C# job market.

Every now and again a language creator has the social skills needed to start an active community. Zig caught my attention when I read that its creator, Andrew Kelley, had quit his job to work full-time on Zig. Two and a-half years later Zig has its own track at FOSEM’21.

Will Zig become the next fashionable language, as Rust/Go popularity fades? I’m rooting for Zig because of its name, there are relatively few languages whose name starts with Z; the start of the alphabet is over-represented with language names. It would be foolish to root for a language because of a belief that it has magical properties (e.g., powerful, readable, maintainable), but the young are foolish.

Andreas Zwinkau (qznc)

3 Ideas How to Communicate Your Architecture February 07, 2021 12:00 AM

Write decision records, write a newsletter, and explain at concrete examples.

Read full article!

February 06, 2021

Patrick Louis (venam)

Making Sense of The Audio Stack On Unix February 06, 2021 10:00 PM

Come see my magical gramophone

Audio on Unix is a little zoo, there are so many acronyms for projects and APIs that it’s easy to get lost. Let’s tackle that issue! Most articles are confusing because they either use audio technical jargon, or because they barely scratch the surface and leave people clueless. A little knowledge can be dangerous.
In this article I’ll try to bridge the gap by not requiring any prerequisite knowledge while also giving a good overview of the whole Unix audio landscape. There’s going to be enough details to remove mysticism (Oh so pernicious in web bubbles) and see how the pieces fit.

By the end of this article you should understand the following terms:

  • ALSA
  • OSS
  • ESD
  • aRts
  • sndio
  • PulseAudio
  • PipeWire
  • GStreamer
  • LADSPA

We’ll try to make sense of their utility and their link. The article will focus a bit more on the Linux stack as it is has more components than others and is more advanced in that respect. We’ll also skip non-open source Unix-like systems.
As usual, if you want to go in depth there’s a list of references at the bottom.

Overall, we got:

  • Hardware layer: the physical devices, input and output
  • Kernel layer: interfacing with the different hardware and managing their specificities (ALSA, OSS)
  • Libraries: used by software to interface with the hardware directly, to manipulate audio/video, to interface with an intermediate layer for creating streams (GStreamer, Libcanberra, libpulse, libalsa, etc..), and to have a standard format (LADSPA).
  • Sound servers: used to make the user facing (user-level) interaction easier, more abstract, and high level. This often acts as glue, resolving the issue that different software speak different protocols. (PulseAudio, ESD, aRts, PipeWire, sndio)

Table of Content

Let me preface this by saying that I am not a developer in any of these tech, neither am I a sound engineer. I am simply regrouping my general understanding of the tech so that anyone can get an overview of what the pieces involved are, and maybe a bit more.

Hardware layer

It’s essential to have a look at the hardware at our disposal to understand the audio stack because anything above it will be its direct representation.
There are many types of audio interfaces, be it input or output, with different varieties of sound cards, internal organizations, and capabilities. Because of this diversity of chipsets, it’s simpler to group them into families when interacting with them.

Let’s list the most common logical components that these cards can have.

  • An interface to communicate with the card connected to the bus, be it interrupts, IO ports, DMA (direct memory access), etc..
  • Output devices (DAC: Digital to analog converter)
  • Input devices (ADC: Analog to digital converter)
  • An output amplifier, to raise the power of output devices
  • An input amplifier, same as above but for input devices (ex: microphones).
  • Controls mechanism to allow different settings
  • Hardware mixer, which controls each devices volume and routing, usually volume is measured in decibel.
  • A MIDI (Musical Instrument Digital Interface) device/controller, a standard unified protocol to control output devices (called synthesizers) — think of them like keyboards for sounds.
  • A sequencer, a builtin MIDI synthesizer (output of the above)
  • A timer used to clock audio
  • Any other special features such as a 3D spatializer

It is important to have a glance at these components because everything in the software layers attempts to make them easier to approach.

Analog to Digital & Digital to Analog (ADC & DAC)

A couple of concepts related to the interaction between the real and digital world are also needed to kick-start our journey.

In the real world, the analog world, sound is made up of waves, which are air pressures that can be arbitrarily large.

Speakers generating sound have a maximum volume/amplitude, usually represented by 0dB (decibels). Volume lower than the maximum is represented by negative decibels: -10dB, -20dB, etc.. And no sound is thus -∞ dB.
This might be surprising and actually not really true either. Decibel doesn’t mean much until it’s tied to a specific absolute reference point, it’s a relative scale. You pick a value for 0dB that makes sense for what you are trying to measure.

vu meter

The measurement above is the dBFS, the dB relative to digital full-scale, aka digital 0. There are other measurements such as dB SPL and dBV.
One thing to note about decibels is that they follow a strictly exponential law, which matches it to human perception. What sounds like a constantly increasing volume is indicated by a constantly rising dB meter, corresponding to an exponentially rising output power. This is why you can hear both vanishingly soft sounds and punishingly loud sounds. The step from the loudest you can hear up to destroying your ears or killing you is only a few more dB.

While decibels are about loudness, the tone is represented as sine waves of certain frequency, the speed. For example, the note A is a 440Hz sine wave.

Alright, we got the idea of decibel and tone but how do we get from waves to our computer or in reverse? This is what we call going from analog to digital or digital to analog.
To do this we have to convert waves into discrete points in time, taking samples per second — what we call sample rate. The higher the sample rate, the more accurate the representation of the analog sound (a lollipop graph). Each sample has a certain accuracy, how much information we store in it, the number of bits for each sample — what we call the bit rate (the higher the less noise). For example, CDs use 16 bits.
Which value you choose as your sample rate and bit rate will depend on a trade-off between quality and memory use.

NB: That’s why it makes no sense to convert from digital low sample rate to digital high sample rate, you’ll just be filling the void in the middle of the discrete points with the same data.

Additionally, you may need to represent how multiple channels play sound — multichannel. For example, mono, stereo, 3d, surround, etc..

It’s important to note that if we want to play sounds from multiple sources at the same time, they will need to agree on the sample rate, bit rate, and format representation, otherwise it’ll be impossible to mix them. That’s something essential on a desktop.

The last part in this equation is how to implement the mechanism to send audio to the sound card. That highly depends on what the card itself supports, but the usual simple mechanism is to fill buffers with streams of sound, then let the hardware read the samples, passing them to the DAC (digital to analog converter), to then reach the speaker, and vice versa. Once the hardware has read enough samples it’ll do an interrupt to notify the software side that it needs more samples. This cyclic mechanism goes on and on in a ring fashion.
If the buffer samples aren’t filled fast enough we call this an underrun or drop-out (aka xruns), which result in a glitch, basically audio stopping for a short period before the buffer is filled again.

The audio can be played at a certain sample rate, with a certain granularity, as we’ve said. So if we call buffer-size the number of samples that can be contained in a cyclic-buffer meant to be read by the hardware, fragment-size or period-size the number of samples after which an interrupt is generated, number-fragments the number of fragments that can fit in a hardware buffer (buffer-size/fragment-size), and sample-rate the number of samples per seconds.
Then the latency will be buffer-size/sample-rate, for example if we can fit 100 samples in a buffer and the samples are played once every 1ms then that’s a 100ms latency; we’ll have to wait 100ms before the hardware finishes processing the buffer.
From the software side, we’ll be getting an interrupt every period-size/sample-rate. Thus, if our buffer-size is 1024 samples, and fragment-size is 512, and our sample-rate is 44100 samples/second, then we get an interrupt and need to refill the buffer every 512/44100 = 11.6 ms, and our latency for this stage is up to 1024/44100 = 23 ms. Audio processing pipelines consist of a series of buffers like this, with samples read from one buffer, processed as needed, and written to the next.

Choosing the values of the buffer-size and period-size are hard questions. We need a buffer big enough to minimize underruns, but we also need a buffer small enough to have low latency. The fragments should be big enough to avoid frequent interrupts, but we also need them small enough so that we’re able to fill the buffer and avoid underruns.
What some software choose to do is to not follow the sound card interrupts but to rely on the operating system scheduler instead, to be able to rewrite the buffer at any time so that it stays responsive to user input (aka buffer rewinding). This in turn allows to make the buffer as big as possible. Though, timers often deviate, but that can be fixed with good real-time scheduling.
There are no optimal solutions to this problem, it will depend on requirements, and these values can often be configured.

So we’ve seen how sound is represented in the real world with strength and tone, to then be converted in the digital world via digital to analog DAC or analog to digital ADC converters. This is done by taking samples at a rate and of a certain accuracy called the bit rate. There can also be other information needed such as the channel, byte ordering, etc.. Sound that needs to be mixed needs to agree on these properties. Lastly, we’ve seen how software has to manage a buffer of samples so that sound plays continuously on the device, while also being responsive to users.

Libraries

Audio related libraries seem to be an alphabet soup of keywords. Here are some examples: alsaplayer-esd, libesd-alsa, alsa-oss, alsaplayer-jack, gstreamer-alsa, gstreamer-esd, lib-alsa-oss, libpulse, libpulse-simple, libao, and so on.

For programs to be able to use audio hardware and the related functionalities, they rely on libraries offering specific APIs. Over time, some APIs get deprecated and new ones appear. Thus, that creates a situation where multiple software speak differently.
To solve this issue, many glue libraries have appeared to interoperate between them. This is especially true when it comes to sound servers such as aRts, eSD, PulseAudio, and backends. For example ALSA supports an OSS layer, that is the role of lib-alsa-oss.

Apart from libraries used to play or record sound and music, there are libraries that have specific usages.

GStreamer is a popular library for constructing chains of media-handling components. It is the equivalent of a shell pipeline for media. This library is used in multiple software in the GNOME desktop environment. For example, cheese (webcam) uses it to add video effects on the fly. Keep this in mind as the creator of GStreamer is now working on PipeWire and applying some of the mindset and functionalities there.

libcanberra is a library that implements a freedesktop.org specs to play event sounds. Instead of having to play event sounds by loading and playing a sound file from disk every time, desktop components should instead use this library which abstract the lower level layer that will handle playing it on the appropriate backend. It’s important considering what we said about them changing over time.
The freedesktop.org event sound files can usually be found in: /usr/share/sounds/freedesktop/stereo/, and you can test by calling on the command line:

canberra-gtk-play -i bell
canberra-gtk-play -i phone-incoming-call

There are also multiple libraries used to abstract the audio backend of any OS, so called cross-platform audio libraries. This includes libraries such as PortAudio (software using it), OpenAL that focuses on 3D audio, libSDL, and libao.

Lastly, there is LADSPA, the Linux Audio Developer’s Simple Plugin API, which is a library offering a standard way to write audio filter plugins to do signal processing effects. Many programs and libraries support the format including ardour, audacity, GStreamer, Snd, ALSA (with plugin), and PulseAudio (with plugin).

We’ve seen multiple usages for libraries, from their use as glue, to them helping in chaining audio, to desktop integration, to cross-platform interaction, and to allow a common format for audio filters.

Audio Driver

For us to be able to use the audio hardware components we mentioned, we need a way to communicate with them, what we call a driver. That job is done dynamically by the kernel which loads a module when it encounters a new device.

Every platform got its device management mechanism, be it devd on FreeBSD, systemd-udev, Gentoo’s eudev, Devuan’s vdev, mdev from BusyBox or Suckless, etc..
For example, on Linux you can take a look at the currently connected components and the driver handling them by executing lspci. Similarly, on FreeBSD this is done with the pciconf -l command.

To be handled properly, the hardware needs an appropriate driver associated with it.

On Linux, the ALSA kernel layer handles this automatically. The driver names start with the prefix snd_. Issuing lsmod should show a list of them. (supported cards)
In case the device doesn’t get associated with the right driver, you can always create specific rules in the device management (udev).

On FreeBSD, the process takes place within the kernel sound infrastructure and is controlled dynamically at runtime using sysctl kernel tunables. We’ll see how to tune drivers settings in another section, as this is how you interact with them on BSD. The process is similar on most BSDs.
If the driver doesn’t load automatically you can always manually activate the kernel module. For example, to load the Intel High Definition Audio bridge device driver on the fly:

$ kldload snd_hda

Or to keep them always loaded you can set it at boot time in /boot/loader.conf:

snd_hda_load="YES"

On BSDs and Linux the drivers, OSS-derived and ALSA in the kernel, then map the components within the file system, they are the reflection of the hardware we’ve seen before. Mostly input, output, controllers, mixers, clocks, midi, and more.

On FreeBSD the sound drivers may create the following device nodes:

  • /dev/dsp%d.p%d Playback channel.
  • /dev/dsp%d.r%d Record channel.
  • /dev/dsp%d.%d Digitized voice device.
  • /dev/dspW%d.%d Like /dev/dsp, but 16 bits per sample.
  • /dev/dsp%d.vp%d Virtual playback channel.
  • /dev/dsp%d.vr%d Virtual recording channel.
  • /dev/audio%d.%d Sparc-compatible audio device.
  • /dev/sndstat Current sound status, including all channels and drivers.

Example of status:

$ cat /dev/sndstat
FreeBSD Audio Driver (newpcm: 64bit 2009061500/amd64)
Installed devices:
pcm0: <NVIDIA (0x001c) (HDMI/DP 8ch)> (play)
pcm1: <NVIDIA (0x001c) (HDMI/DP 8ch)> (play)
pcm2: <Conexant CX20590 (Analog 2.0+HP/2.0)> (play/rec) default

On OpenBSD it’s similar, a SADA-like driver (Solaris Audio API), that has a different and much simpler mapping:

  • /dev/audioN Audio device related to the underlying device driver (for both playback and recording)
  • /dev/sound same as /dev/audioN, for recording and playback of sound samples (with cache for replaying samples)
  • /dev/mixer to manipulate volume, recording source, or other mixer functions
  • /dev/audioctlN Control device, accept same ioctl as /dev/sound
  • /dev/midiN Control device

On Linux, the ALSA kernel module also maps the components to operational interfaces under /dev/snd/. The files in the latter will generally be named aaaCxDy where aaa is the service name, x the card number, and y the device number. For example:

  • pcmC?D?p pcm playback devices
  • pcmC?D?c pcm capture devices
  • controlC? control devices (i.e. mixer, etc.) for manipulating the internal mixer and routing of the card
  • hwC?D? hwdep devices
  • midiC?D? rawmidi devices - for controlling the MIDI port of the card, if any
  • seq sequencer device - for controlling the built-in sound synthesizer of the card, if any
  • timer timer device - to be used in pair with the sequencer

The devices will mostly be mapped as either PCM devices, pulse-code modulation — the digital side of the equation, or as CTL devices, the controller and mixer, or as MIDI interface, etc..

The driver status and configuration interface is in the process information pseudo-filesystem under /proc/asound (instead of kernel tunable like on most BSDs).
The following long list should give you an idea of what’s available:

  • /proc/asound/
  • /proc/asound/cards (RO) the list of registered cards
  • /proc/asound/version (RO) the version and date the driver was built
  • /proc/asound/devices (RO) the list of registered ALSA devices (major=116)
  • /proc/asound/hwdep (RO) the list of hwdep (hardware dependent) controls
  • /proc/asound/meminfo (RO) memory usage information this proc file appears only when you build the alsa drivers with memory debug (or full) option so the file shows the currently allocated memories on kernel space.
  • /proc/asound/pcm (RO) the list of allocated pcm streams
  • /proc/asound/seq/ the directory containing info about sequencer
  • /proc/asound/dev/ the directory containing device files. device files are created dynamically; in the case without devfs, this directory is usually linked to /dev/snd/
  • /proc/asound/oss/ the directory containing info about oss emulation
  • /proc/asound/cards info about cards found in cardX sub dir
  • /proc/asound/cardX/ (X = 0-7) the card-specific directory with information specific to the driver used
    • id (RO) the id string of the card
    • pcm?p the directory of the given pcm playback stream
    • pcm?c the directory of the given pcm capture stream
    • pcm??/info (RO) the pcm stream general info (card, device, name, etc.)
    • pcm??/sub?/info (RO) the pcm substream general info (card, device, name, etc.)
    • pcm??/sub?/status (RO) the current of the given pcm substream (status, position, delay, tick time, etc.)
    • pcm??/sub?/prealloc (RW) the number of pre-allocated buffer size in kb. you can specify the buffer size by writing to this proc file

For instance we can issue:

$ cat /proc/asound/cards
 0 [HDMI           ]: HDA-Intel - HDA ATI HDMI
                      HDA ATI HDMI at 0xf0244000 irq 32
 1 [Generic        ]: HDA-Intel - HD-Audio Generic
                      HD-Audio Generic at 0xf0240000 irq 16
 2 [LX3000         ]: USB-Audio - Microsoft LifeChat LX-3000
                      C-Media Electronics Inc. Microsoft LifeChat LX-3000 at usb-0000:00:12.0-4, full

That gives us an idea of how different Unix-like OS dynamically load the driver for the device, and then maps it to the filesystem, often also giving an interface to get their status and configure them. Now let’s dive into other aspects of OSS and ALSA, more on the user-side of the equation.

We now have an overview of:

  • The basic logical components a card can have (input/output devices, mixers, control mechanisms, etc..)
  • How to go from analog to digital and vice-versa
  • Some libraries and why software use different ones
  • The mapping of hardware devices to the filesystem when discovered

Advanced Linux Sound Architecture (ALSA)

Now let’s dive into ALSA in particular and see what’s the deal with it.

If you want to get dizzy you can look at this spaghetti diagram. It does more to confuse you than to clarify anything, so it fails as far as meaning is concerned. Linux Audio Dizzy Diagram

ALSA, the Advanced Linux Sound Architecture is an interface provided by the Linux kernel to interact with sound devices.
We’ve seen so far that ALSA is a kernel module and is responsible for loading drivers for the appropriate hardware, and also maps things in the filesystem on /proc/asound and in /dev/snd. ALSA also has a library, a user-facing API for real and virtual devices, and configuration mechanisms that let you interact with the internal audio concepts. Historically, it was designed to replace OSS (Open Sound System) on Linux, which we’ll see in the next section. ALSA provides an OSS emulation if needed.

Some features that are often cited:

  • Up to 8 audio devices at the same time, modularized
  • MIDI functionality like Hardware-based MIDI synthesis.
  • Perform hardware mixing of multiple channels
  • Full-duplex operation.
  • Multiprocessor-friendly
  • thread-safe device drivers

Let’s see the following: How ALSA represents devices, what are PCM and CTL, plugins, configurations, and tools.

ALSA is good at doing automatic configuration of sound-card hardware. It does that by grouping different cards based on “chipset” and families, — similar cards will have similar interfaces. It also fills the gap by using plugins when it comes to the name of controls by deliberately keeping them similar. For example, the master volume is always called “Master Volume”, even when not physically there, the abstraction will exist as a software control plugin.
This grouping allows developers to interact with sound devices in a unified way which makes it much simpler to write applications.

We’ve previously seen how ALSA maps devices in entries in /dev/snd (pcm, control, midi, sequencer, timer) with their meta-information in /proc/asound. Moreover, ALSA split devices into a hierarchy.
ALSA has the concept of cards, devices, and subdevices.

A card is any audio hardware, be it a USB audio headset, an audio chip, or virtual sound card, etc.. Real hardware are backed by kernel drivers while virtual ones live in user-space. It has 3 identifiers:

  • A number, which is incremental after each new insertion (so could change after reboot)
  • An ID, which is a text identifier for a card. This is more unique and consistent
  • A name, another text identifier, but not a useful one.

Devices are subdivision of a card, for playback or capture. For example it could be “analog input + output”, “digital output”, etc.. It dictates the type of device that the card is, what it can do and is capable of processing. A sort of “profile” for the card. Same as with cards, devices have three identifiers: Number, ID, and Name.
Only one device is active at a time, because the device is the current “function” that the card takes.

Devices themselves have at least one subdevice. All subdevices share the same playback (output) or recording (input) stream. They are used to represent available slots for hardware mixing, joining audio in hardware. However, hardware mixing is rarely used so there is usually a single subdevice unless it is a surround sound system.

Overall, that gives us this type of notation.

card CARD_NUMBER: CARD_ID [CARD_NAME], device DEVICE_NUMBER: DEVICE_ID [DEVICE_NAME]
Subdevice #SUBDEVICE_NUMBER: SUBDEVICE_NAME
Example:
card 2: LX3000 [Microsoft LifeChat LX-3000], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

Alternatively, you can go directly to the /proc tree, which we’ve seen previously, and list cards with cat /proc/asound/cards.

 2 [LX3000         ]: USB-Audio - Microsoft LifeChat LX-3000
                      C-Media Electronics Inc. Microsoft
                      LifeChat LX-3000 at usb-0000:00:12.0-4,
                      full

A common notation for ALSA devices looks like hw:X,Y where X is the card id and Y the subdevice id. You do not need the device id because only one can be active at a time.

You can list playback devices by issuing aplay -l and recording devices with arecord -l (and MIDI devices with amidi -l).
Another useful script to dump info about the system is alsa-info.sh.

All ALSA clients have to interface with objects in the ALSA world, the most important two are the PCM (pulse code modulation) and the CTL (control) which we’ve briefly mentioned before.
PCM objects are the representation of a sound stream format used for data flow. These PCMs can be chained and are typically attached to a hardware at one end, but can also be attached to other things such as the filesystem, a server, or even dropping audio completely. When PCMs are chained we call them slave PCM, a virtual PCM, which is an extra step of indirection. The PCM streams connected to the hardware need to follow its characteristics: that is they need to have the right sample rate, bit rate (sample width), sample encoding (endianess), and number of channels (mono, stereo, etc..). List them using aplay -L.
CTL objects are control objects telling ALSA how to process non-audio data. That includes things such as volume controls, toggle controls, multiple-choice selections, etc.. These controls can be put in one of 3 categories: playback control (for output device), capture control (for input device), and feature control (for special features).
There are other ALSA objects such as MIXER and UCM (Use Case Manager - for presets separation, like notifications, voice, media, etc..) but they are not important to get the concept across so we’ll skip them.

These objects can be defined, modified, and created in ALSA configurations and are often templatized and manipulated through plugins. A lot of them are automatically created by ALSA itself to create “profiles” for cards. Most of ALSA processing is delegated to plugins.
Clients will then read the configuration and most often use the default PCMs and CTL, if not selected explicitly by the user. Practically, that means the software will write or read audio from a stream (PCM) and control it (usually volume) through the CTL or MIXER interfaces.

For example:

  • aplay and other players use the PCM interface
  • alsactl uses the ctl (control) interface
  • amixer uses the mixer interface
  • amidi the rawmidi interface and so on.
  • alsaucm the ucm interface

As far as configuration location is concerned, clients will load alsa.conf from the ALSA’s data directory, so /usr/share/alsa/alsa.conf, and in turn this configuration will load system- and user-wide configurations in /etc/asound.conf and ~/.asoundrc or ~/.config/alsa/asoundrc respectively.
Change will take place as soon as clients re-read the configuration, normally when they are restarted.

ALSA configuration format is notoriously complex, almost Turing complete. It consists of a dictionary containing key/value pairs of names and object of a given type.
For example, the pcm key will contain the list of all PCM definitions, and the ctl key will contain the list of all CTL definitions.

The statements are of the form:

KEY1.KEY2.KEY3... VALUE

KEY1 being one of the object mentioned (pcm, ctl, and others).

The configuration format supports different value types, they could be either string, number, compound (using braces{}), or reference another value.

Additionally, the configuration is hyper flexible, allowing different ways to define the dictionary, which ALSA will later on resolve internally by adding them to its internal global dictionary that has all the keys it accumulated while reading the confs.
For instance, these are equivalent notations, from multiline definition, to using the = sign between params and values, comma or semicolon between consecutive param assignments. Like so:

pcm.a.b 4
pcm.a.c "hi"

is equivalent to

pcm.a {
    b 4
    c "hi"
}

is equivalent to

pcm.a = {
    b = 4;
    c = "hi";
};

The configuration format has special statements that begin with @ such as @func, @hooks, and @args which have different behavior. @func is used to call functions, @hooks to load files, and @args to define internal variables that can be used within a compound variable type.

Example:

{ @func getenv vars [ ENVVAR1 ENVVAR2 ... ] default VALUE }

Will turn into a string from the specified environment variable. Each environment variable is queried and the first to match is used, otherwise VALUE.

Additionally, you can control how ALSA will act when it finds conflicting entries in the dictionary, how it will merge them. This is done by prefixing the key with one of the following:

  • ! the exclamation mark will cause the previous definition to be overridden instead of adding new values, removing all of the param and sub-param. (!pcm would delete all that is under pcm)
  • ? the question mark will ignore the assignment if the param exists
  • + and - respect the type of any earlier assignment, + creates a new param when necessary, - causes error if param didn’t previously exist.

Alright, so now we know how to create a gigantic dictionary of ALSA sound-related objects, how do we actually make use of them?
What we do is create a name under one of these object and give it a type. This type is a plugin that will dictate what to do with this object. The plugins take different configurations depending on what they do, so you’ll have to consult the docs. That gives rise to something like this:

pcm.NAME {
    type TYPE
    ...
}

ctl.NAME {
    type TYPE
    ...
}

slave_pcm.NAME {
    pcm PCM
    ...
}

So ALSA consists mostly of plugins. You can find the external ones that were installed in: /usr/lib/alsa-lib, and the others are internal. For example, check the documentation for the internal pcm plugins.

For obvious reasons, the most important pcm plugin is the hw hardware one, which is used to access the hardware driver. This plugin takes as parameters things that we mentioned such as the card number, the device, subdevice, the format, rate, channels, etc..

Now things should start to make sense: We have clients reading ALSA configurations and manipulating objects having a common interface, which are handled in the backend by plugins, which often end up on the hardware.

Another important plugin is plug which performs channel duplication, sample value conversion, and resampling when necessary. That is needed if a stream has the wrong sample rate for a hardware. You can use aplay -v to make sure the resampling actually happens.

Yet another one is the dmix pcm plugin, the direct mix plugin, which will will merge multiple pcm streams into an output pcm. This is software mixing, which is much more prominent these days compared to hardware mixing.

There really is a long list of fun plugins that can be used in different scenarios, so take a look.

Example:

pcm.plugger {
	type plug
	slave {
		pcm "hw:0,0"
	}
}

This creates a device called plugger that respect the object interface of pcm. Whatever is written or read from this plugger will be handled by the plug plugin, which in turn will use a slave PCM device hw:0,0.
Notice how we used the word “device”, that is because any pcm connected to a hardware corresponds to an ALSA device. It should start making sense now. These pcm, for instance, are the ones that get listed when you issue aplay -L or arecord -L and these are the objects that clients will interact with — they don’t know if these are even connected to a card or not.

The special name default is used to specify the default object interface. So to set, and override, the default playback device you can do:

pcm.!default "hw:CARD"

ALSA provides a lot of these objects preconfigured, as generic device templates. However, sometimes it requires a bit of fiddling to get right and this isn’t obvious to everyone considering the complexity of ALSA configs.
Sometimes it’s also hard to find a plugin for your case. For example, projects like alsaequal creates an audio equalizer, and project alsa_rnnoise creates a pcm device that will remove noise.
These difficulties are part of the reasons why we use sound servers, which we’ll see in their own sections.

So, we’ve seen ALSA’s representation of components, the common objects such as PCM and CTL, how to create them in the flexible configuration format, how the configuration is mostly plugins, how these plugins will use the sound components, and how clients use the PCM without caring what’s happening under the hood.

Open Sound System (OSS) and SADA

OSS, the Open Sound System, is the default Unix interface for audio on POSIX-compatible systems. Unfortunately, like such standards, it isn’t compatible everywhere. It can be perplexing to understand because different systems have branched out of it.

Untill OSS version 3 Linux was using OSS. The company developing it, 4Front Technology, chose in 2002 to make OSSv4 a proprietary software, then in 2007 they re-released it under GPL.
For that reason OSSv4 isn’t used as the default driver of any major OS these days, but can still be installed manually. However, not many applications have support for it and it might requires a translation layer.

In the meantime, Linux had switched to ALSA because of the license issues and the shortcomings of OSS, namely it couldn’t play multiple sounds simultaneously, allocating the sound device to one application at a time, and wasn’t very flexible.
Similarly, in the BSD world some chose to continue to extend OSS and some only got inspired by it to do something else. FreeBSD continued with its fork of OSS by reimplementing the API and drivers and improving it along the way. They added in-kernel resampling, mixing, an equalizer, surround sound, bit-perfect mode (no resampling or mixing), and independent volume control per application.
On the other side, NetBSD and OpenBSD chose to go with their own audio API that is Sun-like (SADA, Solaris Audio API aka devaudio), with an OSS compatibility mode to keep backward compatibility.
Solaris an OpenSolaris use a fork of OSSv4 called Boomer that combines OSS together with Sun’s earlier SADA API, similar to what OpenBSD does.

Due to this arduous history, it is not guaranteed that any of these OS will use a compatible OSS version or OSS layer.

Like ALSA, OSS audio subsystem provides playback, recording, controller, MIDI, and others. We’ve seen that these are mapped to special files by the driver, all starting with /dev/dsp*. /dev/sndstat can be used to list which driver controls which device.
Unlike ALSA where clients interact with PCM and CTL objects, in OSS there is a common API that is used to interact with the special files that were mapped on the filesystem. That means that developers will rely on a more Unix/POSIX like model using the common system calls like open, close, read, write, ioctl, select, poll, mmap, instead of custom library functions.

What these functions do depends on the OS and OSS version. For example, ioctl lets you interact with the generic device type features, as can be seen here. That gives rise to much simpler programs, check this example

One thing the FreeBSD audio frameworks support is the use of mmap to allow applications to directly map the audio buffer, and use ioctl to deal with head/tail synchronization. ALSA on Linux does the same.

Similarly to OSS, on OpenBSD and NetBSD, the 3 exposed devices /dev/audioN, /dev/audioctlN, and /dev/mixerN can be manipulated with read, write, and mostly ioctl. You can take a look in the man 4 audio to get an idea.

When it comes to configuring specific OS related functionalities such as how to mix sound, selecting the sample rates and bit rates, choosing the default output and input hardware, etc.. That depends entirely on the implementation of the particular operating system.

On FreeBSD, audio configurations are available by configuring kernel tunables via sysctl or set statically at boot. For example, dev.pcm.0 is the first instance of the pcm driver and hw.usb.uaudio is the usb audio hardware settings. You’ll known which one is which by consulting /dev/sndstat.
Setting the default sound device on FreeBSD:

sysctl hw.snd.default_unit=n

Where n is the device number.

You can also set the default value for the mixer:

sysctl hint.pcm.0.vol="50"

As you can notice the mixer is part of the pcm driver. This driver supports Volume Per Channel (VPC), that means you can control the volume of each application independently.

As for OSSv4, it offers configuration files for each driver, along with its own set of tools like ossinfo, ossmix, vmixctl, etc..
The configurations can be found under /usr/lib/oss/conf/, the $OSSLIBDIR. It contains audio config files for different drivers with their tunables. It can help set basic settings such as: virtual mixers, quality of sound, sample rate, etc.. You can consult the man page of each driver to check their settings man 7 oss_usb.
Note that OSSv4 isn’t as flexible and requires turning the sound on and off for the settings to take effect.

On OpenBSD and NetBSD, similarly to FreeBSD, settings are done through kernel tunable. The SADA-like system also has multiple tools such as mixerctl and audioctl to make it easier to interact with the audio driver variables.
For example, you can change the sample rate on the fly with:

audioctl -w play.sample_rate=11025

OpenBSD stays true to its security and privacy aspect by disabling recording by default, which can be re-enabled with:

$ sysctl kern.audio.record=1
$ echo kern.audio.record=1 >> /etc/sysctl.conf

As you can see, multiple systems using different APIs and configurations isn’t practical and very limiting. That is why OpenBSD created a layer on top called sndio, a sound server that we’ll discover in a bit.

Overall, we’ve seen the idea of the OSS-like systems. They expose devices in the file system and let you interact with them through the usual Unix system calls. To configure these devices you have to use whatever way the OS gives you, mostly kernel tunables.
While kernel tunables don’t offer the crazy configurability that ALSA does, these audio stacks can still give more or less the same basic features by having them in the kernel. Like on FreeBSD, where the mixing of streams and volume per application control happens out of sight.
As with anything, the lower in the stack it is, the less flexible it is, but the more stable, efficient, and abstract it is.

complexity of system

Sound Servers

Sound servers are neither kernel modules nor drivers, but are daemons that run in user-space to provide additional functionalities. They are there to both raise the flexibility and to have a more abstract and common layer.

The operations a sound server allow range from transferring audio between machines, to resampling, changing the channel count, mixing sounds, caching, adding audio effects, etc.. Having these operations done in a modular way in a sound server is more advantageous than having them in the kernel. Moreover, having a sound server could mean having the same API for all audio regardless of the underlying kernel module or API used. So you won’t have to worry about the interoperability of running on OSS or ALSA or anything else.

There are many sound servers available, some are deprecated like aRts and ESD, others are in use such as sndio, JACK, and PulseAudio, and new ones are coming out and being pushed into distributions like PipeWire. Every one of these has different features, supports for all kinds of protocols, and run on multiple operating system flavors.

sndio

sndio is the default sound server on BSDs today. It is a small, compact, audio and MIDI framework and user-space server developed by OpenBSD. The server is so simple it doesn’t even need a configuration.

sndio’s main roles are to abstract the underlying audio driver and to be a single point of access instead of requiring each application to get raw access to the hardware.
Having a sound server solves the issue of the fracture between all OSS implementations. It creates a new standardized layer.

Let’s mention some of sndio features:

  • Change the sound encoding to overcome incompatibilities between software and hardware. (can change sample rate and bit rate)
  • Conversions, resampling, mixing, channel mapping.
  • Route the sound from one channel to another, join stereo or split mono.
  • Control the per-application playback volume as well as the master volume.
  • Monitor the sound being played, allowing one program to record what other programs play.
  • Use of ticking mechanism for synchronization (maintained after underruns, when buffer isn’t filled fast enough) and latency control
  • Support network connections

sndiod, the server, operates as follows: it creates a sub-device that audio programs connect to as if it was one of the device created by the driver (a real hardware). Thus, during playback or recording, sndiod will handle the audio streams and commands for all programs and take care of the result on the fly.
In sum, sndiod acts as a proxy while giving a similar interface as the kernel API on BSD (read, write, ioctl).

All programs connected to the same sub-device will be part of the same group, which gives a way to process and configure sound according to which sub-device is used.
These sub-devices are defined in a similar string format:

type[@hostname][,servnum]/devnum[.option]

Which, as you can see, allows to connect to a remote host.

Here are some examples:

snd/0
    Audio device of type snd referred by the first -f option passed
    to sndiod(8) server
snd/0.rear
    Sub-device of type snd registered with "-s rear" option
default
    Default audio or MIDI device.

The server sndiod doesn’t have any configuration files, so everything is passed on the command line as an argument. Here are a couple examples of how to do that.

Start the server with a 48kHz sample rate, 240 frame block size (fragment-size), and 2-block buffers (240*2) (buffer-size) (See previous Analog to Digital & Digital to Analog (ADC & DAC) for more info on what these mean), this creates a 10ms latency.

$ sndiod -r 48000 -b 480 -z 240

Start sndiod by creating the default sub-device with low volume (65) and an additional sub-device called max with high volume (127). These will map to snd/0 and snd/0.max respectively.

$ sndiod -v 65 -s default -v 127 -s max

This example create the default sub-device plus another sub-device that outputs to channels 2:3 only (the output speaker will depend on the card). These will map to snd/0 and snd/0.rear respectively.

$ sndiod -s default -c 2:3 -s rear

The sndioctl utility is a helper tool for audio device status and control through sndio. For example, you can change the output level and mute certain sub-devices.

$ sndioctl output.level=+0.1
$ sndioctl output.mute=1
$ sndioctl -f snd/0 output[0].level+=0.1

The commands passed on the right depend on the actual audio device.

sndio is not only available for BSD, it also has a backend for ALSA, so it can run on top of it.
It is generally well supported by major softwares like media players and web-browsers. However, if a program cannot interface with sndio there are ALSA plugins that provice a PCM that can connect to a sndiod server.

In this sections we’ve seen sndio, a very simple sound server that creates sub-devices on the filesystem for any type of audio device, output, input, midi, control, etc.. By arbitraging resources, it is able to calibrate sound streams to fit the hardware sampling and bit rate support. We’ve also seen how to start the server that has no config, and how to use sndioctl to interact with it.

aRts (analog Real time synthesizer) and ESD or ESounD (Enlightened Sound Daemon)

aRts and ESD or ESounD are two deprecated sound servers (audio daemons). Like all sound servers, they accept audio from applications and feed it to the hardware, while manipulating the stream format so that it fits it (resampling and others, you know the drill).

aRts was part of the KDE project and its main big cool feature was that it had a simulation of analog synthesizer.

EsounD was the sound server for the Enlightenment desktop and GNOME. It had similar functionality as any sound server but additionally it had two special cool features: desktop events sound handling, and a mechanism to pipeline audio and videos.

The different teams partnered to synchronize on their projects and have a single sound server. This split EsounD and aRts into pieces: the desktop events sound handling is now libcanberra (see the Libraries section), the pipeline of audio and video is now GStreamer (see the Libraries section), and the sound server was extracted unto PulseAudio (see next section on PulseAudio).

PulseAudio

PulseAudio

PulseAudio tends to trigger online flame wars, which are non-constructive.
Let’s see what PulseAudio is, what features it has, some examples of it, its design, the definition of its internal objects, the sinks and sources, how it represents things we’ve seen such as cards/devices/profiles, how it is configured, how the server and clients start, the modular mechanism, what some modules/plugins do, the compatibility with different protocols, the desktop integration, some of the common frontends, how supported it is, and how to temporarily suspend it.
There’s a long road ahead!

PulseAudio — What Is It?

PulseAudio is a sound server for POSIX OSes, like sndio and others, so its job is similar: abstracting the interaction with the lower layers, whichever backend that is, and offering flexibility in audio manipulation — a proxy for sound applications.
Additionally, PulseAudio’s main focus is toward desktop users, as it was primarily created to overcome the limitations of EsounD and aRts. It is heavily influence by Apple’s CoreAudio design.

So far, PulseAudio is mainly targeted at the Linux desktop but there exists ports to other operating systems such as FreeBSD, Solaris, Android, NetBSD, MacOS X, Windows 2000, and Windows XP. However, some of the features require integration with the system and so are platform-specific, particularly: timer-scheduler, hot-plug features, and bluetooth interaction. The features in PulseAudio come as “external components” or also called “plugins”, so these functionalities aren’t inherent to the core server.

Let’s have a look at a list of features that PulseAudio provides.

  • Extensible plugin architecture (a micro-kernel arch with dynamically loadable modules via dlopen)
  • A toolset to be able to manipulate the sound server on the fly
  • Support for multiple input and output streams
  • Flexible, implicit sample type conversion and resampling
  • Ability to fully synchronize multiple playback streams
  • Support interacting with audio streams of various protocols and backends, be them local or on a network
  • Per application independent volume control
  • Automatic management and setup of audio device, hotplug, via policies and restoration mechanism (mostly ALSA backend only)
  • Sound processing ability and creation of audio pipeline chains: custom modules, mixing, sample rate conversion, echo cancellation, etc..
  • Sample cache: in-memory storage for short sounds, useful for desktop events
  • Low and accurate latency behaviour: uses features such as clocking and rewinding to keep the buffer responsive and avoid glitches. This is done via a timer-based scheduler per-device. (See previous Analog to Digital & Digital to Analog (ADC & DAC) for more info on what these mean) (ALSA backend only)
  • Power saving: due to the use of default latency and timer-based scheduler per-device, there is no need to have a high number of interrupts. (ALSA backend only)
  • Other desktop integration: X11 bells, D-Bus integration, Media role, hardware control, GConf, etc..

In practice that allows to do things like:

  • Automatically setting up a USB headset when it’s connected, remembering the configuration it was in the last time it was used.
  • A GUI for controlling the sound of specific applications and deciding wether to move the audio stream from one device to another on the fly.
  • Dynamically adding sound processing and filters to a currently running application, such as noise cancellation.

Pulseaudio — Overall Design

PulseAudio Engine Layer

The PulseAudio server consists of 3 logical components:

  • A daemon: the piece that configures the core, loads the modules, and starts the main loop
  • A core: based on libpulsecore this is a building block and shared environment for modules.
  • Modules: dynamically loaded libraries to extend the functionality of the server, relying on the libpulsecore library.

Inside the PulseAudio server lives different types of objects that we can manipulate, understanding these objects means understanding how PulseAudio works:

  • format info
  • source
  • source output
  • sink
  • sink input
  • card
  • device port
  • module
  • client
  • sample cache entry

Pulseaudio — Sink, Sink Input, Source, and Source Input

The source output and sinks inputs are the most important concepts in PulseAudio. They are the representation of audio streams. A source device generates a stream that is read/receive to a source output, like a process generating sound or a capture device, and a sink device is written/sent to via a sink input, like a sound card, a server, or a process.
In sum, sinks are output devices and sources are inputs devices: a source will be read unto a “source output” stream and the “sink input” stream will write to the sink device.
There can be virtual devices and virtual streams, and the sink input and source output can be moved from one device to another on the fly. This is possible because of the rewinding feature, each stream having its own timer-scheduler.

Additionally, sink monitors are always associated with a sink and get written to when the sink device reads from its sink inputs.

sink and source

PulseAudio manages these stream in the ALSA backend using a timer-based scheduler to make them efficient for desktop. (See previous Analog to Digital & Digital to Analog (ADC & DAC) for more info on what these mean)
Every source, sink, source output, and sink input can have their own audio parameters, be it sample format, sample rate, channel map, etc.. The resampling is done on the fly and PulseAudio allows to select between different resamplers in its configuration and modules (speex, ffmpeg, src, sox, trivial,copy, peaks, etc..).
It’s good to note that when multiple sink inputs are connected to the same sink then they automatically get mixed.
Each of these components can have their own volume. A configuration called “flat volume” can be set so that the same volume will be used for all sink inputs connected to the same sink.

This flexible concept of streams and devices can be used effectively with a couple of modules that allow juggling with them. For example, the module-loopback forwards audio from a source to a sink, it’s a pair of source output and sink input with a queue in between. If you load it and the source is your microphone you’ll then be able to hear your voice as echo.

pactl load-module module-loopback
pactl unload-module module-loopback # to unload it

Another example is the module-null-source and module-null-sink which will drop data. As with other sinks it will have a monitor associated with it, thus you can convert a sink-input to source-output, basically turning the audio that was supposed to be written to a device back as a readable stream.
We’ll see more examples in the module section, but this is enough to whet your appetite.

You can use the tool pacmd to check each of these and the properties associated with the object:

pacmd list-sources
pacmd list-source-output
pacmd list-sinks
pacmd list-sink-inputs

For now, in the info shown from the above commands, you should at least understand a couple of the information shown such as the latency, the volume, the sample format, the number of channels, the resampling method, and others.

NB: the default source and sink are often abbreviated as @DEFAULT_SOURCE@ and @DEFAULT_SINK in PulseAudio configs and commands.

Another example, changing the volume via pactl of a sink by its index:

pactl set-sink-volume 0 +5%
pactl set-sink-mute 0 toggle

Pulseaudio — Internal Concepts: Cards, Card Profile, Device Port, Device

We got the idea of streams, but we still need to understand the mapping of actual devices into PulseAudio. This is represented by the concept of cards which could be any sound card or bluetooth device. A card has a card pofiles, device ports, and devices.

A card correspond to a mapping related to the driver in use. When using ALSA, this is the same card as an ALSA card.
For example:

aplay -l
card 2: LX3000 [Microsoft LifeChat LX-3000], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

pacmd list-sinks | grep card
  driver: <module-alsa-card.c>
  card: 5 <alsa_card.usb-C-Media_Electronics_Inc._Microsoft_LifeChat_LX-3000-00>
    alsa.card = "2"
    alsa.card_name = "Microsoft LifeChat LX-3000"
    alsa.long_card_name = "C-Media Electronics Inc. Microsoft LifeChat LX-3000 at usb-0000:00:12.0-4, full"

The card has profiles which are, when using ALSA, equivalent to the list of pcm objects of ALSA that are attached to the hardware, which you can list using aplay -L.
You can see the list of profiles discovered by PulseAudio by doing pacmd list-cards and checking the profiles and active profile sections. Note that only one profile can be active for a card at a time.

In practice, a profile is an opaque configuration set of a card, defining the backend-specific configuration of the card, the list of currently available device ports, devices, configurations, and audio parameters. These are all templatized by ALSA as we’ve seen before.
ALSA manages this when it’s the drivers, however PulseAudio doesn’t take the profiles as they are, it sometimes uses a mapping to bind them into a new format that will result in a sink or source being created for the card. This is done via a configuration found in /usr/share/alsa-card-profile and /usr/share/pulseaudio/alsa-mixer/. You can have the mapping of different ALSA objects to PulseAudio ones in there.

For example, I have the following for the MIXER interface, which tells PulseAudio I can use mute, and capture:

/usr/share/alsa-card-profile/mixer/paths/iec958-stereo-input.conf
/usr/share/alsa-card-profile/mixer/paths/iec958-stereo-output.conf

After finding the right profile we can now know what are the device ports. They correspond to the single input or output associated with the card, like a microphone or speaker. Multiple device ports may belong to a card.
Then device, be it a source or sink, is the representation of the currently active producer or consumer, that is a card+a device port. For example, playing audio as digital stereo output on a USB headset.

Another mapping is possible using ALSA UCM (Use Case Manager) to group cards, but we’ll skip over it. Use case management is used to abstract some of the object configuration like the MIXER (higher level management of CTL) so that you can play the same type of sounds together: notifications, media, video, VOIP, etc..

In summary, that gives the following relation between ALSA backend and PulseAudio objects.

  • The PulseAudio ALSA backend will automatically create the cards, card profiles, device ports, sources, and sinks.
  • A PulseAudio card is an ALSA card
  • A PulseAudio card profile is an ALSA configuration for a certain card, this will dictate the list of available device ports, sources, and sinks for PulseAudio. These can be mapped using configs in a dir.
  • A PulseAudio device port defines the active inputs and outputs for a card and other options, it’s the selection of one profile function.
  • Finally, the source and sink are associated with an ALSA device, a single pcm attached to the hardware. A source or sink get connected to a device port and that defines its parameters (sample rate, channels, etc..)

In general, whatever the backend, be it ALSA, OSS, or Bluetooth, PulseAudio’s goal is to find out what inputs and outputs are available and map them to device ports.

Pulseaudio — Everything Is A Module Thinking

As far as the PulseAudio server is concerned, it only manipulates its internal objects, provides an API, and doesn’t do anything else than host modules. Even the backends like ALSA are implemented as modules.
That gives rise to a sort of micro-kernel architecture where most of the functionalities in the server are implemented in modules, and there’s a module for everything. Most of what we’ve mentioned already is done via a module. Let’s still show a list of some of them.

  • Device drivers
  • Protocols
  • Audio routing
  • Saving information
  • Trivia like x11 bell
  • Volume control
  • Bluetooth
  • Filters and Processing

Some modules are autoloaded in the server like the module-native-protocol-unix, which is PulseAudio’s native protocol, and others are loaded on the fly.
Even the protocol to load modules and interact with the server via the command-line interface is a module in itself: module-cli-protocol-unix/tcp.
If you are interested in knowing about the ALSA backend integration, it is done by the module-alsa-card.

There’s really a lot of modules, the list of the ones that come with the PulseAudio server default installation can be found here and here. There are also many user contributed modules, which you can place on disk in your library path /usr/lib/pulse-<version>/modules/.

To list the currently loaded modules use the following:

pacmd list-modules

NB: It can be quite eery to see that PulseAudio has its own module mechanism while we’ve seen earlier that ALSA does a similar thing through its configuration. However, keep in mind that PulseAudio is relatively easier to use, can work on top of different backends, not only ALSA, and has a different concept when it comes to audio streams (source-output and sink-input).

Now let’s see how to load new modules and configure them.

Pulseaudio — Startup Process And Configuration

Before we see how to load modules into the server, we first need to check how to run the server.
The PulseAudio server can either run in system-wide mode or per-user basis. The latter is preferred as it is better for desktop integration because some modules use the graphical desktop. It is usually started during the setup of the user session, which is taken care of by the desktop environment autostart mechanism. These days, with the advent of the systemd framework project, PulseAudio is often launched as a user service.

$ systemctl --user status pulseaudio
● pulseaudio.service - Sound Service
     Loaded: loaded (/usr/lib/systemd/user/pulseaudio.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-02-06 14:36:22 EET; 19h ago
TriggeredBy: ● pulseaudio.socket
   Main PID: 2159374 (pulseaudio)
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/pulseaudio.service
             ├─2159374 /usr/bin/pulseaudio --daemonize=no --log-target=journal
             └─2159380 /usr/lib/pulse/gsettings-helper

Another way to start the PulseAudio server is to not start it all. That’s surprising, but with the default configuration clients will autospawn the server if they see that it’s not running.
The reverse is also true, there is a configuration to autoexit when no clients have used the server for a certain period.

This is how clients start:

  • Initialization: finding the server address from somewhere (environment variable, X11 root window, per-user and system-wide client conf files)
  • connect: depending on the protocol used (native, tcp localhost, or remote)
  • autospawn: if enable spawn a server automatically
  • authenticate: using cookies found somewhere (environment variable, X11 root window, explicit, per user or system wide conf, per-use home dir)

The server starts by reading the server configurations, and then loading the modules found in the configuration associated with its running stance (system mode or per-user).
The server configurations are found by first looking in the home directory ~/.config/pulse, and if not found then by looking in the system-wide config in /etc/pulse. The directory will contain the following configuration files: daemon.conf default.pa system.pa and client.conf.

daemon.conf: contains the settings related to the server itself, things like the base sample rate to be used by modules that will automatically do resampling, the realtime scheduling options, the cpu limitation, if flat-volume will be used or not, the fragment size, latency, etc.. These cannot be changed at runtime.
You can consult pulse-daemon.conf(5) manpage for more info.

client.conf, this is the file that will be read by clients, which we mentioned above. It contains runtime options for individual clients.
See pulse-client.conf(5) manpage for more info on this one.

default.pa and system.pa are the per-user and system-wide startup scripts to load and configure modules. Once the server has finished initializing, it will read and load the modules from this file.
You can also load and manipulate the modules using tools such as pactl and pacmd, see pulse-cli-syntax(5) manpage for more info.
The .conf files are simple key-value formatted files while the .pa are real command scripts following the CLI protocol format of PulseAudio.

Example:

load-sample-lazy x11-bell /usr/share/sounds/freedesktop/stereo/bell.oga
load-module module-x11-bell sample=x11-bell

When it comes to realtime scheduling, you can either integrate PulseAudio by giving it priority at the OS level, or you can rely on its integration with RealtimeKit (rtkit), which is a D-Bus service that changes the scheduling policy on the fly.
Realtime policy will be applied to all sink and source threads so that timer-based scheduling have lower latency. This is important if you want to play audio in bit-perfect mode, that is about not applying any resampling or mixing to the audio but playing it directly as is.

Pulseaudio — Interesting Modules And Features

Let’s now have a look at a couple features and modules. We can’t list them all as there are so many but let’s try to do a roundup of the most interesting ones.

PulseAudio can talk over many protocols by using different plugins, that includes:

  • Native protocols over different transport (fd, unix, tcp)
  • mDNS (Zeroconf)
  • RTP/SDP/SAP
  • RAOP
  • HTTP
  • DLNA and Chromecast (Digital Living Network Alliance)
  • ESound

It also offers control protocols to manage the server itself and audio streams:

  • D-Bus API
  • CLI protocol

There are many modules for post-processing and effects because it’s easy to create a chain of sound. Though only two types of connections are allowed: source output are connected to source, and sink input to sink. That means you’ll sometimes need to create indirect adapters to have the scenario you want.
If you need more advanced chains you are probably better off going to another sound server that specializes in these like JACK.

PulseEffects is a popular software to add audio effects to stream, but note that it is getting deprecated in favor of PipeWire.
The LADSPA plugin called module-ladspa-sink allows to load the audio processing effects in the common format we’ve seen, and apply them to a sink.
There are a couple different equalizers such as the integrated one and others like prettyeq. An equalizer works by becoming the default output/sink.
There are also noise cancellation filters such as the builtin one module-echo-cancel and NoiseTorch

Some cool desktop integration features:

  • sample cache, basically loading a small sound sample in the server and cache it (pacmd list-samples)
  • multimedia buttons
  • publish address on X11 root window properties
  • x11 bell integration, using XKB bell events and played from sample cache
  • Use of GNOME registry to load modules instead of .pa configuration files.
  • Hotplug based on udev/jackbus/coreaudio/bluetooth/bluez5/bluez4 so that cards are automatically detected.

Let’s go over two less known but cool features of PulseAudio: The restoration DB and the routing process.

PulseAudio keeps track and restores the parameters used for cards, devices, and streams. When a new object appears the server tries to restore the previous configuration and might move the streams to another device based on what it has seen before.
This neat automatic setup is done via an embedded db, which the user can choose the format. This is done via the following plugins: module-device-restore, module-stream-restore, and module-card-restore. You’ll find the files ending in *.tdb in your ~/.config/pulse/ if you are using the default configuration. (You can use tdbtool to inspect them if you’re interested)

The decision regarding this automatic setup is influence by the media role and other properties associated with the stream and device such as application id and name.
This information is set programmatically on the stream when communicating with the server, for example a stream can be set as video, music, game, event, phone, animation, etc.. (Sort of like the use case scenario)
So based on this information in the restore db, the routing will select which source or sink is best for a new stream.

The actual algorithm, that takes care of this isn’t obvious, I advise looking at these two flow charts for more details.

initial route new device route

This can be confusing if you are trying to set a default device because the default device is only used as fallback when the restore db is in place.

Pulseaudio — Tools

There are many tools that can be used to interface with PulseAudio, some are full front-ends and some are more specific.

We’ve seen pacmd and pactl which both are used to reconfigure the server at runtime.
paplay, parec, pacat, pamon, and others are mini-tools used to test features of PulseAudio.
There are GUIs like pamixer, paprefs (useful to setup simultaneous output), pavucontrol.
There are TUI like pulsemixer.

Longer lists can be found here for GUI and here for CLI.

Pulseaudio — Suspending

Sometimes it is useful to temporarily suspend PulseAudio. The utility pasuspender has this purpose. It is especially useful when running JACK in parallel with PulseAudio.

Another way is to use the D-Bus reservation API to allocate a card to a certain application. This can be done more easily when you include the module for JACK within PulseAudio.

JACK

catarina

JACK is a sound server, just like sndio, ESD, PulseAudio, and others, but is designed for professional audio hardware and software. JACK can run on top of many drivers, including on top and along PulseAudio. While PulseAudio is designed for consumer audio for desktop and mobile, JACK is made for music production.

JACK recursive acronym stands for JACK Audio Connection Kit, and as the name implies it specializes in connecting multiple hardware and virtual streams together. As you remember, this was not so simple in PulseAudio. It allows to setup real-time, low-latency connections between streams, and ease the configuration regarding parameters like buffer size, sample rate, fragment size, and others.

The biggest strength of JACK is its integration with professional tooling and its emphasis on MIDI and professional hardware. In a professional environment you often have to deal with multiple devices such as mixers, turntables, microphones, speakers, synthesizer, etc.. The graphical interfaces that come around the JACK server allow to handle this easily but you have to be knowledgeable in the audio world to understand the complex configuration.

JACK also has support for specific professional hardware drivers, like a FireWire driver (IEEE 1394) that PulseAudio doesn’t have.

qjackctl

JACK frontends and software using it are really where it shines, there are so many interfaces and GUIs for professionals. The most widely used software to configure it being qjackctl, a connection manager making links between streams and devices. That’s because JACK separates the concerns: one part is about managing the connections, in a graph-like fashion, and the other part is only concerned with passing the audio stream around. This is especially important when there’s a lot of equipment and should be easily doable via a GUI.
Let’s mention some professional audio engineer software:

  • mpk - Virtual MIDI Piano Keyboard
  • KX.studio Cadence - A studio with multiple sub-tools like Cadence and Claudia
  • Patchage - visual connection manager
  • Catia - anoter visual connection manager
  • ardour
  • Qtractor
  • Carla
  • QASMixer
  • bitwig studio
  • drumstick
  • QSynth
  • Helm
  • Calf Studio Gear
  • LMMS

Here is a longer list.

There are also a bunch of specialized Linux distributions:

The audio domain, for sound engineers and musicians, is gigantic and it’s not really my place to talk about it, so I’ll keep it at that.

PipeWire

PipeWire is a relatively new sound-server but is also much more. It not only handles audio streams but video streams too, it is meant as a generic multimedia processing graph server.
On the audio side, it will try to fill the need of both desktop users, like PulseAudio, and professional audio engineers, like JACK.

Initially the project was supposed to be named PulseVideo and do a similar job to PulseAudio but for Video. The rational to handle all multimedia streams together is that it makes no sense to handle video streams without the audio counter-part to sync them together.

The project was started by GStreamer’s creator. The library, as you may remember, already handles audio and video streams.
So in sum, that creates an equation that includes GStreamer+PulseAudio+PulseVideo+JACK-like-graph. The integration with GStreamer means that applications that already use it will automatically be able to interoperate with PipeWire.

The project is still in early development, not so stable, and the server side currently only supports video and has integration layers with PulseAudio and JACK. Otherwise, it can interfaces with ALSA clients directly, through a new ALSA pcm device that redirects to PipeWire (Like PulseAudio does).

Some of the new goals of PipeWire is that it will give access to media to sandboxed Flatpack applications and allow Wayland compositors to access media streams securely via a mechanism like PolKit for granular access control. This is called a policy/session manager.

PipeWire takes ideas from multiple places. In itself it only cares about creating a graph of nodes that will process and move audio streams around via IPC. Meanwhile, another process, like with JACK, will take care of managing the connections, device discovery, and policies.
So you get a division of roles, a processing media graph and a policy/session manager.

The innovation lies in this graph model taken from JACK, combined with integration of policy management and desktop usage. Each node in the graphs has its own buffer mechanism and dynamic latency handling. That leads to a lower CPU usage because of the timer-based scheduling model that wakes up nodes only when they need to and can dynamically adapt buffers depending on the latency needed.

What’s all this talk about nodes and graphs about, what does this actually mean?

pipewire and WirePlumber pipewire

To understand this we have to get 2 concepts: the PipeWire media graphs and the session management graphs.

PipeWire is a media stream exchange framework, it embodies this concept through the media graph that is composed of nodes that have ports which are connected through directed links. The media streams flows from node to nodes passing by their ports via their links to reach the port of another node.
A node is anything that can process media, that either consumes it or produces it. Each node has its own buffer of data and personal preferences and properties such as media class, sample rate, bit rate, format, and latency. These nodes can be anywhere, not limited to inside the PipeWire daemon, but can also be external inside clients. That means the processing of streams can be delegated to other software and passed around from node to node without PipeWire touching the stream. Nodes can be applications, real devices, virtual ones, filters, recording application, echo-cancellation, anything that dabbles with media streams.
To interact with the rest of the system, a node can have ports, which are interfaces for input (sink) or output (source).
A link is a connection between 2 ports, one as the source the other as the sink.

So far that’s a very generic concept, nodes that handle media and have ports that can generate media stream or consume it.

Whenever some media needs to be handled by a node, they are woken up by a timer-scheduler mechanism. The connected nodes form a graph, and one of these nodes usually “drives” the graph by starting the processing for all other nodes joined in it. This timer dynamically manages the desired latency for each node to negotiate the most appropriate one within a range.
When two nodes in a graph need to communicate with one another, they have to negotiate a common preferred format and minimum latency based on their buffer size. That’s why the nodes are normally wrapped in an adapter that will automatically do the conversion (sample rate, sample format, channel conversion, mixing, volume control). This is all done dynamically which is good for desktop usage, but not so much for pro-audio.
More info about the buffer model here

In practice, that gives a graph that looks like the following, dumped using pw-dot(1) utility:

pw-dot

So far so good, we have nodes that exchange streams, a graph processing media, but how do we get these nodes in there in the first place and how do we choose which one links to which one? We need a way to attach these nodes, decide what is connected to what. Like when a client attaches and asks to play an audio stream, how is that handled?
That’s where the second piece of the PipeWire equation comes in: the session connector and policy controller along with its session management graph.

This piece of software is external to PipeWire, it can have multiple implementations. The default one that comes with the installation is called pipewire-media-session, another one that is still a work-in-progress is called wireplumber. There are talks about including it in desktop environment session managers such as gnome-session-daemon.

The role of this software is to keep track of the devices available, their priorities, keeping track of which application uses which device, ensuring policy control and security, keep a device and stream restoration database (not implemented), share global properties of the system, find the default endpoints used by clients when asked to play sound, etc..

When clients connect to PipeWire they announce their session information, what they’d like to do (playback or capture), the type of media they want to handle (video, playback, VOIP, etc..), their preferred latency and sample rate, etc..
Based on this information, the session manager can shortlist which device the client needs.

As soon as the client connects, if its session information is new (PID,GID,UID), it will first be frozen until its permissions are acknowledged. PipeWire default session manager pipewire-media-session comes with a series of modules that take care of this called module-portal and module-access. The portal module will, through desktop integration (via D-Bus, like PolKit), open a pop-up to ask for user confirmation (read,write,execute). After that, the session manager will configure the client permissions in its client object.
So clients are not able to list other nodes or connect to them until the session manager approves.

The session manager can then choose to restore connections based on previous usage of this stream — decide how to connect it, make sure it’s linked to the appropriate device and follows the peering rules for its use case (depending on media type). Then this new node can get connected and configured in the media graph.

That is as far as clients are concerned, however, PipeWire doesn’t open any device by default either, and it is also the role of the session manager to load devices, configure them, and map them on the media graph.

To achieve this flexibly, some session managers can use what is called a session management graph. In practice, this is the equivalent to how PulseAudio manages devices through the concept of cards and profiles that can create sink and source nodes but with the extra addition of routing based on use case. Internally, the session manager actually reuses the related PulseAudio code and config for device management.

The session management graph is a representation of this, the high-level media flow from the point of view of the session manager. As far as I can see, these graphs are hosted within PipeWire along other objects but they have different types.

+---------------------+                                +----------------------+
|                     |                                |                      |
|            +----------------+  Endpoint Link  +----------------+            |
|  Endpoint  |Endpoint Stream |-----------------|Endpoint Stream |  Endpoint  |
|            +----------------+                 +----------------+            |
|                     |                                |                      |
+---------------------+                                +----------------------+

Endpoints are where media can be routed to or from (laptop speaker, USB webcam, Bluetooth headset mic, amplifier, radio, etc..). They then get mapped to nodes in the media graph, but not always.
They can be mutually exclusive, this is the equivalent of device ports, which as you remember correspond to a single input or output associated with the card. So the Endpoint is a card+a device port in theory.

The Endpoint Stream are the logical path, the routing associated with a use case (Music, Voice, Emergency, Media, etc..). They are equivalent to PulseAudio sink/source on the device side, and sink-input/source-output on the client side.
These can be used to change the routing in the media graph.

The Endpoint Link is what connects and creates the media flow, it can only exist if there are actual links in the media graph or if the link exists physically (real hardware connection).

The session manager is then responsible of knowing which devices are present, what they support, what kind of linking information is there, and if streams need compatibility between them, and share that information if needed.

Additionally, internally the session manager can put on the graph objects of type Device which map to the ALSA cards, JACK clients, or others. Like cards in PulseAudio.

Now let’s see how to configure PipeWire and its session/policy manager.

When the PipeWire daemon starts it reads the config file located at $PIPEWIRE_CONFIG_FILE, normally /etc/pipewire/pipewire.conf. It contains sections, some to set server values, some to load plugins and modules, some to create objects, and some to automatically launch programs.
The goal of this configuration file is to make it easy to configure how the processing happens in the media graph.

The execution section is normally used to automatically launch the session manager.

There are configurations related to how the graph will be scheduled such as the global sample rate used by the processing pipeline, which all signal will be converted to: default.clock.rate. The resampling quality can be configured server side too (even though the node are wrapped in an adapter that does that) in case it needs to be done. This resampler is a custom highly optimized one. Moreover, you can control the buffer size minimum and maximum value through a min-quantum, max-quantum, and default quantum, which are going to be used to dynamically change the latency.

default.clock.quantum =		1024
default.clock.min-quantum =	32
default.clock.max-quantum =	8192

NB: PipeWire relies on plugins that follow the SPA, Simple Plugin API, based on GStreamer plugin mechanism but lighter.
Most of them can be found in /usr/lib/spa-<version>.

Now as far as the session manager is concerned, it highly depends on the implementation. It is about modules for policy and matching rules to associate them with specific actions to do in the media graph or the session management graph. Monitor subsystems watch when a new device or stream appear (new node), or when the system creates a new object, and decides what to do with it based on the endpoint configuration.

For example, I have the following rule for WirePlumber in 00-default-output-audio.endpoint-link:

[match-endpoint]
media_class = "Stream/Output/Audio"

[target-endpoint]
media_class = "Audio/Sink"

Which will attach a new endpoint of class “Stream/Output/Audio” to the default endpoint with class “Audio/Sink”.

However, this all depends on the session manager implementation.

At this point it’s easy to picture that this system would be fantastic to create filters and effects streams, however currently this is still very hard to do. So far, the only way to achieve this is with the help of PulseAudio tools such as pactl.

You can create sinks and sources with specific media classes so that they map within PipeWire.
For example:

pactl load-module module-null-sink object.linger=1 media.class=Audio/Sink sink_name=my-sink channel_map=surround-51

pw-cli can also be used instead:

pw-cli create-node adapter { factory.name=support.null-audio-sink node.name=my-mic media.class=Audio/Duplex object.linger=1 audio.position=FL,FR }

It remains that PipeWire is missing the interface toolset to easily interact with it. There aren’t any good sound configuration tool that permits to inspect and manipulate it so far. Moreover, the ones that will have to do this will need to be able to portray the internal connection mechanism, similar to JACK’s many connection managers.

I quote:

There is currently no native graphical tool to inspect the PipeWire graph but we recommend to use one of the excellent JACK tools, such as Carla, catia, qjackctl, … You will not be able to see all features like the video ports but it is a good start.

PipeWire comes with a set of mini debug tools similar to what PulseAudio provides, they start with the pw-* prefix:

  • pw-cli - The PipeWire Command Line Interface
  • pw-dump - Dump objects inside PipeWire
  • pw-dot - The PipeWire dot graph dump in graphviz format
  • pw-mon - The PipeWire monitor
  • pw-cat - Play an Record media with PipeWire
  • pw-play - Like pw-cat but for play only
  • pw-metadata - The PipeWire metadata
  • pw-profiler - The PipeWire profiler
  • pw-top - Acts like top but for devices nodes inside PipeWire

The most useful tools are pw-cli, pw-dump and pw-dot.

Try:

pw-cli info 0

Here’s an extract from pw-dump showing an Endpoint of class “Audio/Source”, a microphone on the boring headset you’ve encountered in this article.

  {
    "id": 53,
    "type": "PipeWire:Interface:Endpoint",
    "version": 0,
    "permissions": [ "r", "w", "x", "m" ],
    "props": {
      "endpoint.name": "alsa_card.usb-C-Media_Electronics_Inc._Microsoft_LifeChat_LX-3000-00.capture.0.0",
      "media.class": "Audio/Source",
      "session.id": 75,
      "client.id": 39,
      "factory.id": 25
    }
  },

Overall, PipeWire is an interesting sound server, combining a media processing graphs framework along with an external policy/session/connection manager that controls it. The timer and dynamic latency mechanism should have a significant effect on CPU usage.

Unfortunately, after testing it you can clearly see that it is still in its early stage but that it integrates well on the audio part through the backward compatibility with PulseAudio.
Additionally, it remains to be seen if the tooling around it will adapt properly to the graph thinking. Will they build around the concept or dismiss it entirely considering most desktop tools today aren’t used to long sequence of media processing, and neither are users.
Finally, on the session/connection manager side we need more innovation. What is currently available seems to be lacking. I couldn’t find much documentation about the restoration DB mechanism, hot-plug, desktop integration, caching of sample sounds for events, and others.

It’s The Same On All OS - Conclusion

Anybody who claims one system offers better audio “quality” is just plain wrong and base their assumption on something non-scientifically proven.

All the low-level stacks are relatively the same speed when running in bit-perfect mode. The big differences between all that we’ve seen relates to the driver support, the ease of use, the desktop integration, and the buffer/latency management.
Some systems are targeted at end users and others at audio engineers.

According to measurements from A Look at Linux Audio (ALSA, PulseAudio) for instance, ALSA performs very well on Linux and keeps up with a Windows machine that is much more performant. Tests with PulseAudio are similar but use 6% more CPU processing.

Whether in the past with Mac OS X, or Windows, and now Linux, there is no evidence that operating systems make any difference to sound quality if you’re playing “bit perfect” to the hardware directly (ie. ALSA to DAC with no software conversion similar to Windows ASIO, Kernel Streaming or WASAPI)

The discussion then rotates around low-latency using real-time scheduling, better IO, using better sampling size, etc.. (See Analog to Digital & Digital to Analog (ADC & DAC) for more info on what these mean).

The audio stack is fragmented on all operating systems because the problem is a large one. For example, on Windows the audio APIs being ASIO, DirectSound and WASAPI.
Perhaps MacOs has the cleanest audio stack, CoreAudio, but nobody can clearly say if they can’t look at the code. PulseAudio was inspired by it.
The stack of commercial operating systems are not actually better or simpler.

Meanwhile, the BSD stack is definitely the simplest, even though there are discrepancies between the lowest layers and lack of driver support, sndio makes it a breeze.

Linux is the platform of choice for audio and acoustic research and was chosen by the CCRMA (Center for Computer Research in Music and Acoustics).

Let’s conclude, we’ve seen basic concepts about audio such as the typical hardware component, how audio is transferred and converted from the real world to the digital world through digital-to-analog-converters. We’ve seen the issue about buffering and fragment size. Then we’ve taken a look at different libraries that can act as translation layers, as processing helper, or as standard format to write filters. After that we went through the drivers: ALSA and OSS, the crazy configuration format and plugins of ALSA and the internal concepts it has to map devices. On the OSS and SADA side, we’ve seen the historical fracture and how things could be done mainly hidden away inside the kernel via tunable to not freak out the users. Finally, we’ve attacked sound servers, from sndio, to the deprecated aRts and ESD, to PulseAudio, to JACK, and lastly PipeWire. Each of them has its specialty, sndio being extremely simple, PulseAudio being great for the desktop integration use case, JACK catering to the audio engineers having too much equipment to connect together and having superb integration with professional tools, and PipeWire that is getting inspired by JACK’s graphs but wants to take it a step further by including video streams, integrating with the desktop and making things more snappy with wake-up node driving the processing graph.

・゜゚・(^O^)/・゜゚・

That’s it, I hope you learned a thing or two in this post. Let me know what you think!






References

Generic:

Libraries:

OSS:

ALSA:

sndio:

Pulseaudio:

JACK:

Pipewire:

It’s all the same:

Gokberk Yaltirakli (gkbrk)

Drawing on the spectrum February 06, 2021 09:00 PM

Software Defined Radio software usually comes with a waterfall view (spectrogram) that lets the user quickly inspect the spectrum. The spectrogram plots the amplitude of frequencies over time. This means, by carefully outputting a signal consisting of multiple frequencies, we can draw shapes and pictures on the spectrogram.

A common NFM walkie-talkie is too limited to do this, but a Software Defined Radio that can transmit arbitrary I/Q samples will do the job perfectly. Fortunately I have a hackrf-one at hand, so I gave this a try.

In order to transmit from the HackRF, I will be using the hackrf_transfer command. This means all I’ll need to do in my modulator is to output I/Q samples to stdout. Let’s make a quick helper method to do this.

Writing samples

Traditionally, DSP samples are kept between -1 and 1, so we will be using this format internally. In order to give them to hackrf_transfer, we need to encode them as a signed 8-bit integer. The format accepted by the program is alternating 8-bit signed I and Q samples.

import struct, os

dsp = os.fdopen(1, 'wb')

def write(i, q):
    i = int(i * 127)
    q = int(q * 127)
    data = struct.pack('bb', i, q)
    dsp.write(data)

Configuration

Let’s also define some constants; such as the output sample rate, the maximum frequency deviation, and how long it should take to transmit the image. The frequency deviation determines how wide our signal will be on the spectrum and the transmission time will determine the height. You should play around with these values until you can get a clear image.

RATE = 2_000_000 # 4M sample rate
TRANSMIT_TIME = 2 # 2 Seconds
FREQ_DEV = 15_000 # 15 KHz

Loading the image

With the configuration out of the way, we are now ready to produce the samples. The first thing we need to do is to read an image file. To do this, I will be using the Pillow library. Let’s get the image file path from the command line arguments, load the image and convert it to a black and white bitmap.

from PIL import Image
import sys

im = Image.open(sys.argv[1])
im.convert('1') # 1 means a 1-bit image

Outputting the image

We need to output the image bottom-to-top because the spectrogram will put the signals received earlier at the bottom, as it scrolls like a waterfall.

t = 0

for y in range(im.height)[::-1]:
    target = t + TRANSMIT_TIME / im.height
    
    while t < target:
        # Output line...
        pass

Every line, we pick a target time. We will be outputting samples for the current line until we reach target. Each line gets TRANSMIT TIMEIMAGE HEIGHT \frac{\text{TRANSMIT TIME}}{\text{IMAGE HEIGHT}} IMAGE HEIGHTTRANSMIT TIME seconds.

First of all, let’s cache the pixels of the current line since Python is not very fast.

line = [im.getpixel((x, y)) for x in range(im.width)]

When we are outputting the line, we’ll pretend that each pixel of the image is a frequency in out output. So for an image with the width of 300 and frequency deviation of 5000 Hz; x=0x = 0x=0 is offset by 0 Hz, x=150x = 150x=150 is offset by 2500 Hz and x=299x = 299x=299 is offset by 5000 Hz.

Using the mapping we described above, let’s accumulate I and Q values for all the pixels.

i = 0
q = 0

for x, pix in enumerate(line):
    if not pix:
        continue
    offs = x / im.width
    offs *= FREQ_DEV
    i += math.cos(2 * math.pi * offs * t) * 0.01
    q += math.sin(2 * math.pi * offs * t) * 0.01

write(i, q)
t += 1.0 / RATE

We can represent a wave of a particular frequency in time using the well-known formula 2π⋅freq⋅time2\pi \cdot \text{freq} \cdot \text{time}2πfreqtime. Since I is the cosine of the value and Q is the sine, our final values become sin⁡(2π⋅f⋅t)\sin \left( 2\pi \cdot f \cdot t \right)sin(2πft) and cos⁡(2π⋅f⋅t)\cos \left( 2\pi \cdot f \cdot t \right)cos(2πft).

We don’t output anything for lines where the pixel value is 0. We multiply the signals we add to I and Q (i.e. dampen them) by 0.1 in order to prevent the signal from excessive clipping. This approach actually has some downsides, as the signal might still clip for certain images, but for a short demo where we can pick the images and change the dampening factors it won’t be a problem.

Now let’s combine the code snippets so far and try to render a signal. I recommend not transmitting this in real-time as Python is slow, and using PyPy as Python is slow.

$ pypy3 ./pic2spec.py btc.png > btc.raw
... Wait a lot
$ hackrf_transfer -f 433000000 -t btc.raw -s 4000000 -a 1

Results

Here’s a video of what our signal looks like on gqrx.

Your device does not support video playback.

Code

Here’s the full code, if you want to try this on your own.

#!/usr/bin/env python3
import struct
import os
from PIL import Image
import sys
import math

dsp = os.fdopen(1, "wb")


def write(i, q):
    i = int(i * 127)
    q = int(q * 127)
    data = struct.pack("bb", i, q)
    dsp.write(data)


RATE = 4_000_000  # 4M sample rate
TRANSMIT_TIME = 2  # 2 Seconds
FREQ_DEV = 15_000  # 15 KHz


im = Image.open(sys.argv[1])
im.convert("1")  # 1 means 1-bit image


t = 0

for y in range(im.height)[::-1]:
    target = t + TRANSMIT_TIME / im.height

    line = [im.getpixel((x, y)) for x in range(im.width)]
    while t < target:
        i = 0
        q = 0

        for x, pix in enumerate(line):
            if not pix:
                continue
            offs = x / im.width
            offs *= FREQ_DEV
            i += math.cos(2 * math.pi * offs * t) * 0.01
            q += math.sin(2 * math.pi * offs * t) * 0.01
        write(i, q)
        t += 1.0 / RATE

February 03, 2021

Maxwell Bernstein (tekknolagi)

Inline caching: quickening February 03, 2021 12:00 AM

In my last post I discussed inline caching as a technique for runtime optimization. I ended the post with some extensions to the basic technique, like quickening. If you have not read the previous post, I recommend it. This post will make many references to it.

Quickening involves bytecode rewriting — self modifying code — to remove some branches and indirection in the common path. Stefan Brunthaler writes about it in his papers Efficient Interpretation using Quickening and Inline Caching Meets Quickening.

The problem

Let’s take a look at a fragment of the caching interpreter from the last post so we can talk about the problem more concretely. You can also get the sources from the repo and open interpreter.c in your preferred editor.

void add_update_cache(Frame* frame, Object* left, Object* right) {
  Method method = lookup_method(object_type(left), kAdd);
  cache_at_put(frame, object_type(left), method);
  Object* result = (*method)(left, right);
  push(frame, result);
}

void eval_code_cached(Frame* frame) {
  // ...
  while (true) {
    // ...
    switch (op) {
      // ...
      case ADD: {
        Object* right = pop(frame);
        Object* left = pop(frame);
        CachedValue cached = cache_at(frame);
        Method method = cached.value;
        if (method == NULL || cached.key != object_type(left)) {
          add_update_cache(frame, left, right);
          break;
        }
        Object* result = (*method)(left, right);
        push(frame, result);
        break;
      }
      // ...
    }
    frame->pc += kBytecodeSize;
  }
}

As I also mentioned last post, the ADD opcode handler has three cases to handle:

  1. Cache is empty
  2. Cache has the wrong key
  3. Cache has the right key

Since Deutsch & Schiffman found that types don’t vary that much, the third case is the fast path case. This means that we should do as little as possible in that case. And right now, we’re doing too much work.

Why should we have to check if the cache slot is empty if in the fast path it shouldn’t be? And why should we then have to make an indirect call? On some CPUs, indirect calls are much slower than direct calls. And this assumes the compiler generates a call instruction — it’s very possible that a compiler would decide to inline the direct call.

Quickening is a technique that reduces the number of checks by explitly marking state transitions in the bytecode.

Removing the empty check

In order to remove one of the checks — the method == NULL check — we can add a new opcode, ADD_CACHED. The ADD_CACHED opcode can skip the check because our interpreter will maintain the following invariant:

Invariant: The opcode ADD_CACHED will appear in the bytecode stream if and only if there is an entry in the cache at that opcode.

After ADD adds something to the cache, it can rewrite itself to ADD_CACHED. This way, the next time around, we have satisfied the invariant.

G ADD ADD ADD_CACHED ADD_CACHED ADD->ADD_CACHED non-int observed ADD_CACHED->ADD_CACHED

Let’s see how that looks:

void eval_code_quickening(Frame* frame) {
  // ...
  while (true) {
    // ...
    switch (op) {
      // ...
      case ADD: {
        Object* right = pop(frame);
        Object* left = pop(frame);
        add_update_cache(frame, left, right);
        code->bytecode[frame->pc] = ADD_CACHED;
        break;
      }
      case ADD_CACHED: {
        Object* right = pop(frame);
        Object* left = pop(frame);
        CachedValue cached = cache_at(frame);
        if (cached.key != object_type(left)) {
          add_update_cache(frame, left, right);
          break;
        }
        Method method = cached.value;
        Object* result = (*method)(left, right);
        push(frame, result);
        break;
      }
    // ...
    }
    frame->pc += kBytecodeSize;
  }
}

Not too different. We’ve shuffled the code around a little bit but overall it looks fairly similar. We still get to share some code in add_update_cache, so there isn’t too much duplication, either.

Now that we’ve moved the empty check, it’s time to remove the indirect call.

Removing the indirect call

Let’s assume for a minute that you, the writer of a language runtime, know that most of the time, when people write a + b, the operation refers to integer addition.

Not many other primitive types implement addition. Frequently floating point numbers use the same operator (though languages like OCaml do not). Maybe strings. And maybe your language allows for overloading the plus operator. But most people don’t do that. They add numbers.

In that case, you want to remove as much of the overhead as possible for adding two numbers. So let’s introduce a new opcode, ADD_INT that is specialized for integer addition.

In an ideal world, we would just be able to pop two objects, add them, and move on. But in our current reality, we still have to deal with the possibility of programmers passing in a non-integer every once in a while.

So first, we check if the types match. If they don’t, we populate the cache and transition to ADD_CACHED. I’ll get to why we do that in a moment.

And if we did actually get an int, great, we call this new function do_add_int.

void do_add_int(Frame* frame, Object* left, Object* right) {
  Object* result = int_add(left, right);
  push(frame, result);
}

void eval_code_quickening(Frame* frame) {
  // ...
  while (true) {
    // ...
    switch (op) {
      // ...
      case ADD_INT: {
        Object* right = pop(frame);
        Object* left = pop(frame);
        if (object_type(left) != kInt) {
          add_update_cache(frame, left, right);
          code->bytecode[frame->pc] = ADD_CACHED;
          break;
        }
        do_add_int(frame, left, right);
        break;
      }
    // ...
    }
    frame->pc += kBytecodeSize;
  }
}

This is a nice opcode handler for ADD_INT, but right now it’s orphaned. Some opcode has to take the leap and rewrite itself to ADD_INT, otherwise it’ll never get run.

I suggest we make ADD do the transition. This keeps ADD_CACHED fast for other types. If ADD observes that the left hand side of the operation is an integer, it’ll call do_add_int and rewrite itself.

G ADD ADD ADD_INT ADD_INT ADD->ADD_INT int observed ADD_CACHED ADD_CACHED ADD->ADD_CACHED non-int observed ADD_INT->ADD_INT int observed ADD_INT->ADD_CACHED non-int observed ADD_CACHED->ADD_CACHED

Let’s see how that looks in code.

void eval_code_quickening(Frame* frame) {
  // ...
  while (true) {
    // ...
    switch (op) {
      // ...
      case ADD: {
        Object* right = pop(frame);
        Object* left = pop(frame);
        if (object_type(left) == kInt) {
          do_add_int(frame, left, right);
          code->bytecode[frame->pc] = ADD_INT;
          break;
        }
        add_update_cache(frame, left, right);
        code->bytecode[frame->pc] = ADD_CACHED;
        break;
      }
    // ...
    }
    frame->pc += kBytecodeSize;
  }
}

Back to “why transition from ADD_INT to ADD_CACHED”. Two thoughts:

  1. We could transition back to ADD. In that case, this code would perform poorly in an environment where the programmer passes multiple different types at this opcode. There would be a lot of bytecode rewriting overhead going on as it goes back and forth between ADD and ADD_INT.

  2. We could also assume it’s a hiccup and not rewrite. This would perform poorly if the first time the argument is an integer, but something else every subsequent operation. There would be a lot of lookup_method calls.

A great extension here would be to add a polymorphic cache. Those are designed to efficiently handle a small (less than five, normally) amount of repeated types at a given point.

Why is this faster?

Even if we leave the interpreter in this state, a small C bytecode interpreter, we save a couple of instructions and some call overhead in the fast path of integer addition. This is a decent win for math-heavy applications.

In the best case, though, we save a great deal of instructions. It’s entirely possible that the compiler will optimize the entire body of ADD_INT to something like:

pop rax
pop rcx
cmp rax, $IntTag
jne slow_path
add rcx
push rcx
jmp next_opcode
slow_path:
; ...

It won’t look exactly like that, due to our object representation and because our push/pop functions do not operate on the C call stack, but it will be a little closer than before. But what if we could fix these issues and trim down the code even further?

Then we might have something like the Dart intermediate implementation of addition for small integers on x86-64. The following C++ code emits assembly for a specialized small integer handler:

void CheckedSmiOpInstr::EmitNativeCode(FlowGraphCompiler* compiler) {
  // ...
  Register left = locs()->in(0).reg();
  Register right = locs()->in(1).reg();
  // Check both left and right are small integers
  __ movq(TMP, left);
  __ orq(TMP, right);
  __ testq(TMP, compiler::Immediate(kSmiTagMask));
  __ j(NOT_ZERO, slow_path->entry_label());
  Register result = locs()->out(0).reg();
  __ movq(result, left);
  __ addq(result, right);
  // ...
}

This example is a little bit different since it is using an optimizing compiler and assumes the input and output are both in registers, but still expresses the main ideas.

See also the JVM template interpreter implementation for binary operations on small integers:

void TemplateTable::iop2(Operation op) {
  // ...
  __ pop_i(rdx);
  __ addl (rax, rdx);
  // ...
}

which pops the top of the stack and adds rax to it. I think this is because the JVM caches the top of the stack in the register rax at all times, but I have not been able to confirm this. It would explain why it adds rax and why there is no push, though.

Exploring further

There are a number of improvements that could be made to this very simple demo. Bytecode rewriting can unlock a lot of performance gains with additional work. I will list some of them below:

  • Make a template interpreter like in the JVM. This will allow your specialized opcodes (like ADD_INT) directly make use of the call stack.
  • Make a template JIT. This is the “next level up” from a template interpreter. Instead of jumping between opcode handlers in assembly, paste the assembly implementations of the opcodes one after another in memory. This will remove a lot of the interpretive dispatch overhead in the bytecode loop.
  • Special case small integers in your object representation. Why allocate a whole object if you can fit a great deal of integers in a tagged pointer? This will simplify some of your math and type checking. I wrote a follow-up post about this!

Maybe I will even write about them in the future.

February 01, 2021

Gonçalo Valério (dethos)

10 years February 01, 2021 07:18 PM

The first post I published on this blog is now 10 years old. This wasn’t my first website or even the first blog, but it’s the one that stuck for the longest time.

The initial goal was to have a place to share anything I might find interesting on the Web, a place that would allow me to publish my opinions on all kinds of issues (if I felt like it) and to be able to publish information about my projects. I think you still can deduce that from the tag line, that remained unchanged ever since.

From the start, being able to host my own content was one of the priorities, in order to be able to control its distribution and ensuring that it is universally accessible to anyone without any locks on how and by whom it should be consumed.

The reasoning behind this decision was related to a trend that started a couple of years earlier, the departure from the open web and the big migration to the walled gardens.

Many people thought it was an inoffensive move, something that would improve the user experience and make the life easier for everyone. But as anything in life, with time we started to see the costs.

Today the world is different, using closed platforms that barely interact with each other is the rule and the downsides became evident: Users started to be spied for profit, platforms decide what speech is acceptable, manipulation is more present than ever, big monopolies are now gate keepers to many markets, etc. Summing up, the information and power is concentrated in fewer hands.

Last week this event set the topic for the post. A “simple chat app”, that uses an open protocol to interact with different servers, was excluded/blocked from the market unilaterally without any chance to defend itself. A more extensive discussion can be found here.

The message I wanted to leave in this commemorative post, is that we need to give another shot to decentralized and interoperable software, use open protocols and technologies to put creators and users back in control.

If there is anything that I would like to keep for the next 10 years, is the capability to reach, interact and collaborate with the world without having a huge corporation acting as middleman dictating its rules.

I will continue to put an effort in making sure open standards are used on this website (such RSS, Webmention, etc) and that I’m reachable using decentralized protocols and tools (such as email, Matrix or the “Fediverse“). It think this is the minimum a person could ask for the next decade.

Gustaf Erikson (gerikson)

January February 01, 2021 05:18 PM

Ponylang (SeanTAllen)

Last Week in Pony - January 31, 2021 February 01, 2021 12:03 AM

Version 0.38.3 of ponyc and 0.4.1 of corral have been released!

January 31, 2021

Derek Jones (derek-jones)

Growth in number of packages for widely used languages January 31, 2021 10:55 PM

These days a language’s ecosystem of add-ons, such as packages, is often more important than the features provided by the language (which usually only vary in their syntactic sugar, and built-in support for some subset of commonly occurring features).

Use of a particular language grows and shrinks, sometimes over very many decades. Estimating the number of users of a language is difficult, but a possible proxy is ecosystem activity in the form of package growth/decline. However, it will take many several decades for the data needed to test how effective this proxy might be.

Where are we today?

The Module Counts website is the home for a project that counts the number of libraries/packages/modules contained in 26 language specific repositories. Daily data, in some cases going back to 2010, is available as a csv :-) The following are the most interesting items I discovered during a fishing expedition.

The csv file contains totals, and some values are missing (which means specifying an ‘ignore missing values’ argument to some functions). Some repos have been experiencing large average daily growth (e.g., 65 for PyPI, and 112 for Maven Central-Java), while others are more subdued (e.g., 0.7 for PERL and 3.9 for R’s CRAN). Apart from a few days, the daily change is positive.

Is the difference in the order of magnitude growth due to number of active users, number of packages that currently exist, a wide/narrow application domain (Python is wide, while R’s is narrow), the ease of getting a package accepted, or something else?

The plots below show how PyPI has been experiencing exponential growth of a kind (the regression model fitted to the daily total has the form e^{10^{-3}days-6.5*10^{-8}days^2}, where days is the number of days since 2010-01-01; the red line is the daily diff of this equation), while Ruby has been experiencing a linear decline since late 2014 (all code+data):

Daily change in the number of packages in PyPI and Rubygems.

Will the five-year decline in new submissions to Rubygems continue, and does this point to an eventual demise of Ruby (a few decades from now)? Rubygems has years to go before it reaches PERL’s low growth rate (I think PERL is in terminal decline).

Are there any short term patterns, say at the weekly level? Autocorrelation is a technique for estimating the extent to which today’s value is affected by values from the immediate past (usually one or two measurement periods back, i.e., yesterday or the day before that). The two plots below show the autocorrelation for daily changes, with lag in days:

Autocorrelation of daily changes in PyPI and Maven-Java package counts.

The recurring 7-day ‘peaks’ show the impact of weekends (I assume). Is the larger ”weekend-effect’ for Java, compared to PyPI, due to Java usage including a greater percentage of commercial developers (who tend not to work at the weekend)?

I did not manage to find any seasonal effect, e.g., more submissions during the winter than the summer. But I only checked a few of the languages, and only for a single peak (see code for details).

Another way of tracking package evolution is version numbering. For instance, how often do version numbers change, and which component, e.g., major/minor. There have been a couple of studies looking at particular repos over a few years, but nobody is yet recording broad coverage daily, over the long term 😉