Ulf Wiger is responsible for the development of the aeternity blockchain based on the Erlang programming language. He was one of Erlang’s first commercial users in the early 1990s, before joining Ericsson to work alongside Erlang inventor Joe Armstrong. As a veteran in this field, the Swedish programmer talks about the beginnings of Erlang, great fault tolerances, the advantages of functional programming languages, his view on blockchain technology and the therapeutic effect of programming. The interview with Ulf Wiger was held via chat.
Bank Frick: When was your first encounter with blockchain technology?
Ulf Wiger: My first encounter with blockchain tech was around 2009 as one of my business associates took a keen interest in Bitcoin. I was busy with other things at the time, so I didn’t heed his advice to get into it. I then started thinking about it again back in 2017, but realized that it wouldn’t be profitable to mine privately for much longer. Shortly after that, I was between projects and saw that Aeternity was looking for Erlang developers. I contacted them and joined their core team soon after. This was when I really started digging into it.
Can you name a current problem you’re working?
Right now I’m looking into improving our Aeternity state channel support, and also thinking about what might be missing in order to support writing exciting applications using state channels. Since updates to the blockchain are very expensive by design, state channels offer a way to “amortize” the cost over time and over a large number of microtransactions. These transactions happen off-chain, in what you could call a “limited-trust” encrypted session.
Erlang’s open source release marks the beginning
You have a background at Ericsson back in the 1990s which means that you’ve almost been there since the creation of Erlang by Joe Armstrong. What has changed since then?
Yeah, when I first got in touch with Erlang back in 1992, I was actually working in Alaska. Ericsson started trying to sell Erlang in 1993, and I bought one of the very first licenses. I later joined Ericsson in 1996 and, among other things, tried to help market Erlang to the outside world. This was difficult, since it wasn’t really a true Ericsson product – that is, there was no real commitment to support it commercially over time. And Ericsson also did back away from endorsing it for in-house development of its own systems. This led to the Open Source release of Erlang in 1998, which turned out to be a very good thing.
What has happened since is actually quite a lot: The world of software has actually moved pretty clearly in a direction towards what Erlang stands for. Back in 1998 at Ericsson, we discussed how we should see a move towards more transaction-oriented systems on the web, which would drive requirements similar to what we already faced in telecom systems. This has happened in a very big way. Now, that doesn’t necessarily mean that developers flock towards Erlang per se, but Erlang is now recognized by many as a mature and powerful tool for those who want to build robust and complex on-line systems. Quite a few other languages have borrowed this concept – directly or indirectly from Erlang.
As said, Erlang has its roots in the development of telecom systems that have to be stable for a long time, despite they have to evolve. How helps Erlang with that?
In a number of different ways. Ericsson has actually been developing programming languages for quite some time, starting most notably with PLEX in the 1970s. Among the things learned from PLEX and other efforts like EriPascal was, that telecoms systems spend a large part of their time spawning processes, context-switching and sending messages. The conclusion was that a proper telecom-related programming language should be focused on these things. Also, one of the major challenges as phone switches became more intelligent was something called “feature interaction”, where different services ended up interacting, or interfering with each other in unexpected ways. A lesson was that it was crucial to be able to control the order of events, e.g. by reordering, so that you didn’t have to contend with all possible permutations of all possible interactions.
Finally, what they learned already with the first digital switches in the 1960s was, that it was wise to copy some of the error handling mechanisms from the mechanical and relay switches. One such thing was called “the ‘C’ wire”: If you grounded that wire, all relays associated with a certain call would automatically release. This became the inspiration for Erlang’s process linking and “cascading exit” functionality.
Plus, let’s not forget the soft upgrade capability. Erlang has unusually good support for code replacement in running systems. While experience has shown that this is a bit too fine-grained for major software upgrades, it is amazingly powerful for hot-patching and debugging.
There are lots of other things: Erlang’s virtual machine is unbelievably robust, and the quality assurance of the Open Telecom Platform (OTP) team is such that most developers don’t hesitate to move over to new releases. Over time, this also cuts down on the amount of legacy environments the OTP team has to support, which benefits quality.
Concurrency orientation affects the whole thinking
Seems like a good match. Speaking briefly, how would you describe Erlang in one sentence against the background of other programming languages?
Erlang is the gold standard for developing complex, always available and highly concurrent systems – a standard that other languages try to emulate, but which is still superior in some areas, not least when it comes to error handling. A slightly long sentence perhaps.
Although you’ve already addressed it superficially, could you clearly state the key strengths of Erlang?
Erlang is primarily a concurrency-oriented language – whereas most other languages only support concurrency as an afterthought. Concurrency is generally thought of as really hard, and something to stay away from if you can, but in Erlang, it is how you model your systems. This doesn’t really make concurrency easy, but Erlang makes concurrency programming quite intuitive. After all, if you’re designing concurrent systems, there is no escaping that class of problems, and handling it correctly can make a huge difference as poorly designed concurrency support can really make your task impossibly difficult.
It practically affects all aspects of your design: Performance, memory management, error handling, upgrades and so forth. What Erlang does, is offer a mental model that makes it all fall into place neatly. So when you reason about the tricky parts of the problems you’re trying to solve, even domain experts who don’t really know Erlang are usually comfortable discussing design ideas with the programmers. This is an incredible advantage. And essentially, this is what all programming languages aim for: You want the experts in the domain to be able to look at code and recognize their own thought patterns.
Vice versa a programming language affects the whole thinking at best?
Yes, or at least it should! You could say that there are two schools of thought here: Some want their next language to be as similar to the previous one, while others feel that there is no point learning a new language, if it doesn’t change the way you think about programming.
So each programming language follows a unique purpose?
Talking to software research guru Simon Peyton-Jones once, we started discussing “opinionated” programming languages: Languages that have a clear idea about how you’re supposed to structure your code. Erlang is such a language in the area of concurrency.
Back in the 1990s, we had “4th-generation languages” (4GL), which were designed to be excellent at solving a certain class of problems – and often useless for other things. Such a well-designed language should clearly indicate what it’s good for, and what you should better not attempt at all. That can also be said about Erlang as this is part of being an opinionated language.
A full tolerant and resource-saving system
We’ll get back to Erlang’s concurrency-orientation later. Let’s first talk about another specific aspect of Erlang-based system – being indicated as full tolerant. Could you explain this expression?
The idea is that when you’re servicing lots of concurrent requests – even thousands of ongoing sessions – a single software error should be contained as much as possible, preferably only affecting the session where it occurred. Also, especially for systems that need to operate autonomously or in remote places, it’s crucial that they can always recover from errors – meaning they’re able to essentially “self-heal” and continue operating.
It’s an important conclusion: Coming from a domain where you’re trying to develop extremely complex systems in the face of constantly shifting requirements, you really have to have a starting assumption that there will be bugs in your system.
That sounds just like a very pragmatic approach…
Yet not everyone makes this assumption. There is a different tradition, where you try to engineer all bugs out of the system from the start. Sometimes you simply have to do that, at least to the extent that certain classes of errors are guaranteed not to exist in the system. But the key to being able to do that is that your requirements have to be extremely well understood, which – as a result – puts a ceiling on the amount of complexity and flexibility you can support.
Erlang is also considered to be very resource-saving. Why is that?
Well, there are a few things. Since the basic design was that the virtual machine (VM) needs to do concurrency right, a very large part of the processing in Erlang-based systems takes place in the VM code. It turns out that Erlang code is highly reusable, so the OTP environment has evolved into a set of patterns that are reused extremely often with fairly small variations. This makes it even more profitable to optimize the VM.
Erlang is also a garbage-collected language, meaning it comes with an automatic memory management to minimize memory requirements. This is another area where – if you do it right – it can give tremendous benefits. The Erlang designers noted, that if you disallow cyclical data structures, thus have all the data pointers pointing in the same direction, you can greatly simplify garbage collection (GC) and even make it re-entrant. This is something that distinguishes Erlang from e.g. Java, where GC is extremely complex.
Plus, when designing a VM for systems that essentially never go down, you have to put lots of effort into fighting memory fragmentation – and the Erlang VM is extremely sophisticated in this regard.
Could Erlang thereby be a solution for cryptocurrencies considering their high consumption of resources?
We’re using Erlang for the Aeternity blockchain and would argue that Erlang fits very well in that space, but I believe that much of the resource consumption in blockchains is by-design, and cannot really be reduced much through the choice of programming language. The resource-saving tricks I described above are applicable to dynamic RAM use, but not really on the persistent blockchain, where each block really has to be an isolated, self-contained entity.
On the other hand, if you want to reduce the amount of disk space taken up by a blockchain, there are things you can do. For example, you can structure the data store so that checkpointing becomes possible. Then you could delete old data and design logic to fetch it from e.g. archive nodes on-demand. We are planning to add this to Aeternity, but of course, it will complicate the design. Insofar as Erlang helps at all, it would be by allowing us to put more focus on the inherent complexity of the problems we’re trying to solve, rather than battling with accidental complexity.
So now there’s full-tolerant software running on a concurrent basis. How are errors solved in this scheme?
It’s sort of a multi-pronged approach: I mentioned “process linking” before. This is used in Erlang e.g. to create supervisor processes, which automatically detect crashing processes and restart them with default parameters.
Erlang belongs to the family of pattern-matching languages, which amounts to tons of structural tests and value tests being done all the time in Erlang code. When coupled with the model of supervisors, a style of programming often called “Let it Crash” programming evolves. Erlang code is usually written that – by default – when it crashes, anytime something unexpected happens. If you also make sure to have extensive test suites, you will see such crashes quite often during testing, and rig your tests so that evidence of any process crash constitutes a test failure. It’s not that unusual for test suites to appear to work, but then notice at the end that crashes had been logged.
A particular feature of functional programming, where most functions are side-effect free and rely solely on the input arguments, is that once you’ve found and corrected the bugs in such functions, they stay correct until the requirements change. This is actually extremely powerful, and means that your software really matures over time.
A feature of Erlang processes is that they don’t share memory – at least logically. This means that when a process crashes, other processes usually stay unaffected. This makes it much easier to develop self-healing systems.
I could mention that back at Ericsson, we saw that “fault density” – the number of faults per line of code – tended to be about the same in different projects. We then noted that for Erlang-based systems, the fault density was initially only slightly better than the others, but the total number of lines of code was about four times smaller – meaning much fewer bugs in total. We then noticed that the fault density went down over time in the Erlang-based systems, while that was not apparent in other systems.
Using Erlang for blockchain projects
Most software is built by addressing a particular pain point. Regarding blockchain it was cutting out the middle man by setting up a consensus-based distributed database. Why is Erlang suitable here?
Naively, you’d think that since Erlang is great for distributed systems – and blockchains fulfill that characteristic – it would be a perfect fit, but we’re actually not using distributed Erlang at all. The blockchain can be seen as distributed database, but it’s more accurate to view it as a network of synchronized systems. A distributed database is one system of nodes that trust each other and cooperate very efficiently. There is no trust between nodes in a blockchain network, so they are better seen as separate systems that interact in very conservative ways.
Why is it then, that Erlang is being used for blockchain cases?
Having said this, blockchain nodes are still highly concurrent: They sync up with other nodes and participate in a gossip network. At the same time, they provide APIs for clients, and possibly do mining as well. So, there is a bunch of concurrency, great potential for DoS-style overload and an unusually great risk of malicious attacks even via the sync and gossip protocols – which makes Erlang’s concurrently and fault-tolerance support quite handy. Also, I’d say that functional programming is extremely helpful, partly because it’s extremely important to get the semantics exactly right, but also because it’s a moving target. The field is evolving, and we need to be prepared to adapt to new theories and better ways to solve known problems.
As Bitcoin was written in object-oriented C++, why the late rise of functional languages like Erlang or Haskell in the blockchain sphere?
There has been a slow trend in the industry as a whole towards functional languages, so it’s not surprising as such to see it also in the blockchain space. Also, this trend is usually stronger in domains that are mathematically challenging. Basically, as things get more complex, it pays to move to functional programming languages. But the thing that really boosted that trend – and definitely revitalized Erlang – was the shift to multicore CPUs.
For the longest time, the industry lived with the conviction that any performance issues would simply be solved with the next generation of faster CPUs, but finally, the chip makers threw up their hands and declared that they had reached the end of the road, at least until the next technological quantum leap – in this case, perhaps even literally with regard to quantum computing. They couldn’t make the circuits (much) smaller and the clocks run (much) faster anymore as all the known CPU optimizations had already been applied.
What remained were multicore chips, with lots of slower CPUs, essentially making a lot of already existing parallelism explicit. This forced the software industry to learn how to write explicitly concurrent software, and not least for large projects using C++ this became an almost insurmountable problem. In contrast, not only Erlang welcomed this development, and actually deplored the fact that the multicore CPU development was slowed by the need to run legacy software well.
But in general, functional programming worked well, since many small side-effect free functions are still a lot easier to make concurrent. Basically, that’s how you have to program nowadays, whether or not your language makes it easy. More specifically, you want small functional snippets that interact asynchronously without having to resort to critical sections and locks, since this kills performance in multicore environments.
Then what are the main requirements – e.g. speaking of performance or productivity – for blockchain applications in terms of code?
There are some discrete components that you simply have to interface efficiently, unless you’re already using C/C++. Most importantly, you need a fast, robust and very up-to-date crypto library. Then you also need very good networking support, and you need to be able to utilize the cores you have. You also have to treat mining as a separate activity, which either runs as a separate program, in GPUs or basically in a pool of GPUs or ASICs. Overall you need to view it as a system of loosely coupled components, but this is not really an unusual requirement, so it doesn’t really limit your selection much.
Does that entail further side effects?
Yes, as it tilts the field towards the more mature environments. Young, sexy languages tend to be a bit weak in this regard, and you may end up spinning your wheels on integration issues. A thing that should not be underestimated is how much support you can get from the community surrounding your language of choice. This has been a slight disadvantage with Erlang – though not perhaps so much for the next team choosing Erlang. It gets a bit more expensive if you can’t call on the community for support, or find good reusable components.
So you couldn’t count on the community so much as there was no experience in using Erlang for blockchain?
A classic example of this is how Erlang has always sucked for graphical user interface (GUI) programming. It’s not that Erlang inherently wouldn’t be good for implementing GUI libraries or writing GUI code, but the community was almost completely made up of embedded-system programmers, where the product usually didn’t even have a monitor, much less a GUI.
So, for Aeternity it was more that while Erlang has a great community, not that many were busy working on blockchains per se and we had to do most things from scratch. Of course, for all other things, related to networking, databases, testing and so forth, community support was great. Just not so much for blockchain specific stuff, where we had to be the pioneers. Yet, you also critically depend on lots of honest users wanting to use your blockchain.
That’s why serious blockchain service providers invest much time in community management?
Since the way blockchains work is that you must have a good spread of mining power and a majority of miners that are honest. Otherwise, it doesn’t matter how good your tech is as people won’t be able to trust your blockchain – unless you run a proof-of-stake system where you are the main trustee. But in order to stay relevant, you will also want to attract developers and researchers that add value to your system or explore ways to improve. You won’t be able to do all that on your own.
Speaking of “young sexy languages”, which other programming languages are used for DLT-projects?
I’m not that well informed on that. I’ve seen mentions of C++, Python, Go and Haskell, but I’d expect at least one or a few projects for basically every significant programming language. The thing is, that programmers tend to want to use the tools they excel at, so if a group of, say, great Rust programmers want to explore a DLT case, you will see one in Rust too.
This goes back to how good programming languages affect the way you think about software, much like spoken languages shape the poetry written in them. Also, if you’ve grown to love a certain programming language, whatever weaknesses it may have for the task at hand, you will want to help address them – unless it falls into the domain of things that are obviously stupid to attempt in that language. Then, you’ll attend conferences giving talks about how your particular language is the best for that particular problem.
The future of Erlang
Emotional incentives always looked like a good bias. But besides blockchain, where do you see the relevance of Erlang in today’s world?
Erlang is still an amazing language for embedded non-stop systems, and as far as I know, Ericsson too uses it in some of its most strategic products. They just don’t talk about it much. There are some notable examples in e.g. online betting, ad serving, online gaming systems, factoring systems and so forth, where the requirements are quite similar to what Erlang was originally designed for.
The thing is that Erlang doesn’t have much of a hype factor, so it doesn’t pay much to advertise that you’re using it. You use it mainly if you want to focus on building really great online messaging or transaction systems. Then you go to Erlang conferences and let the small pool of Erlang programmers know what you’re up to. In a small language community, the good programmers will find you, if you have something good going.
Yet, to point to a new niche where I think Erlang might be interesting, it would be device programming.
Like Internet-of-things (IoT) devices?
Yes. This has been a bit slow on the uptake, partly because IoT hasn’t really caught on for real yet. But the thing is that these devices are becoming harder and harder to program, and also become more multicore capable. Erlang doesn’t require so much in terms of resources, and is also eminently multicore savvy as well as event-driven, which means you can write battery-efficient, but still complex code.
After all your years of experience and effort, would Erlang again be your language of choice?
You know what they say about old dogs JI have always been quite interested in other languages, but at the same time, I’ve managed to stay busy full-time doing Erlang programming pretty much since 1996, so as long as that continues, I’ll be coding in Erlang. If I have to take a longer break from Erlang projects, I’ll be sure to get up to speed on some other languages.
Fun fact: While at Ericsson, I actually had to learn so much about UML that I was invited along to IBM in order to discuss their upcoming releases. I did try to develop in it, but it raised my blood pressure too much.
Then it’s better to stay with Erlang…
Yep, it’s therapeutic.