Component Simplicity

There is a common metaphor for hyperspace travel in science fiction, where some scientist type explains how the FTL works by taking a piece of paper, drawing a line on it to demonstrate “normal” travel, then folding the paper to bring the origin and destination together.

Imperative code affords straight line approaches to problem. I have some code. It does 73 things. I need it to do a 74th thing, in the middle of one of the things it currently does, which is itself a complicated mixture of the other 72 things going on. The most direct approach is just to slap some code in the middle of the thing that does the new thing. I have seen many code bases that are the result of person-centuries of this style of programming. Despite superficially having functions and modules, it is architecturally just a list of things to do, often multiple lists haphazardly mashed together.

Imperative code affords this style of programming because it is often in the moment the locally easiest thing to do, and imperative programming won’t stop you. If you can’t do the thing you want right where you want to do it, it is generally not too difficult to bash on the code until you can. Do you need to add a few more global variables, add a parameter to a dozen functions, and add some more feature flags to interact with the dozen flags already present? No problem. Some imperative programming cultures positively celebrate the number of ways their language offers to bash code into compliance in this way. They are welcome to that celebration, but I part ways with them philosophically on that point.

If you try to program Haskell this way, it will at the very best be an inferior imperative language, and at worst, it’ll beat you with a crowbar as a temporary measure while it finds something to really put you down with.

Instead, you need to think like the scientist with the FTL demo. Sadly, you won’t be able to fold your way straight to the destination in one shot, but one way of looking at a well-structured Haskell program is to see it as a collection of ways of folding the problem space, shaving a bit of dimensionality off here and a bit more there, doing some folds that you really can’t get away with in imperative languages backed by the safety of the strong type system, until eventually your whole program collapses into a handful of lines taking the input and transforming it to the output, each token deeply imbued with exactly the semantics and meaning you need to operate on this domain.

Imperative code often devolves into writing things that exponentially expand the state space of your program and hoping you can find a happy path through your newly-expanded space without wrecking up too many of the other happy paths. Functional programming generally involves slicing away the state space until it is just what you need.

It isn’t that imperative code forbids this approach, and indeed there are many imperative code bases that are written on the “fold the problem space” principle, at least in the imperative languages that provide tools for doing the slicing. (Dynamic scripting languages are hard to use this way; they were born on day one as a rejection of the restriction-based mindset, so they fight you on a profound level if you try to restrict the state space.) But imperative languages will at best permit such an approach, whereas Haskell all but demands it. The meandering, plodding, exhaustive and exhausting list of instructions almost linearly written out and bashed into compliance by years of effort and bugs reported from the field that so many imperative programs naturally devolve into doesn’t work very well in Haskell.

Haskell programs generally acheive this by building small things from the bottom up, and then creating rich ecosystems for combining them. This is, for instance, one of the main virtues of “monads” and why Haskell uses them everywhere; it isn’t because of what any specific monad interface implementation is, it is because once you have conformed a data structure to the monad interface, you get the entire monad ecosystem “for free”, just as implementing an iterator gets you the entire iterator ecosystem for free for that implementation.

Note in my previous section when I showed a “free monad” that the data structure only had to implement the particular aspects of the specific driver that we wanted. Once we had that, the “program” I wrote with that driver did many things other than just fetch web pages. You get the whole world of Control.Monad, you get monad transformers and all sorts of other ways of combining small pieces into large structures. As small of an example as that may be, it demonstrates how that code does not simply “use IO”, but can be combined with IO, or combined with software transactional memory, or combined with many other things at a very fine level.

Of course imperative code allows combining code. We would not have gotten very far in programming if we were literally forced to write nothing but straight-line scripts that do one thing at a time in exhaustive detail as I impugned imperative programming with above. But again, Haskell permits and/or forces you to weave code together at a much more granular level than imperative languages. One of my favorite type signatures in Haskell is STM (IO a), not because I used it a lot but because of what it implies, “truly” pure code blending together with guaranteed-safe transactionally pure imperative-looking code to finally produce instructions to produce some real-world effect, all able to weave together very intimately, yet safely.

I have to remind myself after writing that that in practice it is somewhat less idyllic than I may have made it sound. Sharp corners start emerging if you want to mix in other effects systems or monads. Still, if that is a failure, it is a failure at a task that imperative programming langauges can’t even conceive of.

Again, if imperative languages can’t get down to the level of fluidity that functional programming languages can, what is the lesson here? The lesson is that it is indeed possible to program like that, even with complicated use cases, and the practical takeaway is that imperative languages can still achieve this at larger scales of architecture.

One of my favorite bits of code I’ve ever written is code that can run against our corporate Wiki, which is XML under the hood, and extract bits of the page as JSON. Big deal. It sounds like the sort of thing you’d get an intern to write. But it is not written as a big script that grabs the XML, does all sorts of ad-hoc things to extract whatever ad-hoc data I want, then outputs it into an ad-hoc JSON format. It is written as a set of operators you can define as a data, then interpret those operators to turn it into JSON, in a format that mirrors the desired JSON file format.

It uses XPath to navigate to some desired set of XML nodes, then specifies how to extract the desired data from those nodes. So, suppose you have a simple table you want to extract from the Wiki page, containing, say, username in the first column and their email address on the right. The whole thing can be specified something like:

object:
  users:
    target: //[id="target_table"]/tr
    for_each:
      array:
        - target: //td:1/
          extract: text
        - target: //td:2/
          extract: text

This creates a JSON object with a key users, which goes and gets the table rows of the table identified by the given ID, and then for each of those rows creates an array, each member of which is an array containing the text of the cell, resulting in something like

{
  "users": [
    ["user1", "user1@company.com"],
    ["user2", "user2@othercompany.com"]
  ]
}

There’s about a dozen ways of extracting and operating on the node sets returned by XPath. Composing these together, together with the general power of XPath, makes it quite easy to build even some relatively complicated extractions out of just these simple parts. I knew I was doing well when I was writing the test cases for this, and even in Go I was starting to prefer embedding one of these programs into my code as a string and executing it rather than writing the equivalent code by hand. Subsequently it has been used in enough ways that it has been easier to write these extractions than to write ad-hoc code to do one. Most importantly, it has been enough easier to make things happen that wouldn’t otherwise have happened.

There’s another code base that I’m currently working on that involves exactly that long meandering list of ad-hoc instruction written over decades, all of which is designed to operate on a particular entity. In the replacement, rather than copying the same approach, it was easy to create a uniform interface (which I will be taking as its own distinct lesson later) and turn it into a large set of discrete tasks that can be understood in isolation rather than a whole big general mishmash of dozens of things being done in quasi-random orders.

An in-the-wild example is the popular concept of “middleware” in the web server world. Extracting out common concerns in such a way that you can create “stacks” of handlers, is a very powerful way to write web servers. I particularly like how in architectures where you can attach middleware to specific components of the paths, it allows you to make solid guarantees like “nobody can access the /admin URL or anything under it unless they are an admin” by structuring the middleware stack correctly, so that even if someone adds a new admin page later they can’t forget to check that the user is an admin.

Having a plugin- or interpreter-based architecture is not news in general. When I wrote in my intro that not everything I covered was going to “brand new”, this is one of the biggest examples of that I had in mind. Imperative code was operating like this before functional programming was much more than a gleam in a researcher’s eye. Programming languages themselves are one of the most pure instantiations of this concept, because the composition cuts down to the core. It is arguably impossible to create a truly large code base that doesn’t incorporate this idea into it just for the sheer sanity of the programmers, so things like “Linux kernel modules” and “image editor plugins” and a standardized format for sound sythesis modularity have been around since just after programming languages were invented.

But functional programming, by forcing its users to use it much more often, forces you to get comfortable with it. I can see by the code I find in the wild that programmers in general take a long time to work this concept out on their own. Many more imperative code bases should have this concept. It also, by forcing it on you in a very restrictive context, forces you to get good at dealing with the pitfalls of this architecture. It is very easy to cut the modules apart in the wrong manner. In imperative languages you might just cut holes through it as needed; functional programming forces you to fix the architecture. If you have plugins that sometimes need to print something, then you have to modify the architecture to accomodate the need for occasional IO. If you have to thread data through the various modules, you will need to account for that quite explicitly in the design.

Functional programming shows that you can indeed write a lot more code as simpler individual modules composed together, with a richer concept of “composition” than you might naturally obtain from decades of imperative-only coding. It is also good at teaching you how to write interpreters of all shapes and sizes, even something as small as the interpreters I showed for the free monad example, so that when you come back to imperative programming you know better what you need to do to have one.