Software is Uniquely Complicated

In 2007, with a well-loaded Linux desktop installation, my /usr/bin is 257 megabytes, with debugging off and dynamically-linked libraries not contributing to that count. My particular copy of the Linux kernel 2.6.19 with certain Gentoo patches has 202,381,268 bytes of C code alone. If I'm computing this correctly, at a constant 100 words per minute (5 chars/word), that's 281 24-hour days just to re-type the C code in the kernel.

One of the projects I was able to work on during my schooling years was a relatively obscure Learning Content Management System with over a decade of history behind it. At the moment, that project contains roughly 3000 files in its CVS repository, nearly 300,000 lines of Perl code in just under 9 megabytes, and still going. One rule of thumb says multiplying by five converts a line count from Perl to something like Java, which would be 1.5 million lines of code. And this is just the project-specific code; it is layered on top of also-complex tools like the Perl implementation, the Apache webserver, the Linux kernel, and numerous other libraries and frameworks of all shapes and sizes. Some of these things, like the library used to support internationalization, are tiny. Others like the Linux kernel or the Apache webserver dwarf this single project.

No matter how you slice it, software has a lot of moving parts, but there's no obvious way to compare source code complexity to mechanical complexity. Trying to do a straight part-count comparison is probably therefore disingenuous, so we can't make a straight quantitative comparison. I'd assert that even relatively simple pieces of software have more parts in them than even relatively complex machines like modern automobiles (minus their software), but I have no way to prove this.

There is a qualitative distinction we can draw between the physical world and the world of software, though: the interaction of the parts of a program qualitatively differ from a real-world device. A real world device's connectivity between parts is limited by physical three-dimensional space; with rare exceptions, parts that are interacting must be physically next to each other. In software, any part can interact with any other part at any time. It's as if they are all right next to each other, a physically untenable situation, the equivalent of zero-dimensional space. (There are some exceptions to physical proximity, like process boundaries, but these are often crossed as well.) Software can also include as a critical component arbitrary pieces from any place in the world, thanks to network communications; the Internet as a machine is the size of the planet. The software stack to run a Google search on the client side is already complex (web browser, OS, graphics driver, text services, graphics renderers, and more), but add in the need for Google's server system to be functioning correctly with it's own mind-boggling complexity, and you start to see why it's a miracle software ever works at all.

It's tempting to dismiss this as hyperbole, but the effects routinely manifest in real life. I've experienced many errors at the highest layer ultimately being traceable back to a bug in something several layers deeper, sometimes all the way down to the kernel. Even the simplest act, like typing a character into a text box in a web browser, will in a fraction of a second call tens or hundreds of software "pieces" into action, starting at the operating system handling the keypress, up through the userspace, into the windowing system, through the layers the window system may have in place to modify the keyboard (such as code for handling international layouts), passing through the code to decide which window gets the keypress, into the program's instance of the widget library it uses, which routes the keypress into an event loop with correct source annotations, into the application code, which then itself creates a Javascript-layer event which may then be hooked into by arbitrary Javascript on the web page which can proceed to do any number of other things that may trigger another cascade that is equally complicated, or perhaps even more complicated (like loading a new image from the network). And that's the simplified version of the simple process of handling a keypress. Everywhere you look in software, you get these sorts of interactions routinely, and a single subtle flaw at any layer can have odd effects at any time.

Fortunately, we can also harness some characteristics of software to reduce the complication that any one person needs to worry about at any one point in time to a reasonable level, or it really would be impossible to write a program that can be counted on to work. Again, it is the unique ease of software that makes all this complexity possible; part of the reason other fields don't deal with the kind of complexity that software can deal with is because they lack the reliability of the basic components, testability, and other such aspects of software. They are forced to keep it simple by the nature of the physical world. The only thing stopping medical doctors from dealing with equal or greater complexity is that they can't see into biological processes as well as we can see into software processes, so they are forced to deal with a simplified model of the human body. As we continue to master the physical world, physical engineers and biologists will begin to experience this complexity too. Programmers may be blazing a trail, and it may be unique today, but someday everybody will get to deal with the complexity of software.

(Mu-hu-ha-ha-ha!)