Programming is Uniquely Difficult

Engineers of other disciplines often take offense at the claim that software is uniquely difficult. They do have a point. As pointed out by Fred Brooks in hyper-classic The Mythical Man Month, one reason software is hard because software is so uniquely easy.

We fundamentally built on top of components that have reliability literally in the 99.999999999% range and beyond; a slow 2GHz CPU that "merely" failed once every trillion operations would still fail on average in eight hours at full load, which would be considered highly unreliable in a server room. Physical engineers would kill for this sort of reliability in their products. Or an equivalent to our ability to re-use libraries. Or how easily we can test most of our functionality with the ability to replicate the tests 100% accurately. Or any number of other very nice things we get in the software domain. Our job is far easier in some ways than any discipline concerned with the physical world, where nothing ever works 100%.

Every library, every new computer, every new programming paradigm, and every other such new thing is designed to make programming easier. Some significant fraction of these things actually do make programming easier, though it can be surprisingly difficult to figure out exactly which. And with every task made easier, we face a choice: We can do the task in less time and then be done, or we can do the task in less time, then take on some other task.

Almost without fail, we choose the latter. This is not unique to software by any means; humans have been making this choice for centuries across a wide variety of fields. What is unique is that this interacts with the unique reliability of software; we can, and therefore do, create huge stacks of software for various purposes. The amount of software involved in running even the simplest of static web sites is well beyond what one human could fully understand. (Full understanding here means the ability to describe the reason for absolutely every design decision, and the ability to then make informed decisions about changes in such a way that minimal adverse affects occur. By this standard, it's likely nobody even fully understands things like the Linux Kernel; even the more central people in kernel development have sometimes made incorrect decisions about core components.)

It's because it's easy to build and build on top of software that the "simple" task of web development requires understanding at least three languages (one a full-fledged programming language), and then at least two more languages (another full-fledged programming language and SQL for the database) on the server, and the server code itself may be even more complicated than that. It's because it's easy to build and build on top of software that this language count is going up, not down, in the future. It's because of this that Windows development tends to get more complicated over time, as more and more abstractions and layers are created.

If these layers could perfectly seal off the layers below them, this wouldn't be so bad, because what really matters is the set of knowledge you have to have in order to do useful work. If the abstractions were perfect, you'd only need to understand the top layer, and that is much simpler than having to understand the whole stack. Unfortunately, since all abstractions leak, the result is increasing complexity over time.

We make these trades for good reasons. I would not trade my software stack for a Commodore 64. I look forward to the next iteration as much as the next programmer. But modern software development is complicated beyond belief, beyond comprehension. Where once a Renaissance Man might know "all there is to know" about the whole of science, today you are an above-average developer if you can stay fully competent in more than one language of the many tens of languages that are viable for doing large projects... and that's just the mainstream general-purpose language count. Go beyond the mainstream or into specialized languages and the count goes into the high hundreds or low thousands.

Ironically, it is exactly the unique ease of software development that ends up making it uniquely complicated.