Was software just always too complex for humans?
I was glued to the Lean chat as the legendary Riemann hypothesis was cracked via 17 million (!) lines of AI-assisted code last week. This has since been certified correct despite no human (yet) understanding how it works. The future of maths, and much science, is clearly going to involve human-AI boosting. However, it also struck me deeply just how broken software engineering has been for my entire life! There's always been an imbalance between our understanding of the theory of computation and our relentless drive to deploy software throughout society. We're paying the piper now as all those decades of human-written code turn out to be full of bugs. Since software essentially drives society now, weird things happen like supermarket fridges needing James Bond to secure them. This has led to people like Tim Gowers joining the Pause AI march. How the hell did we get here? Have we just completely overestimated the human ability to write software? Or will software engineering 'catch up' once LLM harnesses get better? Software was reasonably under control in pre-AI times when attackers were roughly symmetric with defenders, but all the recent mysteriously correlated outages and disclosed issues (Java, PHP, Linux) all point to an acceleration of security pain in 2027. I only see two ways this can go... 1 Derive everything from specifications In April this year, I couldn't believe we'd ever formally specify every layer of hardware and software: I find it hard to believe that we will formally specify all layers of hardware and software and roll them out into production...ever. A Fire Upon the Deep's software archeologists are a profession that's here today and not a scifi trope. -- The Internet needs an antibotty immune system, stat, April 2026 I find it hard to believe that we will formally specify all layers of hardware and software and roll them out into production...ever. A Fire Upon the Deep's software archeologists are a profession that's here today and not a scifi trope. -- The Internet needs an antibotty immune system, stat, April 2026 How a few months makes a difference. With the pace of frontier model advances, translating the vast body of existing legacy code seems entirely plausible if we first speedrun towards defining formal specifications for all of our services, and derive all new software from those as well. That means rebuilding operating system stacks from the ground up and even reengineering instruction sets to match. There are terrible legacy interfaces throughout the stack, largely because our software layers grew so that human groups could scale working on different parts. Other approaches do make it easier though. Unikernels with a libOS linking discipline plus shared memory interconnects are remarkably composable and allow agents to specialise the whole stack at link time, with fewer interfaces making the specification task less daunting. Having good specs also conveniently takes humans out of the security response patch loop. Attackers are currently bounded only by their desire to break in somewhere, while defenders have to wait on maintainer triage. However, fixes re-derived from a formal spec and machine-checked could work more reliably in a timely fashion. The human job remains writing the interfaces iteratively with an AI. 2 Engineer diversity in deliberately Historically, software has tended towards a monoculture through reuse, so one bug can screw over vast numbers of hosts. Agents could instead force diversity via novel 'genetic' mechanisms so digital services aren't all simultaneously infected. This works well with formal specifications, since any decent spec allows multiple valid implementations. The Internet's entire philosophy is based around Postel's Law and encouraging hosts to be liberal in what we receive and strict in what we transmit. The catch, of course, is having just a handful of gated frontier models generating all our software takes us out of the frying pan into an AI controlled fire. The diversity has to extend beyond the code to the models and harnesses too. We seem to have done ok with Internet RFCs not splintering too much, so perhaps formal specs can remain singular. Once that's done, it's time to go let hosts mutate and inoculate each other and generally maximise unpredictability at scale! 3 Digital society is going to have to change To answer my opening question, I don't think better harnesses and models alone will let software engineering catch up, as they just produce mountains of slop code. We need both of the above; formal specifications that let hosts finally be strict about what they receive instead of liberally accepting whatever shows up, while diversity ensures that things that the specs (inevitably) don't capture doesn't break everywhere, everywhere all at once. The real upheaval here will be to social coding though. Open source communities have run on social contracts of human review and trust built up over decades, and every single one I'm a member of is struggling under matters of AI disclosure and trust. At least humans aren't out of a job yet though! Somebody still has to decide what our specs contain, as we have done for decades with RFCs. There's also plenty of new computer science research needed! We need to figure out how to debug diversity when no two nodes run the same code, and on how to force simplicity on systems that agents maintain, and ultimately how we settle the blame games when things do go wrong... (Thanks to Dave Scott, KC Sivaramakrishnan, Thomas Gazagnaire and Sadiq Jaffer for helpful conversations about this post, and to Yaron Minsky for triggering this line of thought with his eBPF observations)