The Floor Keeps Rising
The scarce skill in data and computing has shifted seven times since the 1880s. The foundation underneath it has never moved.
In 1925, Ronald Fisher was working on experimental design for agricultural trials at Rothamsted Research Station. Small plots of land, crop yields, soil variations. The tools he built – analysis of variance, randomised controlled trials, the logic of statistical significance – were so specialised that almost nobody outside mathematics and biology had heard of them.
A hundred years later, every clinical trial, every A/B test, every machine learning model uses the same foundations. Not because the field moved on from Fisher. Because it couldn't.
That is the thing about old tools that survive. They survive because they correspond to something the newer eras keep needing.
The pattern nobody is teaching
Every decade since the 1880s, the scarce skill in data and computing has shifted. A new bottleneck appears. The previous one dissolves into the baseline – into what everyone is assumed to have.
It has happened seven times.
Statistical intuition was the bottleneck until software made it easy to run the calculations. Programming was the bottleneck until databases and cloud platforms abstracted the infrastructure. SQL was the bottleneck until ORMs and no-code tools handled the queries. Distributed systems were the bottleneck until AWS commoditised the infrastructure. Classical ML was the bottleneck until sklearn and AutoML handled the model-building. Deep learning was the bottleneck until PyTorch and Keras abstracted the architecture.
Each time, the people who mastered the scarce skill built the tools that replaced it. Each scarce skill digs its own grave.
This is the pattern. Not a coincidence, not a trend – a mechanism. Every time a skill gets tooled, the value moves to whatever the tool cannot yet do.
The number that changes the argument
Here is how long each skill stayed scarce before the next tool arrived.
A distinction before going further. The coverage and prerequisite findings earlier in this series are measurements – checkable against a 100-university dataset and admissions pages. This table is not a measurement. It is a reading of history: the era boundaries are constructed, and reasonable people can draw them differently. And the conclusion built on it – mathematics as the layer that holds – is a bet, not a finding. I think it is the right bet. But it belongs in a different category from “30 of 31 programmes require Mathematics”, and it should be read that way.
Two companion pages go further than this table can. For the full detail behind each era – what was abundant, what was scarce, who was elite, and a longer look at whether the next layer up (domain judgment) can compress the same way – see The Scarcity Ladder — A Timeline. For a single chart showing how each era's floor builds cumulatively on the last, see The Rising Floor.
The implication is uncomfortable. Choosing what to study based on what is currently scarce is a bet against a curve that has won every time.
What the old foundation actually is
Here is the part that looks like a coincidence but isn't.
Across every transition – across seven different bottlenecks – one capacity has never been commoditised. Not statistical intuition as a set of procedures. Not programming. Not SQL. Something underneath all of them: the ability to reason rigorously about the limits of a system – whether that system is a statistical sample, a cluster of servers, or a neural network. They knew how things break, and why.
Fisher had it. The distributed systems engineers who built Hadoop had it. The ML researchers who designed transformers had it. At every era, the people who were genuinely elite were not just the ones who could use the tool – they were the ones who understood the limits of the tool. Who knew when the output was wrong. Who could see what the model couldn't see.
That capacity kept being needed because each new era produced outputs that could fail in ways that looked like success. A regression can be statistically significant and meaningless. A distributed system can be highly available and consistently wrong. A language model can be fluent and confidently incorrect. The tool cannot evaluate itself. Something outside the tool has to.
The old foundation – the thing Fisher was building in 1925 – is not a historical curiosity. It is what every subsequent era rests on, and what gets more exposed as the implementation above it gets automated.
The three layers
That capacity – reasoning rigorously about the limits of a system – is the first of three layers that make someone genuinely irreplaceable as AI automates the implementation beneath them.
.fit() actually does, not just that it works. Without this layer, you cannot validate the output.The first layer is built early, at the point of choosing a degree or an early career path. The other two follow from years of practice, and they compound on top of whatever foundation was laid first. Which is why the foundation matters so much – not because it is sufficient, but because without it, the other two have nothing to stand on.
The same force, two different moments
The compression curve hits differently depending on where you are in your career. But it is the same force, and it raises the same question for everyone: which of the three layers do you actually have?
Don't choose based on what is currently scarce. The subjects that build the capacity underneath the tools – rigorous reasoning about uncertainty, mathematical foundations, the ability to frame a problem correctly – are not the ones that feel most immediately relevant. They never have been. In 1990, the immediately relevant skill was databases. In 2010, it was Python. In 2024, it is prompting. None of these is the foundation. All of them rest on something older.
The degree you choose is where the first layer gets built – or doesn't. The previous article in this series examines which subjects build that foundation most durably – and why the answer has been stable for longer than most people expect.
A data engineer who has spent eight years building pipelines opens Claude one morning and watches a junior analyst do in twenty minutes what used to take them a day. That is not a hypothetical. It is happening now, in most companies, unremarked.
The question is not "will I be replaced?" It is: what proportion of your value has been in the implementation layer, and what proportion has been in the layers above it?
If it was primarily implementation – the SQL, the pipeline, the model, the code – that position is dissolving faster than any previous transition. The moat is draining. If it was in knowing which problem is worth solving, knowing when the output is wrong, knowing what the model cannot know – that layer did not compress. It has never compressed.
The person who feels most exposed – who has spent years developing deep domain expertise and can see the disruption coming – is often the one with the most durable position. The person who feels fine is often the one who should worry: purely technical roles are exactly what the compression is replacing, and the demand signal just hasn't turned negative yet.
The open question
One thing the pattern doesn't settle: will domain judgment itself compress?
Every previous scarce skill could be encoded because it had a verifiable correct answer. Judgment in domains where being wrong is irreversible – a doctor deciding whether to operate, an engineer deciding whether a design is safe enough, a researcher deciding which hypothesis is worth five years of their life – may be different. These involve accountability that AI does not carry, trust that is earned in a specific domain over years, and the particular kind of knowledge that comes from having been wrong before.
Or it may not be different. The counterargument is that people may simply defer to AI anyway – which might already be happening in radiology and legal research.