The Scarcity Ladder
As each layer of data work gets democratised, the bottleneck shifts upward. What was once elite expertise becomes baseline literacy — and a new scarce skill emerges above it.
This page walks through each of the seven scarcity eras in detail — what was abundant, what was scarce, and who was elite — plus a speculative eighth era. It expands on the compression argument made in "The Floor Keeps Rising" (Article 5). See also: The Rising Floor — a stacked chart of how each era builds on the last.
▼
What happened
Data was precious and scarce — collected by hand from experiments, censuses, agricultural trials. The challenge wasn't handling volume; it was extracting reliable signal from tiny samples rife with noise. Galton, Pearson, and Fisher invented the foundational toolkit: correlation, regression, ANOVA, hypothesis testing, experimental design.
What was abundant
- Time and patience
- Physical ledgers & records
- Domain expertise (biology, agriculture, economics)
What was scarce (= valuable)
Statistical intuition
Experimental design
Probability theory
Who was elite
Academic statisticians, government actuaries, scientific researchers. A tiny priesthood with access to rigorous mathematical methods.
▼
What happened
Computers made statistical calculations feasible at scale — but required programming. Punch cards, FORTRAN, COBOL. Data lived in batch-processed files. The bottleneck shifted from mathematical intuition to: can you write code to run the analysis at all? Turing, von Neumann, Knuth defined this era.
What got democratised
- Basic statistical calculation (tables replaced by computers)
- Arithmetic at scale
- Census & survey processing
What was scarce (= valuable)
Programming
Algorithm design
Systems architecture
Who was elite
Computer scientists and engineers at IBM, Bell Labs, government agencies. Programming was a deep specialist skill — most people never touched a terminal.
▼
What happened
VisiCalc (1979) and Excel (1985) put computation in everyone's hands. Relational databases (Oracle, SQL Server) organised business data systematically. Programming was still specialist, but analysing data in Excel became a mass skill. The bottleneck shifted to: can you design and query databases? Can you build systems that store and retrieve data reliably?
What got democratised
- Basic arithmetic & tabular analysis (Excel)
- Statistical reporting
- Word processing & document creation
What was scarce (= valuable)
Database design
SQL & data modelling
Enterprise architecture
Who was elite
Database administrators, enterprise architects, ERP consultants (SAP, Oracle). These roles commanded high salaries precisely because the infrastructure was new and complex.
▼
What happened
Google, Amazon, Facebook generated petabytes. Relational databases couldn't cope. Hadoop (2006), MapReduce, Cassandra, Kafka emerged. The challenge: data was now so large and fast that just moving and storing it was a hard engineering problem. "Big Data" became a buzzword. The bottleneck was distributed systems engineering, not analysis.
What got democratised
- SQL querying (near-universal for analysts)
- Basic database administration
- Web development at small scale
What was scarce (= valuable)
Distributed systems
Data engineering
Scale & reliability
Who was elite
Site Reliability Engineers, Hadoop architects, infrastructure engineers at Google, Facebook, Amazon. These people earned exceptional salaries for keeping systems running at scale.
▼
What happened
Cloud platforms (AWS, GCP, Azure) commoditised infrastructure. Suddenly you didn't need to manage Hadoop clusters — you rented compute. The bottleneck shifted: now the hard part was building models that extracted value from all this stored data. Python's scientific ecosystem exploded. The "Data Scientist" emerged: half statistician, half software engineer, half domain expert.
What got democratised
- Infrastructure provisioning (cloud)
- Distributed storage & processing
- Basic data pipelines
What was scarce (= valuable)
ML model building
Statistical modelling
Python + Stats combined
Feature engineering
Who was elite
Data Scientists at Google, Netflix, Airbnb. Kaggle grandmasters. People who could go from raw data to a deployed model. Starting salaries surpassed most engineering roles.
▼
What happened
AlexNet (2012) showed deep learning beat everything else on vision tasks. Transformers (2017) did the same for language. AutoML and sklearn made classical ML nearly a commodity. The new frontier: training large neural nets, understanding architecture choices, scaling laws. The bottleneck shifted to: can you think in tensors and gradient flows?
What got democratised
- Classical ML (AutoML, sklearn pipelines)
- Model deployment (MLflow, SageMaker)
- Data visualisation & BI tools
What was scarce (= valuable)
Deep learning architecture
Large-scale training
GPU/TPU systems
NLP & CV research
Who was elite
ML researchers at DeepMind, OpenAI, Google Brain, FAIR. People who could design and train large models. Compensation reached $1M+ at the frontier.
▼
What happened
LLMs made coding itself accessible to non-programmers. A lawyer can now write a Python script. A doctor can build a simple ML model. The floor rose dramatically: tasks that required a junior data scientist in 2020 can now be done by a domain expert with good prompting skills. The bottleneck is shifting — away from "can you write code" toward "do you know what you're actually trying to achieve." This cycle is not finished; the duration below is current, not final.
What got democratised
- Writing code (Copilot, Claude Code)
- Building standard ML pipelines
- Data cleaning and transformation
- Report writing & visualisation
What is scarce (= valuable now)
Domain judgment
Problem framing
AI system design
Evaluating model outputs
Uncertainty reasoning
Who is elite now
People who combine deep domain expertise with the ability to direct and evaluate AI systems — not just use them. The bottleneck has moved above the technical layer.
▼
What might happen
As AI systems get better at problem framing, the bottleneck may shift further up — toward ethical judgment, accountability, and the ability to navigate ambiguity in contexts where being wrong has real consequences. The skills that are hardest to automate are those requiring trust, responsibility, and long-term relationships. This is an open question — and the first genuinely uncertain one in this ladder. It is a hypothesis, not a finding.
Will get democratised
- Many knowledge work tasks
- Standard analysis & reporting
- Code review & debugging
- Possibly: basic research synthesis
Hypothesis: next scarce skills
Ethical accountability
Institutional trust
Deep domain mastery
Cross-domain synthesis
The honest answer
The one thing that has never been automated across 150 years of this ladder is rigorous reasoning about uncertainty. A strong Statistics or Mathematics foundation is the longest-duration bet precisely because it underpins every layer of the stack. Whether Domain Judgment itself compresses is not yet knowable — and this page does not claim to know.
Part of The Long Bet series
← The Floor Keeps Rising (Article 5)
The full argument: the compression pattern, the three layers, and why mathematics is the layer that holds.
The Rising Floor →
A stacked chart showing how each era's floor builds cumulatively on the last.