The Physics of Artificial Intelligence
Scaling laws describe the empirical relationship between model size, dataset size, compute, and performance. From the early days of "Big Models" to the modern "Thinking Models," explore how the equations governing AI progress have evolved.
Kaplan Era (2020)
Focus on parameter count. Performance improves as a power law of model size ($N$).
Chinchilla Era (2022)
Compute optimality. Scale data ($D$) and parameters ($N$) equally.
Inference Era (2024+)
Test-time compute. Trading inference time for reasoning capability (System 2).
1. The Power Law Foundation
In 2020, OpenAI researchers (Kaplan et al.) demonstrated that test loss ($L$) scales as a power law with compute ($C$), dataset size ($D$), and parameter count ($N$).
Insight: The line is remarkably straight on a log-log plot. This predictability allowed labs to invest millions in training runs, knowing in advance the performance they would achieve.
Visualizing the Power Law (Log-Log Scale)
Simulated data illustrating the linear relationship in log-log space.
2. The Chinchilla Correction (Compute Optimality)
In 2022, DeepMind's Hoffmann et al. challenged the Kaplan laws. They found that most large models (like GPT-3) were under-trained. To be "compute optimal," you should double your training data for every doubling of model parameters.
🧮 Compute Optimal Calculator
Enter a hypothetical compute budget (in FLOPs) to see the optimal split between Model Size and Training Tokens according to Chinchilla scaling.
Notice how newer models (Llama 3, Chinchilla) sit "lower and further right" (smaller models, vastly more data) compared to older giants (MT-NLG, GPT-3).
3. Inference-Time Scaling
Pre-training scaling is hitting diminishing returns (and massive costs). The new paradigm (exemplified by OpenAI's o1) is Test-Time Compute.
By allowing the model to "think" (generate Chain-of-Thought tokens) before answering, performance on complex reasoning tasks scales with the amount of compute spent during generation.
- ✓ System 1: Instant response. Fast, intuitive, prone to hallucination.
- ✓ System 2: "Thinking" response. Slow, deliberate, self-correcting.
Try the Simulator
Adjust "Thinking Time" to see accuracy impact on Math Benchmarks.
Inference Scaling Curve
Accuracy: 35%Limits & Future Challenges
📚 The Data Scarcity Wall
We are running out of high-quality public human text. Models trained on synthetic data (model-generated) can suffer from "Model Collapse" unless carefully managed.
⚡ The Energy Equation
Training a frontier model costs ~$100M+. Inference at scale costs significantly more. Economic viability requires models to become efficient, not just smart.