The Bitter Lesson: Simplicity Beats Complexity in AI

In his famous 2019 essay, AI researcher Rich Sutton argued that the biggest AI advances came from methods that leverage computation, not human knowledge. Hand-crafted rules and complex architectures have repeatedly lost to simple algorithms like deep learning that scale with more data and compute.

From chess (DeepBlue vs. human knowledge) to Go (AlphaGo vs. human patterns), every time researchers try to encode human expertise, they eventually hit walls. Meanwhile, simple search and learning that exploit massive compute break through.

If computation is the key, then progress hinges on hardware and energy. Notably, advances like GPT-4 and reinforcement learning from human feedback (RLHF) show that even alignment can be tackled by scaling feedback, not hand-crafted rules.

The Bitter Lesson doesn't mean human knowledge is useless. It means we should focus on building systems that can learn from data and compute, rather than trying to program intelligence by hand. The future belongs to scalable methods.