Trim model complexity

Prefer the simplest model of a situation that still fits all the evidence.

Why it works

Adding explanatory variables can always improve a model’s fit to past data, but each extra variable increases overfitting — the model starts explaining noise, not signal, and its predictions deteriorate. The parsimony principle in statistics (formalized as AIC or BIC) penalizes complexity to find the model that generalizes, not just memorizes.

How to do it

1. When building any model or theory, ask: can I remove any variable without losing explanatory power?
2. Test the simplified model against new data rather than the data it was built on.
3. Use fit + simplicity as a combined criterion, not fit alone.
4. Update the model only when a new variable demonstrably improves out-of-sample prediction.

Evidence

Overfitting is a well-documented failure mode in statistics and machine learning. Information-theoretic criteria (AIC, BIC) that penalize model complexity are standard tools for selecting models that generalize rather than memorize. Hawkins gives a direct account of why unpenalized added parameters degrade out-of-sample prediction, making the cost of unnecessary complexity concrete rather than merely formal. (mechanistic)

Statistical parsimony criteria assume a specific loss function and data-generating model; they are guides, not oracles. Domain knowledge still matters.

Sources

• Akaike (1974), AIC criterion; Schwarz (1978), BIC criterion
• Akaike, Hirotugu (1974). "A New Look at the Statistical Model Identification." IEEE Transactions on Automatic Control, 19(6), 716–723.
• Schwarz, Gideon (1978). "Estimating the Dimension of a Model." The Annals of Statistics, 6(2), 461–464.
• Hawkins, Douglas M. (2004). "The Problem of Overfitting." Journal of Chemical Information and Computer Sciences, 44(1), 1–12.

Common mistake

Adding complexity until the explanation covers every observed quirk, then mistaking the coverage for understanding.