Track your estimates and calibrate

Compare your Fermi estimates to actual figures when you can, and use the gap to improve future estimates.

Why it works

Fermi estimation skill is built through feedback loops — comparing estimates to actual values reveals systematic biases (e.g., consistently underestimating population or overestimating rates) that can be corrected with awareness. Without feedback, estimation skill plateaus at the level of intuition; with feedback, it improves measurably toward expert calibration.

How to do it

1. Keep a log of estimates made and the actual values when they become available.
2. Identify systematic directional biases: are your estimates consistently high or low on certain types of quantities?
3. Apply a correction factor to domains where you have identified a persistent bias.
4. Review the log quarterly and note where calibration has improved and where it has not.

Evidence

Forecasting training research consistently shows that feedback on prediction accuracy, combined with structured reflection, produces substantially better-calibrated estimators. Superforecasters’ advantage is significantly explained by their practice of reviewing and updating based on prediction outcomes. (observational)

Calibration improvement requires enough predictions in a domain for the feedback to be statistically meaningful; in low-frequency domains, calibration gains are slow.

Sources

• Tetlock & Gardner (2015), Superforecasting — tracking records as the engine of calibration improvement

Common mistake

Checking an estimate against reality once and concluding you are well calibrated — calibration is a statistical property visible only across many predictions.