Practice calibration by tracking your confidence predictions

Your "80% confident" beliefs should come true about 80% of the time — check that they do.

Why it works

Calibration is the match between stated confidence and actual accuracy. Most people are overconfident: they assign high probabilities to beliefs that are true far less often than claimed. Deliberate calibration practice — predicting, then tracking outcomes — has been shown to meaningfully improve the accuracy of confidence estimates over time, because it provides the feedback loop that intuition normally lacks.

How to do it

1. Each week, write down five to ten predictions with explicit confidence levels (e.g., "75% confident the project ships on time").
2. After the outcome is known, record whether you were right or wrong.
3. Every month, compute your calibration for each confidence bucket: what fraction of your 70% predictions came true?
4. Adjust your confidence levels based on the pattern — if your 80%s come true 60% of the time, you are overconfident.

Evidence

Calibration training is one of the few documented interventions that reliably improves the accuracy of probabilistic predictions. Work by Phil Tetlock and colleagues on superforecasters found that explicit calibration practice was among the distinguishing features of the most accurate forecasters. Lichtenstein and Fischhoff demonstrated the causal link directly — a brief training regimen with outcome feedback measurably improved calibration, establishing that the feedback loop, not innate skill, is what closes the confidence-accuracy gap. (observational)

Superforecaster research focuses on geopolitical and macroeconomic predictions; generalization to personal and professional decisions is plausible but less directly studied.

Sources

• Tetlock & Gardner (2015), Superforecasting, Crown Publishers
• Lichtenstein, S., & Fischhoff, B. (1980). Training for calibration. Organizational Behavior and Human Performance, 26(2), 149-171.
• Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., Murray, T., Stone, E., & Tetlock, P. E. (2014). Psychological strategies for winning a geopolitical forecasting tournament. Psychological Science, 25(5), 1106-1115.

Common mistake

Using vague language ("probably," "likely") instead of numbers, making it impossible to ever check whether your calibration has improved.