Track your calibration across many predictions

Score your confidence levels against outcomes over dozens of predictions to find your systematic biases.

Why it works

A single decision tells you almost nothing about your decision-making quality — outcomes are too noisy. Calibration — the match between stated confidence and actual accuracy — only becomes visible across many predictions. If you say "80% confident" across 20 decisions and are right only 50% of the time, you are systematically overconfident at that level, and the journal gives you the data to see it.

How to do it

1. When logging each decision, assign a probability to your expected outcome (e.g. 70%).
2. After outcomes are resolved, record whether the outcome matched the prediction.
3. After 20–30 entries, group predictions by confidence band (50–60%, 70–80%, 90%+) and calculate your accuracy rate in each band.
4. If your accuracy lags your confidence consistently, revise your confidence levels downward in that domain.

Evidence

Calibration training — making probabilistic predictions and scoring them against outcomes — is one of the few interventions with evidence of genuinely improving forecasting accuracy. The superforecasting research showed that explicit tracking and feedback narrows overconfidence. Lichtenstein and Fischhoff demonstrated that even brief training with outcome feedback measurably improved calibration, establishing that the confidence-accuracy gap is trainable rather than fixed. (observational)

Superforecasting evidence comes from geopolitical prediction tournaments; transfer to personal and professional decisions is plausible but the effect sizes in ordinary life decisions are not directly established. Calibration improves with practice, but overconfidence is persistent even in experienced practitioners.

Sources

• Tetlock & Gardner (2015), Superforecasting: The Art and Science of Prediction
• Lichtenstein, S., & Fischhoff, B. (1980). Training for calibration. Organizational Behavior and Human Performance, 26(2), 149–171.
• Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., Ungar, L., & Tetlock, P. (2015). Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspectives on Psychological Science, 10(3), 267–281.
• Chang, W., Chen, E., Mellers, B., & Tetlock, P. (2016). Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments. Judgment and Decision Making, 11(5), 509–526.

Common mistake

Logging decisions without assigning probabilities, which makes calibration tracking impossible and leaves you with only a narrative record instead of a statistical one.