Score your own past predictions to calibrate your outside-view use

Keep a forecast log and score it — you cannot improve calibration without feedback on where you were over- or under-confident.

Why it works

Calibration is the alignment between stated confidence and actual accuracy. Most people never find out how well-calibrated they are because they do not track their predictions. Without this feedback loop, outside-view adoption remains a good intention rather than a skill. Scoring forces the system to produce legible error signals and creates the conditions for genuine improvement.

How to do it

1. Start recording significant predictions with probability estimates: "70% chance this takes less than three months."
2. Set a calendar reminder to record the outcome when it resolves.
3. Compute your calibration over time: for everything you said had a 70% probability, did roughly 70% happen?
4. Identify domains where you are systematically over- or under-confident and adjust your base-rate anchoring accordingly.

Evidence

Tetlock’s forecasting tournament research found that tracking and scoring predictions is the feedback mechanism that separates superforecasters from the rest — those who tracked and reflected improved substantially, while those who did not stagnated. (observational)

Calibration scoring requires a sufficient volume of scored predictions to be statistically meaningful; individual life decisions are not resolved frequently enough to produce clean calibration curves in the short term.

Sources

• Tetlock & Gardner (2015), Superforecasting

Common mistake

Recording predictions in natural language rather than explicit probabilities, making retrospective calibration impossible — "I thought it was likely" cannot be scored.