Score your own past predictions to calibrate your outside-view use
Keep a forecast log and score it — you cannot improve calibration without feedback on where you were over- or under-confident.
Why it works
Calibration is the alignment between stated confidence and actual accuracy. Most people never find out how well-calibrated they are because they do not track their predictions. Without this feedback loop, outside-view adoption remains a good intention rather than a skill. Scoring forces the system to produce legible error signals and creates the conditions for genuine improvement.
How to do it
- Start recording significant predictions with probability estimates: "70% chance this takes less than three months."
- Set a calendar reminder to record the outcome when it resolves.
- Compute your calibration over time: for everything you said had a 70% probability, did roughly 70% happen?
- Identify domains where you are systematically over- or under-confident and adjust your base-rate anchoring accordingly.
Evidence
Tetlock’s forecasting tournament research found that tracking and scoring predictions is the feedback mechanism that separates superforecasters from the rest — those who tracked and reflected improved substantially, while those who did not stagnated. (observational)
Calibration scoring requires a sufficient volume of scored predictions to be statistically meaningful; individual life decisions are not resolved frequently enough to produce clean calibration curves in the short term.
Sources
- Tetlock & Gardner (2015), Superforecasting About this work
Common mistake
Recording predictions in natural language rather than explicit probabilities, making retrospective calibration impossible — "I thought it was likely" cannot be scored.
Practice this with IX Coach
More practices for The Outside View
- Invoke the outside view before committing to any significant plan
Before finalizing a plan or forecast, ask "How did similar things go?" as the first checkpoint.
- Actively seek disconfirming cases
When researching base rates, specifically look for cases where things went badly — failure cases are underrepresented in natural memory.
- Separate motivational optimism from your forecast
Let your ambition be honest about what it is — a desired outcome — without contaminating your probability estimate.
- Run a pre-mortem from the outside-view perspective
Imagine the project has failed — then ask which base-rate failure type caused it.
- Defer heavily to base rates when entering a domain where you lack experience
In unfamiliar territory, the class distribution should almost entirely govern the forecast.
Related concepts
Free to reuse under CC BY 4.0: cite or embed this practice · more practices graded Observational studies