Calibration Training
Building the feedback loop between confidence and accuracy
How to train well-calibrated confidence
Calibration training teaches you to match your stated confidence levels to your actual accuracy: if you say "70% confident," roughly 70% of those claims should turn out to be correct. Tetlock’s forecasting research and probability training studies show calibration is trainable with practice and feedback — the evidence here is specific and replicable.
Most people are poorly calibrated in a specific direction: overconfident in domains where they feel competent, underconfident in new ones, and systematically bad at expressing rare-event probabilities. Calibration training is the deliberate practice of measuring the gap between stated confidence and actual accuracy, then closing it with feedback. It is not about being uncertain — it is about being accurately uncertain. Below are the practices that make the training work.
Practices
- Practice confidence interval estimation
Estimate ranges for factual quantities and check how often the true value falls within your range.
- Maintain a scored prediction log
Record predictions with explicit probabilities and score them when they resolve.
- Identify the specific domains where you are most overconfident
Calibration is domain-specific — find where your confidence most exceeds your accuracy.
- Anchor your confidence level on base rates, not gut feel
Start your confidence estimate from the historical hit rate for this type of prediction, not from how confident you feel.
- Pre-commit to resolution criteria before making a prediction
Define exactly what counts as "I was right" before the outcome happens.
- Build calibration reps with low-stakes trivia and almanac questions
Use factual trivia questions as a practice ground for calibration — outcomes resolve immediately and the stakes are zero.
- Distinguish uncertainty (quantifiable) from ignorance (unquantifiable)
Know when you can assign a probability and when the situation is so novel that a number would be fabricated.
Practice this with IX Coach
Related concepts
How this pairs with other practices
- Why Your Time Log Feels Right But Isn't
A time log can feel accurate simply because you wrote it — calibration training is built to test that feeling against what a tracked guess predicts.
- Why Your First Reaction and First Guess Mislead You
The 90-second rule and calibration training both interrupt an automatic snap response before you mistake it for the truth.
- ACT and Calibration Training: What Blocks Good Forecasts
Fear of being visibly wrong wrecks calibration training before any decomposition skill matters. ACT is built to handle exactly that fear.
- Use Calibration Training to Pressure-Test If-Then Plans
An if-then plan assumes the cue will work. Calibration training gives you a number for how sure you actually are that it will.