How does "Practice confidence interval estimation" work?

A 90% confidence interval should contain the true value 90% of the time. Most people’s 90% intervals are far too narrow — real values fall outside them 30–50% of the time, reflecting severe overconfidence in precision. Practicing interval estimation with immediate feedback recalibrates the internal sense of "how much I know," which transfers to more honest uncertainty expression in decisions.

How does "Maintain a scored prediction log" work?

Calibration requires a feedback loop: you state a probability, the outcome happens, and you learn whether your stated confidence matched the base rate of your predictions. Without recording and scoring, confidence remains a feeling rather than a measurable quantity. The Brier score is the standard metric: lower is better, rewarding confident correct predictions and penalizing confident wrong ones proportionally.

Calibration Training

Q: How does "Identify the specific domains where you are most overconfident" work?

People are not uniformly overconfident; they are typically more overconfident in domains where they have experience but limited feedback, where the domain uses jargon they understand, or where they have made public commitments. Diagnosing which domains produce your worst calibration focuses debiasing effort where it will have the largest return.

Q: How does "Anchor your confidence level on base rates, not gut feel" work?

Gut-feel confidence is contaminated by how fluently the answer came to mind, how motivated you are for the outcome, and how recently a similar event happened. None of these is a reliable proxy for accuracy. Base rates from a defined reference class provide a hard external anchor that separates the confidence estimate from the emotional and cognitive noise that gut feel mixes in.

Q: How does "Pre-commit to resolution criteria before making a prediction" work?

Vague predictions survive almost any outcome — "I said things might get complicated" is consistent with disaster or success. This outcome-flexibility is a form of motivated reasoning that prevents calibration feedback from ever arriving. Pre-committing to specific, verifiable resolution criteria closes the escape hatch: the prediction either resolves correctly or it doesn’t, and you can’t post-hoc reconstruct what you "really" meant.

Q: How does "Build calibration reps with low-stakes trivia and almanac questions" work?

Calibration skill, like any skill, requires repetition with feedback. Real-world decisions resolve slowly, have high emotional stakes that distort honest scoring, and occur too infrequently for rapid iteration. Trivia and almanac questions provide high-volume, immediate-feedback practice in a domain with no emotional interference — making them an ideal training environment for the underlying metacognitive skill that transfers to high-stakes predictions.

Q: How does "Distinguish uncertainty (quantifiable) from ignorance (unquantifiable)" work?

Knight’s distinction between risk (quantifiable uncertainty) and uncertainty (unquantifiable) matters practically: assigning a 60% probability to an event requires a reference class or a model. When neither exists, the number is a feeling dressed as a statistic. Distinguishing the two prevents false precision from masking genuine ignorance, and prompts the honest response: "I don’t have enough information to put a calibrated number on this."

How do you train yourself to have well-calibrated confidence — neither overconfident nor underconfident?

Calibration training teaches you to match your stated confidence levels to your actual accuracy: if you say "70% confident," roughly 70% of those claims should turn out to be correct. Tetlock’s forecasting research and probability training studies show calibration is trainable with practice and feedback — the evidence here is specific and replicable.

Most people are poorly calibrated in a specific direction: overconfident in domains where they feel competent, underconfident in new ones, and systematically bad at expressing rare-event probabilities. Calibration training is the deliberate practice of measuring the gap between stated confidence and actual accuracy, then closing it with feedback. It is not about being uncertain — it is about being accurately uncertain. Below are the practices that make the training work.

Practices

Practice confidence interval estimation

Estimate ranges for factual quantities and check how often the true value falls within your range.

Maintain a scored prediction log

Record predictions with explicit probabilities and score them when they resolve.

Identify the specific domains where you are most overconfident

Calibration is domain-specific — find where your confidence most exceeds your accuracy.

Anchor your confidence level on base rates, not gut feel

Start your confidence estimate from the historical hit rate for this type of prediction, not from how confident you feel.

Pre-commit to resolution criteria before making a prediction

Define exactly what counts as "I was right" before the outcome happens.

Build calibration reps with low-stakes trivia and almanac questions

Use factual trivia questions as a practice ground for calibration — outcomes resolve immediately and the stakes are zero.

Distinguish uncertainty (quantifiable) from ignorance (unquantifiable)

Know when you can assign a probability and when the situation is so novel that a number would be fabricated.

Practice this with IX Coach

Reading about a practice changes nothing on its own. IX Coach turns these into a guided, adaptive routine — discerning where you are in real time and walking the practice with you, session after session.

Practice this with IX Coach

IX Coach: 7 days free, then $40/month (about $1.30/day).

Calibration Training

Practices

Practice confidence interval estimation

Maintain a scored prediction log

Identify the specific domains where you are most overconfident

Anchor your confidence level on base rates, not gut feel

Pre-commit to resolution criteria before making a prediction

Build calibration reps with low-stakes trivia and almanac questions

Distinguish uncertainty (quantifiable) from ignorance (unquantifiable)

Practice this with IX Coach

Related