Practice confidence interval estimation

Estimate ranges for factual quantities and check how often the true value falls within your range.

Why it works

A 90% confidence interval should contain the true value 90% of the time. Most people’s 90% intervals are far too narrow — real values fall outside them 30–50% of the time, reflecting severe overconfidence in precision. Practicing interval estimation with immediate feedback recalibrates the internal sense of "how much I know," which transfers to more honest uncertainty expression in decisions.

How to do it

1. Take a calibration quiz: for each factual question, state a range you’re 90% confident contains the true answer.
2. After each batch of questions, score how often the true value was inside your range.
3. If your 90% intervals contain the right answer less than 90% of the time, widen your ranges on the next round.
4. Repeat weekly until your 90% intervals genuinely land 90% of the time — this is calibration.

Evidence

Multiple studies on probability training found that confidence interval exercises with feedback significantly reduce overconfidence in calibration tests, with effects that persist over weeks. This is among the more directly supported forms of debiasing. Alpert and Raiffa’s foundational assessor-training study documented the tell-tale signature — people’s supposed 98% intervals miss the true value far more than 2% of the time — and showed the gap narrows once assessors get feedback on where their ranges failed. (observational)

Calibration improvements in training tasks do not fully transfer to novel domains; calibration is partly domain-specific and requires feedback in each domain to be reliable.

Sources

• Lichtenstein, Fischhoff & Phillips (1982), "Calibration of subjective probabilities," in Judgment Under Uncertainty: Heuristics and Biases
• Tetlock & Gardner (2015), Superforecasting — calibration training modules
• Alpert, M., & Raiffa, H. (1982). A progress report on the training of probability assessors. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment Under Uncertainty: Heuristics and Biases (pp. 294–305). Cambridge University Press.
• Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982). Calibration of probabilities: The state of the art to 1980. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment Under Uncertainty: Heuristics and Biases (pp. 306–334). Cambridge University Press.

Common mistake

Widening all ranges uniformly rather than learning to discriminate — a well-calibrated person has narrow ranges when they genuinely know something and wide ones when they don’t, not uniformly wide ranges.