Actively seek disconfirming cases

When researching base rates, specifically look for cases where things went badly — failure cases are underrepresented in natural memory.

Why it works

Availability bias causes people to build base rates from the cases most easily recalled. Successful projects are more salient than failed ones (they are written up, celebrated, and shared). This means naturally constructed base rates are optimistic by design. Deliberately seeking failure cases corrects the sampling bias in the reference class, producing an estimate closer to the actual historical distribution.

How to do it

1. When collecting comparison cases, set a quota: at least half your cases must be failures or significant underperformances.
2. Explicitly search for "projects like X that failed" rather than only "projects like X that succeeded."
3. In post-mortems and retrospectives, record failure conditions alongside success conditions.
4. Note what "failure" looks like in each case — is it scope creep, market miss, team breakdown? Each type has different lessons.

Evidence

Availability bias and its effects on probability estimation are among the most replicated findings in judgment research (Tversky & Kahneman, 1973). Its application to base-rate construction — that salient cases distort frequency estimates — is well established. Buehler, Griffin, and Peetz (2010) trace how this motivated, memory-driven sampling is a root cause of the planning fallacy, which is precisely why deliberately sampling failure cases is corrective. (observational)

Deliberately seeking failure cases can produce pessimistic distortion in the other direction if done without representativeness discipline. The goal is an accurate base rate, not a pessimistic one.

Sources

• Tversky & Kahneman (1973), "Availability: A heuristic for judging frequency and probability," Cognitive Psychology
• Tversky, A., & Kahneman, D. (1973). Availability: A heuristic for judging frequency and probability. Cognitive Psychology, 5(2), 207-232.
• Buehler, R., Griffin, D., & Peetz, J. (2010). The planning fallacy: Cognitive, motivational, and social origins. Advances in Experimental Social Psychology, 43, 1-62.

Common mistake

Finding the most famous failure ("companies like X fail 90% of the time") without checking whether that famous failure is actually representative of the relevant class.