Treat small samples with explicit skepticism

A short sequence can look representative without being statistically reliable — adjust confidence for sample size.

Why it works

The representativeness heuristic is insensitive to sample size: people judge the probability of a sequence of results with 4 observations the same way they judge one with 40. But the smaller the sample, the higher the variance — extreme or patterned results are far more likely by chance. Recognizing that small samples are unreliable corrects the tendency to read meaning into early results and make premature generalizations.

How to do it

1. When drawing a conclusion from data, write down the sample size explicitly.
2. Ask: "If the underlying truth were random, how often would I see a result this extreme by chance in a sample this small?"
3. Withhold confident conclusions until you have a sample that makes chance explanations implausible.
4. If you can’t get a larger sample, widen your confidence interval rather than narrowing your conclusion.

Evidence

The "law of small numbers" (Tversky & Kahneman, 1971) describes the erroneous belief that small samples are as representative as large ones. This is well-replicated in both lay and expert populations, including psychologists reasoning about statistical power. (observational)

Small-sample skepticism can be overcorrected — not every early result should be dismissed; Bayesian updating using a reasonable prior is more calibrated than blanket skepticism.

Sources

• Tversky & Kahneman (1971), Belief in the law of small numbers, Psychological Bulletin

Common mistake

Widening confidence intervals numerically but still acting on the point estimate — the adjustment must change the decision, not just the verbal acknowledgment of uncertainty.