Beware false precision in forecasts and models

Treat any precise probability or quantitative forecast with explicit suspicion about whether the model fits the domain.

Why it works

Numbers create a confidence effect: a model that says there is a 2.3% probability of failure feels more credible than "roughly 1 in 50" — even though both express the same probability and the precision may be entirely manufactured. In game-like domains, precision is justified; in wild-randomness domains, the precision is a property of the model’s assumptions, not of reality. Questioning precision redirects attention to model validity.

How to do it

1. When you see a precise numerical forecast, ask: "What assumptions is this precision built on, and do those assumptions hold in this domain?"
2. Re-express the forecast as a wide range: "This model says 2.3%, but given the assumptions, the real range could be 0.5%–15%."
3. Weight the decision accordingly.

Evidence

Consistent with evidence on model uncertainty and calibration in forecasting. Professional forecasters’ confidence intervals are systematically too narrow across domains from weather to finance, indicating that stated precision routinely exceeds warranted precision. (observational)

Not all precision is false — in stable, well-understood domains with large data, model precision can be justified. The check is whether the domain warrants it.

Common mistake

Dismissing any quantitative analysis because precision seems suspicious — the goal is to widen the confidence interval, not to abandon quantitative thinking.