Coaching practices for When Probability Models Fail
Describe almost anything you are working through and IX Coach finds the practices whose real-world fit is closest. For When Probability Models Fail, these are the strongest matches in the current practice library.
Does this sound like the set of challenges you might be facing?
- Someone hands me a forecast that says 2.3% and that crisp little number makes me trust it way more than I should
- I’m about to bet a big plan on the way things have always worked, and a quiet voice is asking whether the ground rules here could just shift out from under me
- My worst-case is basically just the worst thing that’s ever happened before, and I keep assuming nothing can be worse than that
- When I look for examples of people who did what I’m about to do, all I find are the success stories
- My theory about why this keeps happening has grown so many moving parts that it explains every little quirk perfectly
Practices that may help
- Beware false precision in forecasts and models
Treat any precise probability or quantitative forecast with explicit suspicion about whether the model fits the domain.
The Ludic Fallacy: When You Mistake Real Life for a Game - Check whether the rules of your domain are actually stable
Before applying any probability model, ask whether the rules governing outcomes could change mid-game.
The Ludic Fallacy: When You Mistake Real Life for a Game - Stress test plans against outcomes beyond the historical range
Ask how your plan holds up if the worst outcome is twice as bad as any historically observed case.
The Ludic Fallacy: When You Mistake Real Life for a Game - Actively seek disconfirming cases
When researching base rates, specifically look for cases where things went badly — failure cases are underrepresented in natural memory.
The Outside View - Trim model complexity
Prefer the simplest model of a situation that still fits all the evidence.
Occam’s Razor: Prefer the Simpler Explanation - Practice probabilistic calibration by tracking your predictions
Assign explicit probability estimates to your predictions and track whether they come true at the right rate.
Base-Rate Neglect: Why We Ignore the Odds - Use maximin reasoning for high-stakes, irreversible decisions under ambiguity
Choose the option whose worst plausible outcome is most survivable — when you can’t compute expected value, optimize the floor.
Ambiguity Aversion — Why Unknown Odds Feel Worse Than Bad Odds - Recognize that random sequences don’t "owe" balance
Random processes have no memory — a run of heads doesn’t make tails more likely.
The Representativeness Heuristic — Judging by Resemblance - Explicitly distinguish coherence from probability
A story can be internally consistent and still be rare — learn to separate "makes sense" from "likely."
The Conjunction Fallacy — When "More Details" Feels More Likely - Diagnose which element — motivation, ability, or prompt — is the bottleneck
When a behavior isn’t happening, ask: Is the person motivated enough? Can they do it? Are they being prompted?
The Fogg Behavior Model, Made Practical
Related concerns
- How To Avoid Probability Mistakes
Assign explicit probability estimates to your predictions and track whether they come true at the right rate.
Practice probabilistic calibration by tracking your predictions
- Game Vs Real World Uncertainty
Ask: would a domain expert still face this uncertainty? If not, the issue is a skill gap — not fundamental ambiguity.
Separate “the world is uncertain here” from “I don’t know enough yet”
- Joint Probability Error
Log your probability estimates and payoff predictions, then compare them to what happened.
Keep a decision journal to score your EV estimates
- Ludic Fallacy
The ludic fallacy, named by Nassim Taleb in The Black Swan, is the mistake of applying the logic of well-defined games (known rules, bounded outcomes, stable probabilities) to domains where those assumptions do not hold — most of real life. The fallacy matters because standard risk models built on game-like distributions systematically underestimate the frequency and magnitude of extreme, unexpected events. This is Taleb’s analytical concept; the supporting evidence is largely observational and historical rather than from controlled experiments.
- What Is The Ludic Fallacy
The ludic fallacy, named by Nassim Taleb in The Black Swan, is the mistake of applying the logic of well-defined games (known rules, bounded outcomes, stable probabilities) to domains where those assumptions do not hold — most of real life. The fallacy matters because standard risk models built on game-like distributions systematically underestimate the frequency and magnitude of extreme, unexpected events. This is Taleb’s analytical concept; the supporting evidence is largely observational and historical rather than from controlled experiments.
- Base Rate Neglect Why We Ignore The Odds After A Loss
Base-rate neglect is the tendency to underweight or ignore prior probabilities (how often things happen in general) when vivid, specific information is available. Identified by Kahneman and Tversky, it is one of the most robustly replicated biases in judgment research, and it leads to systematic overconfidence in predictions about specific cases. Correcting it requires actively looking up or estimating base rates before evaluating individual information.
Describe your situation in your own words to search the complete practice library.