Coaching practices for Small Sample Size Bias
Describe almost anything you are working through and IX Coach finds the practices whose real-world fit is closest. For Small Sample Size Bias, these are the strongest matches in the current practice library.
Does this sound like the set of challenges you might be facing?
- A handful of early results point one clear direction and I’m already drawing a firm conclusion
- I’m sure about my read on this, but I only ever notice the things that confirm what I already think
- I keep hearing about the handful of people this worked out for and it’s pulling me in
- I’d jump on this in a heartbeat if it were the familiar version, but because it’s in a world I don’t know I’m demanding way more proof before I’ll touch it
- When I look for examples of people who did what I’m about to do, all I find are the success stories
Practices that may help
- Treat small samples with explicit skepticism
A short sequence can look representative without being statistically reliable — adjust confidence for sample size.
The Representativeness Heuristic — Judging by Resemblance - Notice which data you selected — and which you ignored
Your conclusions are built on a sample of the available data — ask what the sample excluded.
The Ladder of Inference - Survivorship Bias: Learning from What You Can’t See
Survivorship bias is the error of drawing conclusions only from the cases that made it through a filter — winners, survivors, visible successes — while the failures that never appear are silently excluded. The clearest historical example is Abraham Wald’s WWII aircraft study: the military wanted to armor the bullet holes they saw on returning planes; Wald showed they should armor where they saw no damage, because planes hit there didn’t return. - Always identify the denominator when evaluating risk or success
When a number or story is striking, ask: "Out of how many total cases?"
Base-Rate Neglect: Why We Ignore the Odds - Check whether you’re demanding an unfair ambiguity premium
Estimate what you’d accept under comparable known-odds risk — if your bar is much higher for unknown odds, that gap is the bias.
Ambiguity Aversion — Why Unknown Odds Feel Worse Than Bad Odds - Actively seek disconfirming cases
When researching base rates, specifically look for cases where things went badly — failure cases are underrepresented in natural memory.
The Outside View - Recognize that random sequences don’t "owe" balance
Random processes have no memory — a run of heads doesn’t make tails more likely.
The Representativeness Heuristic — Judging by Resemblance - Choose the right reference class for any prediction
Find the statistical base rate for the category your decision belongs to — not just the inspiring examples.
Survivorship Bias: Learning from What You Can’t See - Steelman the strategy opposite to the successful one
Before adopting a lesson from a success story, build the best possible case for the opposite approach.
Survivorship Bias: Learning from What You Can’t See - Name the prototype you’re comparing against before deciding
Make the prototype you’re using as a reference explicit — hidden templates bias decisions without scrutiny.
The Representativeness Heuristic — Judging by Resemblance
Related concerns
- Data Selection Bias
Your conclusions are built on a sample of the available data — ask what the sample excluded.
Notice which data you selected — and which you ignored
- How Many Tried Survivorship
Look for the people who tried the same thing and didn’t make it through.
Actively seek out the failures you aren’t seeing
- How To Check Survivorship Bias
Survivorship bias is the error of drawing conclusions only from the cases that made it through a filter — winners, survivors, visible successes — while the failures that never appear are silently excluded. The clearest historical example is Abraham Wald’s WWII aircraft study: the military wanted to armor the bullet holes they saw on returning planes; Wald showed they should armor where they saw no damage, because planes hit there didn’t return.
- Availability Bias Examples
Estimate what you’d accept under comparable known-odds risk — if your bar is much higher for unknown odds, that gap is the bias.
Check whether you’re demanding an unfair ambiguity premium
- Avoid Representativeness
The representativeness heuristic is the mental shortcut of judging probability by how closely something resembles a prototype or stereotype. It is fast and often useful, but it reliably misfires when it overrides base rates, produces the conjunction fallacy, or treats random-looking sequences as unlikely.
- Law Of Small Numbers
A short sequence can look representative without being statistically reliable — adjust confidence for sample size.
Treat small samples with explicit skepticism
Describe your situation in your own words to search the complete practice library.