Coaching practices for Data Selection Bias
Describe almost anything you are working through and IX Coach finds the practices whose real-world fit is closest. For Data Selection Bias, these are the strongest matches in the current practice library.
Does this sound like the set of challenges you might be facing?
- I’m sure about my read on this, but I only ever notice the things that confirm what I already think
- My whole feed and reading list is wall-to-wall success stories
- I start "researching" a question but I notice I’m really just typing in searches that hand me the answer I already wanted
- I’ve decided the lesson from this success is “focus narrow” or “go big”
- A choice I made blew up and I’m tearing into myself
Practices that may help
- Notice which data you selected — and which you ignored
Your conclusions are built on a sample of the available data — ask what the sample excluded.
The Ladder of Inference - Survivorship Bias: Learning from What You Can’t See
Survivorship bias is the error of drawing conclusions only from the cases that made it through a filter — winners, survivors, visible successes — while the failures that never appear are silently excluded. The clearest historical example is Abraham Wald’s WWII aircraft study: the military wanted to armor the bullet holes they saw on returning planes; Wald showed they should armor where they saw no damage, because planes hit there didn’t return. - Audit whether your information sources systematically favor survivors
Check whether the media, advice, and communities you consume are filtered toward successes.
Survivorship Bias: Learning from What You Can’t See - Structure the analysis before searching for evidence
Define what you’re looking for and what would count as evidence before starting your search.
Confirmation Bias: Seeing What You Expect to See - Steelman the strategy opposite to the successful one
Before adopting a lesson from a success story, build the best possible case for the opposite approach.
Survivorship Bias: Learning from What You Can’t See - Evaluate decisions by process, not outcome
Judge a decision by the quality of the reasoning at the time, not by what happened.
Hindsight Bias: Why Everything Seems Obvious in Retrospect - Audit person-judgments for prototype substitution
Check whether a judgment about a person is based on their actual behavior or on resemblance to a type.
The Representativeness Heuristic — Judging by Resemblance - Evaluate each option against your criteria before comparing options to each other
Score options independently first — so the comparison set can’t retroactively redefine what good looks like.
The Decoy Effect — How an Irrelevant Option Changes Your Choice - Choose the right reference class for any prediction
Find the statistical base rate for the category your decision belongs to — not just the inspiring examples.
Survivorship Bias: Learning from What You Can’t See - Treat small samples with explicit skepticism
A short sequence can look representative without being statistically reliable — adjust confidence for sample size.
The Representativeness Heuristic — Judging by Resemblance
Related concerns
- Selection Bias Decisions
Survivorship bias is the error of drawing conclusions only from the cases that made it through a filter — winners, survivors, visible successes — while the failures that never appear are silently excluded. The clearest historical example is Abraham Wald’s WWII aircraft study: the military wanted to armor the bullet holes they saw on returning planes; Wald showed they should armor where they saw no damage, because planes hit there didn’t return.
- Small Sample Size Bias
A short sequence can look representative without being statistically reliable — adjust confidence for sample size.
Treat small samples with explicit skepticism
- How Many Tried Survivorship
Look for the people who tried the same thing and didn’t make it through.
Actively seek out the failures you aren’t seeing
- How To Check Survivorship Bias
Survivorship bias is the error of drawing conclusions only from the cases that made it through a filter — winners, survivors, visible successes — while the failures that never appear are silently excluded. The clearest historical example is Abraham Wald’s WWII aircraft study: the military wanted to armor the bullet holes they saw on returning planes; Wald showed they should armor where they saw no damage, because planes hit there didn’t return.
- How To Design An Unbiased Search
Define what you’re looking for and what would count as evidence before starting your search.
Structure the analysis before searching for evidence
- Media Survivorship Bias
Check whether the media, advice, and communities you consume are filtered toward successes.
Audit whether your information sources systematically favor survivors
Describe your situation in your own words to search the complete practice library.