Coaching practices for Sample Size Judgment
Describe almost anything you are working through and IX Coach finds the practices whose real-world fit is closest. For Sample Size Judgment, these are the strongest matches in the current practice library.
Does this sound like the set of challenges you might be facing?
- A handful of early results point one clear direction and I’m already drawing a firm conclusion
- Someone fits the picture of a type so perfectly that I just assume that’s what they are
- I size up what something’s worth and then commit right at that number as if my read is exactly right
- I’m about to launch into something and I keep estimating how it’ll go off the handful of vivid winners in my head
- When I’d swear I know roughly where a number lands, the real answer is constantly outside the range I’d have bet on
Practices that may help
- Treat small samples with explicit skepticism
A short sequence can look representative without being statistically reliable — adjust confidence for sample size.
The Representativeness Heuristic — Judging by Resemblance - Look up base rates before forming a resemblance judgment
Before deciding "this looks like X," ask how common X actually is in the relevant population.
The Representativeness Heuristic — Judging by Resemblance - Discount your estimate to create a margin
If you think something is worth X, only commit at a meaningful discount to X.
Margin of Safety - Choose the right reference class for any prediction
Find the statistical base rate for the category your decision belongs to — not just the inspiring examples.
Survivorship Bias: Learning from What You Can’t See - Practice confidence interval estimation
Estimate ranges for factual quantities and check how often the true value falls within your range.
Calibration Training - Ask the base rate before evaluating the specific case
Before judging any individual instance, first establish how often this kind of thing happens in general.
Base-Rate Neglect: Why We Ignore the Odds - Distinguish uncertainty (quantifiable) from ignorance (unquantifiable)
Know when you can assign a probability and when the situation is so novel that a number would be fabricated.
Calibration Training - Name the prototype you’re comparing against before deciding
Make the prototype you’re using as a reference explicit — hidden templates bias decisions without scrutiny.
The Representativeness Heuristic — Judging by Resemblance - Sanity-check against known extremes
Test your estimate against the clearly too-high and too-low bounds to calibrate your range.
Fermi Estimation - Always identify the denominator when evaluating risk or success
When a number or story is striking, ask: "Out of how many total cases?"
Base-Rate Neglect: Why We Ignore the Odds
Related concerns
- Base Rate Decisions
Base-rate neglect is the tendency to underweight or ignore prior probabilities (how often things happen in general) when vivid, specific information is available. Identified by Kahneman and Tversky, it is one of the most robustly replicated biases in judgment research, and it leads to systematic overconfidence in predictions about specific cases. Correcting it requires actively looking up or estimating base rates before evaluating individual information.
- Small Sample Size Bias
A short sequence can look representative without being statistically reliable — adjust confidence for sample size.
Treat small samples with explicit skepticism
- Stereotype Probability
When assessing probability or quality, ask whether you’re really judging similarity to a prototype.
Interrogate whether similarity is doing the work
- Avoid Representativeness
The representativeness heuristic is the mental shortcut of judging probability by how closely something resembles a prototype or stereotype. It is fast and often useful, but it reliably misfires when it overrides base rates, produces the conjunction fallacy, or treats random-looking sequences as unlikely.
- Base Rate First
Base-rate neglect is the tendency to underweight or ignore prior probabilities (how often things happen in general) when vivid, specific information is available. Identified by Kahneman and Tversky, it is one of the most robustly replicated biases in judgment research, and it leads to systematic overconfidence in predictions about specific cases. Correcting it requires actively looking up or estimating base rates before evaluating individual information.
- Base Rate First Reasoning
Base-rate neglect is the tendency to underweight or ignore prior probabilities (how often things happen in general) when vivid, specific information is available. Identified by Kahneman and Tversky, it is one of the most robustly replicated biases in judgment research, and it leads to systematic overconfidence in predictions about specific cases. Correcting it requires actively looking up or estimating base rates before evaluating individual information.
Describe your situation in your own words to search the complete practice library.