Notice which data you selected — and which you ignored

Your conclusions are built on a sample of the available data — ask what the sample excluded.

Why it works

Attention is finite, and selection is always happening, usually in the direction of information that confirms existing beliefs. The data you selected feels like "the data" — the rest does not feel excluded, it feels absent. Deliberately searching for the data you did not attend to is the countermeasure to selection bias operating at the bottom of the ladder.

How to do it

1. After reaching a conclusion, ask: what did I notice, and what was I not attending to?
2. Seek out data that a person who reached the opposite conclusion would have selected.
3. Ask someone who knows the same situation whether they weighted the same events you did.
4. Treat any data point you feel the urge to dismiss as worth examining more carefully.

Evidence

Selective attention and confirmation bias are among the most replicated findings in cognitive psychology. People attend to and remember information consistent with existing beliefs more reliably than inconsistent information. (observational)

Awareness of confirmation bias provides only partial protection; the selection process operates partly below consciousness and is not fully correctable by attention alone.

Sources

• Nickerson (1998), "Confirmation Bias: A Ubiquitous Phenomenon in Many Guises", Review of General Psychology

Common mistake

Selecting data to support the opposite conclusion as the "corrective" — the goal is to find data you genuinely weighted too low, not to perform the opposite bias.