Transfer testing with novel instances

Periodically test your perceptual skill on examples you have never seen to verify genuine learning.

Why it works

Perceptual learning is genuine when it transfers to novel instances — it has extracted the category structure, not memorized specific examples. Testing on held-out examples that were never shown during training is the only reliable check. Without this, a learner can accumulate high in-training accuracy while their actual discrimination depends on memorized details that will not recur in the field.

How to do it

1. Hold back 20% of your example set before training and never show it during practice.
2. After training reaches a criterion (e.g., 85% accuracy), test on the held-out set cold.
3. If held-out accuracy lags training accuracy by more than 10%, you have memorized rather than generalized.
4. Add more varied examples and repeat the cycle.

Evidence

Transfer testing is methodologically standard in Kellman’s PLM research as the criterion for genuine perceptual learning. Studies that omit transfer tests systematically overestimate learning by conflating familiarity with discrimination. (mechanistic)

The 10% gap threshold is a practical heuristic, not an empirically derived standard.

Sources

• Kellman & Garrigan (2009), "Perceptual learning and human expertise," Physics of Life Reviews

Common mistake

Measuring progress only on seen examples, which inflates confidence and leaves the learner unprepared when real-world instances differ from their training set.