Repeatedly unreliable, and unreliably repeatable: the trials-to-criterion fallacy

Trials-to-criterion (TTC) is a common method used to quantify learning speed and estimate intelligence. Often, individuals are presented with tasks that have two choices: one has a reward (pass), while the other does not (fail). The task is repeatedly presented until a pre-determined criterion is met (e.g., five passes in a row). The total number of trials to reach the criterion purportedly estimates intelligence, with lower scores reflecting higher intelligence. However, in any given trial, the probability of guessing correctly and passing is 50%, meaning that lucky streaks can readily result in individuals reaching the task criterion by chance. Although the criterion is intended to minimise lucky streaks and ensure that < 5% of tasks are completed by random chance, this is a long recognised, but underappreciated, fallacy of TTC. Using simulations, I show the extent to which random runs of passes, rather than deterministic reaching of the task completion criterion via learning, conflate 'smart' with 'lucky'. Moreover, occasional errors by intelligent individuals can dramatically inflate scores, conflating 'smart-less' with 'luck-less'. The extreme variability in outcomes generated by TTC for a fixed level of intelligence exposes its unreliability and suggests that inferences based on TTC data may often be misleading. I offer considerations for future studies that aim to reduce the risk of artefactual results, but ultimately caution against TTC.