The utility of set-loss error scores in the general population

Although the measurement of cognitive performance usually relies on achievement sum scores, a growing body of research suggests that the analysis of errors made may have a predictive validity beyond that provided by the number of items correct. This study examined the validity related to one such kind of error scores--the set-loss errors--in the general population of 8- to 11-year-old children. Set-loss errors (also called rule violations) can be conceptualized as a breakdown in the adherence to task-specific rules, and in clinical populations, the propensity to make these errors has shown some specificity for identifying disorders connected with frontal lobes dysfunction. The results, however, indicate that set-loss errors derived from distinct tests could not be effectively explained by a single latent dimension; hence, they do not tap a single construct that could be called set loss or the ability to maintain set. At the same time, there were only few weak associations between various kinds of error scores as well as between the set-loss error scores and relevant constructs such as the ability to learn, attentional control, working memory, fluid and crystallized intelligence, and executive functions-related real-world behaviors, indicating an overrepresentation of construct-irrelevant variance in these kinds of scores. These indications were further accentuated by the analysis of sensitivity and specificity where any elevated number of set-loss error scores was unable to classify individuals on theoretically relevant constructs beyond chance levels. The evidence thus speaks against the use of set-loss error scores in the general population of 8- to 11-year-old children.