Related Experiment Video
Updated: Jul 16, 2025
A Behavioral Test Battery for the Repeated Assessment of Motor Skills, Mood, and Cognition in Mice
Published on: March 2, 2019
Performance Validity Testing on the NIH Toolbox Cognition Battery: Base Rates of Failed Embedded Validity Indicators
Justin E Karr1, Cristina N Pinheiro1, Jordan P Harp2
1Department of Psychology, College of Arts and Sciences, University of Kentucky, Lexington, KY, USA.
Objective:
The goal of this study was to determine the base rates of failing proposed embedded validity indicators (EVIs) for the National Institutes of Health Toolbox Cognition Battery (NIHTB-CB) in the normative sample.
Method:
Participants included adults in the NIHTB-CB normative sample with data to calculate age-adjusted standard scores (n = 855; ages: M(SD) = 46.9(17.3), range: 18-85; 65.0% women; education: M(SD) = 14.1(2.5) years) or demographically adjusted T-scores (n = 803; ages: M(SD) = 47.3(17.3), range: 18-85; 65.3% women; education: M(SD) = 14.2(2.5) years) for all tests. The NIHTB-CB includes two tests of crystallized cognition and five tests of fluid cognition. Individual norm-referenced test performances were categorized as falling above or below liberal and conservative cutoffs based on proposed univariate EVIs. The number of univariate EVI failures was summed to compute multivariable EVIs. EVI failure rates above 10% were considered high false-positive rates, indicating specificity < .90. Using chi-square analyses, the frequencies of EVI failures were compared based on gender, race/ethnicity, education, and crystallized composite.
Results:
The multivariable EVIs had predominantly low false-positive rates in the normative sample. EVI failure rates were most common among participants with low crystallized composites. Using age-adjusted standard scores, EVI failure rates varied by education, race/ethnicity, and estimated premorbid intelligence. These differences were mostly eliminated when using demographically adjusted T-scores.
Conclusions:
Multivariable EVIs requiring ≥ 4 failures using liberal cutoffs or ≥ 3 failures using conservative cutoffs had acceptable false-positive rates (i.e., < 10%) using both age-adjusted standard scores and demographically adjusted T-scores. These multivariable EVIs could be applied to large data sets with NIHTB-CB data to screen for potentially invalid test performances.
