Development and application of novel performance validity metrics for computerized neurocognitive batteries

Area of Science:

• Neuroscience and Cognitive Psychology
• Psychometrics and Data Science

Background:

• Neurocognitive assessment data can be compromised by factors like disengagement or underperformance.
• Existing performance validity tests (PVTs) have limitations, including being burdensome and not scalable for large datasets.

Purpose of the Study:

• To develop novel performance validity measures using psychometric concepts within the Penn Computerized Neurocognitive Battery (PennCNB).
• To address limitations of traditional PVTs, aiming for scalability, reduced burden, and dimensional assessment.

Main Methods:

• Developed validity measures using simulations of invalid response patterns based on real data.
• Applied and validated these measures in two large, independent samples: children/adolescents (n=9498) and adult servicemembers (n=1444).

Main Results:

• The novel metrics successfully detected subtle invalid responding patterns in simulated data.
• A combination of these metrics significantly predicted established validity rules in both developmental and adult datasets.
• Most clinical diagnostic groups did not exhibit reduced validity estimates.

Conclusions:

• Proof-of-concept for multivariate, data-driven performance validity metrics.
• These metrics offer a scalable, dimensional, and less burdensome approach to assessing performance validity across neurocognitive tests.
• Further research is warranted to explore their full application.