Research methods in performance validity testing studies: Criterion grouping approach impacts study outcomes.

Area of Science:

• Neuroscience
• Psychology

Background:

• Performance validity tests (PVTs) are crucial in assessing response accuracy in clinical and forensic evaluations.
• Known-groups designs are common in PVT research, but the specific methods for defining these groups vary widely.
• The impact of these varying criterion grouping approaches on PVT classification accuracy remains unclear.

Purpose of the Study:

• To investigate how different criterion grouping approaches influence the classification accuracy statistics of a specific PVT, the Word Choice Test (WCT).
• To compare the sensitivity and specificity of the WCT across three distinct criterion grouping methods.

Main Methods:

• The study employed three criterion grouping approaches: (1) failure of 2+ PVTs vs. failure of 0 PVTs, (2) failure of 2+ PVTs vs. failure of 0-1 PVTs, and (3) failure of a stand-alone PVT (Test of Memory Malingering) vs. passing it.
• Classification accuracy statistics, including sensitivity and specificity, were calculated for the WCT under each grouping condition.

Main Results:

• When specificity was set at ≥.90, WCT cutoff scores varied between 41 and 44.
• Associated sensitivity values ranged from .64 to .88, demonstrating variability based on the criterion grouping approach used.
• Classification accuracy was higher than anticipated when a stand-alone PVT defined the criterion groups, potentially due to high correlations with the WCT.

Conclusions:

• The choice of criterion grouping approach significantly affects PVT classification accuracy rates and the resulting cutoff scores.
• Using a stand-alone PVT as the criterion can inflate accuracy, even with potential misclassification issues.
• Approaches using multiple PVT failures yielded more consistent accuracy rates with expectations, highlighting the importance of methodological transparency in PVT research.