A multi-study examination of performance validity in undergraduate research participants

Objective: Performance validity testing is a necessary practice when conducting research with undergraduate students, especially when participants are minimally incentivized to provide adequate effort. However, the failure rate on performance validity measures in undergraduate samples has been debated with studies of different measures and cutoffs reporting results ranging from 2.3 to 55.6%. Method: The current study examined multiple studies to investigate failures on performance validity measures in undergraduate students, and how these rates are influenced by liberal and conservative cutoffs. Failure rates were calculated using standalone performance validity tests (PVTs) and embedded validity indices (EVIs) from eight studies conducted at two universities with over one thousand participants. Results: Results indicated that failure rates in standalone PVTs were up to four times greater when using liberal versus conservative cutoffs. EVI rates varied for conservative versus liberal cutoffs with some measures showing almost no difference and others showing 10 times greater failure rates. Conclusions: Findings provide further descriptive data on the base rate of validity test failure in undergraduate student samples and suggest that EVIs might be more sensitive to alterations made in cutoff scores than standalone PVTs. Overall, these results highlight the variability in failure rates across different measures and cutoffs that researchers might employ in any individual study.