Does performance invalidity undermine replication? Evidence from an undergraduate student participant sample

Performance invalidity is well-documented in clinical and forensic neuropsychology, yet its potential influence on data quality in psychological research has received little attention. This raises concerns for reproducibility, particularly given widespread use of undergraduate students as research participants. The present study examined whether incorporating Performance Validity Tests (PVTs) into a research protocol provides a practical method for improving data quality and interpretation.

As part of a larger replication initiative, PVTs were embedded into a direct replication of a 2008 prescribed optimism study. 87 undergraduate students were classified as credible or non-credible based on PVT performance, and analyses were repeated within subgroups.

The effect replicated successfully across the sample (d = 0.98), consistent with the original study (d = 0.93). 17.2% of participants failed one or more PVTs. The credible subgroup yielded a larger effect (d = 1.02) than the full sample, whereas the non-credible subgroup produced a smaller effect (d = 0.87), and a directional reversal in the secondary analysis - a pattern inconsistent with prior literature and suggestive of systematic noise that disengaged responding may introduce into data. Further, credible and non-credible subgroups differed significantly on the primary outcome despite being demographically and experimentally equivalent.

Performance invalidity occurs at a non-trivial rate in undergraduate research samples and can influence study outcomes. Incorporating PVTs into research protocols offers a practical, scalable safeguard to improving data quality and transparency, highlighting how neuropsychology's established tools could contribute solutions to reproducibility challenges in psychological science.