Does performance invalidity undermine replication? Evidence from an undergraduate student participant sample.

Area of Science:

• Psychological Science
• Neuropsychology
• Research Methodology

Background:

• Performance invalidity is a known issue in clinical and forensic neuropsychology.
• Its impact on psychological research data quality, especially with undergraduate participants, is understudied.
• This raises concerns for the reproducibility of research findings.

Purpose of the Study:

• To investigate if Performance Validity Tests (PVTs) can practically improve data quality in psychological research.
• To assess the influence of performance invalidity on study outcomes in a research setting.
• To explore the utility of established neuropsychological tools in addressing reproducibility challenges.

Main Methods:

• Performance Validity Tests (PVTs) were integrated into a direct replication of a 2008 prescribed optimism study.
• 87 undergraduate students participated and were categorized as credible or non-credible based on PVT performance.
• Analyses were conducted on the full sample and then repeated within these subgroups.

Main Results:

• The replication was successful, with a large effect size (d=0.98).
• 17.2% of participants failed at least one PVT, indicating performance invalidity.
• Credible participants showed a larger effect size (d=1.02) than the full sample, while non-credible participants showed a smaller effect (d=0.87), with a reversal in secondary analyses.
• Significant differences were found between credible and non-credible subgroups on the primary outcome, despite demographic and experimental equivalence.

Conclusions:

• Performance invalidity is prevalent in undergraduate research samples and can significantly impact study results.
• Integrating PVTs into research protocols is a practical and scalable method to enhance data quality and transparency.
• Neuropsychology's established tools, like PVTs, can offer solutions to improve the reproducibility of psychological science.