On reconciling conflicting meta-analytic findings regarding integrity test validity

Area of Science:

• Industrial-Organizational Psychology
• Psychometric Measurement
• Research Methodology

Background:

• Two meta-analyses by Ones et al. (1993) and Van Iddekinge et al. (2012a) reported different findings on the relationship between integrity test scores and work-related criteria.
• Subsequent responses and rebuttals (Harris et al., 2012; Ones et al., 2012; Van Iddekinge et al., 2012b) further debated these discrepancies.

Purpose of the Study:

• To analyze the reasons for the divergent findings between the two meta-analyses.
• To identify specific methodological factors contributing to the differing conclusions regarding integrity test validity.

Main Methods:

• Focused comparison of studies using predictive designs, applicant samples, and non-self-report criteria across both meta-analyses.
• Evaluated potential explanations for discrepancies, including study exclusion criteria, artifact correction, and second-order sampling error.

Main Results:

• Study exclusion criteria, artifact correction, and second-order sampling error were found to be unlikely explanations for the observed differences.
• Lack of detailed documentation of effect size estimates in both meta-analyses prevented a definitive determination of the causes for the divergent findings.

Conclusions:

• The discrepancies between the meta-analyses are not adequately explained by the examined methodological factors.
• There is a critical need for increased detail in meta-analytic reporting and improved information sharing among researchers to ensure transparency and reproducibility.