Implications of empirical Bayes meta-analysis for test validation

Area of Science:

• Psychometrics
• Statistical Modeling

Background:

• Test validation is crucial in psychometrics.
• Meta-analysis synthesizes findings from multiple studies.
• Situational specificity hypothesis posits that test validity varies across contexts.

Purpose of the Study:

• To explore the utility of Empirical Bayes meta-analysis in test validation.
• To present Bayesian tests for the fixed-effects and random-effects cases.
• To investigate the implications for validity generalization and test transportability.

Main Methods:

• Empirical Bayes meta-analysis framework.
• Fixed-effects model (single rho value).
• Random-effects model (sigma2rho > 0).
• Simulated data for illustration.
• Analysis of published meta-analyses.

Main Results:

• The fixed-effects case provides a powerful test of rho = 0, aiding significance testing in test validation.
• The random-effects case offers a method to assess the relative importance of local validity studies versus prior meta-analyses.
• Local validation's importance increases with sigma2rho, as demonstrated with published data.

Conclusions:

• Empirical Bayes meta-analysis is a valuable tool for test validation and validity generalization.
• The Bayesian approach offers practical significance testing.
• Understanding the random-effects case is key for inferring test transportability.