High-Stakes Testing Case Study: A Latent Variable Approach for Assessing Measurement and Prediction Invariance

Area of Science:

• Psychometrics and Educational Measurement
• Sociology of Education
• Quantitative Psychology

Background:

• Disparities in test score prediction systems across demographic groups remain a significant challenge.
• High-stakes testing, such as the SAT, is crucial for college admissions and scholarships.
• Understanding measurement and prediction invariance is key to equitable assessment.

Purpose of the Study:

• To jointly assess measurement and prediction invariance in high-stakes testing using a novel approach.
• To examine group differences in test scores based on latent versus observed scores.
• To investigate the role of measurement error in observed group differences in prediction.

Main Methods:

• Utilized a two-stage least squares (2SLS) estimator for joint assessment of measurement and prediction invariance.
• Analyzed data from 176 colleges and universities, focusing on SAT Mathematics (SAT-M) subtest scores.
• Compared latent scores with observed scores to identify group-based differences.

Main Results:

• Measurement invariance was rejected for SAT-M in a majority of cohorts for Black vs. White and Hispanic vs. White comparisons.
• Black students, on average, had SAT-M scores nearly a third of a standard deviation lower than comparable White students.
• Group differences in SAT-M measurement intercepts partially explained observed prediction intercept differences; 2SLS reduced significant observed differences.

Conclusions:

• The study highlights significant measurement invariance issues in the SAT-M subtest across racial groups.
• Latent score analysis provides a more nuanced understanding of group differences than observed scores alone.
• A new research agenda is proposed to investigate causal mechanisms underlying score disparities in high-stakes testing.