Penalized Best Linear Prediction of True Test Scores

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Best linear prediction (BLP) is used to estimate true test scores from observed item scores and ancillary data.
• A potential fairness issue arises if BLP exhibits disparate impact across different demographic groups.
• Existing methods may not adequately balance prediction efficiency with population invariance.

Purpose of the Study:

• To propose a modified approach, penalized best linear prediction (pBLP), to improve population invariance in test score estimation.
• To enhance fairness by reducing subgroup biases while preserving the efficiency of BLP.
• To evaluate the proposed methodology on high-stakes writing assessments.

Main Methods:

• Development of penalized best linear prediction (pBLP) methodology.
• pBLP weights both mean square error of prediction and a quadratic measure of subgroup biases.
• Application and validation of pBLP on three high-stakes writing assessment datasets.

Main Results:

• The proposed penalized best linear prediction (pBLP) method offers improved population invariance compared to traditional BLP.
• pBLP effectively reduces subgroup biases while maintaining a significant portion of the prediction efficiency.
• Empirical application demonstrates the practical utility of pBLP in real-world assessment scenarios.

Conclusions:

• Penalized best linear prediction (pBLP) provides a viable solution for addressing fairness concerns in educational measurement.
• The method offers a practical way to balance prediction accuracy and subgroup fairness in test score estimation.
• pBLP is a valuable tool for developing more equitable high-stakes assessments.