Improving our understanding of predictive bias in testing.

Area of Science:

• Psychometrics
• Educational Measurement
• Sociology of Education

Background:

• Predictive bias occurs when regression models yield different performance predictions for various demographic groups.
• This differential prediction violates principles of fairness, equal treatment, and opportunity in selection and admissions.
• Understanding the nature and causes of predictive bias is crucial for equitable decision-making.

Purpose of the Study:

• To demonstrate the existence of different types of predictive bias (intercept and slope differences).
• To investigate the causes of predictive bias by analyzing statistical and psychological mechanisms.
• To reorient future research towards understanding the 'why' behind predictive bias.

Main Methods:

• A Monte Carlo simulation was used to analyze out-of-sample predictions for understanding bias types.
• A college admissions study analyzed data from Black, Latinx, and White students.
• Analytical work was conducted to explain the statistical causes of predictive bias.

Main Results:

• Out-of-sample predictions offer a precise understanding of intercept- and/or slope-based predictive bias.
• Evidence of both intercept- and slope-based predictive bias was found in college admissions data.
• Statistical causes of bias were mapped to psychological and contextual mechanisms.

Conclusions:

• Predictive bias manifests in different forms, including intercept and slope differences.
• Understanding the underlying causes is essential for addressing bias in predictive models.
• Future research should focus on the mechanisms driving predictive bias to promote fairness.