Detecting Differential Rater Functioning in Severity and Centrality: The Dual DRF Facets Model.

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Human ratings are crucial for performance assessments but prone to errors and bias.
• Differential Rater Functioning (DRF) threatens assessment fairness through rater-examinee or rater-context interactions.
• Existing DRF research primarily focuses on rater severity, overlooking more complex effects.

Purpose of the Study:

• To extend the DRF framework by investigating simultaneous dual DRF effects in rater severity and centrality.
• To propose and validate a new statistical model, the Dual DRF Model (DDRFM), for detecting and measuring these dual effects.
• To assess the impact of dual DRF on measurement quality and demonstrate its practical implications in large-scale assessments.

Main Methods:

• Developed the Dual DRF Model (DDRFM) using a facets modeling approach.
• Conducted two simulation studies to evaluate the model's ability to detect and compensate for dual DRF effects.
• Applied the DDRFM to a large-scale writing assessment dataset (N=1,323) to examine real-world measurement consequences.

Main Results:

• Dual DRF effects were found to negatively impact the quality of performance assessment measurements.
• The proposed DDRFM reliably detected and enabled compensation for these dual DRF effects in simulation studies.
• Analysis of the writing assessment data revealed significant practical measurement consequences attributable to dual DRF.

Conclusions:

• Dual DRF represents a complex threat to the validity and fairness of performance assessments.
• The DDRFM offers a viable statistical approach for identifying and mitigating dual DRF in rating processes.
• Findings underscore the importance of accounting for dual DRF to ensure psychometric integrity in educational and psychological measurement.