Rater Model Using Signal Detection Theory for Latent Differential Rater Functioning.

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Differential rater functioning (DRF) is when raters exhibit bias, scoring examinees differently across subgroups.
• Prior research used manifest variables (e.g., gender, ethnicity) to identify rater bias.
• Ideally, rater severity should be consistent across all subgroups.

Purpose of the Study:

• To propose and examine an extension of the latent class signal detection theory (LC-SDT) model for identifying DRF.
• To investigate DRF within latent subgroups based on rater scoring behavior, not just manifest factors.
• To provide a more nuanced understanding of rater characteristics and potential biases.

Main Methods:

• Developed an extended latent class signal detection theory (LC-SDT) model.
• Applied the model to real-world scoring data.
• Conducted simulations with varied sample sizes and rater precision levels.

Main Results:

• Real-world data confirmed the signal detection approach effectively identifies latent DRF.
• Simulations demonstrated adequate parameter recovery across different conditions.
• The model proved robust even with varying sample sizes and rater precision.

Conclusions:

• The extended LC-SDT model offers an effective method for detecting latent differential rater functioning.
• This approach can identify subtle rater biases not captured by traditional methods.
• Findings support the model's utility for analyzing rater characteristics and enhancing rater training programs.