Incorporating calibration errors in oral reading fluency scoring

Area of Science:

• Educational Measurement
• Psychometrics
• Reading Fluency Research

Background:

• Oral reading fluency (ORF) assessments are crucial for identifying struggling readers and evaluating educational interventions.
• Current scoring methods often treat calibrated passage parameters as fixed, potentially overlooking calibration errors.
• Unaccounted calibration errors can lead to biased ORF scores and underestimated standard errors (SEs).

Purpose of the Study:

• To develop and evaluate a method for incorporating calibration errors into latent variable scores for ORF assessments.
• To derive accurate SEs for ORF scores that account for calibration uncertainty.
• To assess the impact of calibration errors on score estimates and SEs under varying conditions.

Main Methods:

• Utilized an approach that integrates calibration errors into latent variable score calculations.
• Employed the delta method to derive SEs for ORF scores, explicitly including calibration uncertainty.
• Conducted a simulation study to examine the recovery of point estimates and SEs for latent variable and ORF scores.

Main Results:

• Ignoring calibration errors resulted in underestimated SEs for both latent variable scores and ORF scores.
• The underestimation of SEs was more pronounced when the calibration sample size was small.
• The proposed method demonstrated improved accuracy in SE estimation by accounting for calibration uncertainty.

Conclusions:

• It is essential to acknowledge and incorporate calibration errors in ORF scoring to obtain reliable estimates.
• Underestimating SEs can have significant implications for intervention evaluation and reader screening.
• Future ORF assessment development should prioritize methods that address calibration uncertainty for enhanced score precision.