Does Sparseness Matter? Examining the Use of Generalizability Theory and Many-Facet Rasch Measurement in Sparse...

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Sparse rating designs are common in performance assessments, but analyzing rater effects in these designs is challenging.
• Existing research often uses non-simulated data, limiting the ability to control for rater effects and data incompleteness.

Purpose of the Study:

• To compare the effectiveness of Generalizability theory (G theory) and Many-Facet Rasch (MFR) measurement in detecting rater effects within sparse rating designs.
• To evaluate these methods using simulated data to better understand the impact of incomplete data.

Main Methods:

• A simulation study was conducted to generate performance assessment data with controlled rater effects.
• Two analytical approaches, G theory and MFR measurement, were applied to the simulated sparse data.
• The ability of each approach to identify rater effects, including centrality and bias, was assessed.

Main Results:

• Both G theory and MFR measurement provided valuable information on rating quality in sparse designs.
• The MFR measurement approach demonstrated a greater ability to detect rater effects, specifically centrality and bias, compared to G theory.
• Simulated data allowed for a robust examination of rater effects under conditions of data sparsity.

Conclusions:

• MFR measurement is a more sensitive tool for identifying specific rater effects in sparse rating designs.
• The findings suggest MFR measurement may be preferable for detailed rater quality analysis in practical performance assessments with incomplete data.
• Further research can build upon simulated data approaches to explore psychometric properties in complex assessment designs.