Accuracy of performance-test linking based on a many-facet Rasch model

Area of Science:

• Educational Measurement
• Psychometrics
• Cognitive Science

Background:

• Performance assessments measure higher-order abilities but face challenges in accuracy due to rater and task variability.
• Item response theory (IRT) models, like many-facet Rasch models (MFRMs), address these challenges by incorporating rater and task parameters.
• Test linking is crucial for unifying scales when applying IRT to multiple performance tests, but optimal designs remain unclear.

Purpose of the Study:

• To empirically evaluate the accuracy of IRT-based performance-test linking.
• To investigate the impact of common rater and task numbers on linking accuracy.
• To provide guidance for designing effective performance assessment linking strategies.

Main Methods:

• Utilized simulation experiments to assess linking accuracy.
• Employed a many-facet Rasch model (MFRM) framework.
• Varied the number of common raters and tasks, alongside other influential factors.

Main Results:

• Linking accuracy is demonstrably influenced by the number of common raters and tasks.
• Specific thresholds for common elements were identified as critical for reliable test linking.
• Simulation results provide empirical evidence on the relationship between design parameters and linking accuracy.

Conclusions:

• The number of common raters and tasks significantly impacts the accuracy of IRT-based test linking in performance assessments.
• Findings offer empirical insights to inform the design of performance tests for improved ability measurement.
• This research contributes to optimizing the application of many-facet Rasch models in complex assessment contexts.