Comparing holistic and analytic scoring for performance assessment with many-facet Rasch model

Area of Science:

• Educational Measurement
• Psychometrics
• Social Sciences

Background:

• Performance assessments are crucial for evaluating student abilities.
• Traditional scoring methods like holistic and analytic approaches have limitations.
• The many-faceted Rasch model offers a robust framework for analyzing complex assessment data.

Purpose of the Study:

• To compare holistic and analytic scoring methods in performance assessment.
• To investigate the impact of different scoring methods on student evaluations.
• To explore the utility of the many-faceted Rasch model for performance assessment.

Main Methods:

• Utilized the many-faceted Rasch model to analyze data from 43 student reports.
• Employed both holistic and analytic scoring methods with four raters.
• Examined rating scale categories for potential improvements.

Main Results:

• Scoring rubrics can be enhanced by analyzing rating scale categories.
• Student score comparisons revealed method-dependent implications for absolute ability assessment.
• Analytic scoring demonstrated superior rater consistency compared to holistic scoring.

Conclusions:

• The choice of scoring method significantly impacts the assessment of absolute student abilities.
• Analytic scoring offers improved rater consistency in performance assessments.
• Findings provide guidance for selecting and refining scoring methods in educational evaluations.