Construction of measures from many-facet data

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Traditional measurement models often overlook the impact of subjective judgment.
• Resolving judge indeterminacy is crucial for reliable and valid assessments.
• Existing frameworks may not adequately capture the nuances of rater variability.

Purpose of the Study:

• To extend the Rasch model to include judge severity as a parameter.
• To explore variants of the proposed model and associated judging plans.
• To demonstrate the model's application and characteristics using empirical data.

Main Methods:

• Development of an extended Rasch model incorporating examinee ability, item difficulty, and judge severity.
• Discussion of model variants and judging plan strategies.
• Application of the model to an empirical testing dataset.
• Comparative analysis with Generalizability Theory.

Main Results:

• The extended Rasch model successfully parameterizes judge severity, offering a more comprehensive measurement framework.
• Empirical application demonstrates the model's utility in analyzing subjective testing situations.
• Comparison highlights distinct approaches to judge indeterminacy between the extended Rasch model and Generalizability Theory.

Conclusions:

• The extended Rasch model provides a robust method for fundamental measurement involving subjective judgments.
• This framework enhances the understanding and quantification of judge effects in assessment.
• The model offers a valuable alternative for addressing judge indeterminacy compared to traditional methods.