Measuring teacher dispositions using the DAATS battery: a multifaceted Rasch analysis of rater effect

Area of Science:

• Educational Measurement
• Teacher Education
• Psychometrics

Background:

• Teacher dispositions are crucial for effective teaching and are often assessed using subjective measures.
• Concerns about rater subjectivity and its impact on the reliability of these assessments are prevalent in teacher preparation.
• Existing methods like Cohen's Kappa may not adequately address rater effects in dispositional assessments.

Purpose of the Study:

• To investigate the influence of rater subjectivity on measures of teacher dispositions using the Dispositions Assessments Aligned with Teacher Standards (DAATS) battery.
• To provide evidence for the validity and reliability of inferences drawn from subjective affective measures in teacher training.
• To demonstrate the advantages of the Multi-Faceted Rasch Model (MFRM) over conventional methods for analyzing rater effects.

Main Methods:

• Utilized the Dispositions Assessments Aligned with Teacher Standards (DAATS) battery, which employs a six-point rating scale based on Krathwohl, Bloom, and Masia's affective taxonomy.
• Applied multifaceted Rasch measurement models (MFRM) to detect, analyze, and understand rater effects within the DAATS data.
• Compared the efficacy of MFRM with traditional statistical methods, such as Cohen's Kappa, for handling rater subjectivity.

Main Results:

• The study successfully detected and characterized rater effects in the DAATS battery, highlighting their potential impact on assessment accuracy.
• Multifaceted Rasch measurement models proved effective in quantifying and understanding the nature of rater subjectivity.
• The findings suggest that MFRM offers a more robust approach to managing rater variability compared to traditional methods.

Conclusions:

• Rater subjectivity in teacher disposition assessments, while a challenge, can be effectively managed and understood using advanced psychometric models like MFRM.
• The DAATS battery, when analyzed with MFRM, provides reliable evidence of teacher dispositions, supporting the use of subjective affective measures in professional preparation.
• MFRM offers a valuable alternative for teacher preparation programs seeking to enhance the reliability and validity of their assessment procedures.