Construct validity of primary trait writing rubrics based on assessment use argument (AUA) validation framework

Area of Science:

• Educational Measurement and Assessment
• Language Testing and Evaluation
• Psychometrics

Background:

• Performance-based language assessments require valid and reliable scoring rubrics to minimize measurement errors.
• Traditional rubric validation relying solely on qualitative data lacks empirical evidence.
• The Assessment Use Argument (AUA) Validation Framework provides a structure for empirical validation.

Purpose of the Study:

• To empirically validate the construct relevance of Primary Trait Writing (PTW) rubrics for student self-assessment activities.
• To gather evidence supporting the claim that PTW rubrics accurately measure the intended writing construct.
• To address warrants and rebuttals concerning the validity of PTW rubrics within the AUA framework.

Main Methods:

• Employed the Many Facet Rasch Model (MRFM) to analyze data from 149 secondary school students and 3 teachers in Malaysia.
• Examined three facets: examinee (students), rater (students and teachers), and writing traits (Content, Format, Cohesive Device, Sentence Fluency).
• Assessed warrants and rebuttals using statistical evidence derived from the MRFM analysis.

Main Results:

• PTW rubrics demonstrated the ability to discriminate between different levels of student writing ability.
• The rubrics met six essential conditions for rating scale effectiveness to a notable extent, with Cohesive Device being a key focus.
• Fit statistics indicated internal consistency across all traits, and a high reliability index confirmed clear differentiation in difficulty levels.

Conclusions:

• Empirical evidence supported the warrants and rejected the rebuttal, confirming the construct validity of PTW rubrics for student self-assessment.
• The study underscores the importance of validating assessment rubrics for internal validity and highlights the utility of MRFM in rubric analysis.
• The findings advocate for the use of PTW rubrics in self-assessment, suggesting future research into their application for other essay types and peer assessment.