Using Corpus Analyses to Help Address the DIF Interpretation: Gender Differences in Standardized Writing Assessment

Area of Science:

• Educational Measurement
• Linguistics
• Psychometrics

Background:

• Differential item functioning (DIF) analysis is crucial for score validity across groups, but interpreting DIF results, especially small effects in writing assessment, remains challenging.
• Conventional DIF methods often lack substantive interpretation, necessitating complementary approaches to understand score disparities.
• Writing assessment frequently relies on external variables for proficiency matching, complicating the analysis of subtle DIF effects.

Purpose of the Study:

• To demonstrate a corpus-based method for interpreting small gender DIF in writing assessment.
• To provide linguistic insights into the sources of observed gender-based score differences.
• To enhance the validity evidence for writing test scores by understanding DIF.

Main Methods:

• A corpus of 826 writing samples was analyzed, matched for reading and listening proficiency.
• Four categories of linguistic features were examined: sentiment/social cognition, cohesion, syntax, and lexicon.
• 123 linguistic features correlated with writing scores were selected for gender comparison.

Main Results:

• Female test takers' writing samples showed higher scores on six linguistic features (cohesion and syntactic categories) with small effect sizes.
• Three of these features positively correlated with writing scores, while three negatively correlated.
• The linguistic findings corroborated the small magnitude of gender DIF previously identified statistically.

Conclusions:

• Corpus-based linguistic analysis offers valuable insights into the sources of small gender DIF in writing assessments.
• The identified linguistic differences, though small, help explain score variations and support the validity of writing test interpretations.
• This approach bridges statistical DIF detection with substantive linguistic interpretation, advancing assessment practices.