Integrating a Statistical Topic Model and a Diagnostic Classification Model for Analyzing Items in a Mixed Format...

Area of Science:

• Educational Measurement
• Psychometrics
• Natural Language Processing

Background:

• Mixed-format tests combine selected-response and constructed-response (CR) items.
• Traditional psychometric models primarily use correctness scores, potentially overlooking richer data in CR items.
• Recent research explores extracting more information from CR items beyond simple accuracy.

Purpose of the Study:

• To develop and apply an integrated statistical approach using a topic model and a diagnostic classification model (DCM).
• To investigate how students' mastery of reading skills influences their writing patterns in CR items.
• To analyze a mixed-item format formative English and Language Arts test.

Main Methods:

• Applied a diagnostic classification model (DCM) to estimate students' mastery of specific reading skills.
• Integrated DCM-derived mastery statuses as covariates into a statistical topic model.
• Used the topic model to predict students' use of latent topics in their written responses to CR items.

Main Results:

• The integrated approach successfully linked reading skill mastery to writing patterns.
• The 'Integration of Knowledge and Ideas' skill was identified as a significant predictor of topic use in student writing.
• This demonstrates the utility of combining DCM and topic modeling for deeper analysis of CR item responses.

Conclusions:

• Student mastery of reading skills, particularly 'Integration of Knowledge and Ideas,' significantly impacts their writing patterns.
• Combining diagnostic classification models and topic modeling offers a powerful method for analyzing complex student writing data.
• This approach provides nuanced insights into the relationship between reading comprehension and written expression in educational assessments.