Exploring examinees' responses to constructed response items with a supervised topic model

Area of Science:

• Educational Measurement
• Natural Language Processing
• Statistical Modeling

Background:

• Textual data is increasingly used in assessments, particularly constructed response (CR) items.
• Natural Language Processing (NLP) techniques enable analysis of large textual datasets.
• Probabilistic topic models, like supervised latent Dirichlet allocation (SLDA), analyze latent topic structures in text.

Purpose of the Study:

• To address the limitation of SLDA's homogeneous relationship assumption in diverse populations.
• To introduce a novel supervised topic model integrating finite-mixture modeling with SLDA.
• To detect latent participant subgroups with distinct relationships between textual responses and scores.

Main Methods:

• Developed a new supervised topic model by incorporating finite-mixture modeling into SLDA.
• Applied the model to analyze textual responses and scores from a middle grades science inquiry assessment.
• Conducted a simulation study to evaluate model performance under practical conditions.

Main Results:

• The proposed model successfully detects latent groups of participants exhibiting different textual response-score relationships.
• Demonstrated the model's utility with an example from a science inquiry knowledge assessment.
• Simulation results provide insights into the model's performance.

Conclusions:

• The finite-mixture SLDA model offers a more nuanced approach to analyzing textual assessment data.
• This method enhances understanding of subgroup differences in the relationship between written responses and performance.
• The model has significant implications for educational measurement and NLP applications.