A multidimensional Bayesian IRT method for discovering misconceptions from concept test data.

Area of Science:

• Educational Measurement
• Cognitive Science
• Physics Education Research

Background:

• Identifying student misconceptions is crucial for effective teaching.
• Traditional methods often require manual analysis of test data.
• Existing psychometric models may not fully capture the nuances of student thinking.

Purpose of the Study:

• To develop and validate an exploratory method for discovering student misconceptions from multiple-choice concept tests.
• To leverage a Bayesian implementation of the Multidimensional Nominal Categories Item Response Theory model (MNCM) for automated misconception identification.
• To demonstrate the method's ability to identify misconceptions without manual content labeling.

Main Methods:

• Utilized a Bayesian MNCM combined with factor-analytic rotation methods.
• Analyzed student responses at the individual distractor level.
• Validated the method on synthetic data and compared it to existing IRT software.

Main Results:

• The Bayesian MNCM accurately recovered multidimensional item parameters from synthetic data.
• The method showed robustness to overfitting and performed automatic dimensionality assessment.
• Applied to the Force Concept Inventory, the method identified 13 additional dimensions beyond overall score, consistent with known Newtonian mechanics misconceptions.

Conclusions:

• The developed method effectively discovers likely student misconceptions from concept test data.
• This approach can refine existing concept inventories or aid in developing new ones.
• Findings support the potential for discovering new misconceptions and enabling targeted instruction.