An EM-Based Method for Q-Matrix Validation.

Area of Science:

• Psychometrics
• Educational Measurement
• Cognitive Diagnosis

Background:

• Accurate Q-matrices are crucial for cognitive diagnosis models.
• Specifying Q-matrices can be challenging for subject matter experts.
• Existing Q-matrix validation methods require further investigation.

Purpose of the Study:

• To investigate and compare the efficiency of three Q-matrix validation methods: Maximum Likelihood Estimation (MLE), Marginal Maximum Likelihood Estimation (MMLE), and Intersection and Difference (ID) method.
• To evaluate these methods against sequential EM-based delta (δ) and its extension (ς²), the gamma (γ) method, and a nonparametric method.
• To assess performance under the deterministic-inputs, noisy "and" gate (DINA) and reduced reparameterized unified model (rRUM).

Main Methods:

• Expectation-Maximization (EM)-based algorithm.
• Simulation studies comparing correct recovery rate, true negative rate, and true positive rate.
• Evaluation under different Q-matrix validation methods and cognitive diagnosis models (DINA, rRUM).

Main Results:

• For rRUM, MLE is superior for low-quality tests, MMLE for high-quality tests.
• For DINA, the ID method yields better Q-matrix estimates with large sample sizes (500-1000).
• Q-matrices are estimated more precisely under discrete uniform distribution than multivariate normal threshold model.
• The ς² and ID methods are better for correcting errors, while MLE is better for retaining correct entries.

Conclusions:

• The choice of Q-matrix validation method depends on the specific cognitive diagnosis model and test quality.
• The ID method shows promise for DINA model with large sample sizes.
• Real data analysis confirms the effectiveness of the MLE method.