Balancing fit and parsimony to improve Q-matrix validation

Area of Science:

• Psychometrics
• Educational Measurement
• Cognitive Psychology

Background:

• Q-matrices are crucial for cognitive diagnosis models, mapping items to attributes.
• Expert-constructed Q-matrices can have errors, impacting classification accuracy.
• Existing validation methods like GDI and Wald use cut-off points, which may be suboptimal.

Purpose of the Study:

• To propose and evaluate the Hull method for Q-matrix validation.
• To address the limitations of existing methods, particularly the use of cut-off points.
• To offer a more flexible and accurate approach to Q-matrix specification.

Main Methods:

• The Hull method was developed to balance model fit and parsimony.
• It can be used with item discrimination measures like PVAF or pseudo-R ².
• A simulation study compared the Hull method with GDI and Wald.

Main Results:

• The Hull method demonstrated superior performance and faster computation times compared to GDI and Wald.
• Performance was particularly strong when the Hull method was used with the proportion of variance accounted for (PVAF).
• The Wald method performed well, but the GDI method showed poor results with a high number of attributes.

Conclusions:

• The Hull method provides a flexible and comprehensive solution for Q-matrix specification problems.
• Its absence of a cut-off point enhances its applicability in real-world settings.
• The method was illustrated using real data, demonstrating its practical utility.