A New Method to Balance Measurement Accuracy and Attribute Coverage in Cognitive Diagnostic Computerized Adaptive...

Area of Science:

• Cognitive Psychology
• Educational Measurement
• Psychometrics

Background:

• Cognitive diagnostic computerized adaptive testing (CD-CAT) is crucial for accurate cognitive diagnosis assessment.
• Measurement accuracy in CD-CAT is influenced by item selection and attribute coverage.
• Existing methods for balancing attribute coverage require evaluation.

Purpose of the Study:

• Introduce and evaluate a new attribute coverage index: the ratio of test length to the number of attributes (RTA).
• Compare the RTA index with existing methods like the original item selection method (ORI) and the attribute balance index (ABI).
• Assess the impact of RTA on measurement accuracy and attribute coverage.

Main Methods:

• Simulations were conducted to compare the RTA index against ORI and ABI.
• The study analyzed measurement accuracy and attribute coverage under different item selection strategies.
• Item pool characteristics, specifically the number of attributes measured by items, were considered.

Main Results:

• The RTA method demonstrated comparable measurement accuracy to the ORI method across most item selection strategies.
• RTA generally yielded higher measurement accuracy than ABI, except when using the mutual information item selection method.
• RTA favors items measuring multiple attributes, unlike ABI which prefers single-attribute items.
• RTA showed superior attribute coverage compared to ORI but was outperformed by ABI with longer tests.

Conclusions:

• The RTA index offers a viable approach for CD-CAT, particularly with item pools containing many multi-attribute items.
• RTA provides a balance between measurement accuracy and attribute coverage, making it a valuable tool in cognitive diagnosis.
• The choice of attribute coverage index should consider test length and item pool characteristics.