Determining the Number of Attributes in Cognitive Diagnosis Modeling

Area of Science:

• Psychometrics
• Educational Measurement
• Cognitive Science

Background:

• Cognitive diagnosis models (CDMs) classify respondents based on attribute profiles, relying on accurate Q-matrix specification.
• Current Q-matrix estimation methods necessitate pre-setting the number of attributes, lacking systematic dimensionality assessment.
• Existing research on dimensionality assessment in CDMs is limited compared to factor analysis.

Purpose of the Study:

• To evaluate dimensionality assessment methods from factor analysis for determining the number of attributes in CDMs.
• To address the gap in systematic studies on CDMs dimensionality assessment.
• To provide evidence-based guidelines for selecting appropriate dimensionality assessment techniques in CDMs.

Main Methods:

• Evaluated parallel analysis, minimum average partial, very simple structure, DETECT, empirical Kaiser criterion, exploratory graph analysis, and a factor forest model.
• Included a model comparison approach using empirically estimated Q-matrices and AIC.
• Conducted a comprehensive simulation study varying number of attributes, item quality, sample size, and attribute correlations.

Main Results:

• Parallel analysis (Pearson correlations, mean eigenvalue), factor forest model, and model comparison (AIC) showed suitability for CDM dimensionality assessment.
• These methods achieved over 76% correct estimates across various conditions.
• Agreement among these three methods increased accuracy to 97%.

Conclusions:

• Parallel analysis, factor forest model, and model comparison are recommended for assessing the number of attributes in CDMs.
• These methods enhance the validity of Q-matrix estimation and the reliability of CDM-derived scores.
• The study offers practical guidelines for applied settings, illustrated with intelligence test data.