Dimensionality Assessment in Forced-Choice Questionnaires: First Steps Toward an Exploratory Framework

Area of Science:

• Psychometrics
• Quantitative Psychology
• Survey Methodology

Background:

• Forced-choice (FC) questionnaires are used to minimize social desirability bias in self-reports.
• Confirmatory models for FC data assume known structures, which may not fit empirical data.
• Exploratory models are often necessary, requiring accurate dimensionality assessment.

Purpose of the Study:

• To systematically evaluate the performance of five dimensionality assessment methods for FC questionnaire data.
• To identify the most accurate and least biased methods for determining the number of dimensions in FC data.
• To provide practical recommendations for FC questionnaire design and analysis.

Main Methods:

• A Monte Carlo simulation study was conducted.
• Five dimensionality assessment methods were examined: Kaiser Criterion, Empirical Kaiser Criterion, Parallel Analysis (PA), Hull Method, and Exploratory Graph Analysis.
• Simulated FC data varied in dimensionality, items per dimension, response formats, block composition, factor loadings, inter-factor correlations, and sample size.

Main Results:

• The Maximal Kaiser Criterion and Parallel Analysis (PA) methods showed the highest accuracy and lowest bias.
• Method performance improved with the inclusion of heteropolar or unidimensional blocks.
• Increased questionnaire length also enhanced the performance of these methods.

Conclusions:

• Parallel Analysis (PA) and Maximal Kaiser Criterion are recommended for dimensionality assessment in forced-choice questionnaires.
• Thoughtful questionnaire design, including block composition, is crucial for accurate dimensionality assessment.
• These findings offer practical guidance for researchers using FC questionnaires.