Classification Accuracy of Mixed Format Tests: A Bi-Factor Item Response Theory Approach

Area of Science:

• Educational Measurement
• Psychometrics
• Cognitive Psychology

Background:

• Mixed format tests (combining multiple-choice [MC] and constructed response [CR] items) are increasingly prevalent in educational assessments.
• The underlying latent structure and implications of these mixed-format tests remain areas of active research and debate.
• Questions persist regarding whether constructed response items assess unique reasoning skills beyond multiple-choice items.

Purpose of the Study:

• To investigate the dimensionality of mixed format tests using bi-factor models.
• To compare the accuracy of scores derived from bi-factor analysis versus unidimensional analysis.
• To evaluate the impact of test dimensionality on the classification accuracy of overall grades.

Main Methods:

• Application of bi-factor models to 10 mixed format tests from the Advanced Placement (AP) Program.
• Comparison of score accuracy between bi-factor and unidimensional analyses.
• Assessment of classification accuracy for overall test grades.

Main Results:

• The degree of multidimensionality in mixed format tests varied across different subjects.
• Subject-specific multidimensionality was influenced by the disattenuated correlation between MC and CR subtest scores.
• Unidimensional analysis showed minimal decrements in classification accuracy when disattenuated correlations exceeded 0.90.

Conclusions:

• The dimensionality of mixed format tests is not uniform and is subject to the interrelationship between item formats within specific content areas.
• For practical purposes, unidimensional scoring approaches can maintain high classification accuracy in mixed format tests under certain conditions of high item format correlation.
• Further research is warranted to understand the nuanced interplay between item formats and dimensionality in diverse assessment contexts.