Reliability for Tests With Items Having Different Numbers of Ordered Categories

Area of Science:

• Psychometrics
• Statistical Modeling
• Educational Measurement

Background:

• Traditional reliability coefficients like coefficient alpha have limitations with items having varying numbers of ordered categories.
• Assessing the reliability of tests with mixed-category items requires advanced statistical approaches.

Purpose of the Study:

• To introduce and evaluate a novel structural equation modeling (SEM) approach for estimating reliability in tests with items having different numbers of ordered categories.
• To compare the performance of the proposed SEM reliability coefficient against coefficient alpha and population reliability.

Main Methods:

• Structural Equation Modeling (SEM) was employed to develop a new reliability coefficient.
• A simulation study was conducted to compare reliability coefficients under various conditions, including different numbers of ordered categories, one-factor and bifactor structures, and score skewness.
• An empirical example using a test with dichotomous and trichotomous items was analyzed.

Main Results:

• The proposed SEM reliability coefficient demonstrated strong performance, closely approximating population reliability across most simulated conditions.
• The simulation results provided insights into the behavior of different reliability coefficients under varying psychometric properties.
• The empirical example highlighted the practical application and performance differences of the coefficients.

Conclusions:

• The proposed SEM-based reliability approach is a viable and accurate method for tests with items having different numbers of ordered categories.
• This study offers a valuable tool for researchers and practitioners needing to assess reliability in complex measurement instruments.
• The findings underscore the importance of using appropriate reliability estimation methods tailored to item characteristics.