Methods for Estimating Item-Score Reliability

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Reliability is typically assessed for total test scores.
• Item-score reliability offers insights into individual item performance and contribution to overall test reliability.
• Applications include identifying unreliable items and selecting single-item measures.

Purpose of the Study:

• To compare four distinct methods for estimating item-score reliability.
• To evaluate the accuracy and performance of these methods under various conditions.
• To identify the most suitable methods for practical application in test development and analysis.

Main Methods:

• The study compared four methods: Molenaar-Sijtsma (MS), Guttman's $$, Latent Class Reliability Coefficient (LCRC), and Correction for Attenuation (CA).
• A simulation study was conducted across six conditions: standard, polytomous items, unequal $$ parameters, two-dimensional data, long tests, and small sample sizes.
• Performance was evaluated based on median bias, variability (IQR), and percentage of outliers.

Main Results:

• Methods MS and CA were found to be the most accurate among the evaluated methods.
• Method LCRC produced nearly unbiased results but exhibited significant variability.
• Guttman's $$ method consistently underestimated item-score reliability, though it showed lower variability (IQR).

Conclusions:

• Methods MS and CA are recommended for accurate item-score reliability estimation.
• The choice of method may depend on specific study conditions and desired balance between bias and variability.
• Further research can explore the practical implications of these findings in test construction and psychometric analysis.