Item-Score Reliability as a Selection Tool in Test Construction

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Item selection is crucial for developing reliable and valid tests.
• Traditional methods like corrected item-total correlation are widely used.
• Exploring alternative item-score reliability criteria can enhance test construction.

Purpose of the Study:

• To evaluate the effectiveness of different item-score reliability criteria for item selection.
• To compare item-assessment methods (MS, λ6, CA) against corrected item-total correlation.
• To determine optimal item selection procedures (bottom-up and top-down) based on population reliability.

Main Methods:

• Investigated MS, λ6, and Cronbach's Alpha (CA) as item-assessment methods.
• Used corrected item-total correlation as a benchmark for comparison.
• Employed Kendall's τ and Kendall's W to assess ordering resemblance and concordance across 1,000 replicated samples.

Main Results:

• Cronbach's Alpha (CA) and corrected item-total correlation most closely matched the ideal item ordering for both bottom-up and top-down procedures.
• All investigated methods showed improved performance with larger sample sizes.
• Greater variance in item discrimination parameters also enhanced method performance and ordering concordance.

Conclusions:

• Item-score reliability, particularly Cronbach's Alpha, is a valuable criterion for item selection in test construction.
• The effectiveness of item selection methods is influenced by sample size and item characteristics.
• These findings support the use of CA and corrected item-total correlation for optimizing test development.