Item-Score Reliability in Empirical-Data Sets and Its Relationship With Other Item Indices

Area of Science:

• Psychometrics
• Statistical analysis

Background:

• Reliability is typically assessed for total scores.
• Item-score reliability offers insights into individual item repeatability within a group.

Purpose of the Study:

• To introduce and compare three novel methods for estimating item-score reliability: Method MS, Method $$, and Method CA.
• To evaluate the performance of these new methods against established item indices such as item-rest correlation, item-factor loading, item scalability, and item discrimination.
• To determine realistic values for item-score reliability in empirical datasets and establish tentative minimum values for item analysis.

Main Methods:

• Three methods for item-score reliability estimation (MS, $$, CA) were developed.
• These methods were compared with four established item indices: item-rest correlation, item-factor loading, item scalability, and item discrimination.
• Empirical datasets were analyzed to monitor realistic values and investigate the relationships between the methods.

Main Results:

• Item-score reliability can be estimated using methods MS, $$, and CA.
• The study investigated the relationships between these novel reliability methods and traditional item indices.
• Realistic values for item-score reliability were observed in empirical data.

Conclusions:

• Item-score reliability estimation provides valuable information beyond total score reliability.
• The study offers a comparison of new methods against existing indices, aiding in their selection for item analysis.
• Tentative minimum values for item-score reliability methods are proposed for practical application in item analysis.