Reliability measures in item response theory: manifest versus latent correlation functions

Area of Science:

• Psychometrics
• Statistical Modeling

Background:

• Item response theory (IRT) models are generalized linear/non-linear mixed models.
• Reliability of observed scores (manifest correlation) is scientifically important but difficult to compute in IRT.
• Existing approximations like Cronbach's alpha and Fisher's information have significant limitations.

Purpose of the Study:

• To derive and evaluate Taylor series based reliability measures for IRT models.
• To compare these new measures against latent correlation, Fisher's information, and exact reliability.
• To provide accurate and computationally efficient reliability estimation for manifest correlations in IRT.

Main Methods:

• Derivation of Taylor series based reliability measures.
• Comparison using algebraic expressions, Monte Carlo simulations, and real data analysis.
• Evaluation of latent correlation, Fisher's information, and proposed Taylor series approximations.

Main Results:

• Latent correlations are consistently higher than manifest correlations.
• Fisher's information measure exhibits unstable and sometimes negative values.
• Taylor series based approximations closely match exact reliability, offering superior performance.

Conclusions:

• Taylor series based reliability measures are accurate and computationally efficient.
• These new measures are recommended for estimating manifest reliability in IRT models.
• Avoid using latent correlations or Fisher's information as substitutes for manifest reliability due to potential inaccuracies.