On the Unreliability of Test-Retest Reliability

Area of Science:

• Psychometrics
• Psychological Measurement
• Statistical Modeling

Background:

• The Test-Retest Coefficient (TRC) is a cornerstone of reliability in Classical Test Theory and psychological assessments.
• TRC relies on assumptions of perfectly stable true scores and independent error scores, which are often untested.
• Violations of these core assumptions can lead to significant biases in reliability estimation.

Purpose of the Study:

• To investigate the impact of violated TRC assumptions on reliability estimates.
• To examine TRC performance under varying conditions of true score stability and error score dependence.
• To assess the interpretability and suitability of TRC in applied psychological measurement.

Main Methods:

• Exploration of the theoretical foundations of TRC assumptions.
• Simulation studies using artificial data to model varying conditions.
• Analysis of TRC performance across different sample sizes, true score stabilities, and error score dependencies.

Main Results:

• Decreased true score stability leads to underestimation of reliability.
• Error score dependence can artificially inflate TRC values, suggesting false reliability.
• Assumption violations render the TRC underidentified, compromising its interpretability.

Conclusions:

• The TRC's validity is questionable when its underlying assumptions are not met.
• Findings challenge the TRC's utility in dynamic traits or uncontrolled measurement contexts.
• Alternative reliability metrics may be more appropriate when measurement assumptions are violated.