Trial and error: A hierarchical modeling approach to test-retest reliability

Area of Science:

• Cognitive Neuroscience
• Psychometrics
• Neuroimaging

Background:

• Test-retest reliability measures measurement consistency over time.
• High reliability is essential for scientific studies, particularly those examining individual differences.
• Growing evidence suggests poor reliability in common behavioral and functional neuroimaging tasks.

Purpose of the Study:

• To propose a hierarchical framework for estimating reliability independent of trial-level variability.
• To compare reliability estimates between a new framework and conventional intraclass correlation methods.
• To investigate factors influencing reliability estimates, such as sample size and cross-trial variability.

Main Methods:

• Developed a hierarchical framework to estimate reliability as a correlation separate from trial-level variability.
• Analyzed how reliability estimates differ between the proposed framework and traditional intraclass correlation methods.
• Assessed the impact of trial and subject sample sizes and cross-trial variability on reliability estimates.

Main Results:

• Conventional intraclass correlation frameworks may underestimate reliability by relying on condition-level modeling.
• Cross-trial variability is substantial in most tasks, necessitating a large number of trials (e.g., >100) for precise reliability estimation.
• Reliability estimation diverges significantly between trial-level and condition-level modeling approaches.

Conclusions:

• The proposed hierarchical framework offers a more accurate estimation of test-retest reliability.
• Researchers may need to increase the number of trials in behavioral and neuroimaging tasks to achieve reliable individual difference measures.
• Tools like TRR and 3dLMEr are recommended for applying trial-level models to enhance data reliability.