Estimating Test-Retest Reliability in the Presence of Self-Selection Bias and Learning/Practice Effects.

Area of Science:

• Psychometrics
• Statistical modeling

Background:

• Test-retest reliability is crucial for assessment validity.
• Estimates often rely on self-selected test repeaters, introducing potential bias.
• Self-selection and time-varying effects can distort reliability measures.

Purpose of the Study:

• To develop and evaluate methods for unbiased test-retest reliability estimation from observational data.
• To address biases arising from self-selection and time-dependent effects in repeated testing.
• To improve the precision and generalizability of reliability estimates.

Main Methods:

• Development of methods including sample weighting, polynomial regression, and Bayesian model averaging.
• Application of these methods to observational data from test repeaters.
• Validation using both empirical and simulated data.

Main Results:

• Demonstrated reduction in bias for test-retest reliability estimates.
• Showcased improvements in the precision of reliability estimations.
• Empirical and simulated data supported the effectiveness of the proposed methods.

Conclusions:

• The developed methods effectively mitigate bias in test-retest reliability estimation.
• These techniques are valuable for settings with self-selected repeaters, such as admissions testing.
• The methods generalize to various repeated measurement scenarios with potential confounding factors.