The effect of missing data and imputation on the detection of bias in cognitive testing using differential item...

Area of Science:

• Psychometrics
• Epidemiology
• Neuroscience

Background:

• Item response theory (IRT) and differential item functioning (DIF) methods detect group differences in cognitive test performance.
• These methods are increasingly used to identify bias related to hearing impairment, race, and education.
• The impact of missing data on DIF detection has not been previously studied, despite substantial missing cognitive data in epidemiologic studies.

Purpose of the Study:

• To investigate the effect of missing data patterns and imputation on differential item functioning (DIF) detection in cognitive tests.
• To explicate how missing data influences bias detection in neurocognitive research.

Main Methods:

• Utilized data from the Atherosclerosis Risk in Communities Neurocognitive Study (ARIC N=3,580).
• Artificially imposed missing data patterns to simulate different levels of missingness (10%, 30%, 50%).
• Compared DIF detection accuracy with and without multiple imputation.

Main Results:

• When missing data was unrelated to cognitive performance or DIF group, no systematic error was observed.
• Systematic error in DIF detection occurred when missing data was related to both cognitive performance and DIF group membership.
• Multiple imputation reduced the median DIF detection error from -0.03, -0.08, -0.14 to -0.02, -0.04, -0.08 SD units for 10%, 30%, and 50% missingness, respectively.

Conclusions:

• Inferences from DIF testing can be incorrect when missing data is present and related to performance and group.
• Multiple imputation can mitigate bias in DIF detection.
• It is crucial to account for missing data when assessing bias in cognitive testing to ensure valid research findings.