When Nonresponse Mechanisms Change: Effects on Trends and Group Comparisons in International Large-Scale Assessments

Area of Science:

• Educational measurement
• Statistical modeling
• Psychometrics

Background:

• Item nonresponse in large-scale assessments can be nonignorable, potentially biasing parameter estimates.
• Missing data mechanisms and their prevalence may change over time in trend analyses.

Purpose of the Study:

• Investigate changes in missing data mechanisms and percentages over time in real data.
• Evaluate the performance of different missing data handling approaches under these conditions.
• Assess the practical implications for trend estimates in large-scale assessments.

Main Methods:

• Reanalysis of Programme for International Student Assessment (PISA) data from 35 OECD countries.
• Simulation study with generated data allowing for time-varying missing data mechanisms and amounts.
• Application of multiple group models for nonignorable missing values.

Main Results:

• Missing data mechanisms and proportions varied significantly across time, countries, and domains in PISA data.
• Trend estimates were biased when time-varying missing data mechanisms were not accounted for.
• Standard methods (scoring omissions as wrong, ignoring omissions, constant mechanism models) led to biased trend estimates.

Conclusions:

• Accurate trend estimates require accounting for changes in missing data mechanisms and amounts over time.
• Multiple group models for nonignorable missing values are recommended when missing data characteristics change.
• Adjusting for changing missing data mechanisms can alter observed trends, as seen in the PISA 2009 reading literacy example for Ireland.