Analyzing Potential Non-Ignorable Selection Bias in an Off-Wave Mail Survey Implemented in a Long-Standing Panel...

Area of Science:

• Survey methodology
• Statistical bias analysis
• Longitudinal data analysis

Background:

• Probability sample weighting methods assume random selection and ignorable nonresponse.
• Violations of these assumptions can lead to selection bias and inaccurate population estimates.
• The 2019 Health Survey Mailer (HSM), a sub-study of the Health and Retirement Study (HRS), may have selection bias due to eligibility criteria.

Purpose of the Study:

• To investigate potential non-ignorable selection bias in the 2019 HSM.
• To compare the effectiveness of different weighting adjustments for mitigating bias.
• To assess the adequacy of standard design-based weighting for HRS off-wave mail surveys.

Main Methods:

• Utilized a novel method for estimating measures of unadjusted bias for proportions (MUBP).
• Incorporated aggregate population data (means, variances, covariances) to inform proportion estimates.
• Compared HSM proportions under three conditions: unweighted, standard design-based weights, and MUBP adjustment.

Main Results:

• Modest differences were observed between standard weighted and MUBP-adjusted estimates for four out of ten outcomes.
• This suggests some evidence of non-ignorable selection bias in the HSM.
• Despite detected bias, standard design-based weighting methods proved sufficient for correction.

Conclusions:

• Standard design-based weighting methods are appropriate and sufficient for correcting potential selection bias in HRS off-wave mail surveys.
• The novel MUBP method provides valuable insights into bias but may not be strictly necessary when standard methods suffice.
• The findings support the continued use of established weighting procedures for Health and Retirement Study data collections.