What Do You Think? Using Expert Opinion to Improve Predictions of Response Propensity Under a Bayesian Framework

Area of Science:

• Survey Methodology
• Statistics
• Data Science

Background:

• Responsive survey designs adapt protocols using accumulating paradata.
• Case-level predictions, like response propensity, tailor data collection for efficiency and quality.
• Predictions using only partial current data can be biased, limiting tailoring effectiveness.

Purpose of the Study:

• To evaluate Bayesian approaches for mitigating bias in response propensity predictions.
• To explore eliciting prior beliefs from expert survey managers for new or data-scarce surveys.
• To compare Bayesian predictions with standard methods using paradata only and historical data.

Main Methods:

• Survey managers provided expected attempt-level response rates for subgroups.
• Prior distributions for response propensity model coefficients were developed from expert responses.
• Response propensity predictions were compared using expert priors, paradata-only, and historical data.

Main Results:

• Bayesian approaches incorporating expert knowledge demonstrated improved response propensity predictions.
• Expert-informed priors offered protection against bias present in paradata-only predictions.
• The performance of expert-informed priors was comparable to or better than historical data-based priors.

Conclusions:

• Eliciting prior beliefs from experienced survey managers is a viable strategy for Bayesian responsive survey designs.
• Bayesian methods offer a robust framework for handling bias in predictive modeling for surveys.
• Incorporating expert knowledge enhances the reliability and effectiveness of adaptive data collection strategies.