Survey Follow-Up in Randomized Trials With Dual Long- and Short-Form Instruments

Area of Science:

• Social Sciences
• Program Evaluation
• Survey Methodology

Background:

• Human services program evaluations frequently use surveys for outcome data collection in randomized controlled trials.
• Low survey response rates, especially in control groups, hinder evaluations from meeting high evidence quality standards set by clearinghouses.
• Traditional methods for boosting response rates are costly and time-consuming, with limited impact on reducing nonresponse bias.

Purpose of the Study:

• To introduce and evaluate a novel approach for increasing survey response rates in human services program evaluations.
• To assess the effectiveness of a shortened data collection instrument for reluctant respondents.
• To determine if the new method reduces nonresponse bias and impacts outcome data.

Main Methods:

• A new survey method was implemented, allowing respondents to complete a shortened version of the data collection instrument.
• The study compared the effectiveness of this new method against traditional approaches for boosting response rates.
• Analysis focused on response rates, cost-effectiveness, and the extent of nonresponse bias reduction.

Main Results:

• The new method successfully increased survey response rates and was more cost-effective than traditional methods.
• Data from reluctant respondents using the shortened instrument likely did not substantially reduce nonresponse bias on program impacts.
• However, the method identified significant differences in outcome means within study arms, valuable for descriptive purposes.

Conclusions:

• While the new method enhances response rates and reduces costs, its ability to mitigate nonresponse bias in impact evaluations is questionable.
• The findings align with previous research suggesting that aggressive pursuit of high response rates may yield diminishing returns regarding bias reduction.
• The method's utility in identifying within-study arm outcome differences makes it potentially useful for descriptive survey research.