A confidence interval approach to investigating non-response bias and monitoring response to postal questionnaires

The aim was to develop an alternative method of investigating non-response bias in postal surveys, including a method of calculating a final full (100%) coverage confidence interval which avoids the wide intervals of existing approaches.

As part of a two stage survey of disablement in the community, a first phase postal questionnaire was sent to 25,168 households in Calderdale, West Yorkshire, England. Confidence intervals were calculated to investigate the precision of estimates using a "no bias" model, where the prevalence in non-responders is assumed to be the same as in responders.

A total of 21,889 postal questionnaires were returned (87%), representing households containing 42,826 people aged 16 years and over. This was achieved by the original post (1st wave, 57% response); two further postal follow ups (2nd and 3rd waves, taking the response to 73% and 81% respectively), the latter including a small personal call back; and a final postal follow up (the 4th wave).

The cumulative estimated prevalence of those with dependence was plotted as the survey progressed. The final wave full coverage estimated prevalence for those aged 16-64 years was 12.8 per 1000 with 95% confidence intervals of 11.3-14.4 per 1000. The integrity of this estimate holds as long as the true prevalence in non-responders is within the calculated non-response confidence interval under the no bias assumption, 9.7-16.0 per 1000 people. This latter interval represents the tolerance of prevalence in non-responders implied by the no bias assumption.

The findings have general implications for monitoring non-response bias in postal screening questionnaires. The confidence interval approach developed in this paper offers an alternative to existing regression based estimates, giving an indication of the range of prevalence amongst non-responders that could be tolerated before the no bias assumption used by the model is breached. It is suggested that this approach can be used to determine both the extent of bias, and to aid decision making about the appropriate juncture to terminate follow up. It highlights the potential, particularly in the context of a computerised survey operation, of methodological investigation occurring simultaneously with survey operation.