Factors influencing the effectiveness of mailed health surveys

The authors investigated sources of bias in health surveys by examining responses to their 1989 questionnaire mailed to 1,255 Massachusetts men who were eligible for dental care provided by the Department of Veterans Affairs. After a maximum of three mailings and one telephone call to nonrespondents, a total of 1,049 veterans had responded out of 1,228 finally determined to be eligible, a response rate of 85 percent. The investigators found that small differences in univariate estimates would have occurred had the field phase been terminated after the first mailing, which had a response rate of 61 percent. To evaluate multivariate distributions, they duplicated their previously published logistic regression model for sources of dental care, using only those who responded to the first and second mailings. Although model fits would have been substantively the same had the field phase been terminated after the first or the second mailings, analysis of parameter estimates and their statistical significances suggested bias that would have led to different substantive conclusions, in some instances. Another potential source of bias in surveys was found to be item omission. Fifty-eight percent of respondents answered all 46 survey questions, and 90 percent answered at least 91 percent of the questions. Fewer questions were answered by those whose responses were received last, but trends regarding missing data by age or education were not statistically significant. Although the survey using this methodology met all objectives, subject nonresponses, the ineligibility of potential respondents, item nonresponses, and skewed distributions of outcome variables combined to reduce the statistical power to detect differences among groups or to alter the analysis of the differences. These factors need to be planned for by investigators undertaking similar surveys.