Strengths and weaknesses in the methodology of survey-based research in surgery: A call for standardization

Survey-based studies are often the basis of policy changes; however, the methodologic quality of such research can be questionable. Methodologic reviews of survey-based studies have been conducted in other medical fields, but the surgical literature has not been assessed.

All citations published in 9 major surgical journals from 2002 to 2019 were screened for studies administering surveys to health care professionals. Descriptive and methodologic data were collected by 2 reviewers who also assessed the transparency and quality of the methodology. Agreement between reviewers was assessed using a weighted κ-statistic. Survey quality metrics were measured, descriptive statistics were calculated, and regression analysis was used to assess the association between subjective overall study quality and objective quality metrics.

We included 271 articles in our analysis; the weighted-κ for reviewer quality assessment was 0.69 and for transparency assessment was 0.71. Deficiencies were identified in questionnaire development methodology and reporting, in which the median number of developmental steps reported was 1 (of 8) and in the reporting of incomplete/missing data where 63% of studies failed to report how incomplete questionnaires were managed; 70% of studies failed to report missing data. Overall subjective quality was positively associated with objective quality metrics.

The deficiencies identified in the surgical literature highlight the need for improvement in the conduct and reporting of survey-based research, both in the surgical literature and more broadly. Adoption of a standardized reporting guideline for survey-based research may ameliorate the deficiencies identified by this study and other investigations.