What is the value of statistical testing of observational data?

Statistical analysis of medical data aims to reveal patterns that can aid in decision making for future cases and, hopefully, improve patient outcomes. Large and bias-free datasets, such as those produced in formal randomized clinical trials, are necessary to make such analyses as reliable as possible. For a host of reasons, randomized trials are, unfortunately, relatively uncommon in veterinary medicine and surgery, implying that less ideal datasets (mostly observational data) must form the basis for much of our decision making regarding treatment of individual patients under our care. In this review, we first describe the common shortcomings of many observational veterinary datasets when viewed in comparison with their optimal counterparts and highlight how the deficiencies can lead to unreliable conclusions. We illustrate how many of the interpretative problems associated with observational data, predominantly various forms of bias, are not solved, and may even be exacerbated, by statistical analysis. We emphasize the need to examine summary data and its derivation in detail without being lured into relying upon P values to draw conclusions and advocate for completely omitting statistical analysis of many observational datasets. Finally, we present some suggestions for alternative statistical methods, such as propensity scoring and Bayesian methods, which might help reduce the risk of drawing unwarranted, and overconfident, conclusions from imperfect data.