The Limits of Inductive Reasoning for Clinical Evidence Appraisal: A Simulation Study

Background To establish the possible likelihood of a body of evidence, inductively judged to be of 'low bias risk'/'high-quality' according to a limited set of appraisal criteria, of actually being error-free. Methods A total of 45 simulation trials were generated and randomly assigned to 0-5 errors out of a total of 65 error domains. The trials were then appraised for errors with a simulated appraisal tool consisting of five pre-specified error domains. Trial appraisal yielded either true positive, true negative, false negative or false positive results. From these values, the negative likelihood ratio (-LR) with 95% confidence interval (CI) was computed. -LR computation was repeated 25 times, each with newly generated random values for all 45 trials. The individual results of all 25 runs were statistically pooled. The pooled -LR result with 95% CI was interpreted as how likely a 'low bias risk'/'high-quality' rated body of evidence is actually error-free. Results The pooled -LR was 0.84 (95% CI: 0.80 - 0.88, I² = 0.0%). The result suggests that error-free evidence is only 1.2 times more likely to be rated as 'low bias risk'/'high-quality' than evidence containing some form of error. Conclusions The likelihood of a 'low bias risk'/'high-quality' rated body of evidence being actually error-free is small and the inductive generalization from any limited, pre-specified set of appraisal criteria rarely justifies a high level of confidence that a 'low bias risk'/'high-quality' rating of clinical evidence reflects the true effect of a certain treatment without being affected by error.