The Maimed Martian, credible intervals and bias against benefit

A sad little story about a maimed Martian astronaut is used to illustrate a method of improving confidence interval (CI) calculations. CIs in medical statistics are currently calculated from the data available in a clinical trial or meta-analysis considered in isolation from all other information available on earth. Likewise, the Martian in the story uses only information available to it, in isolation from further information from earth. However, there is further objective knowledge available to people on earth to improve the Martian's estimate. In the same way, we have objective prior knowledge available to us outside of the current clinical trial results which we can use to improve CI calculations. This prior knowledge is incorporated into the CI calculations using Bayesian methods. The objective prior knowledge that is available is the fact that there were researchers who felt it worthwhile to conduct the trial and journal editors who felt it worthwhile publishing the results. It is shown here that the use of this information contracts the width of the log CI by a factor of about three quarters on average. Unlike standard CIs, these new intervals also have the advantage of being directly interpretable in terms of probabilities. These probabilities also enable calculation of improved point estimates. These calculations are applied to 100 randomly selected Cochrane systematic reviews and show serious problems in assessing medical treatments. For treatments not involving new drugs or devices, it is shown that there is evidence of a bias towards a negative assessment. The calculations here make a quantitative adjustment for publication bias. They show that the proportion of negative assessments do not reflect an appropriate adjustment for publication bias.