Systematic reviews of observational studies frequently conclude based on meta-analyses of biased results: standards...

Systematic reviews of observational studies are common, yet the methodology of these systematic reviews and meta-analyses based on observational data are not standardized.

We reviewed systematic reviews, including observational studies published in JAMA, Lancet, BMJ, or Annals of Internal Medicine journals in 2024. We extracted information on the methodology and reporting of these reviews. We proposed minimal standards for conducting and reporting of systematic reviews, including observational studies.

We included 63 systematic reviews that compiled observational studies. Nearly all (51/63, 80.9%) performed meta-analyses of crude, unadjusted, results from observational studies. Only 22 of 63 (34.9%) addressed anywhere in the article or supplement adjusted association estimates from observational studies. Adjusted and unadjusted meta-analyses were presented in 9 of 63 (14.3%) reviews. The study designs included in the review were not presented in the abstract or in the methods section in about one-third of the reviews. These systematic reviews frequently concluded strongly on effects that cannot be assessed in observational data. We analyzed the methodology used in the systematic reviews addressing adjusted results from observational studies and proposed standards for analysis and reporting of systematic reviews of observational studies.

The use of crude data is common in systematic reviews, including observational studies, and ignores bias by indication, immortal time bias, and other biases. These reviews are misleading, and better standards are needed.

The method of proving treatments in medicine is through randomized controlled trials, where people are asked to participate in the trial and are then randomly allocated to one or another treatment. Observational studies look at what people are receiving in the routine clinical care and compare patients treated with one or another treatment. The problem with the comparisons in observational studies is that the patients receiving the different treatments are different. There is a reason why 1 treatment was selected over the other for the specific patient. When conducting observational studies, researchers use statistical methods to correct for this difference between patients receiving different treatments that are compared, an adjusted analysis. Systematic reviews summarize multiple studies on the same topic. They are influential because they compile the totality of knowledge on a topic. We highlight the problem that systematic reviews, including observational studies, usually look simply at the 2 groups of people receiving 1 or another treatment. Of 63 systematic reviews published in top journals in 2024, 51 (80.9%) presented a simple analysis of events in each group. The systematic reviews typically did not use the results of the adjusted analysis reported in the observational studies, which are the results that take into consideration the differences between patient groups. The results of these systematic reviews might exaggerate the effects of the treatments investigated. We call for better methods in systematic reviews, including observational studies, and mandate the use of results that are adjusted for the differences between patient groups that are inherent to observational studies comparing treatments.