Can We Trust Observational Studies Using Propensity Scores in the Critical Care Literature? A Systematic Comparison...

Area of Science:

• Critical Care Medicine
• Epidemiology
• Biostatistics

Background:

• Randomized clinical trials (RCTs) are the gold standard for evaluating therapeutic interventions.
• Propensity score (PS) methods are increasingly used in critical care research to emulate RCTs using observational data.
• The agreement between PS studies and RCTs in critical care is not well-established.

Purpose of the Study:

• To assess the degree of agreement between propensity score studies and randomized clinical trials in critical care research.
• To compare treatment effect estimates derived from PS studies and RCTs on mortality outcomes.

Main Methods:

• Systematic search for PS studies and corresponding RCTs in critical care and general medicine literature.
• Matching of PS studies and RCTs based on patient characteristics, interventions, and mortality outcomes.
• Meta-analysis was performed for topics with multiple studies to obtain summary treatment effects.

Main Results:

• 21 PS studies were matched with 58 RCTs across 18 distinct comparisons for mortality.
• Only one significant difference in treatment effect was found (hyperoncotic albumin vs. crystalloid fluids).
• Estimates from PS studies and RCTs differed by over 30% in one-third of comparisons, with no systematic over- or underestimation.

Conclusions:

• Propensity score studies published in high-impact journals generally yield results consistent with RCTs in critical care.
• Caution is advised when interpreting PS studies due to occasional contradictions with RCT findings.
• Predicting disagreements between PS studies and RCTs remains challenging.