When do confounding by indication and inadequate risk adjustment bias critical care studies? A simulation study

Area of Science:

• Critical care medicine
• Observational study design
• Health services research

Background:

• Observational studies in critical care often confound treatment comparisons with patient illness severity.
• This confounding can lead to incorrect conclusions about treatment effectiveness or harm.

Purpose of the Study:

• To investigate the risk of incorrect conclusions in observational studies due to inadequate risk adjustment for illness severity.
• To assess how confounding by severity of illness impacts treatment effect estimation.

Main Methods:

• Monte Carlo simulations of observational studies evaluating a hypothetical treatment's effect on mortality.
• Varied treatment effect, confounding strength, study size, and risk-adjustment accuracy (AUROC).
• Measured rates of inaccurate conclusions and odds ratios for mortality.

Main Results:

• Adequate risk adjustment generally allowed accurate estimation of true treatment effects.
• Worsening risk adjustment increased incorrect conclusions, particularly in large studies (n=10,000).
• Low accuracy risk adjusters (AUROC < 0.66) falsely indicated harm from beneficial treatments, with odds ratios up to 1.4.

Conclusions:

• Large, confounded observational studies risk incorrect findings even with "acceptable" risk adjustment.
• Large effect sizes may be falsely attributed to true associations.
• Reporting AUROC of risk adjustment can help evaluate confounding risk.