Retrospective comparative effectiveness research: Will changing the analytical methods change the results?

Area of Science:

• Biostatistics
• Health Services Research
• Oncology

Background:

• Retrospective cohort studies are crucial for comparing medical treatments.
• Biostatistical analytic choices can heavily influence outcomes in comparative effectiveness research.
• Inconsistent conclusions may arise from variations in analytical approaches.

Purpose of the Study:

• To investigate the impact of biostatistical analytic choices on outcomes in retrospective comparative effectiveness research.
• To assess how different statistical methods affect conclusions regarding cancer treatment efficacy.

Main Methods:

• Utilized a clinical scenario of survival in metastatic prostate, breast, and lung cancer.
• Ran over 300,000 regression models with varied analytic choices (propensity score matching, left truncation adjustment, landmark analysis, covariate combinations).
• Included data from 72,549 lung, 14,904 prostate, and 13,857 breast cancer patients.

Main Results:

• Basic analyses (without advanced adjustments) favored the addition of local therapy (Hazard Ratios <1, P-values <.001).
• Advanced adjustments (propensity score matching, left truncation, landmark analysis) produced results with non-significant P-values or Hazard Ratios >1, favoring systemic therapy alone.
• Increased statistical adjustments to reduce selection bias led to Hazard Ratios approaching 1.0.

Conclusions:

• Varying analytic choices in comparative effectiveness research can generate contradictory outcomes.
• Retrospective observational studies may report conflicting results on treatment effectiveness due solely to analytical decisions.
• These findings highlight the potential for bias and inconsistency in retrospective comparative effectiveness research.