Replication of Randomized, Controlled Trials Using Real-World Data: What Could Go Wrong?

Area of Science:

• Health Research Methodology
• Regulatory Science
• Data Science

Background:

• Growing interest in utilizing real-world evidence (RWE) for regulatory decision-making.
• Randomized controlled trials (RCTs) are the established gold standard for research quality.
• A key approach involves replicating RCT findings using non-randomized real-world data (RWD).

Discussion:

• The "efficacy-effectiveness gap" highlights differences in medical intervention performance between controlled trials and real-world practice.
• Discrepancies between RCT findings and RWE are anticipated, even with high-quality RWD and robust analytical methods.
• Attributing discrepancies solely to RWD quality or analytical methods is inappropriate and hinders RWE acceptance.

Key Insights:

• Replication of RCT findings using RWD is a common but potentially misleading approach to assess RWE credibility.
• The inherent differences between experimental and real-world settings necessitate expecting variations in study outcomes.
• Focusing on RWD/RWE validation should not solely rely on direct congruence with RCT results.

Outlook:

• Re-evaluating the objectives of RCT replication efforts in the context of RWE assessment.
• Promoting a nuanced understanding of RWE validity that accounts for the efficacy-effectiveness gap.
• Encouraging the appropriate use of RWE in regulatory and decision-making processes by avoiding premature invalidation based on RCT discrepancies.