Online randomized controlled experiments at scale: lessons and extensions to medicine.

Area of Science:

• Digital Health
• Health Services Research
• Data Science

Background:

• Technology firms conduct numerous online randomized controlled experiments (RCTs) daily, often called A/B tests, on millions of users.
• Unlike traditional medical RCTs, technology experiments have near-zero marginal cost and high recognized value for data-driven decision-making.

Purpose of the Study:

• To provide an overview of key scaling lessons learned from technology's use of online randomized controlled experiments.
• To explore the applicability of these lessons to healthcare decision-making and service delivery.

Main Methods:

• Focus on key metrics for evaluation and debugging, automatically computed for each experiment.
• Implementation of quick release cycles with automated experiment ramp-up and shut-down for agile and safe testing.
• Cultivating a 'test everything' culture due to the high failure rate of ideas and the potential for significant impact from minor changes.

Main Results:

• Online randomized controlled experiments at scale have enabled accelerated innovation and clear improvements in user experience and revenue in technology.
• Systematic series of randomized trials, even with numerous failures, lead to sustainable improvement in the technology sector.
• Lessons from technology's large-scale experimentation are applicable to healthcare as it increases digital interactions with patients.

Conclusions:

• Healthcare can benefit from considering and applying designs similar to technology's simple RCTs for decision-making and service delivery.
• Continuous and repeated evaluation of changes through randomized trials is crucial for accelerated healthcare improvements.