Simulating A/B testing versus SMART designs for LLM-driven patient engagement to close preventive care gaps

Area of Science:

• Health Services Research
• Biostatistics
• Health Communication

Background:

• Population health initiatives frequently use outreach to address gaps in preventive care, like overdue screenings and immunizations.
• Personalizing outreach messages for diverse patient groups is difficult, as traditional A/B testing needs large sample sizes for limited message variations.

Purpose of the Study:

• To compare the statistical power and false positive rates of A/B testing versus Sequential Multiple Assignment Randomized Trials (SMART) for personalized patient communication.
• To evaluate the cost-effectiveness and net benefit of A/B testing and SMART designs in patient engagement strategies.

Main Methods:

• Microsimulations were employed to model and compare A/B testing and SMART designs.
• Analyses considered various effect sizes and sample sizes to assess statistical power and false positive rates.
• Cost-effectiveness and net benefit were evaluated across different simulation scenarios.

Main Results:

• Sequential Multiple Assignment Randomized Trials (SMART) demonstrated better cost-effectiveness and net benefit across all simulated scenarios.
• SMART designs showed superior power for detecting heterogeneous treatment effects (HTEs) in later randomization stages.
• A/B testing requires larger sample sizes and is less efficient for nuanced message optimization.

Conclusions:

• SMART trials are a more efficient and cost-effective method for developing personalized patient communications compared to traditional A/B testing.
• The findings support the use of SMART designs in population health initiatives aiming for optimized patient engagement and resource allocation.
• SMART's advantage in detecting HTEs becomes more pronounced as patient populations become more homogeneous during outreach.