Scaling up behavioral science interventions in online education

Area of Science:

• Educational Technology
• Behavioral Science
• Online Learning

Background:

• Online education is expanding, but student completion rates remain a challenge.
• Prior studies suggested behavioral science interventions could improve online student persistence.
• Evidence for intervention effectiveness across diverse online contexts was limited.

Purpose of the Study:

• To test established behavioral science interventions across a large-scale online learning environment.
• To identify conditions influencing the effectiveness of these interventions.
• To investigate the potential of machine learning for predicting and addressing achievement gaps.

Main Methods:

• A 2.5-year study involving 247 online courses from Harvard, MIT, and Stanford, with approximately 250,000 students globally.
• Iterative hypothesis testing with cyclical preregistration between data collection waves.
• Analysis of self-regulation and value-relevance interventions, and machine learning approaches.

Main Results:

• Interventions showed reduced average effectiveness when scaled across diverse contexts.
• Self-regulation interventions boosted early engagement but not final completion.
• Value-relevance interventions closed achievement gaps in developing countries for specific courses.
• Machine learning had limited success in forecasting gaps or personalizing interventions.

Conclusions:

• Scaling behavioral science interventions can significantly decrease their average effectiveness.
• Context, individual student characteristics, and timing are crucial for intervention success.
• Iterative research is essential for understanding the nuanced application of behavioral science in online education.