Conversations that count: Lessons from evaluating a men's digital mental health response during COVID-19

Area of Science:

• Digital Health
• Program Evaluation
• Public Health Interventions

Background:

• The COVID-19 pandemic significantly disrupted established practices in digital health intervention (DHI) evaluation.
• Movember's men's digital health intervention, Movember Conversations, required a rapid product pivot to remain relevant during the pandemic.

Purpose of the Study:

• To detail the disruption and innovation in digital evaluation practices at Movember.
• To examine the implications of a product pivot on DHI evaluation methods.
• To outline principles for redesigning evaluations in agile settings.

Main Methods:

• Focus on the adaptation of evaluation methods to a modified digital health product.
• Implementation of a user-centered approach to evaluation design.
• Utilization of proportionate data collection and mixed, flexible methodologies.

Main Results:

• The evaluation was redesigned to align with the modified Movember Conversations product.
• Real-time, practical insights were delivered through adapted evaluation methods.
• Four key principles for agile evaluation redesign were identified and applied.

Conclusions:

• The study provides key lessons and reflections on agile evaluation in digital health settings.
• The findings support other evaluators planning digital health evaluations, particularly in response to unforeseen disruptions.
• Successful adaptation involved a user-centered approach, flexible methodologies, and agile reporting.