Leveraging machine learning to examine engagement with a digital therapeutic

Area of Science:

• Digital health interventions
• Machine learning in healthcare
• Human-computer interaction

Background:

• Digital Therapeutics (DTx) are software-based interventions for medical conditions.
• DTx generate objective data on patient engagement, including interaction quality.
• Assessing the quality of engagement is crucial for treatments like cognitive interventions.

Purpose of the Study:

• To present a novel technique for real-time measurement of user interaction quality with digital treatments.
• To evaluate the effectiveness of this technique in classifying user engagement.
• To explore applications in clinical decision-making and research.

Main Methods:

• Developed a machine learning model trained on expert-labeled data.
• Classified user interactions during a multitasking cognitive training task (mission).
• Evaluated the model's performance on a held-out test set.

Main Results:

• The machine learning classifier achieved high accuracy (0.94) and F1 score (0.94) in predicting expert-derived labels.
• The technique reliably distinguishes between users 'using as intended' and 'not using as intended'.
• The approach provides evaluations at the mission (approx. 4-minute) level.

Conclusions:

• This technique offers a valuable tool for assessing the quality of patient engagement with DTx.
• Findings support enhanced shared decision-making and communication among patients, caregivers, and healthcare providers.
• The method has potential applications in clinical trials and personalized digital interventions.