Ecological Momentary Assessment as a Measure of Intervention Change: Evaluation in 4 Digital Mental Health Trials

Ecological momentary assessment (EMA) is increasingly being incorporated into intervention studies to acquire a more fine-grained and ecologically valid assessment of change. The added utility of including relatively burdensome EMA measures in a clinical trial hinges on several psychometric assumptions, including that these measure are (1) reliable, (2) related to but not redundant with conventional self-report measures (convergent and discriminant validity), (3) sensitive to intervention-related change, and (4) associated with a clinically relevant criterion of improvement (criterion validity) above conventional self-report measures (incremental validity).

This study aimed to evaluate the reliability, validity, and sensitivity to change of conventional self-report versus EMA measures of rumination improvement.

We conducted a secondary analysis of data from 4 trials of app-based meditation interventions (N=412). Participants included adolescents (samples 1-2; n=232, 56.3%; aged 12-15 years), college students (sample 3; n=88, 21.4%; aged 18-21 years), and adults (sample 4; n=92, 22.3%; aged 18-80 years). All participants completed validated conventional self-report rumination measures (Children's Response Styles Questionnaire, Response Styles Questionnaire, or Perseverative Thinking Questionnaire) at baseline and the postintervention time point and EMA rumination assessments throughout 3- to 4-week intervention periods (mean compliance 72.2%-80.2%). We examined the reliability of conventional self-report and EMA measures, computed correlations between measurement approaches, compared rumination improvement using standardized effect sizes, and tested criterion validity by predicting depression symptom improvement.

Reliability of conventional self-report rumination was high at baseline (Cronbach α=0.89-0.94) and the postintervention time point (Cronbach α=0.90-0.95) but decreased for residualized change scores (ρ= 0.71-0.90). EMA rumination showed high mean-level reliability (ρ=0.89-0.96) but substantially lower reliability for change scores (ρ=0.50-0.77). Conventional self-report and EMA measures showed medium correlations at individual time points (r=0.28-0.47; P<.01 in all cases) but nonsignificant correlations between change scores (Child and Adolescent Research in Emotion samples: r=0.03 and P=.66; Healthy Minds Program sample: r=0.20 and P=.06). Conventional self-report detected larger intervention effects than EMA measures (Cohen d=0.37 vs 0.14 for group differences; Cohen d=0.77 vs 0.17 for pretest-posttest change). Despite modest intercorrelations, both measurement approaches demonstrated incremental predictive validity for depression improvement. In the Child and Adolescent Research in Emotion samples, improvements in both conventional rumination (b=0.04; SE 0.017; P=.03) and EMA rumination (b=0.03; SE 0.014; P=.04) significantly predicted reduced depression symptoms until the 12-week follow-up. Similarly, in the Healthy Minds Program sample, both conventional rumination change (b=0.23; SE 0.060; P<.001) and EMA rumination change (b=20.49; SE 9.94; P=.04) predicted depression improvement.

Conventional self-report and EMA measures of rumination provide distinct and clinically meaningful information. When deciding to use EMA in intervention studies, researchers should carefully consider the psychometric properties of their measures and the precise construct they intend to capture.