Applying multiverse analysis to experience sampling data: Investigating whether preprocessing choices affect robustness of conclusions

The experience sampling method (ESM) has revolutionized our ability to conduct psychological research in the natural environment. However, researchers have a large degree of freedom when preprocessing ESM data, which may hinder scientific progress. This study illustrates the use of multiverse analyses regarding preprocessing choices related to data exclusion (i.e., based on various levels of compliance and exclusion of the first assessment day) and the calculation of constructs (i.e., composite scores calculated as the mean, median, or mode) by reanalyzing established group differences in negative affect, stress reactivity, and emotional inertia between individuals with and without psychosis. Data came from five studies and included 233 individuals with psychosis and 223 healthy individuals (in total, 26,892 longitudinal assessments). Preprocessing choices related to data exclusion did not affect conclusions. For both stress reactivity and emotional inertia of negative affect, group differences were affected when negative affect was calculated as the mean compared to the median or mode. Further analyses revealed that this could be attributed to considerable differences in the within- and between-factor structure of negative affect. While these findings show that observed differences in affective processes between individuals with and without psychosis are robust to preprocessing choices related to data exclusion, we found disagreement in conclusions between different central tendency measures. Safeguarding the validity of future experience sampling research, scholars are advised to use multiverse analysis to evaluate the robustness of their conclusions across different preprocessing scenarios.