An explorative multiverse study for extracting differences in P3 latencies between young and old adults

It is well established that P3 latencies increase with age. Investigating these age-related differences requires numerous methodological decisions, resulting in pipelines of great variation. The aim of the present work was to investigate the effects of different analytical pipelines on the age-related differences in P3 latencies in real data. Therefore, we conducted an explorative multiverse study and varied the low-pass filter (4 Hz, 8 Hz, 16 Hz, 32 Hz, and no filter), the latency type (area vs. peak), the level of event-related potential analysis (single participant vs. jackknifing), and the extraction method (manual vs. automated). Thirty young (18-21 years) and 30 old (50-60 years) participants completed three tasks (Nback task, Switching task, Flanker task), while an EEG was recorded. The results show that different analysis strategies can have a tremendous impact on the detection and magnitude of the age effect, with effect sizes ranging from 0% to 88% explained variance. Likewise, regarding the psychometric properties of P3 latencies, we found that the reliabilities fluctuated between r_tt  = .20 and 1.00, while the homogeneities ranged from r_h  = -.12 to .90. Based on predefined criteria, we found that the most effective pipelines relied on a manual extraction based on a single participant's data. For peak latencies, manual extraction performed well for all filters except for 4 Hz, while for area latencies, filters above 8 Hz produced desirable results. Furthermore, our findings add to the evidence that jackknifing combined with peak latencies can lead to inconclusive results.