Lost in a large EEG multiverse? Comparing sampling approaches for representative pipeline selection.

Area of Science:

• Neuroscience
• Data Science
• Computational Biology

Background:

• The multitude of data analysis pipelines contributes to low replicability in scientific research.
• This issue is particularly prominent in electroencephalography (EEG) data analysis.
• Multiverse analyses aim to assess result robustness across various pipelines, but computing all is often infeasible, leading to reliance on sampling methods and potential multiverse sampling uncertainty.

Purpose of the Study:

• To develop and evaluate an open-source tool for comparing the representativeness of pipeline samples in multiverse analyses.
• To assess how different sampling methods impact the estimation of robustness in EEG data analysis.
• To mitigate bias introduced by sampling in large-scale neuroimaging studies.

Main Methods:

• A 528-pipeline multiverse was computed using EEG recordings for an emotion classification task (predicting extraversion from the Late Positive Potential).
• Three sampling methods (random, stratified, active learning) were applied to select 5% (26 pipelines) of the full multiverse.
• The representativeness of model fit distributions across these samples was evaluated.

Main Results:

• Variability in the representativeness of model fit distributions was observed across different sampling methods.
• Active learning and stratified sampling demonstrated the highest representativeness of the full multiverse.
• Replicability was assessed via cross-validation, and reproducibility was explored concerning pipeline sample sizes.

Conclusions:

• Representative pipeline sampling is essential for mitigating bias in large multiverse analyses.
• The choice of sampling strategy significantly influences the reliability of robustness estimates.
• The developed tool aids researchers in selecting appropriate sampling methods for more dependable neuroimaging research.