Comparing perturbation models for evaluating stability of neuroimaging pipelines

Area of Science:

• Neuroimaging
• Computational Neuroscience
• Numerical Analysis

Background:

• Growing concerns about the reproducibility of analytical software tools in science.
• Numerical instabilities are a key factor contributing to the lack of reproducibility.
• Previous observations of unexpected deviations in neuroimaging analyses due to variations in operating systems, software, or noise.

Purpose of the Study:

• To compare different perturbation models for introducing instabilities into neuroimaging pipelines.
• To assess the significance and impact of these instabilities on resulting data derivatives.
• To provide recommendations for evaluating tool stability in analytical settings.

Main Methods:

• Implementation of three perturbation models: targeted noise, Monte Carlo Arithmetic, and operating system variation.
• Application of these models to a typical neuroimaging pipeline, specifically structural connectome estimation.
• Analysis of the resulting derivatives to quantify the impact of numerical instabilities.

Main Results:

• Even low-order neuroimaging models, such as structural connectome estimation, are sensitive to numerical instabilities.
• Observed heterogeneity across participants, highlighting the interaction between tools and specific datasets.
• Demonstrated that stability is a critical comparison criterion for software tools, alongside accuracy and efficiency.

Conclusions:

• The stability of neuroimaging tools must be evaluated in the context of specific datasets and participant cohorts.
• Perturbation methods serve distinct use cases, including quality assurance, error detection, and sensitivity analysis.
• Further research is needed to understand how these findings scale to more complex tools and datasets.