Using Multiverse Analysis to Highlight Differences in Convergent Correlation Outcomes Due to Data Analytical and...

Area of Science:

• Neuropsychology
• Psychometrics

Background:

• Neuropsychological research offers numerous study design possibilities.
• Ensuring the reliability of analytical choices in psychometric test validation is crucial.

Purpose of the Study:

• To demonstrate the multiverse/specification curve technique for assessing analytical pathway robustness.
• To examine how choices in study design affect convergent validation correlations for executive function tests.

Main Methods:

• Utilized the multiverse/specification curve technique.
• Analyzed data from 87 healthy adults and 117 stroke survivors.
• Conducted 2,220 different analyses varying sample groups, sizes, metrics, and covariates.

Main Results:

• The choice of sample group, sample size, and test metric influenced validation outcomes.
• Covariate inclusion did not significantly affect the observed correlation coefficients.
• Demonstrated variability in validation results based on analytical decisions.

Conclusions:

• Highlights the critical need for a priori justification of all psychometric validation study components.
• Emphasizes considering the intended purpose and application of a new tool during validation design.
• Advocates for rigorous methodological transparency in neuropsychological research.