Assessing the performance of statistical validation tools for megavariate metabolomics data

Area of Science:

• Biostatistics
• Bioinformatics
• Omics Data Analysis

Background:

• Statistical model validation tools (e.g., cross-validation, permutation tests) are crucial for assessing model performance and stability.
• Their reliability for megavariate datasets, where variables significantly exceed subjects, remains largely unexamined.

Purpose of the Study:

• To evaluate the performance of statistical validation tools on megavariate metabolomics data.
• To determine the impact of sample size relative to the number of variables on validation reliability.

Main Methods:

• Partial Least Squares Discriminant Analysis (PLS-DA) models were constructed using LC-MS lipidomic datasets.
• Model performance and predictability were assessed using 10-fold cross-validation, permutation testing, and independent test datasets.

Main Results:

• Cross-validation error rates varied widely (0% to 60%), increasing as the number of subjects decreased.
• Test error rates ranged from 5% to 50%.
• Validation tool reliability diminished significantly with smaller subject-to-variable ratios.

Conclusions:

• Validation tools are less trustworthy for megavariate data when the number of variables is much larger than the number of subjects.
• The results are highly dependent on the specific sample composition.
• These tools may not serve as reliable warnings for sample size issues or sampling representativity.