Algorithm-agnostic significance testing in supervised learning with multimodal data.

Area of Science:

• Biostatistics
• Computational Biology
• Machine Learning

Background:

• Statistical inference is vital for decision-making, yet challenging with multimodal data (clinical, genomic, imaging).
• Black-box algorithms common in multimodal learning hinder traditional significance testing.
• Developing reliable methods for analyzing complex, high-dimensional datasets is essential.

Purpose of the Study:

• To introduce COvariance MEasure Tests (COMETs) for valid statistical inference in supervised learning with multimodal data.
• To demonstrate the application of COMETs for variable significance testing and modality selection.
• To provide a robust and powerful testing framework applicable to diverse high-dimensional datasets.

Main Methods:

• COMETs are presented as calibrated and powerful statistical tests.
• The tests can be integrated with any sufficiently predictive supervised learning algorithm.
• Implementation is available via the cometsR R package and pycomets Python library.

Main Results:

• COMETs were applied to high-dimensional, multimodal datasets for variable significance testing (e.g., drug-activity modulation by mutations).
• Modality selection was demonstrated for survival prediction using multiomics data and combined clinical/imaging data.
• Results aligned with domain knowledge, without invalidating type I error control through data-driven pre-processing.

Conclusions:

• COMETs offer a powerful and robust solution for statistical significance testing with multimodal data.
• The method facilitates discovery in complex biological and clinical research by identifying relevant variables and data sources.
• COMETs provide reliable inference capabilities, overcoming limitations of traditional methods with black-box algorithms.