Evaluating generalizability of artificial intelligence models for molecular datasets.

Area of Science:

• Computational Biology
• Machine Learning
• Genomics

Background:

• Deep learning models excel at molecular sequencing data but their generalizability to novel sequences is poorly understood.
• Existing benchmarks using metadata-based (MB) or sequence-similarity-based (SB) splits fail to capture the full spectrum of cross-split overlap, potentially mischaracterizing model performance.

Approach:

• Introduced Spectra, a spectral framework for evaluating model generalizability across a spectrum of cross-split overlaps.
• Spectra plots model performance against decreasing overlap and uses the area under the curve as a generalizability metric.
• Applied Spectra to 18 diverse sequencing datasets and 19 state-of-the-art deep learning models, including large language models and graph neural networks.

Key Points:

• Traditional SB and MB splits provide an incomplete assessment of deep learning model generalizability.
• Spectra demonstrates that decreasing cross-split overlap consistently reduces model performance in a task- and model-dependent manner.
• No single model dominated across all tasks, but deep learning models showed task-specific generalization capabilities.

Conclusions:

• Spectra offers a more comprehensive evaluation of deep learning model generalizability in biological sequence modeling.
• The findings underscore the importance of considering sequence similarity and overlap in benchmark design.
• This work advances the understanding of how foundation models generalize within biological applications.