Multimodal foundation transformer models for multiscale genomics.

Area of Science:

• Computational Biology
• Genomics
• Artificial Intelligence

Background:

• Transformer models are increasingly vital for analyzing complex biological data.
• Recent advancements have led to multimodal foundation models integrating diverse data types.

Purpose of the Study:

• To review transformer architectures for multiscale biological data analysis.
• To categorize and evaluate transformer models for genomics tasks.
• To provide a roadmap for their application and development.

Main Methods:

• Categorization of transformer models into three tiers.
• Evaluation of capabilities in structural learning, representation transfer, and tasks like cell annotation.
• Discussion of challenges (tokenization, interpretability, scalability) and emerging approaches (masked modeling, contrastive learning, large language models).

Main Results:

• Transformers show promise across genomic sequences, single-cell transcriptomics, and spatial data.
• Emerging methods address current challenges in model application.
• A novel modular 'Super Transformer' architecture is proposed for integrating heterogeneous modalities.

Conclusions:

• Transformer models are foundational for multiscale, multimodal genomics.
• Practical guidance and open-source resources facilitate adoption.
• Future directions include developing integrated architectures for complex biological data analysis.