mmContext: an open framework for multimodal contrastive learning of omics and text data

Area of Science:

• Computational Biology
• Bioinformatics
• Machine Learning in Biology

Background:

• Multimodal approaches are crucial for integrating omics data with biological text.
• Existing frameworks lack standardization for comparing omics representations with various text encoders.
• A unified workflow is needed for systematic analysis of multimodal biological data.

Purpose of the Study:

• To introduce mmContext, a novel, accessible, and standardized framework for multimodal learning.
• To enable systematic comparison of omics data embeddings generated with different text encoders.
• To facilitate reproducible and accessible multimodal learning for omics-text integration.

Main Methods:

• Developed mmContext, a lightweight and extensible multimodal embedding framework using Sentence Transformers.
• Integrated omics data (numeric representation in AnnData.obsm) with text data using Hugging Face text encoders.
• Provided pipelines for training, evaluation, and data preparation for multimodal models.

Main Results:

• Trained and evaluated models for RNA-Seq and text integration.
• Demonstrated utility through zero-shot classification of cell types and diseases across four independent datasets.
• Showcased mmContext's capability in multimodal learning for biological data.

Conclusions:

• mmContext provides a standardized framework for multimodal omics-text integration.
• The framework enables reproducible research through open-source models, datasets, and tutorials.
• Facilitates advanced biological insights by combining diverse data types effectively.