Cross-Modal Graph Contrastive Learning with Cellular Images | JoVE Visualize

Area of Science:

Computational chemistry
Bioinformatics
Machine learning in drug discovery

Background:

Learning molecular representations is crucial for drug discovery, chemistry, and medicine.
Current graph neural network and self-supervised learning methods primarily use molecular structures, limiting their effectiveness in complex biological processes.
There is a need for methods that integrate molecular data with biological context.

Purpose of the Study:

To develop a unified framework for cross-modality pre-training by integrating molecular structures with phenotypic cell microscopy images.
To improve the learning of molecular representations by incorporating biological context from cell imaging data.
To enable mutual retrieval of molecules and corresponding cell images and infer functional molecules from cellular phenotypes.

Main Methods:

A unified framework was constructed for cross-modality pre-training using graph neural networks and self-supervised learning.
Multiple contrastive loss functions were employed to align molecular structures with high-content cell microscopy images.
The model was evaluated on tasks including mutual retrieval of molecules and images, inference of functional molecules from cellular images, and molecular property/clinical outcome predictions.

Main Results:

The proposed framework effectively aligns molecular structures with phenotypic cell images through contrastive learning.
The model demonstrated success in mutual retrieval tasks between molecules and their corresponding cellular images.
Significant improvements were observed in predicting molecular properties and clinical outcomes, outperforming existing methods.

Conclusions:

Cross-modality learning by integrating molecular structures with cell imaging data enhances molecular representation learning.
This approach bridges the gap between molecular information and biological phenotype, offering significant potential for drug discovery.
The model's ability to infer functional molecules and predict clinical outcomes highlights its utility in pharmaceutical research.