ITMol: a molecular image-text foundation model bridging the semantic gap for property prediction and retrieval

Multimodal foundation models demonstrate remarkable understanding ability to perform accurate molecular representation, including SMILES sequence-text and molecular graph-text representations. Molecular images represent another molecular modality that provides a more intuitive way to capture the spatial structural relationships of molecules. Existing models suffer from integrating image-text modalities for molecular property prediction, largely due to (1) the scarcity of high-quality image-text paired datasets, as existing molecular databases predominantly contain structural representations (SMILES, graphs) with limited descriptive text tailored for visual molecular understanding, and (2) the fundamental semantic gap between molecular images and textual descriptions, where complex visual features must be precisely aligned with domain chemical terminology and functional descriptions.

In this study, we present a molecular image-text foundation model, named ITMol, pretrained on 500k molecular image-text pairs. Specifically, ITMol addresses the scarcity of high-quality image-text paired datasets by constructing a comprehensive dataset through collection from chemical databases and automated generation using MolT5 to create high-quality image-text pairs, and tackles the fundamental semantic gap between molecular images and textual descriptions by implementing a sophisticated cross-attention mechanism designed specifically for molecular modalities, coupled with three complementary self-supervised learning strategies. We demonstrate the high performance of ITMol in molecular property prediction across 8 benchmark datasets, achieving an ROC-AUC score of 0.885 on the BBBP dataset. ITMol shows high R@5 scores on a dataset of 100k molecules in the cross-modal retrieval task. We further illustrate interpretability of ITMol using key chemical structures and functional groups.

ITMol provides an effective framework for integrating 2D molecular images and molecular textual descriptions. The results show that image-text multimodal pretraining can improve molecular representation learning for property prediction and retrieval, offering a useful direction for multimodal molecular modeling.