Geometry-Aware Line Graph Transformer Pretraining for Molecular Property Prediction

Molecular property prediction with deep learning approaches has gained much attention over the past years. Due to the scarcity of labeled molecules, there has been growing interest in self-supervised learning (SSL) methods that learn generalizable molecular representations from unlabeled data through pretraining. Molecules are typically modeled as 2-D topological graphs. However, their 3-D geometry is also important in determining molecular functionalities, and such 3-D information can enhance 2-D molecular representation learning. The 2-D-3-D pretraining has two key challenges: 1) how to construct a high-capacity backbone network to encode the dual-modality molecular information and 2) how to design effective pretraining tasks to learn both intermodality and intramodality relations simultaneously. To tackle these two challenges, we propose a Geometry-aware line graph transformer (Galformer) pretraining framework, a novel SSL method that aims to enhance molecular representation learning from a dual-modality perspective. Specifically, we first design a dual-modality line graph transformer backbone to adaptively encode a molecule's 2-D topological and 3-D geometric line graphs. The designed backbone has a high capacity and can capture critical structural information from both 2-D and 3-D modalities. Then, we devise two complementary pretraining tasks to achieve a comprehensive understanding at both intermodality and intramodality levels. These tasks provide properly supervised information and effectively extract discriminative 2-D and 3-D knowledge from unlabeled molecules. We evaluate Galformer against ten state-of-the-art baselines on 15 property prediction benchmarks via downstream fine-tuning. The experimental results show that Galformer consistently outperforms all baselines on both classification and regression tasks, demonstrating its effectiveness. Our source code is available at GitHub: https://github.com/peizhenbai/Galformer.