Transformers for molecular property prediction: domain adaptation efficiently improves performance

Area of Science:

• Cheminformatics and computational drug discovery.
• Development and application of deep learning models, specifically transformer architectures, for molecular property prediction.

Background:

• Molecular transformer models are increasingly used in drug discovery, typically pre-trained on vast unlabeled datasets (e.g., ZINC, ChEMBL).
• The actual benefit of large-scale pre-training for molecular property prediction accuracy remains insufficiently understood.
• Current limitations in transformer model performance for predicting ADME (Absorption, Distribution, Metabolism, Excretion) properties necessitate further investigation.

Purpose of the Study:

• To systematically evaluate the impact of pre-training dataset size on molecular transformer model performance for ADME property prediction.
• To investigate the effectiveness of domain adaptation strategies, using chemically informed objectives, to enhance model performance.
• To benchmark adapted transformer models against existing large-scale models and traditional machine learning approaches.

Main Methods:

• Trained transformer models on varying sizes of pre-training datasets (up to ~400K molecules).
• Applied domain adaptation using multi-task regression on small, domain-relevant datasets (<4K molecules) focusing on physicochemical properties.
• Evaluated model performance across seven datasets covering five ADME endpoints: lipophilicity, permeability, solubility, microsomal stability, and plasma protein binding.
• Compared adapted models against large-scale transformers (MolFormer, MolBERT) and baseline methods (RDKit descriptors, Morgan fingerprints with Random Forest).

Main Results:

• Increasing pre-training dataset size beyond 400K-800K molecules showed no significant improvement in predictive performance for the studied ADME endpoints.
• Domain adaptation on small, relevant datasets (<4K molecules) led to significant performance improvements across all evaluated datasets (P < 0.001).
• A model pre-trained on ~400K molecules and domain-adapted outperformed larger models like MolFormer and matched MolBERT's performance.
• Incorporating chemically, physically, and topologically informed features consistently improved performance, irrespective of the underlying architecture (traditional or transformer-based).

Conclusions:

• Domain adaptation is a critical strategy for enhancing molecular transformer models, offering substantial performance gains over large-scale pre-training alone.
• Aligning model training and adaptation with chemically meaningful tasks and domain-specific data is crucial for advancing molecular property prediction.
• The study provides practical guidelines and open-source models (available on HuggingFace) for developing more accurate and robust cheminformatics tools.