Transformers for Molecular Property Prediction: Lessons Learned from the Past Five Years

Area of Science:

• Computational Chemistry
• Machine Learning
• Drug Discovery

Background:

• Molecular Property Prediction (MPP) is crucial for drug discovery, crop protection, and environmental science.
• Computational techniques for MPP have evolved from classical machine learning to deep learning.
• Transformer models represent a recent advancement in MPP.

Purpose of the Study:

• To review and distill insights on the application of transformer models for MPP.
• To analyze existing transformer models and identify key considerations for their training and fine-tuning.
• To highlight underexplored research areas and challenges in the field.

Main Methods:

• Analysis of current research on transformer models for MPP.
• Exploration of critical questions in transformer model training and fine-tuning.
• Identification of challenges in model comparison and standardization.

Main Results:

• Key questions regarding pretraining data, architecture, and objectives for transformer models in MPP were analyzed.
• Gaps in current research were identified, suggesting avenues for future exploration.
• Challenges in comparing different MPP models were highlighted, emphasizing the need for standardization.

Conclusions:

• Transformer models offer significant potential for advancing MPP.
• Further research is needed to optimize pretraining strategies and address model comparison challenges.
• Standardized methodologies are essential for robust evaluation and progress in transformer-based MPP.