Relative molecule self-attention transformer.

Area of Science:

• Computational Chemistry
• Machine Learning
• Drug Discovery

Background:

• Predicting molecular properties is vital for cost-effective drug discovery.
• Machine learning (ML) is increasingly used for molecular property prediction.
• Challenges remain in optimizing ML for small datasets common in drug discovery.

Purpose of the Study:

• Introduce a novel ML architecture for molecular representation learning.
• Address limitations in current ML approaches for drug discovery datasets.
• Improve the efficiency and accuracy of molecular property prediction.

Main Methods:

• Developed the Relative Molecule Self-Attention Transformer (RMST) architecture.
• Utilized relative self-attention and 3D molecular representations.
• Implemented a two-step pre-training procedure for model optimization.

Main Results:

• RMST effectively captures atom and bond interactions using domain-specific inductive biases.
• The pre-training strategy requires tuning minimal hyperparameters.
• Achieved performance comparable to state-of-the-art models across various tasks.

Conclusions:

• RMST offers a powerful new tool for molecular representation learning in drug discovery.
• The proposed pre-training method enhances ML model performance on small datasets.
• This work contributes to more efficient and accurate drug design processes.