Knowledge Distillation for Molecular Property Prediction: A Scalability Analysis.

Area of Science:

• Computational Chemistry
• Machine Learning
• Materials Science

Background:

• Knowledge distillation (KD) is a model compression technique.
• Graph neural networks (GNNs) are used for molecular property prediction.
• Reducing computational costs in molecular modeling is crucial.

Purpose of the Study:

• Investigate KD's effectiveness in molecular property prediction.
• Evaluate KD across domain-specific and cross-domain tasks.
• Utilize state-of-the-art GNNs: SchNet, DimeNet++, and TensorNet.

Main Methods:

• Applied KD to QM9 dataset for quantum mechanical properties.
• Conducted cross-domain evaluations using ESOL (logS) and FreeSolv (ΔGhyd) datasets.
• Analyzed student-teacher model alignment using cosine similarity.

Main Results:

• KD improved regression performance on QM9, with DimeNet++ student models showing up to 90% R² improvement.
• Smaller student models achieved comparable or better R² gains, with 2x size reduction.
• Cross-domain KD enhanced ESOL and FreeSolv predictions (SchNet ≈65% logS gain).
• Embedding analysis showed significant student-teacher alignment.

Conclusions:

• KD is a robust strategy for enhancing molecular representation learning.
• KD improves efficiency and predictive performance in molecular property prediction.
• Findings have implications for cheminformatics, materials science, and drug discovery.