Relating Model Performance to Embedding Distributions in Molecular Machine Learning.

Area of Science:

• Machine Learning
• Computational Chemistry
• Data Science

Background:

• Effective molecular representation is crucial for molecular machine learning, often determined by trial-and-error.
• Model selection typically relies on predictive performance, potentially overlooking deeper structural relationships.
• Representational alignment techniques, like centered kernel alignment (CKA), offer a principled way to compare models beyond performance.

Purpose of the Study:

• To investigate the link between representational alignment and performance differences in molecular machine learning models.
• To develop actionable insights for selecting molecular representations and optimizing model training strategies.
• To introduce a dataset-level statistic predicting a dataset's position in the alignment-performance space.

Main Methods:

• Theoretical analysis of representational alignment for linear regression to establish performance bounds.
• Empirical validation of alignment-performance relationships across 661 classification datasets.
• Introduction and application of the mean minimum class distance (MMCD) statistic across 23 molecular representations and ten datasets.

Main Results:

• Representational alignment theoretically upper-bounds performance gaps in linear regression, predicting an "exclusion zone" for highly aligned models.
• Empirical validation confirmed this exclusion zone across numerous datasets.
• A strong correlation was observed between high model alignment and low MMCD, indicating dataset-specific structure influences alignment.

Conclusions:

• Representational alignment is fundamentally linked to performance differences, providing insights beyond predictive metrics.
• Low alignment suggests exploring alternative molecular representations for performance gains.
• High alignment indicates that increasing training data size is a more effective strategy for improvement.