Clever materials: when models identify good materials for the wrong reasons.

Area of Science:

• Materials Science
• Computational Chemistry
• Machine Learning

Background:

• Machine learning (ML) shows promise for accelerating materials discovery.
• High performance on benchmarks does not guarantee models have learned underlying chemical principles.

Purpose of the Study:

• To investigate if bibliographic confounding, rather than chemical understanding, drives ML model performance in materials property prediction.
• To assess the impact of metadata (author, journal, publication year) on predictive accuracy.

Main Methods:

• Evaluated ML models on five materials science tasks: MOF stability, perovskite solar cell efficiency, battery capacity, and TADF emitter wavelength.
• Trained models using standard chemical descriptors and analyzed prediction accuracy.
• Tested a hypothesis where bibliographic metadata ('bibliographic fingerprints') were used as sole input for prediction.

Main Results:

• Models trained on chemical descriptors accurately predicted metadata like author, journal, and publication year.
• Using only bibliographic fingerprints as input, prediction performance was sometimes competitive with traditional descriptor-based methods.
• Demonstrated that many existing datasets are susceptible to non-chemical explanations for ML model success.

Conclusions:

• Current materials discovery datasets may contain spurious correlations, leading to inflated performance metrics.
• Routine falsification tests, such as group/time splits and metadata ablation, are crucial for validating ML models.
• Distinguishing between predictive utility and genuine chemical understanding is essential for advancing the field.