Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet.

Area of Science:

• Materials Science
• Machine Learning
• Computational Chemistry

Background:

• Data-driven approaches are rapidly advancing materials science.
• Consistent benchmarking of model performance is essential for reliability and applicability.
• Existing benchmarks may not fully capture model nuances.

Purpose of the Study:

• To benchmark the Materials Optimal Descriptor Network (MODNet) against the MatBench v0.1 dataset.
• To evaluate MODNet's performance, particularly with smaller datasets.
• To highlight the importance of diverse metrics and uncertainty quantification in model evaluation.

Main Methods:

• Benchmarking MODNet against 13 tasks in the MatBench v0.1 dataset.
• Analyzing performance across various metrics.
• Utilizing ensemble MODNet models for uncertainty quantification.

Main Results:

• MODNet outperformed existing methods on 6 out of 13 tasks.
• MODNet achieved comparable performance on 2 additional tasks.
• MODNet demonstrated superior performance with datasets containing fewer than 10,000 samples.
• Ensemble modeling successfully quantified prediction uncertainty.

Conclusions:

• MODNet is a highly effective method for materials science.
• Diverse metrics and uncertainty assessment are critical for robust benchmarking.
• Addressing data imbalance and bias is key for successful real-world ML deployment in materials science.