Evaluation guidelines for machine learning tools in the chemical sciences

Area of Science:

• Chemistry
• Computer Science
• Data Science

Background:

• Machine learning (ML) offers significant potential for advancing chemical research and hypothesis generation.
• Current ML applications in chemistry suffer from diverse and non-standardized evaluation methodologies.
• This heterogeneity hinders the comparison and assessment of novel ML algorithms, potentially delaying the digitalization of chemistry.

Purpose of the Study:

• To critically discuss guidelines for developing and evaluating ML-based methods in chemistry, with a focus on supervised learning.
• To address the heterogeneity in evaluation techniques and metrics that impedes the comparison of ML algorithms.
• To promote transparency and credibility in ML applications within the chemical sciences.

Main Methods:

• A critical discussion of method development and evaluation guidelines tailored for ML in chemistry.
• Inclusion of diverse examples from various chemical disciplines to illustrate recommendations.
• Proposal of a checklist of retrospective and prospective tests to enhance ML credibility.

Main Results:

• Identified heterogeneity in evaluation study designs and metrics across ML applications in chemistry.
• Highlighted the difficulties in comparing and assessing the relevance of new ML algorithms due to inconsistent evaluation.
• Provided recommendations focusing on reporting completeness and standardized comparisons.

Conclusions:

• Standardized evaluation practices are essential for the reliable adoption of ML in chemistry.
• Implementing proposed guidelines and checklists can improve ML transparency, credibility, and comparability.
• Widespread adoption of best practices will foster informed ML utilization for real-world chemical problems.