REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Area of Science:

• Cross-disciplinary application of machine learning in scientific research.

Background:

• Widespread adoption of machine learning (ML) methods in science is hindered by frequent failures in validity, reproducibility, and generalizability.
• These failures impede scientific progress, foster consensus around incorrect findings, and damage the credibility of ML-driven research.
• Consistent patterns of ML application failures are observed across diverse scientific disciplines.

Purpose of the Study:

• To provide actionable recommendations for the rigorous application and transparent reporting of machine learning in scientific research.
• To address the common pitfalls encountered in ML-based scientific studies across various fields.

Main Methods:

• Development of the REFORMS checklist (Recommendations for Machine-learning-based Science) based on an extensive literature review.
• Consensus-building process involving 19 researchers from computer science, data science, mathematics, social sciences, and biomedical sciences.
• Checklist comprises 32 questions and accompanying guidelines.

Main Results:

• The REFORMS checklist offers a structured approach to enhance the quality of ML-based scientific research.
• Guidelines are designed to improve validity, reproducibility, and generalizability of ML applications.
• The checklist was developed through a rigorous, interdisciplinary consensus process.

Conclusions:

• The REFORMS checklist serves as a vital resource for researchers designing and executing studies.
• It aids peer reviewers in evaluating the methodological soundness of ML-based research.
• Journals can utilize REFORMS to enforce higher standards for transparency and reproducibility in scientific publications.