Evaluating automated machine learning platforms for use in healthcare.

Area of Science:

• Health Informatics
• Machine Learning in Healthcare
• Clinical Decision Support

Background:

• Automated machine learning (Auto ML) platforms offer potential for developing clinical machine learning (ML) models.
• Selecting the appropriate Auto ML platform is critical for successful implementation in healthcare settings.
• Existing selection processes may not adequately address the unique requirements of clinical ML model development.

Purpose of the Study:

• To describe the development and application of a checklist for selecting automated machine learning (Auto ML) platforms.
• To provide a structured approach for healthcare organizations to choose Auto ML tools for clinical ML model creation.
• To ensure selected Auto ML platforms meet the specific needs of local health districts and clinical applications.

Main Methods:

• A three-step process was employed: identification of key requirements, a market scan of Auto ML platforms, and an assessment process with defined desired outcomes.
• A multidisciplinary team including clinicians, data scientists, and stakeholders contributed to developing the evaluation criteria.
• The developed checklist, comprising 21 functional and 6 non-functional criteria, was applied to vendor submissions.

Main Results:

• A comprehensive checklist with 21 functional and 6 non-functional criteria was finalized for Auto ML platform selection.
• The checklist was successfully applied in selecting a platform for a use case involving the creation of a machine learning heparin dosing model.
• The evaluation process facilitated informed decision-making for acquiring a suitable Auto ML platform.

Conclusions:

• A validated checklist can aid healthcare organizations in selecting appropriate Auto ML platforms for clinical ML model development.
• The developed checklist is adaptable to the diverse ML needs of various healthcare organizations.
• Further validation in larger, multi-site studies is recommended to confirm the checklist's broader applicability and effectiveness.