A Graphical Toolkit for Longitudinal Dataset Maintenance and Predictive Model Training in Health Care.

Area of Science:

• Clinical Informatics
• Health IT
• Machine Learning Operations (MLOps)

Background:

• Predictive analytic models, including machine learning (ML) models, are increasingly integrated into electronic health record (EHR)-based decision support tools.
• These models offer potential care improvements but face challenges in validation, implementation, and long-term maintenance.
• Machine learning operations (MLOps) principles can guide the development of infrastructure for the entire ML lifecycle.

Purpose of the Study:

• To present conceptual prototypes for a novel predictive model management system.
• To evaluate the acceptability of the proposed system among diverse end-user groups.

Main Methods:

• Developed web-based MLOps interface prototypes using user-centered design, human-computer interaction, and ethical design principles.
• Conducted semistructured user interviews with health informaticians, clinical/data stakeholders, and chief information officers.
• Assessed preliminary usability via the System Usability Scale (SUS) and revised prototypes based on feedback.

Main Results:

• Prototypes feature design frameworks for ML lifecycle management: feature selection, training, deployment, maintenance, visualization, and collaboration.
• Users completed 71% of prompted tasks independently.
• The initial prototype achieved an average SUS score of 75.8, rated as 'good' (B grade).

Conclusions:

• The developed MLOps system prototypes are preliminarily usable and address a significant unmet need in clinical informatics.
• This work highlights the potential of MLOps to streamline the management of ML models in healthcare settings.