A method for rapid machine learning development for data mining with doctor-in-the-loop.

Area of Science:

• Health Informatics
• Machine Learning in Healthcare
• Clinical Data Management

Background:

• Classifying historical clinical free-text data into research formats is challenging for clinicians.
• Existing methods may lack adaptability and customizability for diverse natural language databases.

Purpose of the Study:

• To develop an interactive machine-learning (ML) model training methodology for classifying free-text clinical notes.
• To ensure the ML model is adaptable to various natural language databases and allows researcher-defined categories.
• To evaluate the accuracy and speed of the developed ML model on distinct clinical datasets.

Main Methods:

• Developed a user interface for medical experts to train and evaluate the ML algorithm.
• Utilized a 'label-train-evaluate' loop with expert-defined categories on two unique datasets: death certificates and aeromedical retrievals.
• Validated the model's performance on separate, blinded datasets.

Main Results:

• Achieved 94.7% macro-accuracy for classifying death certificate records (stroke types).
• Achieved 92.4% macro-accuracy for classifying aeromedical retrieval records (medical, surgical, trauma, obstetric, psychiatric).
• Model development and training took approximately 2-2.5 hours per dataset.

Conclusions:

• The developed interactive ML training methodology efficiently classifies free-text clinical notes.
• The model demonstrates high accuracy and speed across different health datasets.
• This approach facilitates health service research by converting unstructured clinical data into coded formats.