A Weakly-Supervised Named Entity Recognition Machine Learning Approach for Emergency Medical Services Clinical Audit

Area of Science:

• Computational medicine
• Machine learning applications in healthcare
• Emergency medical services research

Background:

• Clinical performance audits in Emergency Medical Services (EMS) are crucial for protocol adherence and training development.
• Current manual chart review for EMS audits is time-consuming and labor-intensive.
• Need for efficient, automated methods to support EMS quality improvement and research.

Purpose of the Study:

• To develop and evaluate a weakly-supervised machine learning approach for automated EMS clinical audits.
• To train a named entity recognition (NER) model for analyzing ambulance incident reports.
• To assess the feasibility of using limited labeled data for effective model training.

Main Methods:

• Utilized a dataset of 58,898 unlabeled ambulance incidents from the Singapore Civil Defence Force.
• Employed a weakly-supervised machine learning strategy to train NER models.
• Compared performance of different models, including BiLSTM-CRF and BERT-based architectures.

Main Results:

• Achieved high F1 scores (around 0.981 for entity type matching, 0.976 for strict evaluation) with only 5% labeled data.
• The BiLSTM-CRF model demonstrated superior efficiency, being 1-2 orders of magnitude lighter and faster than BERT models.
• Successfully demonstrated a proof-of-concept for automated EMS clinical auditing.

Conclusions:

• Weakly-supervised NER models can automate EMS clinical audits, improving efficiency and reducing manual workload.
• The proposed approach offers a viable method for EMS database research and quality improvement.
• Further external validation is recommended to confirm the generalizability of the findings.