Traditional Machine Learning, Deep Learning, and BERT (Large Language Model) Approaches for Predicting...

Area of Science:

• Health Informatics
• Clinical Natural Language Processing
• Machine Learning in Healthcare

Background:

• Predicting patient hospitalization from nurse triage notes can improve care.
• Model selection for this task requires careful consideration of computational infrastructure and budget constraints.
• Health systems face varying resource limitations that impact model implementation.

Purpose of the Study:

• To compare the performance of a deep learning model (Bio-Clinical-BERT) with a traditional machine learning model (BOW-LR-TF-IDF).
• To evaluate models based on differing computational requirements for predicting hospitalization from nurse triage notes.
• To inform model selection for health systems with diverse resource availability.

Main Methods:

• Retrospective analysis of 1,391,988 emergency department visits (2017-2022).
• Models trained on data from 4 hospitals within the Mount Sinai Health System.
• External validation performed on data from a fifth hospital.

Main Results:

• Bio-Clinical-BERT demonstrated slightly higher predictive performance (AUCs of 0.82-0.85) than BOW-LR-TF-IDF (AUCs of 0.81-0.84).
• Both models effectively utilized triage notes for hospitalization prediction.
• Performance differences were modest across varying training set sizes (10,000 to ~1,000,000 patients).

Conclusions:

• Simpler models like BOW-LR-TF-IDF may be sufficient in resource-limited environments.
• The choice of model should align with available computational resources and budget.
• Further research into alternative models is needed to optimize predictive performance for patient care and resource management.