Enhancing Adverse Event Reporting With Clinical Language Models: Inpatient Falls

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Patient Safety

Background:

• Inpatient falls are frequently underreported, with self-reporting mechanisms missing up to 91% of incidents.
• Accurate tracking of adverse events like falls is crucial for patient safety and quality improvement.
• Existing methods for fall detection often rely on manual reporting, which can be burdensome and incomplete.

Purpose of the Study:

• To develop and evaluate a computational method for detecting patient fall events using clinical language models.
• To assess the performance of various language models, including Bidirectional Encoder Representations from Transformers (BERT) and Generative Pretrained Transformer (GPT)-4, in identifying falls from clinical notes.
• To compare the effectiveness of prompt programming with standardized prompts for GPT-4 in fall detection.

Main Methods:

• A retrospective observational study utilizing unstructured nursing notes from electronic health records and national patient safety reports.
• Data preprocessing involved anonymization, English translation, and semantic validation of 34,480 records (January 2015 - December 2019).
• Five language models, including fine-tuned BERT and GPT-4 with prompt programming, were explored, with performance measured by F1 scores and error analysis.

Main Results:

• Fine-tuned BERT models achieved the highest performance, with Bio+Clinical BERT and Korean BERT both reaching an F1 score of 0.98.
• GPT-4 with prompt programming demonstrated significantly improved performance (F1 score of 0.94 for Korean data, 0.85 for English data) compared to standardized prompts.
• Common errors included misclassification due to fall history, homonyms (false positives), and implicit expressions or missing context (false negatives).

Conclusions:

• Clinical language models offer a promising approach to computationally detect patient falls, significantly improving the identification of unreported incidents.
• This method can enhance adverse event tracking accuracy and reduce the reliance on manual self-reporting by nurses.
• Integrating language model-based fall detection with existing self-reporting systems can lead to more comprehensive and accurate patient safety monitoring.