Automatically Identifying Event Reports of Workplace Violence and Communication Failures using Large Language Models.

Area of Science:

• Healthcare safety
• Natural Language Processing
• Artificial Intelligence in Medicine

Background:

• Safety event reporting is crucial for patient and staff well-being.
• Current reporting systems face challenges with variability and resource limitations, hindering timely analysis and trend identification.
• These limitations delay improvements in healthcare workplace safety.

Purpose of the Study:

• To explore the utility of large language models (LLMs) in automating the classification of safety event report narratives.
• To assess the effectiveness of LLMs in identifying specific safety event categories like workplace violence and communication failures.
• To lay the groundwork for automated labeling systems to enhance workplace safety.

Main Methods:

• Utilized large language models to analyze the text of safety event reports.
• Trained and evaluated LLMs for their ability to classify narratives into predefined categories.
• Measured classification performance using F1 scores for specific event types.

Main Results:

• LLMs demonstrated high accuracy in classifying safety event narratives.
• Achieved an F1 score of 0.80 for physical violence and 0.94 for verbal abuse.
• Attained an F1 score of 0.94 for communication failures.

Conclusions:

• Large language models show significant promise for automated safety event report classification.
• This technology can expedite the identification of critical safety trends, including workplace violence and communication issues.
• Automated labeling through LLMs can ultimately contribute to a safer healthcare environment.