Related Experiment Video
Updated: Sep 16, 2025
10:31
Intraoperative Gastroscopy for Tumor Localization in Laparoscopic Surgery for Gastric Adenocarcinoma
Published on: August 9, 2016
12.9K
Language Models for Multilabel Document Classification of Surgical Concepts in Exploratory Laparotomy Operative
Jeremy A Balch1,2, Sasank S Desaraju3, Victoria J Nolan1
1Department of Surgery, University of Florida College of Medicine, Gainesville, FL, United States.
JMIR Medical Informatics
|July 9, 2025
Summary
Large language models (LLMs) significantly improve data extraction from surgical notes, outperforming traditional methods. Further refinement is needed for reliable use in surgical research and quality improvement.
Area of Science:
- Natural Language Processing (NLP)
- Machine Learning in Healthcare
- Surgical Data Science
Background:
- Operative notes are crucial for clinical care, research, and billing, but manual data extraction is time-consuming.
- Traditional NLP methods like bag-of-words (BoW) and tf-idf have limitations for complex surgical note analysis.
- Large Language Models (LLMs) show promise in augmenting or replacing traditional NLP for surgical text mining.
Purpose of the Study:
- To develop and evaluate LLMs for expediting data extraction from surgical operative notes.
- To compare the performance of LLMs against traditional NLP techniques for multilabel classification of surgical concepts.
- To assess the utility of different LLM architectures and the impact of context on classification accuracy.
Main Methods:
- A dataset of 388 exploratory laparotomy notes was annotated for 21 surgical concepts.
- Compared traditional BoW and tf-idf models with encoder-only (Clinical-Longformer) and decoder-only (Llama 3) transformer models.
- Evaluated multilabel classification using 5-fold cross-validation, F1-score, and Hamming Loss (HL), with and without contextual information.
Main Results:
- The decoder-only Llama 3 model achieved the highest performance (micro F1-score 0.88, HL 0.11), significantly outperforming BoW, tf-idf, and Clinical-Longformer.
- Incorporating context improved Llama 3's F1-scores by an average of 0.16.
- Performance varied across concepts, with challenges in classifying contamination and handling complex cases like prior or simultaneous operations.
Conclusions:
- Off-the-shelf autoregressive LLMs demonstrate superior performance over traditional NLP and encoder-only models for classifying surgical operative notes.
- LLMs offer a potential solution for streamlining retrospective reviews in surgery.
- Further development is necessary to address semantic nuances and edge cases for reliable application in research and quality improvement.
