Feasibility of extracting key elements from thoracic surgical operative notes using foundational large language...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine

Background:

• Operative notes are crucial for patient care but manual data extraction is time-consuming.
• Large language models (LLMs) show potential for automating this extraction process.
• This study evaluates LLMs for extracting key elements from thoracic surgical operative notes.

Purpose of the Study:

• To assess the feasibility of using LLMs for automatic extraction of critical information from thoracic surgical operative notes.
• To compare the accuracy and efficiency of different LLMs in this task.

Main Methods:

• Three LLMs (Gemini, ChatGPT, LLaMA) were used to extract data from 116 thoracic surgery operative notes.
• A blinded thoracic surgeon evaluated accuracy.
• Blinded physicians timed manual vs. LLM-based extraction of specific elements (approach, purpose, complications, devices left).

Main Results:

• Gemini achieved the highest accuracy (95.3%), followed by ChatGPT (93.1%) and LLaMA (87.7%).
• LLM use significantly reduced extraction time: 4.1s vs. 22.1s (emergency physician) and 3s vs. 30s (radiologist).
• P-value <0.001 indicates statistical significance.

Conclusions:

• LLMs can reliably extract key information from thoracic surgical notes with human oversight.
• LLM implementation offers significant time savings for clinicians without sacrificing accuracy.
• Comparative evaluations of LLMs are recommended to suit specific institutional needs.