Feasibility of extracting key elements from thoracic surgical operative notes using foundational large language...

Operative notes contain key information vital to subsequent care of the patient; however, manual extraction of this information can be time-consuming. Large language models (LLMs) offer promise in this regard. The aim of this study was to evaluate the feasibility of using LLMs to automatically extract key elements from thoracic surgical operative notes.

We instructed three LLMs, Gemini, ChatGPT, and LLaMA to extract key information from 116 thoracic surgery operative notes. A blinded thoracic surgeon assessed accuracy, while two blinded physicians (an emergency physician and a radiologist) recorded the time needed to extract the key elements ("approach", "purpose of the procedure", "complications", and "devices left") manually and using LLM construct.

Gemini demonstrated the highest overall accuracy (95.3%), followed by ChatGPT (93.1%) and LLaMA (87.7%). The mean time to extract key elements was significantly lower when using LLMs: 22.1 vs. 4.1 seconds for the emergency physician and 30 vs. 3 seconds for the radiologist (P<0.001).

With appropriate human supervision, LLMs can be used to automatically extract key information from thoracic surgical operative notes. Using LLMs for this task can save clinician time significantly, without compromising accuracy. Due to the availability of several LLM choices, including open- and closed-source, comparative evaluation is beneficial to address specific needs and constraints.