Flexible Large Language Models for Multi-variable Extraction in Sarcoma Pathology Notes: A Pilot Study

Bone and soft tissue sarcomas are heterogeneous cancers. We developed a large language model (LLM)-based extraction workflow to extract six variables from surgical pathology reports of patients with musculoskeletal sarcoma: tumor diagnosis, greatest dimension, closest margin, tumor grade, necrosis, and viability. We hypothesized that the pipeline could achieve clinically acceptable accuracy and serve as a scalable solution to augment or replace manual chart abstraction in sarcoma registry development.

We retrospectively collected 1106 surgical pathology notes of patients who underwent surgical resection of bone or soft tissue sarcoma at three sites of a single institution. Notes were split into a training set (n = 353) and a testing set (n = 753). Prompt engineering and fine-tuning were used to query the open-source LLMs Llama3-70B-Instruct and Gemma4-31B-Instruct. Accuracy, F1 score, and specificity were calculated to evaluate the performance of extracted points.

On the testing set, both models achieved accuracy of at least 0.85 on extracting all data points and specificity of at least 0.84 on predicting "unknown" when data were not available in the note. The Gemma model achieved higher sensitivity on detecting positive margins than the Llama model: 0.92 versus 0.78. Both LLMs took less than 1 second on average to extract each data point from each pathology note.

Registry development for heterogeneous cancers such as sarcoma is limited by the labor-intensive nature of manual data abstraction, and our pipelines achieved promising results. These findings support the feasibility of LLM-based workflows as scalable and accurate alternatives to manual data abstraction for sarcoma clinical registries.