Fully Automated Abstraction of Longitudinal Breast Oncology Records with Off-The-Shelf Large Language Models

Area of Science:

• Oncology
• Medical Informatics
• Computational Biology

Background:

• Manual chart abstraction is a significant bottleneck in clinical research, particularly for oncology studies relying on narrative data.
• Key outcomes like disease recurrence and treatment history are often confined to clinical notes, limiting study scale and quality.
• Developing automated methods for data abstraction is crucial for advancing observational and epidemiologic oncology research.

Purpose of the Study:

• To assess the performance of large language models (LLMs) in abstracting diverse variables from complex, longitudinal oncology records.
• To determine if LLM-based abstraction can achieve performance comparable to expert medical oncologists.
• To evaluate the feasibility of using LLMs for creating research-grade retrospective datasets.

Main Methods:

• An open-source, HIPAA-compliant pipeline was developed using commercially available LLMs for data abstraction.
• Key variables including diagnosis/recurrence dates, clinical stage, biomarkers, and systemic therapies were extracted from unstructured clinical notes, pathology reports, and other records.
• A reference standard was created by expert breast oncologists abstracting the same variables; LLM performance was compared against this standard and inter-oncologist variability.

Main Results:

• The best-performing LLM achieved high concordance with expert oncologists for critical variables (e.g., 99% for recurrence status, 100% for BRCA1/2 variants, 96% for HER2 status).
• LLM performance for anti-cancer drug extraction approached inter-oncologist variability, outperforming research coordinators.
• Survival and hazard ratio estimates derived from LLM-abstracted data were similar to those from expert-derived datasets.

Conclusions:

• General-purpose LLMs, integrated into a fixed retrieval pipeline, can effectively abstract variables from complex oncology records with performance near expert levels.
• This approach eliminates the need for model fine-tuning or institution-specific retraining, offering a practical solution.
• The study demonstrates a viable method for scaling the creation of research-grade retrospective datasets from narrative medical records.