Related Experiment Video
Updated: Jun 14, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Phenotyping Prostate Cancer in a National Health System Using Large Language Models
Michael P Dykstra1,2, Phoebe A Tsao3,4,5, Megan E V Caram3,4,5
1Department of Radiation Oncology, Veterans Affairs Ann Arbor Healthcare System, Ann Arbor, MI.
JCO Clinical Cancer Informatics
|June 12, 2026
Summary
Large language models (LLMs) accurately extract prostate cancer data from clinical reports, outperforming traditional methods. Ambiguous language in radiology reports is the main challenge for optimal LLM performance.
Area of Science:
- Medical Informatics
- Natural Language Processing
- Oncology
Background:
- Extracting prognostic variables from unstructured clinical text is crucial for prostate cancer management.
- Traditional rule-based natural language processing (NLP) methods have limitations in accurately capturing complex clinical information.
- Large language models (LLMs) show promise in advancing automated information extraction from clinical narratives.
Purpose of the Study:
- To evaluate the efficacy of LLMs in extracting key prostate cancer phenotypes from diverse unstructured clinical reports.
- To compare LLM performance against traditional NLP methods for prognostic variable extraction.
- To identify challenges and limitations in LLM-based clinical text analysis.
Main Methods:
- Iterative prompt engineering with few-shot examples was used to develop LLM prompts for 30 phenotypes.
- Data were sourced from pathology (biopsy, radical prostatectomy, TURP) and radiology (MRI, CT, bone scan, PSMA PET/CT) reports from over 130 VA facilities.
- LLM inference was performed using Llama 3.3 70B or GPT-4o, with performance assessed using accuracy, sensitivity, PPV, NPV, and macro-F1 metrics on independent test sets.
Main Results:
- LLMs achieved near-perfect accuracy in pathology extraction tasks, including total and involved cores from biopsy reports (accuracy >95%).
- Excellent performance was observed for extracting PIRADS scores, lesion locations, and dimensions from pelvic MRI reports (accuracy >98%).
- High positive predictive values (PPVs) were achieved for nodal and bone metastases extraction from PSMA PET/CT (PPV >97.9%), with comparable Tc-99m bone scan performance. Lower PPVs were noted for MRI and CT due to ambiguous language.
Conclusions:
- LLMs demonstrate high reliability in extracting critical prostate cancer phenotypes across various pathology and radiology report types.
- The study highlights the potential of LLMs to significantly improve the extraction of prognostic variables from unstructured clinical text.
- Ambiguous or indeterminate language within radiology reports presents the primary obstacle to achieving optimal LLM performance in clinical data extraction.
