Related Experiment Video
Updated: Aug 6, 2026
Machine Learning Algorithms for Early Detection of Bone Metastases in an Experimental Rat Model
Published on: August 16, 2020
Evaluating Locally Deployed Large Language Models for Ga-68 PSMA PET/CT Report Mining in Prostate Cancer
Jagrati Chaudhary1, Param Dev Sharma2, Sanjay Kumar3
1Department of Nuclear Medicine, All India Institute of Medical Sciences, New Delhi, Ansari Nagar, 110029 India.
Purpose:
Large Language Models (LLMs) extracts structured data from unstructured Ga-68 Prostate Specific Membrane Antigen Positron Emission Tomography/Computed Tomography (PSMA PET/CT) reports by interpreting context and semantics, enabling reliable conversion of free-text into structured, queryable data. We evaluated whether locally deployed Llama 3.2:3b and Gemma 2:2b models could accurately perform zero-shot extraction of key diagnostic features.
Methods:
We retrospectively selected 50 de-identified Ga-68 PSMA PET/CT reports of histologically confirmed prostate cancer (01 January 2020-30 June 2024), ensuring rigorous expert annotation and inter-reader agreement. Reports were batch-processed using standardized zero-shot and few-shot prompts applied to locally deployed Llama 3.2:3b and Gemma 2:2b models, with model outputs benchmarked against an adjudicated dual-expert ground truth reference derived from independent annotations and consensus resolution of discrepancies using standard performance metrics.
Results:
Both Llama 3.2:3b and Gemma 2:2b achieved rapid, reliable extraction of key diagnostic features from Ga-68 PSMA PET/CT reports, with excellent inter-reader agreement (average κ = 0.882). Llama 3.2:3b offered superior sensitivity (83.7%) and negative predictive value (NPV) (97.3%), while Gemma 2:2b demonstrated greater overall accuracy (86.2%) and specificity (88.5%). Model performance varied with question prevalence and complexity, highlighting distinct strengths for clinical integration.
Conclusion:
Open-weight Llama 3.2:3b and Gemma 2:2b LLMs enable rapid extraction of key PET/CT findings (16-20x faster than manual review), with Llama excelling in sensitivity and Gemma in specificity. However, modest performance on rare but clinically important findings and limited positive predictive value indicate these models are best suited for initial screening or human-assisted workflows rather than autonomous extraction.
Supplementary Information:
The online version contains supplementary material available at 10.1007/s13139-026-00991-z.
