Metastasis Extraction From Non-Small Cell Lung Cancer Clinical Notes Using Large Language Model-Based Classification:...

Identification of metastasis status in non-small cell lung cancer (NSCLC) is a critical part of understanding disease prognosis, treatment courses, trial eligibility, and population-level cancer surveillance. However, metastasis status is inconsistently recorded in structured cancer registry fields, since manual abstraction of clinical notes is often an error-prone process. This challenge highlights an opportunity for leveraging large language models (LLMs) to conduct high-scale metastasis extraction from real-world clinical documentation.

We conducted a retrospective, multicohort comparative evaluation of 3 distinct LLMs for 2 independent classification tasks: overall metastasis presence at any site and brain/central nervous system (CNS) metastasis presence. We evaluated model performance on 2 independent NSCLC cohorts: one with registry labels and another with manual annotations. We further explored whether our methods could analyze clinical documentation and recover missing information in structured registry fields.

Patient cohorts were derived from the Winship Cancer Institute. Cohort 1 (n=579 patients; 24,887 notes across 69 note types; 2023-2025) used registry-linked metastasis fields as the reference standard. Cohort 2 (n=22 patients; 644 radiology notes; 2010-2021) was drawn from 2 completed randomized trials and used dual-annotated labels (Cohen κ: 0.93 overall metastasis; 0.88 CNS metastasis) as the reference standard. We fine-tuned the GatorTron-base encoder model for each independent binary classification task. We evaluated MedGemma-27B-text and Llama 3.1-70B using zero-shot prompting. A separate cohort of 675 patients with missing registry labels was used for an exploratory missingness-recovery analysis, validated against annotations of a random subsample. Our best-performing model was used to conduct further site-specific metastasis classification, based on available registry data.

More than half (675/1260, 54%) of initially identified cohort 1 patients had missing or unknown registry metastasis labels. MedGemma demonstrated the best performance in overall metastasis classification (cohort 1: F₁-score=0.80; cohort 2 patient-level: F₁-score=1.0; cohort 2 note-level: F₁-score=0.93). For brain/CNS metastasis, Llama3 performed best in both cohorts (cohort 1: F₁-score=0.79; cohort 2 patient-level: F₁-score=0.93; cohort 2 note-level: F₁-score=0.86). The fine-tuned GatorTron model showed strong performance for classification of overall metastasis in cohort 1 (F₁-score=0.72). Error analysis indicated that several model errors reflected incomplete registry labels, ambiguous clinical language, or missing documentation. In the exploratory recovery analysis, model predictions agreed with manual annotations at an accuracy of 0.90 and F₁-score of 0.89. Additional site-specific metastasis classification by MedGemma demonstrated high performance for liver metastasis (F₁-score=0.82) and bone metastasis (F₁-score=0.79).

All models demonstrated relatively high performance. The zero-shot generative models were more robust to nuanced documentation and context-dependent brain/CNS metastasis extraction. The fine-tuned encoder model demonstrated strong classification performance but may have been limited by potential inaccuracies in the registry reference standards during model training. This study further demonstrated the potential of LLMs in recovering plausible structured labels from narrative text, complementing cancer registries for metastasis ascertainment.