Large Language Model and Natural Language Processing Approach to Identify Cancer Recurrence From Pathology Reports
Diego Bayona1, Daniel K Ebner1, Lydia Ekama1
1Department of Radiation Oncology, Mayo Clinic, Rochester, MN.
Purpose:
Cancer recurrence is a critical outcome for patients and physicians. Retrospective cancer recurrence data can evaluate recurrence-directed treatment and generate novel interventions targeting recurrent cancers. However, large cancer databases do not provide recurrence-related information, stymying study at scale, and consequently require significant manual record review. Automated evaluation of records may allow for the rapid generation of easily analyzed data sets, accelerating the evaluation of recurrence altogether.
Methods:
Patients treated with radiation therapy at one tertiary referral center from 2010 to 2018 with a verified cancer status (cancer recurrence v no cancer recurrence) were identified. Patients with recurrent disease were initially identified through manual record review, and the associated pathology report was collected. Google Automated Machine Learning with Natural Language Processing (AutoNLP) and Google Gemini 1.5 Pro were used to generate a model for binary classification, with comparison to the gold-standard manually developed data set.
Results:
A total of 7,054 patients were identified. 3,431 (48.6%) were female, with a median age of 64 years. Head and neck (1,482, 21%), breast (1,480, 21%), upper GI (1,307, 18.5%), and lung/thorax (1,107, 15.7%) were the most common disease sites. Recurrence was verified for 1,546 patients (21.9%) using pathology reports, of which 1,249 positive cases were paired with 651 negative pathology reports for model development. Google Gemini 1.5 Pro consistently outperformed AutoNLP across all measurements of accuracy, generating a greater absolute difference in precision, recall, negative predictive value, and specificity, and a higher likelihood of correct classification at the individual level, rendering Gemini superior in recurrence status extraction.
Conclusion:
AutoNLP and Google Gemini 1.5 Pro are promising tools for identifying recurrence from pathology reports, with the latter demonstrating superior overall performance, making it particularly suitable for clinical translation.
