Precision Oncology in Non-small Cell Lung Cancer: A Comparative Study of Contextualized ChatGPT Models

The growing adoption of Large Language Models (LLMs) in medicine has raised important questions about their potential utility for clinical decision support within oncology. This study aimed to evaluate the effects of various contextualization methods on ChatGPT's ability to provide National Comprehensive Cancer Network (NCCN) guideline-aligned recommendations on managing non-small cell lung cancer (NSCLC).

GPT-4o, base GPT-4, and GPT-4 models contextualized with prompts and PDF documents were asked to identify preferred chemotherapies for twelve advanced lung cancers given molecular profiles derived from the 2024 NCCN Clinical Practice Guidelines in Oncology for NSCLC. GPT responses were subsequently compared to NCCN guidelines using readability scores and qualitative reviewer assessments of (1) recommendation of specific targeted therapy, (2) agreement with NCCN-guideline-preferred therapies, (3) recommendation of guideline non-concordant therapies, and (4) provision of supplementary information.

The PDF+Prompt contextualized model demonstrated elevated agreement scores of 23/24 versus 17/24 for GPT-4 (P = 0.040) and 18/24 for GPT-4o (P = 0.089). No PDF+Prompt model responses contained guideline non-concordant therapies in contrast to 4/12 responses for GPT4 (P = 0.093) and 5/12 responses for GPT4o (P = 0.037). Comparison of response readability between the PDF+Prompt model and GPT-4 or GPT-4o showed a lower mean word count (both P < 0.001), Simple Measure of Gobbledygook (SMOG) score (both P < 0.001), and Gunning Fog readability score (P < 0.001 for GPT-4, P = 0.002 for GPT-4o). Prompting alone did not significantly improve agreement or reduce the rate of non-concordant therapy recommendations.

The performance gains observed following contextualization suggest that broader applications of LLMs in oncology may exist than current literature indicates. This study provides proof of concept for the use of contextualized GPT models in oncology and showcases their accessibility. Future studies validating this application within additional cancer types or real-life patient encounters could provide an important bridge to eventual adoption.