Comparative Analysis of Generative Pre-Trained Transformer Models in Oncogene-Driven Non-Small Cell Lung Cancer:...

Area of Science:

• Oncology
• Artificial Intelligence
• Biomarker Discovery

Background:

• Precision oncology in non-small cell lung cancer (NSCLC) is vital but faces challenges.
• Lack of genomic training, nonstandardized reporting, and evolving treatments impede progress.
• Generative AI offers potential for clinical decision support in NSCLC.

Purpose of the Study:

• To evaluate ChatGPT versions' accuracy in generating NSCLC next-generation sequencing reports and treatment recommendations.
• To assess AI models' propensity for hallucinations and incorrect information.
• To introduce a novel Generative AI Performance Score (G-PS) for evaluating AI utility.

Main Methods:

• ChatGPT versions were queried for first-line NSCLC treatment recommendations using a zero-shot prompt approach.
• Responses were evaluated against National Comprehensive Cancer Network (NCCN) guidelines for accuracy, relevance, and hallucinations.
• The G-PS was calculated, incorporating a base score for correct recommendations and penalties for hallucinations.

Main Results:

• GPT-4 demonstrated superior performance compared to GPT-3.5 in generating NSCLC treatment recommendations.
• GPT-4 achieved a higher base score (90% vs. 60%) and exhibited fewer hallucinations (34% vs. 53%).
• GPT-4's overall G-PS (0.34) was significantly higher than GPT-3.5 (-0.15).

Conclusions:

• Generative AI is rapidly improving in matching NSCLC treatments with biomarkers.
• While GPT-4 shows improvement, high accuracy with minimal hallucinations is crucial for clinical AI adoption.
• The G-PS offers a novel metric for quantifying generative AI utility in healthcare.