Exploratory task-specific evaluation of large language models in lung cancer clinical scenarios: A comparative study

Lung cancer remains the leading cause of cancer-related mortality worldwide. Large language models (LLMs), including ChatGPT, DeepSeek, and Grok, have shown promise in clinical decision support, but differences in training and alignment may lead to variable performance. Current evaluations often rely on aggregate metrics or isolated tasks, which may not capture real-world clinical complexity.

We conducted a structured evaluation of three LLMs using nine simulated lung cancer cases across five clinical domains. LLMs' outputs were anonymized, randomized, and independently scored by five senior lung cancer specialists under a double-blind design using a five-point Likert scale evaluating accuracy, comprehensiveness, relevance, and clinical applicability. Qualitative error analysis was also performed.

Inter-rater agreement was moderate (Fleiss' κ = 0.463; ICC (2, k) = 0.675). All LLMs achieved high scores across evaluation dimensions without statistically significant differences (P > 0.05). Given the limited number of simulated cases, these findings should be interpreted cautiously. Descriptive analyses suggested context-dependent performance patterns across clinical domains: Grok tended to show more consistent performance in diagnosis and treatment decision-making, DeepSeek showed comparatively lower descriptive performance in therapeutic decisions but higher applicability in prognosis and rehabilitation, and GPT exhibited relatively stable intermediate performance. No single LLM consistently outperformed others across all clinical scenarios.

LLMs demonstrate substantial potential in supporting lung cancer clinical workflows, but their performance appears to be context-dependent. The present findings are exploratory and suggest that task-specific evaluation may provide a more clinically informative framework than overall model ranking. Continued validation using larger and more diverse clinical datasets, together with appropriate governance and specialist oversight, remains essential for the safe integration of LLMs into clinical practice.