Related Experiment Video
Updated: Oct 27, 2026
Endobronchial Ultrasound-guided Intratumoral Injection of Cisplatin for the Treatment of Isolated Mediastinal Recurrence of Lung Cancer
Published on: February 12, 2017
AI-Generated Treatment Recommendations in Newly Diagnosed Lung Cancer: A Single-Center Feasibility Study in a
Andrei Popescu1,2, Mara Jidveian-Popescu3, Beatrice Mahler4,5
1Department of Medical Oncology, Colentina Clinical Hospital, 020125 Bucharest, Romania.
Background:
How treatment recommendations generated by large language models (LLMs) compare with real-world decisions in oncology remains poorly characterized, particularly in resource-limited settings.
Methods:
In this single-center feasibility study, four LLMs (ChatGPT, Claude, Gemini, and Grok, accessed via consumer interfaces in June 2026) generated treatment recommendations for 27 consecutive newly diagnosed lung cancer cases treated at a Romanian oncology unit (June 2024-May 2026). Each model's output and the treatment actually delivered were formatted identically (135 recommendations total), presented in randomized order, and scored for appropriateness (1-5) by a reviewer blinded to the source, with three quality flags. A second independent reviewer re-scored all recommendations to assess inter-rater reliability. Scores were compared using the Friedman test.
Results:
Mean appropriateness was highest for the real-world decision (4.52), followed by ChatGPT (4.48), Claude (4.44), Grok (4.41), and Gemini (4.37), with no significant difference (Friedman χ24 = 1.99, p = 0.74). No recommendation scored below 3. Inter-rater agreement was high in raw terms (93.3% within one point) but only fair after chance correction (weighted κ = 0.26), reflecting a ceiling effect.
Conclusions:
AI-generated recommendations were rated similarly to real-world decisions. The evaluation method proved feasible in a resource-limited setting; the small sample and ceiling effect preclude equivalence claims, and the findings are hypothesis-generating.
