Related Experiment Video
Updated: Jan 10, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Assessing the Accuracy of Large Language Models on European Guidelines for Cervical Cancer: An In Silico Benchmarking
Matteo Pavone1,2,3,4, Chiara Innocenzi1,2, Nicola Macellari1
1UOC Ginecologia Oncologica, Dipartimento di Scienze per la Salute Della Donna e del Bambino e di Sanità Pubblica, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy.
Objective:
Although large language models are increasingly used in clinical and research settings, the validity of the information they provide remains uncertain. This study aimed to evaluate the accuracy, consistency, and reliability of three large language models-ChatGPT 4.0, DeepSeek R1, and Gemini 2.0-in answering cervical cancer-related questions based on the ESGO/ESTRO/ESP guidelines.
Design:
Prospective, comparative in silico benchmarking study.
Setting:
Fondazione Policlinico Universitario A. Gemelli, Rome, Italy.
Population Or Sample:
Fifty questions derived from the ESGO/ESTRO/ESP (European Society of Gynaecologic Oncology/European Society for Radiotherapy and Oncology/European Society of Pathology) Guidelines for Cervical Cancer.
Methods:
Each question was submitted simultaneously to ChatGPT 4.0, DeepSeek R1, and Gemini 2.0, and re-entered twice to assess response repeatability. Answers were evaluated for accuracy using a Global Quality Score (GQS) from 1 (poor) to 5 (completely accurate). Consistency (intra-model response stability) and reliability (alignment with guidelines) were assessed using binary classification.
Main Outcome Measures:
Median GQS, percentage of GQS 5 responses, consistency between repeated answers, and reliability.
Results:
ChatGPT 4.0 achieved the highest performance, with 42% of responses rated GQS 5, followed by Gemini 2.0 (30%) and DeepSeek R1 (28%). DeepSeek R1 and Gemini 2.0 scored lower in median GQS (3.50) compared to ChatGPT 4.0 (4.00). Response consistency varied significantly, with ChatGPT 4.0 and DeepSeek R1 showing differences from Gemini 2.0 (p = 0.034 and p = 0.044, respectively). No significant difference was observed in reliability (p = 0.602).
Conclusion:
All models demonstrated suboptimal accuracy in aligning with clinical guidelines. ChatGPT 4.0 was the most accurate and consistent whereas DeepSeek R1 underperformed. Despite similar reliability across models, expert oversight remains essential to ensure safe clinical application and prevent misinformation.
More Related Videos
07:13Comparison of Predictive Performance of Three Lymph Node Staging Systems in Colorectal Signet Ring Cell Carcinoma Based on Machine Learning Model
Published on: April 18, 2025
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
Related Concept Videos
02:07Improving Translational Accuracy
02:07Improving Translational Accuracy