Related Experiment Video
Updated: May 24, 2025
A Mouse Model of Incompletely Resected Soft Tissue Sarcoma for Testing Neoadjuvant Therapies
Published on: July 28, 2020
Improving Accuracy and Source Transparency in Responses to Soft Tissue Sarcoma Queries Using GPT-4o Enhanced with
Cheng-Peng Li1,2, Wei-Wei Jia2, Yuan Chu1
1Department of Surgery, Medical Faculty Mannheim, Mannheim School of Medicine, Heidelberg University, Mannheim, Germany.
Introduction:
This study aimed to evaluate the effectiveness of GPT-4o, with and without retrieval-augmented generation (RAG), in responding to soft tissue sarcoma (STS)-related queries.
Methods:
The study used a 20-question dataset derived from clinical scenarios related to adult STS. The responses were generated by GPT-4o with and without the RAG approach. The RAG system incorporated the English version of German evidence-based S3 guidelines through an embedding-based retrieval system. Two sarcoma experts evaluated the responses for accuracy, comprehensiveness, and safety using a Likert scale. Statistical analyses were conducted to compare the performances.
Results:
GPT-4o with RAG outperformed the model without RAG across all evaluated areas (p < 0.05). GPT-4o without RAG had a 40% error rate, which was reduced to 10% by the RAG approach. In 90% of the questions, the pages with the relevant information that addressed the questions were correctly cited using the retrieval system.
Conclusion:
The RAG approach significantly enhanced the performance of GPT-4o in answering STS-related questions. However, the model still produced incorrect responses in certain complex scenarios. GPT-4o, even with RAG, should be used cautiously in clinical settings, particularly for rare diseases like sarcoma. Human expertise remains irreplaceable in medical decision-making.
