[Can ChatGPT Responses Be Improved by Incorporating S3 Clinical Practice Guidelines?]

Large Language Models such as ChatGPT are increasingly being discussed as tools to support medical decision-making. The aim of this study was to investigate whether the quality of responses generated by ChatGPT (GPT-4) to guideline-based ENT questions improves when the underlying evidence-based S3 guideline is provided as a PDF together with the respective question.

Thirty guideline-based questions were derived from five current S3 guidelines. ChatGPT answered each question in two runs: once without and once with provision of the respective guideline as a PDF together with the question. Three ENT specialists independently and anonymously evaluated the responses regarding content accuracy and length/conciseness (scale 1-3).

Providing the guideline as a PDF together with the respective question resulted in a significant improvement in accuracy (1.77 vs. 1.26; p = 0.003) and conciseness (1.96 vs. 1.50; p = 0.004). A significant correlation between accuracy and conciseness was observed (p < 0.001). Interrater reliability was fair to moderate.

The targeted provision of evidence-based guidelines as a PDF together with the respective guideline-based ENT question significantly improves response quality in ChatGPT. These findings highlight that the integration of medically sound primary sources is crucial for the accuracy and reliability of LLMs. LLMs therefore appear to be a potentially supportive tool in clinical and scientific assistance but do not replace medical expertise. In particular, potential "hallucinations" require continued critical medical and scientific oversight.