Evaluation of Large Language Models' Concordance With Guidelines on Olfaction

Area of Science:

• Otolaryngology
• Artificial Intelligence
• Medical Informatics

Background:

• The 2022 International Consensus Statement on Allergy and Rhinology: Olfaction (ICAR-O) provides current clinical guidelines.
• Artificial intelligence (AI) tools are increasingly used for information retrieval in healthcare.

Purpose of the Study:

• To evaluate the accuracy and concordance of AI-generated responses with established olfaction guidelines.
• To compare the performance of ChatGPT (version 4.0) and Google Gemini in responding to olfaction-related queries.

Main Methods:

• Extracted 42 guidelines from the ICAR-O and formulated them as questions.
• Submitted questions to ChatGPT and Google Gemini, assessing response accuracy, concordance, and credibility.
• Utilized a Likert scale for accuracy grading and statistical analysis to compare AI platforms.

Main Results:

• ChatGPT responses were significantly more accurate (mean 1.85/2) than Gemini (mean 1.48/2).
• ChatGPT achieved 78.57% concordance with ICAR-O guidelines, while Gemini achieved 66.67%.
• Both platforms cited credible resources, with no significant difference in concordance or credibility between them.

Conclusions:

• ChatGPT provided more accurate olfaction information than Gemini.
• Neither AI platform consistently aligned with clinical olfaction guidelines.
• Further evaluation of AI tools is crucial before clinical implementation or use as educational aids.