Accuracy, completeness and hallucinations in large language model generated statements and references for common...

Large language models (LLMs) are increasingly adopted for medical documentation, clinical decision-making, and scientific writing. Although these applications demonstrated impressive capabilities, concerns persist regarding their accuracy, fabricated references, and ethical implications (1-3). The Massive Multitask Language Understanding (MMLU) test was introduced as a comprehensive benchmark to evaluate how well LLMs perform across a diverse range of tasks (4). This prospective comparative two part study evaluated the accuracy and completeness of ChatGPT (CGPT)-3.5-generated 100 word summaries of four common rhinologic disorders (epistaxis, allergic rhinitis, odontogenic sinusitis, and chronic rhinosinusitis) including background, investigations, and magement as assessed by international rhinology society members. Furthermore, we evaluated the accuracy of references generated by contemporary LLM versions for the same prompts (CGPT-5 thinking (5), non-thinking (6), Claude Sonnet 4.5 non-thinking (6) and Gemini 2.5 (7)). Otolaryngologists evaluated each statement using standardized scales for accuracy, completeness, and overall reference quality.