Reference Accuracy in Large Language Model Chatbots: A Metric for Inherent Misinformation?

BACKGROUND We suggest that testing a large language model (LLM) chatbot in terms of the accuracy of the references it provides could be a powerful, quantifiable means of rating its inherent degree of misinformation, since the accuracy of the bibliographic data can be directly verified. Given the growing reliance on artificial intelligence (AI) tools in academic research and clinical decision-making, such a rating could be extremely useful. MATERIAL AND METHODS In this study, we compared 3 versions of ChatGPT and 3 versions of Gemini by asking them to provide references about 25 highly cited topics in otorhinolaryngology (those with "guidelines" in the title). Answers were sought on 3 consecutive days to assess the variability and consistency of responses. In total, the 6 chatbots returned 1947 references, which were carefully checked against PubMed, Web of Science, and Google Scholar, and rated according to accuracy. Ratings were given based on correct authorship, complete bibliographic details, and proper DOI numbers. RESULTS Common discrepancies noted were wrong author names and erroneous DOI numbers. Across the 6 chatbots, ChatGPT-4.1 (with web search enabled) achieved the best accuracy, with a score of 51%, with Gemini 2.5 Pro being second at 41%. The 2 versions with a web search facility performed better than the 4 versions without. Topics having higher citation counts were associated with lower error rates, suggesting that more widely disseminated scientific findings result in more accurate references. CONCLUSIONS Our findings provide a solid benchmark for rating AI-driven bibliographic retrieval and underline the need for further refinement before these tools can be reliably integrated into academia and clinical applications.