How do the Available LLM Platforms Fare as On-the-Go Orthopaedic Referencing Source? A Comparative Analysis

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• On-the-go (OTG) referencing is crucial for timely clinical decision-making.
• Traditional references (textbooks, journals) are evolving towards digital and AI-powered platforms.
• Large Language Models (LLMs) are increasingly utilized for rapid information retrieval in clinical settings.

Purpose of the Study:

• To compare the relevance and evidence quality of LLM-generated answers against human clinician references for OTG queries.
• To evaluate the performance of ChatGPT (GPT-4), Google Bard, and Microsoft Bing Chat in providing clinical information.
• To assess the suitability of AI-LLMs for supporting clinical decision-making.

Main Methods:

• Three AI-LLM platforms (ChatGPT-4, Bing Chat, Google Bard) were tested.
• 250 orthopaedic clinical queries with established answers and references were used.
• LLM responses were compared to human answers for relevance and Level of Evidence (LOE).

Main Results:

• No significant difference in answer relevance was found among the tested LLMs (p=0.110).
• ChatGPT-4 showed superiority for numerical queries (p=0.006).
• A significant difference in the LOE of answers was observed (p<0.001), with human references ranking highest, followed by Bing Chat, ChatGPT-4, and Bard.

Conclusions:

• Microsoft Bing Chat utilized a relatively better LOE for OTG questions compared to other LLMs.
• All evaluated AI-LLMs demonstrate potential for OTG referencing in healthcare.
• Customization for the medical domain and adherence to regulatory policies are recommended prior to clinical implementation.