Large language models for diabetes training: a prospective study.

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Endocrinology

Background:

• Diabetes knowledge gaps exist among healthcare professionals globally.
• Effective diabetes training is crucial for improving patient outcomes.
• Large Language Models (LLMs) offer potential new avenues for medical education, but their efficacy in specialized fields like diabetes care, especially in non-English languages, requires evaluation.

Purpose of the Study:

• To evaluate the performance of ten LLMs on diabetes-related queries using standardized examinations in both Chinese and English.
• To assess the utility of LLMs, specifically ChatGPT-4.0, as a supplementary tool for primary care physician (PCP) training in diabetes management.
• To determine the reliability of LLMs as medical assistants in the context of physician education.

Main Methods:

• Ten LLMs were tested on diabetes knowledge using the Chinese National Certificate Examination for Primary Diabetes Care (NCE-CPDC) and the UK Royal College of Physicians' Endocrinology and Diabetes examination.
• The impact of ChatGPT-4.0 on PCP performance in the NCE-CPDC examination was assessed by comparing scores with and without AI assistance.
• Performance metrics included accuracy and pass rates for each LLM and the improvement in PCP scores.

Main Results:

• ChatGPT-4.0 demonstrated superior performance in the English examination (62.50% passing accuracy), outperforming other models like Google Bard and LLaMA.
• For the Chinese NCE-CPDC examination, ChatGPT-4.0, Ali Tongyi Qianwen, Baidu ERNIE Bot, Google Bard, MedGPT, and ChatGPT-3.5 passed, while LLaMA2-7B, HuatuoGPT, Chinese LLaMA2-7B, and LLaMA-7B failed.
• ChatGPT-4.0 achieved an 84.82% score in the NCE-CPDC, surpassing all PCPs and improving the scores of most assisted PCPs by 1%-6.13%.

Conclusions:

• LLMs exhibit significant competence in answering diabetes-related questions across both Chinese and English languages.
• ChatGPT-4.0 shows particular promise as a high-performing AI tool for diabetes knowledge assessment and physician training.
• LLMs hold substantial potential to enhance global physician training programs for diabetes care.