Performance of open-source large language models on nephrology self-assessment program.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Nephrology

Background:

• Large Language Models (LLMs) show potential in medical question-answering and education.
• LLM effectiveness in specialized fields like nephrology is underexplored.
• This study assesses open-source LLMs on nephrology-specific questions.

Purpose of the Study:

• To evaluate the performance of open-source LLMs in answering nephrology multiple-choice questions.
• To understand the capabilities and limitations of LLMs in this specialized domain.
• To compare different LLM architectures and training data impacts.

Main Methods:

• Five open-source LLMs (PodGPT, Llama 3.2-11B, Mistral-7B-Instruct-v0.2, Falcon3-10B-Instruct, Gemma-2-9B-it) were evaluated.
• Models answered questions from the Nephrology Self-Assessment Program (NephSAP).
• Performance was measured by accuracy and NLP metrics; expert review assessed error rates.

Main Results:

• PodGPT achieved the highest accuracy (64.77%), while Llama performed lowest (45.08%).
• PodGPT demonstrated the lowest factual error rate (0.017).
• Llama and Falcon exhibited the lowest reasoning error rates (0.038).

Conclusions:

• STEMM-based training enhances LLM reasoning and reliability in clinical contexts.
• Findings support developing AI-driven decision-support tools for nephrology.
• LLMs show potential but require specialized training for clinical accuracy.