Grounding large language models in clinical diagnostics.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Medical Informatics

Background:

• Large Language Models (LLMs) possess extensive medical knowledge but struggle with dynamic, iterative clinical diagnosis.
• Existing LLMs often commit clinical errors in real-world diagnostic workflows.
• There is a need for specialized LLMs trained for complex diagnostic procedures.

Purpose of the Study:

• To develop and evaluate ClinDiag-GPT, a specialized LLM for full diagnostic procedures.
• To assess the performance of ClinDiag-GPT against existing LLMs using a real-world clinical dataset.
• To investigate the impact of physician-LLM collaboration on diagnostic accuracy and efficiency.

Main Methods:

• Fine-tuning a specialized LLM (ClinDiag-GPT) on a dataset of 4,421 real-world clinical cases (ClinDiag-Benchmark).
• Utilizing the ClinDiag-Framework for evaluating diagnostic accuracy and procedural performance.
• Comparing ClinDiag-GPT against baseline LLMs (GPT-4o-mini, GPT-4o, Claude-3-Haiku, Qwen2.5 models) and physician-only performance.

Main Results:

• ClinDiag-GPT significantly outperformed all baseline LLMs in diagnostic accuracy and procedural performance.
• General LLMs demonstrated proficiency in static tasks but faltered in dynamic diagnostic workflows.
• Collaboration between physicians and ClinDiag-GPT resulted in superior diagnostic accuracy and efficiency compared to individual performance.

Conclusions:

• ClinDiag-GPT represents a significant advancement in applying LLMs to clinical diagnosis.
• Specialized LLMs trained on clinical data can overcome limitations of general-purpose LLMs in healthcare.
• ClinDiag-GPT shows promise as an effective clinical assistant, enhancing physician capabilities and patient care.