Comparative Performance of agentic AI and Physicians in Taking Clinical History across Leading Large Language Models...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Natural Language Processing

Background:

• Comprehensive clinical history taking is fundamental for accurate diagnosis and effective patient care.
• Current methods of history collection can be time-consuming and may miss crucial details.
• The integration of advanced AI, specifically large language models (LLMs), offers potential for optimizing this process.

Purpose of the Study:

• To evaluate the efficacy of an agentic framework utilizing LLMs for efficient and comprehensive clinical history taking.
• To assess the accuracy and completeness of patient histories obtained through the developed LLM system.
• To determine the potential of LLM-driven tools in generating clinical summaries, differential diagnoses, and investigation recommendations.

Main Methods:

• Development of an iterative prompting system employing LLMs (GPT-4o, Gemini-2.5-Flash-Lite, Grok-3) within a structured agentic framework.
• Evaluation using 52 published case reports and 20 simulated patient interactions.
• Assessment of captured history elements for relevance and completeness by blinded physicians.
• Generation of EHR-ready summaries, differential diagnoses, and recommended investigations post-interaction.

Main Results:

• The LLM framework achieved >85% accuracy and F1 scores in capturing relevant history elements across all tested models.
• Physician assessments confirmed the high quality and completeness of the collected clinical histories.
• Recommended investigations generated by the system aligned with those used for final diagnoses.

Conclusions:

• Agentic LLM systems demonstrate significant potential for structured and efficient clinical history collection.
• The developed framework can accurately extract clinically meaningful information, aiding in diagnosis and care planning.
• Further prospective clinical evaluation is warranted to validate these findings in real-world healthcare settings.