AutoMedic: An Automated Evaluation Framework for Clinical Conversational Agents with Medical Dataset Grounding

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Computational Linguistics

Background:

• Evaluating large language models (LLMs) for medical applications is crucial for safety and trustworthiness.
• Existing static benchmarks inadequately assess LLMs in dynamic, multi-turn clinical conversations.
• Standardized evaluation of interactive clinical scenarios is challenging due to complexity.

Purpose of the Study:

• To introduce AutoMedic, a novel multi-agent simulation framework for automated evaluation of LLMs as clinical conversational agents.
• To enable realistic, multi-turn clinical dialogues using transformed static QA datasets.
• To establish a multi-faceted evaluation metric (CARE) for clinical conversational agents.

Main Methods:

• Developed AutoMedic, a multi-agent simulation framework.
• Transformed static medical question-answering datasets into virtual patient profiles.
• Assessed LLM performance using the CARE metric (Clinical Conversational Accuracy, Efficiency/Strategy, Empathy, Robustness).

Main Results:

• AutoMedic facilitates automated, realistic multi-turn clinical dialogues.
• The CARE metric provides a comprehensive evaluation of clinical conversational agents.
• Findings were validated by human experts, confirming AutoMedic's utility.

Conclusions:

• AutoMedic is a valid automated evaluation framework for clinical conversational LLMs.
• The framework supports practical guidelines for developing medical LLMs.
• AutoMedic enhances the safe and trustworthy application of LLMs in healthcare.