AgentMD: Empowering Language Agents for Risk Prediction with Large-Scale Clinical Tool Learning

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Computational Health

Background:

• Clinical calculators are crucial for evidence-based predictions but face usability and dissemination challenges.
• Manual curation of clinical calculators for large language models is not scalable.
• Integrating AI with clinical tools can enhance healthcare analytics and patient care.

Purpose of the Study:

• To introduce AgentMD, a language agent for automated curation and application of clinical calculators.
• To develop RiskCalcs, a large collection of executable clinical calculators.
• To evaluate AgentMD's performance in selecting and applying relevant calculators for patient descriptions.

Main Methods:

• AgentMD automatically curated 2,164 clinical calculators from published literature, forming the RiskCalcs collection.
• RiskCalcs tools were manually evaluated for accuracy on three quality metrics.
• AgentMD's performance was assessed on the RiskQA benchmark and applied to real-world clinical notes.

Main Results:

• RiskCalcs tools achieved over 80% accuracy on quality metrics.
• AgentMD significantly outperformed GPT-4 on the RiskQA benchmark (87.7% vs. 40.9%).
• AgentMD successfully analyzed population-level and risk-level patient characteristics from clinical notes.

Conclusions:

• Language agents augmented with clinical calculators offer a scalable solution for healthcare analytics.
• AgentMD demonstrates the potential to improve clinical workflow efficiency and patient care.
• Automated curation and application of clinical calculators by AI can overcome existing limitations in healthcare settings.