Autonomous Agents for Auditable Cardiovascular Artificial Intelligence Development

Area of Science:

• Artificial Intelligence in Medicine
• Machine Learning for Healthcare
• Clinical Decision Support Systems

Background:

• Clinical artificial intelligence (AI) models are typically static, representing limited exploration of potential development pathways.
• Existing AI model development relies heavily on human expertise and manual iteration, which can be inefficient.
• Optimizing AI models requires extensive search across architectures, inputs, loss functions, optimizers, and training recipes.

Purpose of the Study:

• To investigate the efficacy of autonomous code-writing agents in performing controlled experiments for clinical AI model development.
• To assess if these agents can improve AI model performance without requiring new data or human intervention.
• To develop and release an open-source toolkit of autonomous agents for AI model improvement.

Main Methods:

• Developed two autonomous agents: an Iteration Agent for sequential search and an Evolution Agent for parallel search using multiple large language models.
• Applied agents to two distinct AI-enhanced electrocardiography (AI-ECG) models for structural heart disease.
• Evaluated agent-optimized variants on held-out, external, and cross-institutional datasets.

Main Results:

• Agent-optimized AI-ECG models demonstrated improved rank discrimination across all evaluation sets, with statistically significant gains in area under the receiver operating characteristic curve (0.006 to 0.039).
• At 90% sensitivity, specificity increased by up to 7.1 percentage points and positive predictive value by up to 4.8 percentage points.
• The agents identified substantive code changes in architecture, data representation, and training recipes.

Conclusions:

• Autonomous code-writing agents can effectively enhance clinical AI model performance through automated, auditable development processes.
• These agents represent a promising approach for continuous improvement of AI models in healthcare settings.
• The findings support the integration of autonomous agents into clinical AI development, emphasizing the need for explicit validation and governance protocols.