Performance of single-agent and multi-agent language models in Spanish language medical competency exams

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Medical Education Technology

Background:

• Large language models (LLMs) show potential in medical decision-making and education.
• LLM performance in Spanish medical contexts is largely unexplored.
• This study assesses LLM strategies on a Chilean medical licensure exam (EUNACOM).

Purpose of the Study:

• To evaluate single-agent and multi-agent LLM strategies for answering Spanish medical exam questions.
• To compare the effectiveness of different prompting and agent collaboration techniques.
• To analyze LLM performance across 21 medical specialties.

Main Methods:

• GPT-4o tested on 1,062 EUNACOM multiple-choice questions.
• Evaluated strategies: Zero-Shot, Few-Shot, Chain-of-Thought (CoT), Self-Reflection, MED-PROMPT, Voting, Weighted Voting, Borda Count, MEDAGENTS, MDAGENTS.
• Assessed accuracy across three temperature settings (0.3, 0.6, 1.2) with statistical analysis.

Main Results:

• MDAGENTS achieved the highest accuracy (89.97%), outperforming all other strategies (p < 0.001).
• MEDAGENTS (87.99%) and CoT with Few-Shot (87.67%) also showed high performance.
• Temperature settings did not significantly impact results; highest accuracies in Psychiatry, Neurology, and Surgery.

Conclusions:

• Multi-agent strategies, particularly MDAGENTS, significantly enhance LLM performance on Spanish medical exams.
• Simpler single-agent strategies suffice for many questions, indicating limited need for complex reasoning.
• LLMs offer scalable tools for Spanish-speaking healthcare, with computational optimization as a future research focus.