Related Experiment Video
Updated: Sep 19, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
PeruMedQA: Benchmarking Large Language Models (LLMs) on Peruvian Medical Exams-Dataset Construction and Evaluation
Rodrigo M Carrillo-Larco1,2, Jesús Lovón-Melgarejo3, Manuel Castillo-Cara4,5
1Hubert Department of Global Health, Rollins School of Public Health, Emory University, Atlanta, GA USA.
Background:
Medical large language models (LLMs) have demonstrated remarkable performance in answering medical examinations. However, the extent to which this high performance is transferable to medical questions in Spanish and from a Latin American country remains unexplored. This knowledge is crucial as LLM-based medical applications gain traction in Latin America.
Aims:
To build a dataset of questions from medical examinations taken by Peruvian physicians pursuing specialty training; to fine-tune a LLM on this dataset; to evaluate and compare the performance in terms of accuracy between vanilla LLMs and the fine-tuned LLM.
Methods:
We curated PeruMedQA, a multiple-choice question-answering (MCQA) dataset containing 8,380 questions spanning 12 specialties (2018-2025). We selected ten medical LLMs, including medgemma-4b-it and medgemma-27b-text-it, and developed zero-shot task-specific prompts to answer the questions. We employed parameter-efficient fine tuning (PEFT) and low-rank adaptation (LoRA) to fine-tune medgemma-4b-it utilizing all questions except those from 2025 (test set).
Results:
Medgemma-27b showed the highest accuracy across all specialties, achieving the highest score of 89.29% in Psychiatry; yet, in two specialties, OctoMed-7B exhibited slight superiority: Neurosurgery with 77.25% and 77.38%, respectively; and Radiology with 76.13% and 77.30%, respectively. Across specialties, most LLMs with < 10 billion parameters exhibited < 50% of correct answers. The fine-tuned version of medgemma-4b-it emerged victorious against all LLMs with < 10 billion parameters and rivaled a LLM with 70 billion parameters across various examinations.
Conclusions:
For medical AI applications and research that require knowledge bases from Spanish-speaking countries and those exhibiting similar epidemiological profiles to Peru's, interested parties should utilize medgemma-27b-text-it.
Supplementary Information:
The online version contains supplementary material available at https://doi.org/10.1007/s40670-026-02692-w.
