GlobalMedQA: A Standardized Multilingual Dataset for Assessing Medical Knowledge in LLMs

Large Language Models usage in the healthcare environment is more noticeable than ever, finding use cases from data extraction and clinical knowledge retrieval. However, current benchmarks are predominantly single language and biased by the country-of-origin framework.

This work presents the creation of a new harmonized multilingual dataset of medical multiple-choice questions from 14 countries, with 13 languages. The questions were standardized and labelled according to the European Union of Medical Specialists classification, resorting to automated annotation, when required. To demonstrate the applicability of the dataset, exploratory benchmark experiments were conducted on a smaller set and a wide selection of models were analyzed to illustrate their performance across different languages.

The results show a great variability dependent on the language and model specifications. These findings highlight the uneven performance of LLMs across linguistic contexts in medical reasoning tasks.

With a standardized and multilingual resource, a reliable foundation is available for future research on globally applicable medical AI.