GlobalMedQA: A Standardized Multilingual Dataset for Assessing Medical Knowledge in LLMs.

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Medical Education

Background:

• Large Language Models (LLMs) are increasingly used in healthcare for tasks like data extraction and knowledge retrieval.
• Existing benchmarks for LLMs in medicine are often single-language and geographically biased.
• There is a need for standardized, multilingual resources to assess AI performance globally.

Purpose of the Study:

• To develop a harmonized, multilingual dataset of medical multiple-choice questions.
• To evaluate the performance of various LLMs across different languages and countries.
• To establish a foundation for research on globally applicable medical AI.

Main Methods:

• Creation of a multilingual dataset comprising medical multiple-choice questions from 14 countries and 13 languages.
• Standardization and labeling of questions using the European Union of Medical Specialists classification, with automated annotation where necessary.
• Conducting benchmark experiments on a subset of the data with diverse LLMs to analyze cross-lingual performance.

Main Results:

• Significant variability in LLM performance was observed, dependent on language and specific model characteristics.
• Uneven performance of LLMs across different linguistic contexts was highlighted in medical reasoning tasks.
• The findings underscore the challenges in achieving consistent AI performance in diverse global healthcare settings.

Conclusions:

• The developed multilingual dataset provides a standardized resource for future research.
• This resource facilitates the development and evaluation of AI models for global medical applications.
• It lays the groundwork for more equitable and reliable AI in international healthcare contexts.