Benchmarking open-source large language models on Portuguese Revalida multiple-choice questions.

Area of Science:

• Artificial Intelligence
• Medical Education
• Natural Language Processing

Background:

• Large language models (LLMs) show promise in various fields.
• Evaluating LLM performance in specialized domains like medicine is crucial.
• The Revalida, Brazil's national medical examination, serves as a rigorous benchmark for medical knowledge.

Purpose of the Study:

• To assess the performance of leading large language models (LLMs) on validated medical knowledge tests in Portuguese.
• To compare the capabilities of open-source and proprietary LLMs in a high-stakes medical examination context.

Main Methods:

• A comprehensive evaluation of 31 large language models (LLMs), including 23 open-source and 8 proprietary models.
• Models were tested on the national Brazilian medical examination, comprising 399 multiple-choice questions.
• Performance was measured by success rates on the Revalida benchmark.

Main Results:

• Proprietary models GPT-4o (86.8%) and Claude Opus (83.8%) achieved the highest success rates.
• Among open-source models, Llama 3 70B (77.5%), Mixtral 8×7B (63.7%), and Llama 3 8B (53.9%) showed varying performance based on size.
• Ten LLMs demonstrated performance exceeding the human level on the Revalida benchmark.

Conclusions:

• Several large language models (LLMs) achieved expert-level performance on the Revalida medical examination.
• Model size generally correlated with performance, but some medium-sized LLMs outperformed larger ones.
• LLMs show significant potential in medical knowledge assessment, though challenges with coherence remain for some models.