MedConceptsQA: Open source medical concepts QA benchmark.

Area of Science:

• Natural Language Processing
• Medical Informatics
• Artificial Intelligence

Background:

• Clinical data integrates standardized medical codes and free-text narratives.
• Effective Clinical Large Language Models (LLMs) must comprehend medical codes and their nuances.
• A novel benchmark is introduced to assess LLM understanding of medical codes.

Purpose of the Study:

• To introduce MedConceptsQA, an open-source benchmark for evaluating LLM comprehension of medical concepts.
• To assess the performance of various LLMs on a diverse range of medical concepts, including diagnoses, procedures, and drugs.
• To categorize question difficulty into easy, medium, and hard levels for granular analysis.

Main Methods:

• Development of MedConceptsQA, a question-answering benchmark focused on medical concepts.
• Inclusion of questions spanning multiple medical vocabularies (diagnoses, procedures, drugs).
• Evaluation of diverse Large Language Models using the MedConceptsQA benchmark.

Main Results:

• Most pre-trained clinical LLMs demonstrated performance near random chance on the benchmark.
• GPT-4 showed a 9-11% improvement over the best-performing clinical LLM (Llama3-OpenBioLLM-70B).
• Few-shot and zero-shot learning evaluations indicated GPT-4's superior performance.

Conclusions:

• The MedConceptsQA benchmark effectively evaluates LLM capabilities in interpreting and differentiating medical concepts.
• Current state-of-the-art clinical LLMs largely perform at random guessing levels.
• General-purpose LLMs like GPT-3.5, GPT-4, and Llama3-70B outperform specialized clinical LLMs on this benchmark.