Context matching is not reasoning when performing generalized clinical evaluation of generative language models

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Clinical capabilities of generative language models (GLMs) are often assessed using multiple-choice question-answer (MCQA) benchmarks from licensing exams.
• The unique characteristics of GLMs raise concerns about the validity of these traditional benchmarks.

Purpose of the Study:

• To validate five MCQA benchmarks using eight GLMs, assessing parameter size and reasoning capabilities.
• To test key assumptions of MCQA generalizability: knowledge application vs. memorization, semantic consistency, and recognition of null-answer scenarios.

Main Methods:

• Eight GLMs were used to validate five MCQA benchmarks.
• Prompt permutation was employed to test three core assumptions of benchmark generalizability.
• Models were analyzed based on parameter size and reasoning abilities.

Main Results:

• All tested GLMs globally invalidated the assumptions underpinning MCQA benchmark generalizability.
• Larger GLMs showed more resilience to perturbations than smaller models, but memorization remained an issue for smaller models.
• All models demonstrated significant failure in identifying scenarios with no correct answers (null-answer scenarios).

Conclusions:

• Current MCQA benchmarks are not fully valid for assessing GLM clinical capabilities due to fundamental assumption violations.
• GLMs, particularly smaller ones, exhibit memorization and struggle with null-answer scenarios, impacting their reliability.
• Adaptations to benchmark designs are necessary for more robust and realistic evaluations of GLMs in clinical settings.