Evaluating clinical competencies of large language models with a general practice benchmark

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Healthcare AI Evaluation

Background:

• Large Language Models (LLMs) show promise for general practice applications.
• Existing LLM evaluation methods lack real-world clinical competency alignment.
• The suitability of LLMs for General Practitioner (GP) roles is currently unverified.

Purpose of the Study:

• To develop a novel evaluation framework for assessing LLM capabilities in general practice.
• To introduce GPBench, a benchmark dataset reflecting real-world clinical standards.
• To evaluate the competency of state-of-the-art LLMs for GP duties.

Main Methods:

• Creation of a competency-based evaluation framework for LLMs in general practice.
• Development of GPBench, a dataset annotated by medical experts.
• Assessment of ten leading LLMs using the GPBench framework.

Main Results:

• Current LLMs demonstrate limitations in fulfilling the comprehensive duties of General Practitioners.
• All evaluated LLMs require ongoing human supervision for clinical deployment.
• Significant optimization is needed for LLMs to effectively support daily GP responsibilities.

Conclusions:

• LLMs are currently unsuitable for autonomous clinical general practice.
• Human oversight is mandatory for any realistic LLM application in primary care.
• Future LLM development must prioritize alignment with specific GP tasks and responsibilities.