Toward expert-level medical question answering with large language models.

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Medical Informatics

Background:

• Large language models (LLMs) show promise in medical question answering, but face challenges in long-form queries and real-world workflows.
• Med-PaLM achieved a passing score on United States Medical Licensing Examination-style questions, indicating early potential.

Purpose of the Study:

• To introduce Med-PaLM 2, an enhanced LLM designed to address limitations in medical question answering.
• To improve reasoning, grounding, and performance on complex medical queries.

Main Methods:

• Utilized base LLM improvements and medical domain fine-tuning.
• Implemented ensemble refinement and chain-of-retrieval strategies for enhanced reasoning.
• Evaluated performance on MedQA, MedMCQA, PubMedQA, and MMLU clinical topics datasets.

Main Results:

• Med-PaLM 2 achieved 86.5% on the MedQA dataset, a 19% improvement over Med-PaLM.
• Demonstrated significant performance gains across multiple medical question-answering benchmarks.
• Physicians preferred Med-PaLM 2 answers over other physicians' responses on eight of nine clinical axes in human evaluations.
• In a pilot study, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time.

Conclusions:

• Med-PaLM 2 represents a substantial advancement in AI-powered medical question answering.
• The model shows comparable safety to physician answers and significant potential for integration into clinical workflows.
• Further development is warranted to fully leverage LLMs in medical applications.