Reasoning with large language models for medical question answering

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show potential in medical question answering.
• Current LLM reasoning methods can be inconsistent and lack refinement.
• Improving LLM reasoning is crucial for reliable medical applications.

Purpose of the Study:

• To investigate LLM reasoning approaches.
• To propose and evaluate a novel prompting technique called ensemble reasoning.
• To enhance medical question answering performance through refined reasoning and reduced inconsistency.

Main Methods:

• Utilized multiple-choice questions from the USMLE Sample Exam.
• Evaluated ensemble reasoning on closed-source (GPT-3.5 turbo, GPT-4 turbo) and open-source (Med42-70B) clinical LLMs.
• Compared ensemble reasoning against zero-shot chain-of-thought with self-consistency.

Main Results:

• Ensemble reasoning outperformed zero-shot chain-of-thought with self-consistency on GPT-3.5 turbo and Med42-70B across multiple exam steps.
• GPT-4 turbo showed mixed results, with ensemble reasoning excelling on Step 1 questions.
• The approach consistently improved response accuracy and demonstrated more reliable reasoning.

Conclusions:

• Iterative ensemble reasoning can significantly improve LLM performance in medical question answering, particularly for less powerful models.
• The method refines LLM reasoning, enhancing response consistency even with advanced models like GPT-4 turbo.
• Human-AI teaming is identified as a future direction to further advance LLM reasoning capabilities.