Two-stage prompting framework with predefined verification steps for evaluating diagnostic reasoning tasks on two...

Area of Science:

• Artificial Intelligence in Medicine
• Medical Informatics
• Clinical Decision Support Systems

Background:

• Large language models (LLMs) show promise in healthcare but struggle with accurate medical diagnostic reasoning.
• Existing LLM prompting methods may not fully leverage their potential for complex clinical tasks.

Purpose of the Study:

• To evaluate a novel two-stage prompting framework (Initial Diagnosis → Verification → Final Diagnosis) for enhancing LLM diagnostic reasoning.
• To assess the impact of this framework on accuracy, uncertainty, consistency, and reasoning errors compared to standard methods.

Main Methods:

• The study utilized GPT-4o and DeepSeek-V3 models on 589 MedQA-USMLE and 300 NEJM cases.
• A two-stage prompting framework with predefined verification steps was implemented and tested.
• Cases were sampled five times and diagnoses were evaluated by blinded, board-certified physicians.

Main Results:

• The two-stage framework improved final diagnosis accuracy by up to 5.2%, reduced uncertainty by 16.0%, and increased consistency by 23.3%.
• This approach led to a 63.0% reduction in incorrect medical knowledge errors.
• Compared to Chain-of-Thought, the framework showed up to 4.0% higher accuracy and 11.0% greater consistency.

Conclusions:

• The two-stage prompting framework with verification steps demonstrates potential for improving LLM diagnostic reasoning in medical applications.
• The findings suggest a viable strategy for enhancing the reliability and accuracy of AI in clinical decision support.
• Further validation across diverse datasets and models is recommended to confirm generalizability.