Human-AI collectives most accurately diagnose clinical vignettes

Area of Science:

• Artificial Intelligence
• Medical Diagnostics
• Collective Intelligence

Background:

• Large language models (LLMs) are increasingly used in high-stakes decisions.
• LLMs have limitations like hallucination, lack of common sense, and bias.
• Sole reliance on LLMs for complex decisions is problematic due to these limitations.

Purpose of the Study:

• To develop and evaluate a hybrid collective intelligence system integrating human physicians and LLMs.
• To mitigate risks associated with LLM use in high-stakes decision-making.
• To improve accuracy in open-ended medical diagnostics.

Main Methods:

• A hybrid system combining physician differential diagnoses with LLM diagnoses was created.
• The system analyzed 40,762 physician diagnoses and 5 state-of-the-art LLMs' diagnoses.
• The study used 2,133 text-based medical case vignettes for evaluation.

Main Results:

• Hybrid collectives of physicians and LLMs significantly outperformed individual physicians and LLMs.
• Performance improvements were observed across various medical specialties and experience levels.
• The enhanced accuracy is attributed to the complementary error patterns of humans and LLMs.

Conclusions:

• Hybrid collective intelligence systems can effectively leverage complementary strengths of humans and LLMs.
• This approach offers a promising solution for improving accuracy in complex, open-ended domains like medical diagnostics.
• Integrating human experience with AI capabilities enhances decision-making safety, quality, and equity.