How does AI compare to the experts in a Delphi setting: simulating medical consensus with large language models

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Collective Intelligence

Background:

• Large language models (LLMs) show potential for enhancing decision-making through simulated debate and collaboration.
• Limited research exists on whether LLM collective intelligence can replicate human expert consensus.

Purpose of the Study:

• To investigate consensus-building processes and outcomes among LLMs using a modified Delphi method.
• To compare LLM consensus results with those from a human expert Delphi study.

Main Methods:

• A three-round Delphi study involving eight LLMs evaluated 135 medical statements.
• LLMs independently assessed statements, refined opinions with feedback, and engaged in pairwise debates.
• Consensus was defined as ≥70% agreement; concordance measured identical outcomes between LLMs and human experts.

Main Results:

• LLMs achieved a higher overall consensus rate (93.3%) than human experts (81.5%).
• Concordance between LLM and human expert consensus was high (78.5% overall, 91.8% for human-agreed statements).
• Significant variation existed in individual LLMs' decision-changing likelihood and persuasion abilities.

Conclusions:

• LLM-based Delphi methods demonstrated high clinical consensus, closely aligning with human expert decisions.
• LLMs simulated deliberative reasoning but adopted more conservative, guideline-driven positions.
• Further research on parameter optimization is needed for LLM reproducibility in medical consensus-building.