Large Language Models for Pre-mediation Counseling in Medical Disputes: A Comparative Evaluation against Human...

Area of Science:

• Medical Law
• Artificial Intelligence
• Health Informatics

Background:

• Medical dispute assessment requires specialized medical and legal knowledge, posing challenges for patients seeking malpractice claim guidance.
• Existing methods for medical dispute counseling can be complex and difficult for patients to navigate.

Purpose of the Study:

• To develop and evaluate a large language model (LLM) chatbot utilizing a chain-of-thought pipeline for medical dispute counseling.
• To compare the chatbot's performance against human expert responses in providing medical dispute counseling.

Main Methods:

• A dataset of 279 retrospective medical dispute counseling cases was utilized, with 50 randomly selected for validation.
• The Claude 3.5 Sonnet model processed counseling requests via a five-step chain-of-thought pipeline.
• Thirty-eight experts evaluated chatbot responses against human experts using a 5-point Likert scale across four dimensions, with statistical analysis via Wilcoxon signed-rank tests.

Main Results:

• The LLM chatbot significantly surpassed human experts in information quality (p < 0.001), understanding and reasoning (p < 0.001), and overall user satisfaction (p < 0.001).
• The chatbot exhibited a greater propensity for generating opinion-driven content (p < 0.001).
• Despite high overall scores, evaluators identified specific limitations and areas where the chatbot faced challenges.

Conclusions:

• A chain-of-thought-based LLM chatbot demonstrates significant potential to enhance medical dispute counseling services.
• The chatbot outperformed human experts on key performance indicators, suggesting its utility in this domain.
• Future research should focus on mitigating inaccuracies related to legal and contextual nuances, assessing patient adoption, and optimizing domain-specific performance.