Advancing Chinese Conversation-based Patient Guidance with a Benchmark and Knowledge-Evolvable Assistant

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing
• Clinical Informatics

Background:

• Chinese Conversation-based Patient Guidance (CCPG) utilizes natural language to direct patients to appropriate hospital departments.
• Despite advancements in Large Language Models (LLMs), CCPG lacks dedicated benchmarks and remains under-explored.
• Existing LLMs demonstrate insufficient performance for practical CCPG applications.

Purpose of the Study:

• To develop the first comprehensive benchmark, PG-Bench, for evaluating CCPG systems.
• To introduce a novel framework, Knowledge-Evolvable Assistant (KEA), to enhance LLM performance in CCPG.
• To establish a rigorous foundation and baseline for future research in conversation-driven patient guidance.

Main Methods:

• Developed PG-Bench, comprising five subsets, 19,814 annotated dialogues, and 98 clinical departments.
• Evaluated 25 representative LLMs on PG-Bench, including state-of-the-art models.
• Introduced the Knowledge-Evolvable Assistant (KEA) framework, augmenting LLMs with experience and reflection banks and an external knowledge base.
• Utilized retrieval-augmented generation for iterative knowledge evolution within KEA.

Main Results:

• PG-Bench revealed uniformly poor performance across all evaluated LLMs, failing to meet practical requirements.
• KEA consistently and significantly improved the CCPG performance of all tested LLMs on PG-Bench.
• Despite improvements, current performance levels still fall short of clinical expectations, highlighting the complexity of CCPG.

Conclusions:

• PG-Bench serves as a crucial resource for advancing CCPG research.
• The KEA framework offers a viable approach to enhance LLM capabilities for CCPG.
• Further research is essential to bridge the gap between current AI capabilities and clinical requirements in patient guidance.