Medical QA dialogue datasets in RAG systems performance evaluation and ChatGPT optimization.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Medical Informatics

Background:

• Clinical question answering (QA) systems aim to provide accurate medical information.
• Retrieval-Augmented Generation (RAG) enhances large language models by incorporating external knowledge.
• Optimizing retrieval sources is crucial for improving RAG performance in specialized domains like medicine.

Purpose of the Study:

• To evaluate the effectiveness of Chinese doctor-patient dialogues as retrieval sources for RAG in clinical QA.
• To compare various retrieval strategies and advanced language models (GPT-4o, GPT-5) for medical QA.
• To identify key factors influencing RAG performance in the medical domain.

Main Methods:

• Utilized Chinese doctor-patient dialogues as a retrieval corpus for RAG.
• Implemented and compared retrieval methods: dense retrieval, Cross-Encoder reranking, Reciprocal Rank Fusion (RRF), and Cascade RRF→Rerank.
• Evaluated performance using automatic metrics (ROUGE, BERTScore) and expert human evaluations across different language models (ChatGPT-3.5, GPT-4o, GPT-5).

Main Results:

• Dialogue-based retrieval significantly improved generation quality over direct prompting (ROUGE-1-f: +12.6%, BERTScore_F1: +1.5%).
• The Rerank-only strategy offered the optimal accuracy-latency balance; cascade pipelines did not provide additional benefits.
• GPT-4o demonstrated superior automatic metrics and lower latency, while GPT-5 achieved slightly higher human preference scores.

Conclusions:

• Chinese doctor-patient dialogues are effective retrieval sources for improving RAG in clinical QA.
• Data representation and metadata structure are more critical for RAG performance than retrieval algorithm complexity.
• Findings provide practical guidance for deploying reliable medical QA systems using RAG.