From raw audio to structure: an agent-based pipeline that boosts medical LLM performance

Area of Science:

• Artificial Intelligence
• Computational Linguistics
• Medical Informatics

Background:

• Large language models (LLMs) require high-quality conversational data for clinical communication applications.
• Real-world clinical recordings are often degraded by noise, transcription errors, and fragmented dialogue, limiting their utility for training LLMs.

Purpose of the Study:

• To develop an agent-based transcription framework to convert raw unstructured clinical conversation transcriptions (RUCT) into structured conversation transcriptions (SCT).
• To evaluate the framework's accuracy, speed, and impact on downstream LLM fine-tuning for medical dialogue.

Main Methods:

• An agent-based system with Planner, Memory, and Executor modules was designed for noise removal, content correction, speaker identification, and dialogue segmentation.
• The framework was applied to Chinese and English clinical recordings, and its performance was compared against other methods.
• An independent LLM (Qwen3-32B) was fine-tuned using agent-generated SCT and RUCT for comparative analysis.

Main Results:

• The agent achieved high reconstruction accuracy: 94.7% denoising, 96.9% content correction, 88.6% speaker identification, and 92.7% segmentation.
• The framework operated 3.6x faster than manual processing and outperformed cascaded deep-learning and end-to-end models.
• Fine-tuning LLMs on agent-generated SCT significantly improved quality scores (3.1 to 3.7) and benchmark performance compared to RUCT fine-tuning.

Conclusions:

• Agent-structured clinical corpora enhance LLM fine-tuning performance for medical dialogue applications.
• The developed framework offers a scalable solution for creating reliable medical conversational AI by improving data quality.