Automatic Speech Recognition and Large Language Models for Multilingual Pathology Report Generation: Proof-of-Concept...

Area of Science:

• Medical Informatics
• Natural Language Processing
• Pathology Reporting

Background:

• Accurate pathology gross examination dictation is crucial for clinical documentation.
• Multilingual dictation, especially mixed Chinese-English, presents transcription challenges for final English reports.

Purpose of the Study:

• To evaluate a Whisper-based automatic speech recognition (ASR) pipeline enhanced with contextual system messages.
• To assess the performance of open-source large language models (LLMs) in generating English pathology reports from multilingual dictations.
• To improve transcription accuracy and clinical appropriateness of pathology reports.

Main Methods:

• A controlled study used 125 simulated mixed Chinese-English pathology dictation audio recordings.
• Whisper ASR was employed with and without contextual system messages for transcription.
• Three open-source LLMs (Qwen2:72b, Llama3.1:70b, Gemma2:27b) converted ASR transcripts into English gross description reports.
• Outcomes included character error rate, automated metrics (BLEU, ROUGE), pathologist rankings, and error categorization.

Main Results:

• Contextual system messages reduced the mean character error rate in ASR from 0.344 to 0.066 (P<.001).
• Qwen2:72b achieved the highest automated metric scores and the lowest pathologist-coded total error rate (16.8%).
• Pathologist agreement on the top-ranked model was high (81.6%, Cohen κ=0.722).

Conclusions:

• Contextual prompting enhances ASR accuracy for multilingual pathology dictation.
• Qwen2:72b demonstrated the highest accuracy among evaluated LLMs for generating pathology reports.
• LLM-generated reports require pathologist verification, and clinical validation is necessary before deployment.