Generative Large Language Models for Detection of Speech Recognition Errors in Radiology Reports.

Area of Science:

• Medical Informatics
• Artificial Intelligence
• Radiology

Background:

• Speech recognition errors in radiology reports can impact patient care.
• Manual error detection is time-consuming and prone to human error.

Purpose of the Study:

• To evaluate the efficacy of generative large language models (LLMs) in detecting speech recognition errors within radiology reports.
• To compare the performance of various LLMs, including GPT-4, in identifying these errors.

Main Methods:

• A dataset of 3233 CT and MRI reports was analyzed for speech recognition errors by radiologists.
• Five generative LLMs (GPT-3.5-turbo, GPT-4, text-davinci-003, Llama-v2-70B-chat, Bard) were tested for error detection.
• Prompt engineering was utilized to optimize LLM performance, with manual detection serving as the reference standard.

Main Results:

• GPT-4 achieved high accuracy, with F1 scores of 86.9% for clinically significant errors and 94.3% for non-clinically significant errors.
• Other LLMs showed varied performance, with Bard exhibiting the lowest accuracy.
• Factors such as longer reports and resident dictation were associated with increased error rates.

Conclusions:

• Advanced generative LLMs, particularly GPT-4, demonstrate significant potential for the automated detection of speech recognition errors in radiology reports.
• This technology could enhance the accuracy and efficiency of radiology reporting quality assurance.
• Further research may explore integrating LLMs into clinical workflows for real-time error flagging.