Intelligent Head and Neck CTA Report Quality Detection with Large Language Models

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Informatics

Background:

• Radiological report quality control is crucial for patient safety and effective diagnosis.
• Automating error detection in radiological reports can enhance efficiency and accuracy.
• Large language models (LLMs) show promise in analyzing medical text.

Purpose of the Study:

• To evaluate the performance of GPT-4, ERNIE Bot, and SparkDesk in identifying common errors in head and neck CT angiography (CTA) reports.
• To assess the potential of LLMs in supporting quality control processes for Chinese radiological reports.
• To compare the efficiency and accuracy of LLMs against manual review for report quality assessment.

Main Methods:

• Collected 15,000 head and neck CTA reports from two datasets.
• Identified six common error types and developed LLM-based detection methods.
• Assessed report quality using a 5-point Likert scale and statistical tests (Wilcoxon rank-sum, Friedman, ICC, ANOVA).
• Evaluated model performance using accuracy, precision, recall, and F1 score.

Main Results:

• LLMs achieved over 95% detection accuracy for six common error types in Dataset 2.
• Error detection rates were lower in final reports compared to preliminary reports.
• GPT-4 demonstrated moderate consistency with manual scores (ICC=0.517), while ERNIE Bot and SparkDesk showed slightly lower consistency.
• LLMs evaluated reports significantly faster than human reviewers.

Conclusions:

• LLMs can effectively identify error types and differentiate report quality in head and neck CTA reports.
• These models offer a significant improvement in the efficiency of quality control reviews.
• LLMs hold substantial research and practical value for enhancing radiological report quality assurance.