Benchmarking Radiology Report Generation From Noisy Free-Texts

Area of Science:

• Artificial Intelligence in Medical Imaging
• Natural Language Processing for Healthcare
• Radiology Informatics

Background:

• Automatic radiology report generation is crucial for diagnostic efficiency but faces challenges with limited clean data.
• Existing radiological texts are often noisy and unsuitable for direct use.
• There's a need for methods to refine noisy free-text into accurate radiology reports.

Purpose of the Study:

• Introduce a novel task, Noisy Report Refinement (NRR), to generate radiology reports from noisy free-texts.
• Propose a report refinement pipeline leveraging large language models (LLMs) with guided self-critique and report selection.
• Develop a new benchmark, NRRBench, with clinically explainable LLM-based metrics for evaluating NRR.

Main Methods:

• Developed a report refinement pipeline using LLMs with guided self-critique and report selection strategies.
• Introduced NRRBench, comprising two online-sourced datasets for NRR.
• Proposed four LLM-based metrics: radiology entity matching, modality-specific template attribute matching, report cleanliness, and overall performance.

Main Results:

• Guided self-critique and report selection strategies significantly enhanced the quality of refined radiology reports.
• The proposed NRRBench metrics demonstrated a higher correlation with report noise and error rates compared to traditional metrics.
• LLM-based metrics effectively evaluated cleanliness, radiological usefulness, and factual correctness in NRR.

Conclusions:

• The proposed NRR pipeline effectively refines noisy radiological texts into usable reports.
• NRRBench and its associated metrics provide a robust evaluation framework for the NRR task.
• This work advances automatic radiology report generation by addressing data noise and improving evaluation methodologies.