Semantic Consistency-Based Uncertainty Quantification for Factuality in Radiology Report Generation

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Automated radiology report generation (RRG) shows promise in assisting radiologists.
• Current generative models, including Vision Large Language Models (VLLMs), struggle with factual accuracy and hallucinations.
• Ensuring the factual correctness of AI-generated reports is a critical challenge.

Purpose of the Study:

• To introduce a novel Semantic Consistency-Based Uncertainty Quantification (SCUQ) framework for RRG.
• To provide both report-level and sentence-level uncertainty measures.
• To develop a plug-and-play module that enhances factual accuracy without altering existing models.

Main Methods:

• Developed a Semantic Consistency-Based Uncertainty Quantification (SCUQ) framework.
• Implemented a method that quantifies uncertainty without model modification or access to inner states.
• Integrated the SCUQ framework as a plug-and-play module with state-of-the-art RRG models.

Main Results:

• The SCUQ framework effectively detects hallucinations and improves the factual accuracy of generated reports.
• Rejecting high-uncertainty reports improved factuality scores by 10% on the MIMIC-CXR dataset using the Radialog model.
• Sentence-level uncertainty successfully flagged low-precision sentences with an 82.9% success rate.

Conclusions:

• The SCUQ framework significantly enhances the factual accuracy of AI-generated radiology reports.
• This plug-and-play approach offers a practical solution for improving the reliability of RRG systems.
• The open-source implementation facilitates further research and adoption in medical AI.