Observe, align, and enhance: a hierarchical retrieval-augmented vision-language model for generating radiology...

Radiology Report Generation (RRG) is designed to automatically generate diagnostic narratives based on radiological image interpretation, supporting clinicians in making diagnoses and relieving radiologists of reporting pressure. Previous approaches mainly use generative architectures developed to change visibly produced features into coherent written results. However, these approaches generally run into problems with precisely aligning written outputs to visible data, especially in the process of producing thorough diagnostic narratives. We propose a new, hierarchical retrieval-enhanced framework called Observe, Align, and Enhancement (OAE) for the creation of the radiology report to get around these problems: Observe, which leverages retrieval techniques to enhance visual feature comprehension by identifying similar images and associated reports; Align, where the retrieved contextual reports guide the generation of an initial diagnostic report ensuring semantic consistency; and Enhance, an iterative refinement process that incorporates additional textual information to improve semantic coherence and diagnostic precision. The proposed framework is shown in a comprehensive assessment of the IU-XRAY and MIMIC-CXR datasets, which shows that it performs better than the current state-of-the-art approaches, including improved diagnostic accuracy and reported quality.