Segmentation-enhanced multi-scale deep hashing for chest X-ray image retrieval

Chest X-ray (CXR) images play a crucial role in diagnosing COVID-19 by facilitating the rapid identification of lung damage. Recently, deep hashing technology has enhanced our capabilities for retrieving large CXR image databases, providing healthcare professionals with more comprehensive tools for pandemic analysis. However, current methods typically depend on single-scale features to represent CXR images, without modeling the multi-scale contextual information present in the original images. Additionally, lung segmentation and image classification from CXR offer anatomical and semantic insights that could enhance the feature extraction of CXR images, but this remains largely unexplored in the field. To this end, we develop a segmentation-enhanced multi-scale deep hashing (SMDH) framework for automated CXR image retrieval, comprising a feature extraction module and an image retrieval module. The feature extraction module comprises a multi-scale neural network architecture, aiming to deeply mine and integrate key semantic information within CXR images through a multi-level feature fusion strategy. In particular, to capture rich anatomical and semantic information in CXR images, we utilize lung segmentation and image classification as auxiliary tasks to guide the feature extraction process. During retrieval, the trained feature extraction module is used to convert each input query CXR image and all CXR images in the retrieval database into binary hash codes, followed by Hamming distance-based ranking for fast image matching. Experiments on the COVID-QU-Ex dataset with 33,920 CXR images suggest that SMDH outperforms several state-of-the-art methods.