Unsupervised Hybrid framework for ANomaly Detection (HAND)- applied to Screening Mammogram

Out-of-distribution (OOD) detection is essential for improving the generalization of of AI models used in mammogram screening, as unknown distribution shifts in external datasets can degrade performance. Identifying OOD samples helps maintain strong model performance across internal external datasets. Given the challenge of limited prior knowledge about OOD samples in external datasets, unsupervised generative learning is a preferable solution which trains the model to discern the normal characteristics of in-distribution (ID) data. Generic hypothesis is that during inference, the model aims to reconstruct ID samples accurately, while OOD samples exhibit poorer reconstruction due to their divergence from normality. Inspired by the hybrid architectures combining CNNs and trans formers, we developed a novel backbone- HAND, for detecting OOD from large-scale digital screening mammogram studies. However, relying solely on reconstruction error is insufficient for reliable OOD detection, as it may not consistently distinguish between ID and OOD samples. To address this, we incorporated synthetic OOD samples and a parallel discriminator in the latent space to distinguish between ID and OOD samples. Additionally, we applied gradient reversal to the OOD reconstruction loss, effectively discouraging the model from accurately reconstructing OODinputs and reinforcing its ability to differentiate them from ID samples. An anomaly score is computed by weighting the reconstruction and discriminator loss. On internal RSNA mammogram held-out test and external Mayo clinic hand-curated dataset, the proposed HAND model outperformed encoder-based and GAN-based baselines, and interestingly, it also outperformed the hybrid CNN+transformer baselines. Therefore, the proposed HAND pipeline offers an automated efficient computational solution for domain-specific quality checks in external screening mammograms, yielding actionable insights without direct expo sure to the private medical imaging data.