Diffusion Prior-Driven Vision-Language Adaptation for Few-Shot Medical Anomaly Detection

Unsupervised medical anomaly detection faces dual challenges of pathological data scarcity and domain gaps when transferring general-purpose vision-language models to fine-grained tasks. This study aims to address the misalignment between global semantic representations and local pathological nuances by proposing a novel hybrid supervision framework. We propose a flexible vision-language adaptation framework in which diffusion-based pseudo-anomaly synthesis serves as an optional training enhancement. For CheXpert and BUSI, region-guided diffusion generates localized pseudo-anomalies to provide additional discriminative supervision. For BrainMRI, the framework is trained exclusively on normal reference images without any synthetic anomaly augmentation, allowing us to assess its effectiveness independently of the diffusion synthesis component. The architecture incorporates a local-enhanced feature rectifier to enhance local textural representation, a zero-initialized hypersphere-aligned residual adapter for stable cross-modal manifold alignment, and a temperature-scaled mapmaker for multi-scale dense anomaly localization. Performance was evaluated across three medical imaging modalities under multiple few-shot configurations. Our method remains highly competitive compared to state-of-the-art approaches in terms of image-level detection and pixel-level localization. On the ultrasound dataset under a 32-shot setting, the framework achieved an image-level area under the curve of 94.2% and a pixel-level area under the curve of 96.6%. These results demonstrate the effectiveness of the proposed framework across the evaluated medical imaging modalities under different training configurations. In particular, its competitive performance on BrainMRI without synthetic anomaly augmentation suggests that the vision-language adaptation framework can remain effective when offline pseudo-anomaly generation is unavailable. The code involved in this study is publicly available on website https://github.com/SWLee777/DiffVLA .