MuCoSA: Multi-contextual similarity assessment for histopathology image search

Histological diagnosis in pathology often begins with visual identification of morphological patterns that resemble known disease entities. Whereas digital pathology and augmented intelligence have enabled reverse image search in histopathology, most existing frameworks rely on single-magnification image patches and slide-level labels, limiting their ability to capture tissue morphologies with at various levels of granularity and scale. To address this, we propose Progressive Regional Image Sequence by Magnification (PRISM), a sequence of images that captures contextual information across multiple magnifications. Building on this, we introduce Multi-Contextual Similarity Assessment (MuCoSA), a PRISM-based image retrieval framework that leverages pre-trained feature encoders without additional fine-tuning. To evaluate MuCoSA's performance, we constructed reference and query PRISM datasets utilizing histological patterns of lung adenocarcinoma. Whole-slide image (WSIs) from Samsung Medical Center were used as reference and internal query, whereas those from The Cancer Genome Atlas served as external query. For each WSI, representative regions of interest were selected and converted into PRISM structure. Similarity between two PRISMs was calculated by averaging the cosine similarities of corresponding patches at the same magnification level across all magnifications. We conducted a comprehensive evaluation across different feature encoders, magnification levels, receptive field sizes, and both internal/external query conditions. As a result, MuCoSA with a multi-magnification outperformed single-magnification methods. For instance, MuCoSA using UNI2-h encoder with eight magnifications achieved an F1-score of 0.8051 (95% CI: 0.7843-0.8267), mAP@5 of 0.7065 (95% CI: 0.6893-0.7250), and mMV@5 of 0.8232 (95% CI: 0.8038-0.8434), all significantly higher than the single-magnification baseline (p < 0.0001). Furthermore, confusion matrices and visual inspection confirmed that search results were closely aligned with the morphological perceptions of pathologists. In conclusion, our study demonstrates that using a simple and efficient framework like MuCoSA with multi-magnification PRISM without fine-tuning can significantly improve image search in histopathology. We anticipate that MuCoSA can assist pathologists in making more accurate and consistent diagnoses, thereby reducing observer variability in histopathological interpretation.