SI-ViT: Shuffle instance-based Vision Transformer for pancreatic cancer ROSE image classification

The rapid on-site evaluation (ROSE) technique improves pancreatic cancer diagnosis by enabling immediate analysis of fast-stained cytopathological images. Automating ROSE classification could not only reduce the burden on pathologists but also broaden the application of this increasingly popular technique. However, this approach faces substantial challenges due to complex perturbations in color distribution, brightness, and contrast, which are influenced by various staining environments and devices. Additionally, the pronounced variability in cancerous patterns across samples further complicates classification, underscoring the difficulty in precisely identifying local cells and establishing their global relationships.

To address these challenges, we propose an instance-aware approach that enhances the Vision Transformer with a novel shuffle instance strategy (SI-ViT). Our approach presents a shuffle step to generate bags of shuffled instances and corresponding bag-level soft-labels, allowing the model to understand relationships and distributions beyond the limited original distributions. Simultaneously, combined with an un-shuffle step, the traditional ViT can model the relationships corresponding to the sample labels. This dual-step approach helps the model to focus on inner-sample and cross-sample instance relationships, making it potent in extracting diverse image patterns and reducing complicated perturbations.

Compared to state-of-the-art methods, significant improvements in ROSE classification have been achieved. Aiming for interpretability, equipped with instance shuffling, SI-ViT yields precise attention regions that identifying cancer and normal cells in various scenarios. Additionally, the approach shows excellent potential in pathological image analysis through generalization validation on other datasets.

By proposing instance relationship modeling through shuffling, we introduce a new insight in pathological image analysis. The significant improvements in ROSE classification leads to protential AI-on-site applications in pancreatic cancer diagnosis. The code and results are publicly available at https://github.com/sagizty/MIL-SI.