Pattern-Aware Transformer: Hierarchical Pattern Propagation in Sequential Medical Images.

Area of Science:

• Computer Vision
• Medical Imaging Analysis
• Artificial Intelligence

Background:

• Sequential image analysis in medical imaging often struggles with effectively capturing contextual information.
• Existing methods using point-wise token encoding have limitations in preserving both local and global details.

Purpose of the Study:

• To develop a novel method for mining contextual information among sequential images in medical imaging tasks.
• To introduce a hierarchical pattern-aware tokenization strategy and a Pattern-Aware Transformer (PATrans) for improved sequence modeling.

Main Methods:

• A hierarchical pattern-aware tokenization strategy is proposed to handle distinct visual patterns independently and hierarchically.
• A Pattern-Aware Transformer (PATrans) with a global-local dual-path pattern-aware cross-attention mechanism is developed for hierarchical pattern matching.
• PATrans is designed as a plug-and-play module for integration with various backbone networks.

Main Results:

• PATrans achieves state-of-the-art performance across multiple benchmarks in video object detection and 3D volumetric semantic segmentation.
• Specific benchmark results include CVC-Video (92.3% detection F1), ASU-Mayo (99.1% localization F1), Lung Tumor (78.59% DSC), Nasopharynx Tumor (75.50% DSC), and Kidney Tumor (87.53% DSC).

Conclusions:

• The proposed hierarchical pattern-aware tokenization and PATrans effectively model sequential images by preserving local and global information.
• PATrans demonstrates broad applicability and superior performance in diverse medical imaging sequence modeling tasks.