Revisiting Challenges in Real-world Video Colonoscopy using End-to-End Two Stream Polyp Detection Transformer...

Accurate polyp detection is essential for the early diagnosis and effective treatment of colorectal cancer (CRC). However, colonoscopy videos in real-world clinical settings present significant challenges, often causing existing algorithms to fail. Compared to single images, videos contain richer temporal and contextual information, making them valuable for developing deep-learning-based detection systems. To address these challenges, we propose an end-to-end Two-Stream Polyp Detection Transformer (TS-PDTR) network. First, our framework uses a two-stream feature extraction network to capture both spatial and temporal features from the RGB frames and optical flow. Then, the proposed Detail-Aware Convolution (DAConv) module enhances fine-grained contextual information in low-level features. Following this, the Detail-Guided Attention (DGA) module generates channel-specific Spatial Attention Maps (SAMs) to refine deep feature maps, improving the model's sensitivity to small and camouflaged polyps. Finally, a Flow Fusion Encoder (FFE) module combines temporal cues from optical flow to increase robustness against poor single-frame image quality. Experiments on three benchmark video colonoscopy datasets show that TS-PDTR consistently outperforms previous state-of-the-art image- and video-based polyp detection methods. Notably, our model achieves a mean Average Precision (mAP) of 33.2 on the most challenging LDPolypVideo dataset. It also improves the mAP to 64.0 and 55.6 on the SUN Colonoscopy Video Database and CVC-VideoClinicDB, respectively. In summary, TS-PDTR is a promising video-based polyp detection method with strong potential for further development and real-world clinical application.