An end-to-end tracking method for polyp detectors in colonoscopy videos

Deep learning based computer-aided diagnosis technology demonstrates an encouraging performance in aspect of polyp lesion detection on reducing the miss rate of polyps during colonoscopies. However, to date, few studies have been conducted for tracking polyps that have been detected in colonoscopy videos, which is an essential and intuitive issue in clinical intelligent video analysis task (e.g. lesion counting, lesion retrieval, report generation). In the paradigm of conventional tracking-by-detection system, detection task for lesion localization is separated from the tracking task for cropped lesions re-identification. In the multi object tracking problem, each target is supposed to be tracked by invoking a tracker after the detector, which introduces multiple inferences and leads to external resource and time consumption. To tackle these problems, we proposed a plug-in module named instance tracking head (ITH) for synchronous polyp detection and tracking, which can be simply inserted into object detection frameworks. It embeds a feature-based polyp tracking procedure into the detector frameworks to achieve multi-task model training. ITH and detection head share the model backbone for low level feature extraction, and then low level feature flows into the separate branches for task-driven model training. For feature maps from the same receptive field, the region of interest head assigns these features to the detection head and the ITH, respectively, and outputs the object category, bounding box coordinates, and instance feature embedding simultaneously for each specific polyp target. We also proposed a method based on similarity metric learning. The method makes full use of the prior boxes in the object detector to provide richer and denser instance training pairs, to improve the performance of the model evaluation on the tracking task. Compared with advanced tracking-by-detection paradigm methods, detectors with proposed ITH can obtain comparative tracking performance but approximate 30% faster speed. Optimized model based on Scaled-YOLOv4 detector with ITH illustrates good trade-off between detection (mAP 91.70%) and tracking (MOTA 92.50% and Rank-1 Acc 88.31%) task at the frame rate of 66 FPS. The proposed structure demonstrates the potential to aid clinicians in real-time detection with online tracking or offline retargeting of polyp instances during colonoscopies.