ES-DETR: Real-time detection transformer with encover and soft-dropout

Real-time Detection Transformer (RT-DETR) exhibits notable advantages in real-time object detection, as evidenced by its enhanced speed and accuracy. However, the performance of RT-DETR is still constrained, due to (a) limited channel-wise correlation of attention mechanism that fails to capture cross-channel feature interactions, (b) information loss caused by dropout that may affect small and overlapped targets, and (c) insufficient robustness and limited diversity of small-scale training set. To solve these problems, this study proposes a real-time Detection Transformer with Encover and Soft-Dropout (ES-DETR). ES-DETR consists of three parts: (a) a spatial attention oriented module called Encover is introduced to replace the traditional flattened attention, and learns global spatial features by capturing cross-view knowledge from images; (b) soft dropout (SD) that replaces traditional dropout with a predetermined number of features being suppressed by Gaussian distribution, and performs stable feature-wise dropout to increase the robustness of the detector; and (c) Grid Noise Augmentation (GNA) that divides the image into grid-like patterns, and pile multiple Gaussian masks on the image to mitigate real-world disturbances. A series of experiments conducted on several datasets show that ES-DETR achieves significant improvement and excels in most of object detection tasks. Source code and pretrained models are available at https://github.com/he13689/ES-DETR.