Mobile real-time detection transformer for computationally constrained devices

The recent state-of-the-art model, Real Time Detection Transformer (RT-DETR), has surpassed YOLOv8 in terms of both accuracy and speed. We present a model based on the RT-DETR architecture, with a lightweight backbone, to achieve compactness and high speed. Our model, Mobile RT-DETR (MRTDETR), utilises a series of MobileNetv3 blocks. Specifically, the layers 7, 13, and 15 of the MobileNetv3 backbone were used as inputs to the encoder. Separable self attention was also explored, but not retained. MRTDETR achieves 44.6% mAP with a train batch size of 4 on NVIDIA GeForce RTX 3050 OEM GPU and 43.1% mAP with a train batch size of 64 on NVIDIA H100 PCIe GPUs, hence surpassing existing implementations of FasterRCNN, MobileFormer, and MobileViT. With 15.23 GFLOPs, MRTDETR comes close to the performance of the state-of-the-art YOLOv8 and YOLOv9 detectors of similar size while having [Formula: see text] and [Formula: see text] less FLOPs correspondingly. The model has been tested with Raspberry Pi 5, alongside other models (including quantised models), to profile edge related metrics. It has an average inference time of 1,967.4 milliseconds with a standard deviation of 71.9 milliseconds, takes 435.86 MB of VmRSS memory and 1,350.27 MB of VmSize memory, and has an ONNX file size of 43.1 MB. The MRTDETR model outperforms all other models in optimality, with respect to the trade-off between accuracy and FLOPs. By supplementing model complexity metrics and performance metrics, with real-world profiling results, a more relevant evaluation of the proposed MRTDETR model is provided.