A multimodal dual-branch attention-centric YOLOv12 framework for real-time integrated pest and weed detection from...

The combination of unmanned aerial vehicle technology and deep learning has revolutionized precision agriculture by facilitating the monitoring of crops using UAVs. The current pest and weed detection methods have limitations in that they use individual neural networks for pest and weed detection, leading to methodological fragmentation. This new framework clearly explains about a new pest and weed detection system dubbed AgriYOLO12-Dual, based on a unified detection framework. The framework employs YOLOv12, the initial YOLO variant to use self-attention as a basic computing unit. The attention-centric YOLOv12 framework system has a dual-branch encoder network that takes RGB and multispectral images from a UAV and passes them separately to a neural network. The outputs of the networks are then combined at a cross-modal fusion point using Area Attention. The attention-centric YOLOv12 framework system has three key innovations. The primary one is the Area Attention mechanism, it has a large receptive field and linear computational cost. The second innovation is the use of Residual Efficient Layer Aggregation Networks (R-ELAN), that allows for the training of large attention models. The third innovation is the use of Flash Attention to reduce memory usage by 38%. The multimodal dual-branch attention-centric YOLOv12 framework was trained on 15,000 annotated images of maize, soybean, and wheat crops at various stages of growth. The results showed that the attention-centric YOLOv12 framework system achieved a weed detection mAP of 90.1% and a pest detection mAP of 93.4%, with an inference time of 28.7 ms on a Jetson Orin device. The small target recall was improved from 67.4% to 79.3%, and the low-light detection mAP was improved from 76.2% to 87.2%. The results of the experiments showed that attention-based models have a significant improvement in pest and weed detection accuracy without additional complexity.