Enhancing Instance Segmentation in Agriculture: An Optimized YOLOv8 Solution

To address the limitations of traditional segmentation algorithms in processing complex agricultural scenes, this paper proposes an improved YOLOv8n-seg model. Building upon the original three detection layers, we introduce a dedicated layer for small object detection, which significantly enhances the detection accuracy of small targets (e.g., people) after processing images through fourfold downsampling. In the neck network, we replace the C2f module with our proposed C2f_CPCA module, which incorporates a channel prior attention mechanism (CPCA). This mechanism dynamically adjusts attention weights across channels and spatial dimensions to effectively capture relationships between different spatial scales, thereby improving feature extraction and recognition capabilities while maintaining low computational complexity. Finally, we propose a C3RFEM module based on the RFEM architecture and integrate it into the main network. This module combines dilated convolutions and weighted layers to enhance feature extraction capabilities across different receptive field ranges. Experimental results demonstrated that the improved model achieved 1.4% and 4.0% increases in precision and recall rates on private datasets, respectively, with mAP@0.5 and mAP@0.5:0.95 metrics improved by 3.0% and 3.5%, respectively. In comparative evaluations with instance segmentation algorithms such as the YOLOv5 series, YOLOv7, YOLOv8n, YOLOv9t, YOLOv10n, YOLOv10s, Mask R-CNN, and Mask2Former, our model achieved an optimal balance between computational efficiency and detection performance. This demonstrates its potential for the research and development of small intelligent precision operation technology and equipment.