Segmentation, Reconstruction, and Grasp-Pose Estimation Using an RGB-D Camera for Robotic Apple Harvesting

Accurate segmentation, three-dimensional reconstruction, and grasp-pose estimation of fruits and surrounding structures are essential for robotic harvesting in complex orchard environments. However, occlusions caused by branches, support wires, and irregular canopy structures remain major challenges for perception and manipulation. This study proposes SimAM-SegFormer-DeepLabV3+, a lightweight semantic segmentation model for real-time recognition of apples, branches, and wires. The model integrates a SegFormer encoder into DeepLabV3+ and incorporates a simple attention module to enhance multiscale feature representation. The segmentation results are fused with RGB-D data for semantic-guided 3D reconstruction. Apples are reconstructed using RANSAC-based sphere fitting, whereas branches and wires are reconstructed using PCA-guided cylinder fitting. Based on the reconstructed scene, a grasp pose estimation strategy is developed to generate obstacle-aware approach vectors while satisfying end-effector constraints. Experimental results showed that SimAM-SegFormer-DeepLabV3+ achieved a mean IoU of 88.6% and a mean pixel accuracy of 95.1%. Reconstruction count accuracy reached 96.4%, 90.0%, and 88.2% for apples, branches, and wires, respectively. The mean dimensionless grasp-direction error was below 15 in three directions. In grasping experiments, the system achieved success rates of 92% and 82% in simple and complex indoor scenes, respectively. This study provides a reference for vision-based apple harvesting using agricultural robotic arms.