Estimating strawberry weight for grading by picking robot with point cloud completion and multimodal fusion network

Area of Science:

• Computer Vision
• Robotics
• Agricultural Technology

Background:

• Automated strawberry grading using robots requires precise weight estimation for classification.
• Manual measurement is labor-intensive and can damage delicate fruit.
• Depth image completion is challenging due to environmental interference affecting depth cameras.

Purpose of the Study:

• To develop a multimodal point cloud completion method for symmetrical objects like strawberries.
• To create a multimodal fusion regression model (MMF-Net) for accurate strawberry weight estimation.
• To enhance automated strawberry grading systems for robotic harvesting.

Main Methods:

• Collected 1521 strawberry RGB-D images and manual weight/size measurements.
• Proposed a multimodal point cloud completion method leveraging RGB images to guide depth image completion for symmetrical objects.
• Developed MMF-Net, integrating features from completed point clouds (PointNet) and RGB images (EfficientNet) using gradient blending.

Main Results:

• The proposed multimodal fusion model, MMF-Net, achieved 87.66% accuracy (PCW@0.2) in strawberry weight estimation.
• MMF-Net significantly outperformed traditional methods (best: SVR at 77.7%) and single-modal deep learning models (EfficientNet at 85%, PointNet++ at 54.3%).
• The multimodal approach effectively combined the strengths of RGB and depth data for superior performance.

Conclusions:

• The developed multimodal point cloud completion and fusion method significantly enhances strawberry weight estimation accuracy.
• MMF-Net offers a robust solution for precise grading in automated agricultural systems.
• This approach paves the way for more efficient and accurate robotic harvesting and fruit classification.