IPENS: Interactive unsupervised framework for rapid plant phenotyping extraction via NeRF-SAM2 fusion

Area of Science:

• Agricultural Science
• Computer Vision
• Plant Biology

Background:

• Advanced plant phenotyping is crucial for crop improvement and intelligent breeding.
• Existing methods often require extensive manual annotation, which is challenging for diverse species and self-occluded grain-level objects.
• Unsupervised methods struggle with precise segmentation of individual plant parts.

Purpose of the Study:

• To develop an unsupervised, interactive method for accurate multi-target point cloud extraction in plant phenotyping.
• To overcome limitations of existing methods in handling species diversity and self-occlusion at the grain level.
• To enable efficient and non-invasive phenotypic trait estimation for crops like rice and wheat.

Main Methods:

• Proposed IPENS (Interactive Plant Extraction using Neural Fields), an unsupervised multi-target point cloud extraction method.
• Utilized radiance field information to lift 2D masks from Segment Anything Model 2 (SAM2) into 3D space.
• Implemented a multi-target collaborative optimization strategy for segmenting multiple objects from single interactions.

Main Results:

• Achieved high segmentation accuracy on rice (mIoU 63.72%) and wheat (mIoU 89.68%).
• Demonstrated excellent phenotypic trait estimation: grain volume R² up to 0.7697 (rice) and 0.9956 (wheat).
• Leaf area, length, and width predictions showed high accuracy (R² up to 0.84 for rice, 1.00 for wheat).

Conclusions:

• IPENS provides a rapid (under 3 minutes), non-invasive solution for grain-level point cloud extraction without manual annotation.
• The method effectively extracts phenotypic traits for rice and wheat, significantly aiding intelligent breeding programs.
• IPENS offers a robust and scalable approach for advanced plant phenotyping, addressing key challenges in current technologies.