Zero-shot instance segmentation for plant phenotyping in vertical farming with foundation models and VC-NMS

Area of Science:

• Computer Vision
• Agricultural Technology
• Plant Science

Background:

• Image instance segmentation is crucial for plant phenotyping in vertical farms.
• Limited annotated data and diverse plant types hinder traditional supervised methods.
• Zero-shot segmentation is needed to segment plants without type-specific training data.

Purpose of the Study:

• To develop a zero-shot instance segmentation framework for plant phenotyping in vertical farms.
• To address the challenge of scarce annotated data in agricultural computer vision.
• To improve segmentation accuracy and generalization without target-specific annotations.

Main Methods:

• A zero-shot instance segmentation framework combining Grounding DINO and Segment Anything Model (SAM).
• Vegetation Cover Aware Non-Maximum Suppression (VC-NMS) with Normalized Cover Green Index (NCGI) for refining box prompts.
• Integration of similarity maps with a max distance criterion for enhancing point prompts.

Main Results:

• The proposed framework outperforms SAM's everything mode and Grounded SAM in zero-shot segmentation.
• Enhanced box and point prompts show superior performance on test datasets.
• The framework achieves the best zero-shot generalization compared to supervised methods like YOLOv11.

Conclusions:

• The developed zero-shot segmentation framework effectively addresses data scarcity in vertical farming.
• Domain-specific indices and optimized prompts offer a robust solution for plant phenotyping.
• Weakly supervised models show significant potential for agricultural computer vision applications.