De-occlusion models and diffusion-based data augmentation for size estimation of on-plant oriental melons

Accurate fruit size estimation is crucial for plant phenotyping, as it enables precise crop management and enhances agricultural productivity by providing essential data for growth and resource efficiency analysis. In this study, we estimated the size of on-plant oriental melons grown in a vertical cultivation system to address the challenges posed by leaf occlusion. Data augmentation was achieved using a diffusion model to generate synthetic leaves to cover existing fruits and create an enriched dataset. Three instance segmentation models-mask region-based convolutional neural network (CNN), Mask2Former, and detection transformer (DETR)-and six de-occlusion models derived from these architectures were implemented. These models successfully inferred both visible and occluded areas of the fruit. Notably, Amodal Mask2Former and occlusion-aware RCNN (ORCNN) achieved average precision scores of 85.92 ​% and 85.35 ​%, respectively. The inferred masks were used to estimate the height and diameter of the fruit, with Amodal Mask2Former yielding a mean absolute error of 5.46 ​mm and 4.20 ​mm and a mean absolute percentage error of 4.86 ​% and 5.33 ​%, respectively. The results indicate enhanced performance of the transformer-based Amodal Mask2Former over CNN architectures in de-occlusion tasks and size estimation. Finally, the enhancement in de-occlusion models compared to conventional models was assessed and demonstrated across occlusion ratios ranging from 0 to 70 ​%. However, generating synthetic datasets with occlusion ratios over 70 ​% remains a limitation.