Accurate orange yield estimation using a novel dataset, fine-tuned deep learning models, and vision-LLM benchmarking

Accurate yield estimation is one of the critical factors for overall management of orchards production and profit estimation. Traditionally, oranges yield estimation is done manually which are not only labor-intensive and time-consuming but also prone to errors, making them inefficient for large-scale agriculture. Modern deep learning techniques have enhanced fruit detection and yield estimation algorithms, helping farmers to implement and mature precision agriculture. Yield estimation of orange trees from the images is still a challenging task due to the multiple factors such as fruit density and overlapping causing occlusion, varying lighting conditions, variation in orange sizes and color, image quality and resolution etc. Most of the existing oranges detection and estimation studies focus on the detection of single oranges. However, in an image, there are overlapping oranges which cause some oranges to be occluded which may cause the model to generate erroneous results. To handle this issue, a standard dataset of orange images needs to be collected which have variation of oranges sizes and are captures in different lighting conditions. The dataset must be annotated with multiple classes of oranges bunches to handle occluded oranges scenario so that different state of the art machine learning and deep learning models can be bench-marked to have high precision of oranges detection and estimation. In this paper, a significant images dataset having 25,541 images has been collected by capturing 41 videos of multiple orange trees at different angles. After doing frame differencing, eventually 9,781 images are selected having variation of orange sizes, colors, occlusion and lighting conditions. A detailed annotation guideline has been defined for the annotation of one orange, bunch of two, three and four oranges separately to ensure the correct estimation of oranges. A total of 205,712 bounding boxes have been annotated having a total of 157,128, 33,208, 10,761 and 5,413 instances of one orange, bunch of two oranges, bunch of three and bunch of four oranges respectively. This dataset will promote research and development in the area of orange yield estimation. In this study yield estimation is done in two passes. Detection of oranges classes using three state-of-the-art multiclass models namely Faster R-CNN, Mask R-CNN, and YOLOv8n by fine-tuning on this dataset. These models are used for the estimation of oranges by aggregating results of detection. Faster R-CNN outperforms for the estimation of oranges, having 18.22 Mean Absolute Percentage Error (MAPE) and [Formula: see text] of 0.95. Furthermore, the zero-shot evaluation of Vision Large Language Models (VLLMs), specifically Gemini and LLaMA, is also assessed. The evaluation results indicate that Gemini and LLaMA achieve MAPE values of 52.05 and 103.77, respectively and [Formula: see text] values of 0.85 and 0.42 respectively.