Multimodal Deep Learning and Visible-Light and Hyperspectral Imaging for Fruit Maturity Estimation.

Area of Science:

• Agricultural Science
• Computer Vision
• Machine Learning

Background:

• Fruit maturity is crucial for supply chains and consumer satisfaction.
• Traditional methods often classify fruit into only two stages: ripe or unripe.
• Accurate maturity assessment is vital for the agriculture industry.

Purpose of the Study:

• To develop a non-destructive, multimodal classification system for estimating six distinct stages of papaya fruit maturity.
• To leverage deep convolutional neural networks (CNNs) for enhanced fruit maturity assessment.
• To explore the efficacy of combining visible-light and hyperspectral imaging for refined maturity classification.

Main Methods:

• Utilized a multimodal approach combining visible-light (RGB) and hyperspectral imaging (400-900 nm).
• Employed feature concatenation of data from both imaging modes.
• Modified and analyzed various deep CNN architectures (AlexNet, VGG16, VGG19, ResNet50, ResNeXt50, MobileNet, MobileNetV2) for multimodal data.
• Performed sensitivity analyses on modified architectures using multimodal data cubes.

Main Results:

• Achieved up to a 0.90 F1 score for classifying six stages of papaya maturity.
• Recorded a low top-2 error rate of 1.45% using multimodal deep learning models.
• Demonstrated that multimodal input significantly improves classification accuracy for refined maturity stages.

Conclusions:

• Multimodal deep learning architectures integrating visible-light and hyperspectral imaging offer high accuracy for classifying fruit maturity into multiple stages.
• This approach shows great potential for real-time, in-field fruit maturity estimation and optimizing harvest times.
• The findings support the application of multimodal imaging in industrial agricultural settings for improved quality control.