Dual-modality fusion for mango disease classification using dynamic attention based ensemble of leaf & fruit images

Area of Science:

• Agricultural Science
• Computer Science
• Plant Pathology

Background:

• Mango cultivation is vital to tropical economies but threatened by diseases like Anthracnose and Red Rust.
• Early and accurate mango disease detection is challenging due to symptom similarity and misclassification.

Purpose of the Study:

• To develop a multimodal deep learning framework for enhanced mango disease classification.
• To improve detection accuracy and generalization by integrating leaf and fruit image data.

Main Methods:

• Trained individual Convolutional Neural Network (CNN) models (ResNet-50, MobileNetV2, EfficientNet-B0, ConvNeXt) on mango leaf and fruit disease datasets.
• Introduced a Modality Attention Fusion (MAF) mechanism to dynamically combine predictions from different image types.
• Implemented a class-aware augmentation pipeline to mitigate overfitting and enhance generalization.

Main Results:

• The attention-based fusion strategy achieved 99.08% test accuracy, 99.03% F1 score, and 99.96% ROC-AUC using EfficientNet-B0.
• The multimodal approach significantly outperformed individual models and static fusion methods.
• A web application demonstrated real-world applicability through out-of-distribution testing.

Conclusions:

• Combining visual data from multiple plant organs (leaves and fruits) improves diagnostic accuracy.
• Adaptive model attention to contextual features is crucial for effective agricultural diagnostics.
• The proposed framework offers a robust solution for early and accurate mango disease identification.