Visual question answering model for fruit tree disease decision-making based on multimodal deep learning.

Area of Science:

• Agricultural Science
• Computer Science
• Artificial Intelligence

Background:

• Current deep learning models for fruit tree diseases rely on single data sources, limiting practical agricultural decision-making.
• Existing methods struggle to integrate diverse data for comprehensive disease analysis.

Purpose of the Study:

• To design a multimodal Visual Question Answering (VQA) model for fruit tree disease diagnosis.
• To fuse image data with question-answering knowledge for improved decision-making in smart agriculture.

Main Methods:

• Developed a multimodal bilinear factorized pooling model using Tucker decomposition to integrate image and question features.
• Explored a deep modular co-attention architecture for simultaneous learning of image and question attention.
• Utilized data augmentation and 10-fold cross-validation for robust performance evaluation.

Main Results:

• Achieved 86.36% accuracy in decision-making with limited data (8,450 images, 4,560k Q&A pairs).
• Outperformed existing multimodal methods in fine-grained identification and decision-making.
• Demonstrated friendly interaction and effective disease region identification.

Conclusions:

• The proposed unified multimodal fusion model offers a significant advancement for intelligent agricultural management.
• The model's ability to integrate visual and textual data enables practical, fine-grained disease diagnosis.
• This VQA approach holds potential for wide deployment in smart agriculture systems.