Cross-Modal Data Fusion via Vision-Language Model for Crop Disease Recognition

Area of Science:

• Agricultural Science
• Computer Science
• Artificial Intelligence

Background:

• Crop diseases significantly threaten global food security and agricultural productivity.
• Accurate and timely disease identification is essential for crop yield and quality management.
• Existing deep learning methods primarily rely on image data, often neglecting valuable textual information.

Purpose of the Study:

• To develop a novel cross-modal data fusion approach for crop disease recognition.
• To enhance disease identification accuracy by integrating visual and textual features.
• To leverage a vision-language model for a more comprehensive understanding of crop leaf diseases.

Main Methods:

• Utilized Zhipu.ai multi-model to generate detailed textual descriptions of crop diseases (global, local lesion, color-texture).
• Encoded textual descriptions and image features into vectors.
• Employed a cross-attention mechanism for iterative fusion of multimodal features across layers.
• Implemented a classification prediction module for disease identification.

Main Results:

• The proposed cross-modal fusion model outperformed state-of-the-art image-only methods on Soybean Disease, AI Challenge 2018, and PlantVillage datasets.
• Achieved high recognition accuracies: 98.74% (Soybean Disease), 87.64% (AI Challenge 2018), and 99.08% (PlantVillage).
• Demonstrated superior performance with a significantly lower parameter count (1.14M).

Conclusions:

• Cross-modal learning effectively integrates visual and textual data for precise and efficient crop disease recognition.
• The developed vision-language model offers a scalable and accurate solution for agricultural disease identification.
• This approach enhances the potential for improving crop yield and global food security through advanced AI techniques.