MMST: A Multi-Modal Ground-Based Cloud Image Classification Method.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Meteorology

Background:

• Convolutional Neural Networks (CNNs) dominate ground-based cloud image classification but have limitations.
• CNNs exhibit inductive bias, lack global modeling capabilities, and show performance saturation with increasing data.
• Limitations hinder optimal performance in complex cloud image recognition tasks.

Purpose of the Study:

• To introduce a novel Multi-modal Swin Transformer (MMST) for enhanced ground-based cloud image recognition.
• To overcome the limitations of CNNs by employing a Transformer architecture for global feature extraction.
• To improve classification accuracy and efficiency in meteorological image analysis.

Main Methods:

• Utilized the Swin Transformer as the visual backbone for attention-based feature extraction.
• Implemented a multi-modal information fusion network with linear layers and residual structures.
• Leveraged pre-trained weights from ImageNet for efficient transfer learning.

Main Results:

• Achieved a classification accuracy rate of 91.30% on the multi-modal ground-based cloud dataset (MGCD).
• Demonstrated superior performance compared to existing state-of-the-art methods.
• Validated the effectiveness of Transformer-based models in ground-based cloud image classification.

Conclusions:

• The proposed MMST method offers a significant advancement in ground-based cloud image classification.
• Transformer architectures provide a powerful alternative to CNNs, enabling better global modeling and higher accuracy.
• MMST shows great potential for improving meteorological observations and climate research through accurate cloud analysis.