MAPE-ViT: multimodal scene understanding with novel wavelet-augmented Vision Transformer

Area of Science:

• Computer Vision
• Machine Learning
• Image Processing

Background:

• RGB-D scene classification faces challenges like sensor misalignment and depth noise.
• Preserving object boundaries is crucial for accurate scene understanding.
• Existing methods struggle with sensor artifacts and noise.

Purpose of the Study:

• To introduce a novel approach for RGB-D scene classification.
• To address challenges of sensor misalignment, depth noise, and object boundary preservation.
• To enhance feature discrimination and classification accuracy.

Main Methods:

• Integration of Maximally Stable Extremal Regions (MSER) with wavelet coefficients for patch embedding.
• Utilizing a Vision Transformer (ViT) with attention mechanisms for high-level feature extraction.
• Employing the Gray Wolf algorithm for feature optimization and a dual-stream architecture (Extreme Learning Machine and Conditional Random Fields) for classification.

Main Results:

• Demonstrated significant improvements in classification accuracy compared to existing methods.
• MAPE-ViT effectively handles sensor misalignment, depth noise, and preserves object boundaries.
• The approach shows robustness in challenging RGB-D scene understanding scenarios.

Conclusions:

• MAPE-ViT offers a robust and effective solution for RGB-D scene classification.
• The proposed method outperforms traditional approaches, especially under noisy conditions.
• This framework advances the field of multimodal scene understanding.