Reconstructing Hyperspectral Images from RGB Images by Multi-Scale Spectral-Spatial Sequence Learning.

Area of Science:

• Computer Vision
• Image Processing
• Deep Learning

Background:

• Transformer models have advanced hyperspectral image reconstruction (spectral super-resolution, SSR).
• Existing transformer methods face challenges in balancing computational efficiency and long-range feature extraction.
• Mamba offers linear complexity for long-range dependencies and shows promise in vision tasks.

Purpose of the Study:

• To propose MSS-Mamba, a novel multi-scale spectral-spatial sequence learning method for hyperspectral image reconstruction from RGB images.
• To enhance Mamba's capabilities for SSR by improving feature extraction and multi-scale information processing.

Main Methods:

• Introduced a continuous spectral-spatial scan (CS3) mechanism to enhance Mamba's cross-dimensional feature extraction.
• Developed a sequence tokenization strategy with a multi-scale information fusion (MIF) module to address hierarchical multi-scale learning limitations.
• The MIF module uses a dual-branch architecture for separate global and local processing, with dynamic feature fusion via an adaptive router.

Main Results:

• The proposed MSS-Mamba method effectively reconstructs hyperspectral images from RGB inputs.
• Experimental results on ARAD_1k, CAVE, and grss_dfc_2018 datasets demonstrate the method's superior performance.
• The approach successfully generates feature maps with both global context and local details for high-fidelity reconstruction.

Conclusions:

• MSS-Mamba offers an efficient and effective solution for spectral super-resolution using Mamba.
• The novel CS3 mechanism and MIF module significantly improve hyperspectral image reconstruction quality.
• This work advances the field of hyperspectral imaging by leveraging efficient sequence modeling for complex reconstruction tasks.