Mosaic Pattern Excavation Transformer for Spectral Imaging.

Area of Science:

• Spectral imaging
• Computer vision
• Deep learning

Background:

• Single spectral image demosaicing is crucial for multispectral filter array (MSFA) imaging.
• Current deep learning methods struggle with intertwined spatio-spectral correlations from spatial sub-sampling and spectral aliasing.

Purpose of the Study:

• To propose a novel deep learning model, MPEFormer, for improved MSFA demosaicing.
• To effectively model and leverage spatio-spectral correlations for enhanced spectral image reconstruction.

Main Methods:

• Developed a three-branch MPEFormer architecture integrating low-frequency, edge, and high-frequency details.
• Introduced the Dual Fusion Self-attention Block (DFSAB) with Mosaic Pattern Excavation Self-attention (MPESA) to capture non-local correlations.
• Incorporated the Mosaic Pattern-guided Spectral Modulation Module (MPSM) for adaptive spectral recalibration.

Main Results:

• MPEFormer effectively models intertwined spatio-spectral correlations inherent in MSFA data.
• The MPESA mechanism captures non-local spatio-spectral dependencies across the entire image.
• MPSM adaptively recalibrates spectral information based on MSFA pattern dependencies.

Conclusions:

• MPEFormer demonstrates superior performance compared to state-of-the-art MSFA demosaicing methods.
• The proposed architecture effectively addresses limitations of existing deep learning approaches.
• The method shows significant potential for advancing spectral imaging reconstruction.