MTC-CSNet: Marrying Transformer and Convolution for Image Compressed Sensing.

Area of Science:

• Computer Vision
• Signal Processing
• Machine Learning

Background:

• Image compressed sensing (ICS) enables image sampling and reconstruction below Nyquist rates.
• Convolutional Neural Networks (ConvNets) excel at local feature extraction in ICS.
• Transformer architectures demonstrate strength in modeling global feature correlations.

Purpose of the Study:

• To propose a novel hybrid network, MTC-CSNet, for enhanced image compressed sensing.
• To leverage the complementary strengths of ConvNets and Transformers for high-quality image recovery.
• To improve the performance of ICS methods by integrating local and global feature modeling.

Main Methods:

• Developed a dual-path framework, MTC-CSNet, incorporating separate ConvNets and Transformer recovery branches.
• Designed a lightweight ConvNets branch for efficient local feature capture.
• Implemented a Transformer branch for iterative modeling of global image patch dependencies.
• Utilized a bridging unit for adaptive feature fusion between the two branches.

Main Results:

• MTC-CSNet demonstrated superior performance compared to state-of-the-art ICS methods.
• The hybrid approach effectively captured both local and global image features.
• High-quality image reconstruction was achieved across various public datasets.

Conclusions:

• The proposed MTC-CSNet hybrid network offers a significant advancement in image compressed sensing.
• Combining ConvNets and Transformers provides a powerful strategy for improving image recovery.
• The method's effectiveness is validated by extensive experimental results and public availability of code and models.