Mobile-friendly under-sampling single-pixel imaging based on a lightweight hybrid CNN-ViT architecture.

Area of Science:

• Computational imaging
• Deep learning for optical systems
• Single-pixel imaging (SPI)

Background:

• Deep learning models, including convolutional neural networks (CNNs) and vision transformers (ViTs), have advanced single-pixel imaging (SPI).
• Existing ViT-based models are computationally intensive and parameter-heavy, limiting their use in mobile SPI applications.
• Current deep learning SPI (DLSPI) methods face challenges in efficiency and hardware compatibility.

Purpose of the Study:

• To develop a lightweight and efficient deep learning model for accurate SPI reconstruction.
• To enable mobile SPI applications by reducing computational and memory requirements.
• To introduce a novel, hardware-friendly modulation pattern scheme for DLSPI.

Main Methods:

• Proposed mobile ViT blocks to decrease the computational cost of traditional ViTs.
• Designed a novel lightweight CNN-ViT hybrid model for efficient SPI reconstruction.
• Introduced a general-purpose differential ternary modulation pattern scheme for DLSPI.

Main Results:

• The proposed lightweight CNN-ViT hybrid model achieved higher imaging quality compared to state-of-the-art DLSPI methods.
• The method demonstrated significantly lower memory consumption and computational burden.
• Simulations and real-world experiments validated the effectiveness of the proposed approach.

Conclusions:

• The novel lightweight CNN-ViT hybrid model offers an efficient and accurate solution for SPI reconstruction.
• The proposed differential ternary modulation scheme is training-friendly and hardware-friendly.
• This work overcomes limitations of existing models, paving the way for practical mobile SPI applications.