ConvShareViT: A Vision Transformer-Like Architecture for Free-Space Optical Accelerators

This article introduces convolutional shared vision transformers (ConvShareViT), a novel deep learning architecture that adapts the vision transformer (ViT) architecture to the 4f free-space optical system. ConvShareViT replaces linear layers in multi-head self-attention (MHSA) and multilayer perceptrons (MLPs) with a depthwise convolutional layer with shared weights across input channels. The effectiveness of the attention mechanism was analyzed systematically in 12 experiments with different Models. Experimental results demonstrated that configurations with valid-padded shared convolutions successfully learned attention, achieving comparable quantitative attention scores to those obtained with standard ViTs. However, same-padded convolutions showed limitations in attention learning and operated like regular convolutional neural networks (CNNs) rather than transformer models. In terms of speed, ConvShareViT can theoretically achieve up to 3.04 times faster inference than GPU-based systems. This potential acceleration makes ConvShareViT an attractive candidate for future optical deep learning applications.