Memory-efficient divide-and-conquer attention for lightweight image super-resolution

Recently, transformer-based methods have achieved significant progress in lightweight image super-resolution (SR). However, most of these approaches primarily aim to improve either inference speed or reconstruction quality, while overlooking memory consumption, thereby limiting their practicality on resource-constrained devices. In this paper, we propose a memory-efficient divide-and-conquer attention (MEDCA) for SR, which substantially reduces memory usage while achieving notable improvements in reconstruction performance and competitive inference speed. To address the high memory and space complexity of standard window-based self-attention (WSA), MEDCA adopts a divide-and-conquer strategy. Specifically, the input features are first split into multiple subspaces along the channel dimension. Each subspace is further partitioned into multiple windows, which are then evenly divided into two parts using distinct asymmetric strategies. Self-attention is independently applied to each part, and the outputs are aggregated to form the final representation. Compared with traditional WSA methods such as SwinIR, MEDCA reduces the space complexity within each subspace by half. Furthermore, we design multiple asymmetric partitioning strategies that allow the model to extract features from a broader spatial context, thereby enabling it to capture richer spatial information and enhance its representation capacity. Extensive experiments demonstrate that MEDCA significantly reduces memory consumption while outperforming existing lightweight state-of-the-art methods and maintaining competitive inference speed across multiple public benchmark datasets. In particular, compared with the state-of-the-art method HiT-SRF, MEDCA improves the average performance by 0.12dB across five public test sets, while maintaining comparable inference time and requiring only 29.7% of the memory used by HiT-SRF. The code and models are provided at https://github.com/hms-source/MEDCA.