Decomposed Structural Pattern Banks for Efficient Screen Content Image Super-Resolution

With the development of electronic devices and contemporary coworking offices, remote offices such as online conferences and cloud desktops have become increasingly widespread. In remote offices, super-resolution is vital to break the limitation of restricted bandwidth for exhibiting screen content images (SCIs) in high resolutions. However, traditional screen content image super-resolution (SCISR) methods are committed to continuous super-resolution tasks by concentrating on the particular distribution of SCIs, but with unacceptable network memory and runtime. In addition, conventional efficient single-image super-resolution (ESISR) algorithms are only designed for natural images, and fail to generalize in constructing readable text regions in SCIs. In this paper, we introduce an efficient screen content image super-resolution framework named DPBSR that swiftly restores SCIs by two decomposed structure priors of natural and text regions. First, we encode two sparse pattern banks by self-reconstructing high-resolution regions of natural and text elements, so that these SCI elements are accurately recovered with different recovery branches. Second, we efficiently decompose the weights for pre-trained pattern priors guided by the text global structure and process a dual-branch refinement using encapsulated banks, where the priors are injected with negligible computational overhead. Moreover, the existing SCI datasets suffer from the issues of inadequate data volume, limited application scenarios, and insufficient attention to text region information. Therefore, we propose a large-scale SCI benchmark with masks (SCIM), which includes region classification masks and text global structure masks. To match the difficulty in the cloud-desktop scenarios, our SCIM dataset is the first dataset with both single-window and multi-window setups. Experimental results on four datasets demonstrate our DPBSR method achieves an efficiency-accuracy trade-off compared with existing SCISR and ESISR models. Extensive comparisons on various train sets reveal the reliability and challenge of SCIM dataset.