LIPT: Latency-Aware Image Processing Transformer.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Deep Learning

Background:

• Transformer models are increasingly influential in image processing.
• Existing lightweight transformers prioritize reducing computations (FLOPs) or parameters, not practical inference speed.
• There is a need for image processing models optimized for real-world inference latency.

Purpose of the Study:

• To present a novel latency-aware image processing transformer, LIPT.
• To achieve practical inference acceleration in image processing tasks.
• To improve both speed and accuracy (PSNR) compared to existing methods.

Main Methods:

• Introduced the low-latency LIPT block, combining self-attention and convolutions to replace memory-intensive operations.
• Developed non-volatile sparse masking self-attention (NVSM-SA) for efficient contextual information capture over larger windows.
• Proposed a high-frequency reparameterization module (HRM) to improve detail reconstruction and model reparameterization.

Main Results:

• LIPT demonstrates significant improvements in both latency and Peak Signal-to-Noise Ratio (PSNR) across various image processing tasks.
• Achieved real-time GPU inference speeds.
• Outperformed existing methods on multiple image super-resolution benchmarks.

Conclusions:

• LIPT offers a practical solution for accelerating image processing tasks using transformers.
• The proposed LIPT block, NVSM-SA, and HRM contribute to efficient and effective image enhancement.
• LIPT sets a new standard for real-time, high-performance image processing with transformer architectures.