In-Loop Filtering Using Learned Look-Up Tables for Video Coding

Neural network-based in-loop filtering (ILF) technologies have achieved remarkable coding gains, emerging as a promising candidate for next-generation video coding standards. However, using deep neural networks (DNNs) brings significant computational and time complexity or high demands for dedicated hardware, making it challenging for general use. To address this limitation, we study a practical ILF solution by adopting look-up tables (LUTs). After training a DNN for ILF with a restricted reference range, all possible inputs are traversed and the output values of the DNN are stored into LUTs. During the coding process, the filtering process is performed by retrieving the filtered pixel through locating the input pixels and interpolating between the stored values, instead of relying on heavy DNN inference. We propose a universal LUT-based ILF framework, termed LUT-ILF++. First, we introduce the cooperation of multiple kinds of filtering LUTs and propose a series of customized indexing mechanisms to enable better filtering reference perception with limited storage consumption. Second, we propose the cross-component indexing mechanism to enable the filtering of different color components jointly. Third, in order to make our solution practical for coding uses, we propose the LUT compaction scheme to enable LUT pruning, achieving a lower storage cost of the entire solution. The proposed framework is implemented in the Neural Network-based Video Coding reference software. Experimental results show that the proposed framework achieves on average 0.82%/2.97%/1.63% and 0.85%/4.11%/2.06% bitrate reduction for common test sequences, under the all-intra and random-access configurations, respectively. Compared to DNN inference-based solutions, our proposed solution has much lower time complexity and modest storage cost. Code is publicly available at https://github.com/milkbrotherli/LUT-family.