Spatio-Temporal Convolutional Neural Network for Enhanced Inter Prediction in Video Coding.

Area of Science:

• Video Compression
• Artificial Intelligence
• Signal Processing

Background:

• Inter prediction is crucial for video compression efficiency in standards like Versatile Video Coding (VVC).
• Conventional methods may lack the signal adaptivity needed for optimal prediction.
• Neural networks offer potential for adaptive signal processing in video coding.

Purpose of the Study:

• To enhance inter prediction in VVC using a novel convolutional neural network (CNN) approach.
• To improve the prediction signal adaptivity by incorporating spatial and temporal reference samples.
• To achieve a better trade-off between computational complexity and coding performance.

Main Methods:

• A residual CNN incorporating spatial and temporal reference samples was developed for inter prediction.
• A polyphase decomposition stage was added to the CNN to optimize complexity and performance.
• A novel signal plane with constrained reference samples was introduced to enable parallel decoding.

Main Results:

• The proposed CNN-based enhancement significantly improves the prediction signal adaptivity.
• Adding a polyphase decomposition stage yielded a better complexity-performance trade-off.
• Experimental results demonstrated average bit rate savings of 4.07% (RA) and 3.47% (LB).

Conclusions:

• The CNN-based inter prediction offers a significant improvement over conventional methods in VVC.
• The novel signal plane effectively enables parallel decoding without compromising compression efficiency.
• This approach provides a promising direction for future video coding standards.