Ultra-Low Bitrate Face Video Compression Based on Conversions from 3D Keypoints to 2D Motion Map.

The rapid expansion of digital connectivity has made the efficient transmission of facial imagery a fundamental challenge for contemporary digital communication platforms like remote education and live broadcasting. Prior research has shown that face-centric sequences contain significant structural patterns that allow for compact representation through deep learning architectures. Traditional codecs often struggle to maintain visual fidelity when operating under extreme bandwidth constraints because they rely on block-based motion estimation. Generative frameworks have emerged as a solution by synthesizing frames rather than transmitting raw pixel data, leveraging the inherent symmetry of human faces. However, current generative approaches frequently encounter a discrepancy between physical 3D facial dynamics and their 2D visual projections, leading to unnatural warping. This absence of evidence motivated the development of a framework that aligns spatial motion descriptions with temporal reconstruction requirements to ensure visual consistency.

This research introduces a novel architecture termed 3D-Keypoint-and-2D-Motion (FVC-3K2M) to resolve inconsistencies in generative video synthesis. The investigators sought to decouple facial dynamics into distinct global and local perspectives using three-dimensional coordinates to improve coding flexibility. Establishing a robust link between 3D motion descriptors and 2D dense flow maps serves as a primary objective for the engineering team. The system aims to facilitate high-quality reconstruction even when the available transmission rate drops to approximately 2 kilobits per second (kbps), which is essential for low-bandwidth environments. Enhancing the perceptual realism of reconstructed frames through a cascade conversion process represents a core goal of the proposed methodology. The project addresses the need for adaptive reference frame selection to accommodate diverse temporal movements in natural speech and varied head poses. By bridging the gap between 3D physical motion and 2D image generation, the study seeks to redefine the limits of ultra-low bitrate communication.

The FVC-3K2M framework utilizes a dual-perspective approach to characterize facial evolution through separate 3D keypoints extracted from the input stream. These spatial markers capture both broad head orientations and fine-grained muscular shifts across the facial surface, providing a comprehensive description of movement. A cascade motion conversion mechanism translates these 3D coordinates into 2D dense motion fields to guide the generative synthesis process within the decoder. The researchers implemented an adaptive reference frame selection scheme to optimize the selection of temporal anchors based on movement intensity and structural change. This selection logic ensures that the reconstruction engine maintains access to the most relevant visual context during the decoding phase, minimizing error accumulation. Performance was benchmarked against established video coding standards like High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC) using multiple objective quality metrics. The experimental setup involved testing the algorithm on diverse datasets to verify its robustness across different subjects and environmental conditions.

The proposed FVC-3K2M system achieved reliable video communication at an extremely restricted bitrate of 2 kbps, representing a significant milestone in compression technology. Experimental evaluations indicated that the cascade conversion process significantly improved the perceptual consistency of the synthesized output compared to baseline models. The model demonstrated superior compression efficiency compared to state-of-the-art video coding standards like High Efficiency Video Coding (HEVC) and the latest generative methods. Quantitative comparisons showed that the 3D-to-2D mapping strategy outperformed existing generative methods across various facial datasets in terms of Peak Signal-to-Noise Ratio (PSNR). The adaptive reference frame selection logic effectively reduced temporal artifacts during rapid head movements or complex expressions by selecting optimal anchor points. Visual assessments confirmed that the reconstructed frames retained high-frequency details and structural integrity despite the low data throughput. These results validate the effectiveness of using 3D keypoints as a compact yet descriptive representation for face-centric video data.

Integrating 3D keypoints with 2D motion maps provides a scalable solution for ultra-low bandwidth facial communication in real-world applications. The findings suggest that separating global and local motion components enhances the flexibility of generative video codecs, allowing for more precise control over synthesis. This architecture offers a viable path for maintaining high-quality video chat and conferencing in regions with limited network infrastructure or high congestion. Future iterations of this technology could further refine the conversion mechanisms to support even more diverse lighting conditions and occlusion scenarios. The success of the FVC-3K2M model highlights the potential of hybrid 3D-2D representations in the field of neural video compression for specialized content. These advancements pave the way for more efficient remote education tools and live broadcasting services globally, ensuring accessibility for all users. Ultimately, the study demonstrates that generative models can overcome the limitations of traditional pixel-based compression through intelligent motion modeling.