An End-to-End Depth-Based Pipeline for Selfie Image Rectification.

Area of Science:

• Computer Vision
• Deep Learning
• Image Processing

Background:

• Close-up portraits and selfies often exhibit perspective distortion.
• Existing methods for distortion correction can be complex and time-consuming.

Purpose of the Study:

• To develop an end-to-end deep learning pipeline for mitigating perspective distortion in facial images.
• To improve the quality and efficiency of image rectification for portraits.

Main Methods:

• A deep convolutional neural network (CNN) predicts facial depth for perspective adjustment.
• A differentiable renderer facilitates end-to-end training of depth estimation and feature extraction.
• An inpainting module reconstructs missing pixels, aided by a camera movement prediction module.

Main Results:

• The proposed pipeline effectively rectifies perspective distortion in full-frame images.
• It outperforms previous methods in both quantitative and qualitative evaluations.
• Achieves comparable results to 3D GAN-based methods but is over 260 times faster.

Conclusions:

• The deep learning pipeline offers a fast and effective solution for perspective distortion correction in portraits.
• Processing full-frame images simplifies the rectification process, eliminating complex post-processing.
• The use of synthetic data generated with Unreal Engine demonstrates the robustness of the approach.