Generalized image outpainting with U-transformer

Area of Science:

• Computer Vision
• Artificial Intelligence
• Deep Learning

Background:

• Existing image outpainting methods primarily focus on horizontal extrapolation.
• Generalized image outpainting requires filling visual context on all sides of an image.
• Complex scenes, buildings, and art images present significant challenges for current outpainting techniques.

Purpose of the Study:

• To introduce U-Transformer, a novel transformer-based generative adversarial neural network for generalized image outpainting.
• To enable extrapolation of visual context on all sides of an image with plausible structure and details.
• To address the limitations of existing methods in handling complex image content.

Main Methods:

• Development of a novel transformer-based generative adversarial neural network named U-Transformer.
• Generator architecture utilizes an encoder-to-decoder structure with Swin Transformer blocks to handle long-range dependencies.
• Inclusion of a U-shaped structure and a multi-view Temporal Spatial Predictor (TSP) module for enhanced reconstruction and prediction.
• The TSP module allows for generating arbitrary outpainting sizes by adjusting the prediction step.

Main Results:

• The proposed U-Transformer method demonstrates superior performance in generalized image outpainting.
• Results show plausible structure and details even for complicated scenery, building, and art images.
• Experimental comparisons confirm visually appealing outcomes against state-of-the-art approaches.

Conclusions:

• U-Transformer effectively addresses the generalized image outpainting problem by extrapolating visual context on all sides.
• The integration of Swin Transformer blocks and the TSP module enhances the network's ability to reconstruct and predict unknown image regions realistically.
• The method offers a flexible solution for generating high-quality outpainted images of arbitrary sizes.