High-Fidelity and Efficient Pluralistic Image Completion With Transformers.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Convolutional Neural Networks (CNNs) excel at texture modeling but struggle with global structures and pluralistic image completion.
• Transformers effectively model long-range dependencies and generate diverse outputs but face computational challenges with high-resolution images.
• Existing auto-regressive sampling for transformers is inefficient for decoding diverse image completion results.

Purpose of the Study:

• To develop an effective and efficient method for pluralistic image completion.
• To combine the strengths of transformers and CNNs for superior image reconstruction.
• To introduce a faster decoding strategy for transformer-based image completion.

Main Methods:

• A hybrid approach using transformers for appearance prior reconstruction and CNNs for texture replenishment.
• Implementation of a novel decoding strategy, temperature annealing probabilistic sampling (TAPS), for efficient inference.
• Design of a texture-aware guided attention module to improve realism and address boundary artifacts.

Main Results:

• The proposed method significantly outperforms state-of-the-art methods in image fidelity and pluralistic completion quality.
• TAPS achieves over 70x inference speedup while preserving high quality and diversity.
• The method demonstrates exceptional generalization on large masks and datasets like ImageNet.
• The texture-aware guided attention module enhances realism and coherence, mitigating boundary artifacts.

Conclusions:

• The hybrid transformer-CNN approach offers a powerful solution for pluralistic image completion.
• TAPS provides a substantial efficiency improvement for transformer-based image generation.
• The developed method achieves superior performance, diversity, and generalization capabilities in image completion tasks.