Self-Supervised Fragment Alignment With Gaps.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Digital Image Processing

Background:

• Traditional image alignment methods require overlapping regions or common boundaries.
• Real-world scenarios, like artifact restoration, often present image fragments with significant gaps and no overlap.
• Unsupervised alignment of such fragmented data is a challenging, yet crucial, problem.

Purpose of the Study:

• To develop an unsupervised, self-supervised method for learning the alignment of image fragments without common boundaries or overlapping regions.
• To address the specific challenge of aligning fragmented archaeological artifacts such as frescoes and mosaics.
• To leverage inner image statistics for accurate alignment in the absence of external guidance.

Main Methods:

• A self-supervised approach generating 'self-examples' from existing image fragments.
• Utilizing an adversarial neural network, specifically a spatial transformer Generative Adversarial Network (GAN).
• Sub-fragmentation of initial fragments to expose new alignment relations and inner feature statistics, feeding these into the GAN.

Main Results:

• Demonstrated capability to learn alignment for image fragments with gaps in a self-supervised manner.
• Successful application on both synthetic datasets and large-scale real-world data (frescoes and mosaics).
• Validation of the hypothesis that inner image statistics are sufficient for accurate alignment.

Conclusions:

• The proposed self-supervised method effectively aligns deteriorated image fragments without requiring overlap or common boundaries.
• The technique offers a viable solution for the unsupervised restoration of fragmented cultural heritage.
• Inner image statistics provide a powerful signal for learning spatial transformations in a self-supervised context.