Stitching Misaligned Multi-Spectral Images by Versatile Transformation: A Streamlined Solution and Applications

Multi-spectral image stitching reconstructs the informative and broader scene by aligning the image pairs captured from different viewpoints. This task is significantly limited by the registered sources. Stitching accuracy deteriorates notably with misaligned multi-modality images. To address the challenges imposed by the registration requirement, we propose robust multi-spectral image stitching for misaligned infrared-visible pairs, in which the intrinsic attributes of multi-modality registration and multi-view alignment are employed concurrently, rather than independent processing. Specifically, the proposed method develops a hierarchical versatile transformation to conduct the correspondence matching progressively, where the sparse foundational homography is utilized for global adjustment and point-flexible splines are applied for local fine-tuning. In the registration of infrared and visible images, a dual consistency-driven modality transfer is introduced to alleviate feature variance and facilitate registration within the shared information matching. During cross-view alignment, multi-spectral image pairs are encoded into a latent domain to exploit the complementary information adaptively, and the planar deformation is conducted through correspondence matching processes identical to those in the registration procedure. In this way, by leveraging the consistency between multi-modality and multi-view matching, we achieve a robust reconstruction of the multi-spectral panorama. Extensive experiments in registration and stitching demonstrate the superiority of our method. Code is available at https://github.com/ZengxiZhang/VTMS.