Three stream fusion network with color aware transformer for image to point cloud registration

Area of Science:

• Computer Vision
• Artificial Intelligence
• Robotics

Background:

• Image-to-point-cloud (I2P) registration faces challenges due to dimensional differences between image and point cloud data.
• Leveraging features from one modality to augment the other is difficult, complicating feature alignment in latent spaces.

Purpose of the Study:

• To propose a novel I2P registration method, TFCT-I2P, that addresses the feature alignment challenges.
• To enhance the integration of image and point cloud features for improved registration accuracy.

Main Methods:

• Introduced a Three-Stream Fusion Network (TFN) to integrate image color and point cloud structural information.
• Developed a Color-Aware Transformer (CAT) to mitigate patch-level misalignments caused by color information integration.

Main Results:

• TFCT-I2P demonstrated superior performance compared to state-of-the-art methods on multiple benchmark datasets (7Scenes, RGB-D Scenes V2, ScanNet V2).
• The proposed method effectively handles the dimensional discrepancies between image and point cloud data.

Conclusions:

• The TFCT-I2P method represents a significant advancement in image-to-point-cloud registration.
• The integration of color and structural information, along with the transformer-based approach, enhances registration accuracy and robustness.