ThermalGaussian++: Improving Alignment and Resolution for ThermalGaussian

Area of Science:

• Computer Vision
• Machine Learning
• Remote Sensing

Background:

• 3D scene reconstruction from thermal and RGB images is crucial for surveillance.
• Neural Radiance Fields (NeRF) methods exist but are slow; 3D Gaussian Splatting (3DGS) offers faster training and real-time rendering.

Purpose of the Study:

• Introduce ThermalGaussian, the first 3DGS approach for high-quality RGB and thermal scene rendering.
• Develop ThermalGaussian++ to enhance multimodal alignment and thermal image resolution.
• Provide new datasets (RGBT-Scenes and RGBT-Scenes++) to advance thermal scene reconstruction research.

Main Methods:

• Calibrate RGB and thermal cameras for accurate alignment.
• Learn multimodal 3D Gaussians with regularization constraints to prevent modality overfitting.
• Implement multimodal pose optimization for direct processing of non-aligned image pairs.
• Utilize a multimodal joint super-resolution reconstruction module for low-resolution thermal fields.

Main Results:

• ThermalGaussian++ achieves photorealistic thermal rendering and improved RGB rendering quality.
• Significant enhancements in multimodal alignment and thermal resolution were demonstrated.
• Multimodal regularization constraints effectively reduce model storage requirements.

Conclusions:

• ThermalGaussian++ represents a significant advancement in 3D thermal scene reconstruction using 3DGS.
• The method offers practical benefits for surveillance and other applications requiring high-quality thermal imaging.
• The released datasets and code will foster further research in this domain.