Photorealistic Learned Landscapes for Augmented Reality

The ability to create photorealistic 3D reconstructions of real-world environments from standard color images has numerous applications in fields such as healthcare, education, and industry. This paper presents a novel system that integrates advanced techniques such as structure from motion, incremental scene registration, and Gaussian Splatting to reconstruct detailed 3D models. These models are seamlessly integrated into virtual reality environments to enable immersive interaction. The pipeline begins with the capture of 360-degree images using a matrix-based pattern for comprehensive coverage. The images are processed using COLMAP to estimate camera poses and generate a sparse point cloud. Gaussian Splatting refines the reconstruction by optimizing the positions, shapes, and appearance of the points to produce highly realistic 3D environments. These models are then rendered in Unity, enabling interaction by VR headsets and supporting additional features such as avatars driven by large language models. One use case demonstrates the versatility of the system. In healthcare, the approach creates controlled, familiar virtual spaces for therapy, particularly beneficial for people with autism spectrum disorders. The immersive environments and interactive avatars provide a safe and comfortable environment for personalized therapy. This system offers significant advantages, including high visual fidelity, ease of image acquisition, and immersive user experiences. However, challenges remain, such as the computational cost of Gaussian Splatting and its reliance on accurate camera pose estimation. By overcoming these limitations, this method has the potential to transform applications in various domains, ranging from therapeutic interventions to industrial design and cultural preservation.