A 129 FPS Full HD Real-Time Accelerator for 3D Gaussian Splatting

Area of Science:

• Computer Graphics
• Hardware Acceleration
• Augmented Reality/Virtual Reality (AR/VR)

Background:

• Rendering large-scale, unbounded scenes on AR/VR devices is computationally expensive due to the demands of 3D Gaussian Splatting (3DGS).
• Existing methods face limitations in computation, bandwidth, and storage, hindering real-time performance on resource-constrained devices.

Purpose of the Study:

• To design and implement a low-power, cost-effective hardware accelerator for real-time 3DGS rendering.
• To develop a hardware-friendly compression pipeline to reduce model size and maintain visual fidelity.

Main Methods:

• A novel hardware accelerator integrating point-based culling, projection, tile-based sorting, and rasterization.
• A compression pipeline featuring iterative Gaussian pruning, progressive Spherical Harmonics (SH) degree reduction, and vector quantization.
• Optimization techniques including skipping zero-Jacobian matrix multiplications and employing comparison-free tile-based sorting.

Main Results:

• Achieved a $51.6\times$ model-size reduction with minimal 0.743 dB PSNR loss.
• The accelerator delivers 1219 Mpixels/J energy efficiency and 267.5 Mpixels/s throughput, enabling 1080p at 129 FPS.
• Compared to prior accelerators, the design is $5.98\times$ smaller, offers $5.94\times$ higher throughput, and $7.5\times$ greater energy efficiency.

Conclusions:

• The proposed hardware accelerator and compression pipeline effectively address the limitations of 3DGS for AR/VR applications.
• Real-time, high-resolution rendering of complex scenes is achievable with significantly reduced power and resource requirements.