Label-aware Gaussian Splatting for Medical Volume Visualization in Mixed Reality

Volume visualization is an important tool in exploring and understanding 3D medical data. Though the recent 3D Gaussian Splatting (3DGS) offers great potential for realtime high-quality medical volume visualization, directly applying 3DGS to visualize medical volume data often overlooks important anatomical structures, especially for vascular structures that are vital in disease detection and surgical planning. This paper presents a new framework called Label-aware Gaussian Splatting (LAGS) to leverage the inherent structural information of medical volumes and optimize the Gaussian representations, especially for fine-grained details of medical volume data. First, we introduce a new label attribute for Gaussian points and propose fine label initialization to enhance subsequent Gaussian training. Then, we devise a Label-aware Rasterizer (LAR) consisting of dual branches of differentiable rasterizers to jointly optimize photorealistic and structural representations. Next, we develop Label-aware Densification (LAD) to adaptively supplement Gaussians in regions with inferior structural representation. The above modules work synergistically during Gaussian training to encourage accurate reconstruction of delicate anatomical structures. We generate a dataset called MedVol-GS including medical scenes from various volumes and evaluate our method on it. Experiments show that LAGS outperforms existing NeRF and GS-based methods in rendering quality, with over 3 dB improvement in PSNR on average. Qualitative results reveal that our method has advantages in reconstructing intricate geometric structures even with occlusion and preventing floaters in nonlabeled areas. In addition, we enable real-time interaction at 72 FPS with labeled structures of reconstructed Gaussians in mixed reality (MR) for intuitive observation. A user study demonstrates that LAGS significantly improves medical volume visualization quality and experience for both clinicians and engineers in MR environments.