SurgTPGS++: Text-promptable Gaussian Splatting with dense features aggregation for semantic 4D surgical scene...

With the growing demand for surgical embodied intelligence, accurate semantic understanding of 3D surgical scenes with language-based interaction has become increasingly important. Such capability can assist surgeons in identifying and interacting with surgical instruments and anatomical structures during pre-operative planning and real-time intra-operative guidance. However, existing methods typically address surgical vision-language modeling, 3D reconstruction, and semantic segmentation as separate tasks, leaving real-time language-assisted 3D querying in dynamic surgical scenes largely unexplored. In this paper, we present SurgTPGS++, a novel Gaussian Splatting pipeline for text-promptable 3D surgical scene understanding. We introduce a semantic feature aggregation (SFA) strategy that extracts and integrates rich vision-language features from VLMs. The aggregated semantic features are embedded into a semantic-aware 3D Gaussian representation, enabling joint reconstruction of appearance, geometry, and semantic information. To handle complex dynamic surgical environments, we further propose semantic-aware deformation tracking (SADT), which models the temporal deformation of both texture and semantic features for more accurate 4D reconstruction. In addition, we introduce a codebook-based query (CQ) module that supports real-time language-assisted 3D segmentation by matching rendered semantic features with text-codebook embeddings. Comprehensive experiments on real-world surgical datasets demonstrate that SurgTPGS++ outperforms state-of-the-art methods in text-promptable 3D surgical scene understanding, highlighting its potential for advancing surgical embodied intelligence. Our code is available at: https://github.com/lastbasket/SurgTPGS.