Related Experiment Video
Updated: Jul 5, 2026
10:25
Technical Approach for Infrared Tracking for Soft Tissue Navigation with a Holographic Head-Mounted Display and Preclinical Validation
Published on: September 2, 2025
Sound source localization for spatial mapping of surgical actions in dynamic scenes.
Jonas Hein1,2, Lazaros Vlachopoulos3, Maurits Geert Laurent Olthof3
1ROCS, Balgrist University Hospital, University of Zurich, Zurich, Switzerland.
Summary
This study introduces a new method for understanding surgical scenes by combining 3D sound localization with visual data. This multimodal approach enhances surgical scene representation for intelligent surgical systems.
Area of Science:
- Computer-aided surgery
- Surgical scene understanding
- Multimodal AI
Background:
- Current surgical scene understanding relies heavily on visual data, limiting contextual modeling.
- End-to-end learning approaches often lack fine-grained spatial and temporal awareness.
Purpose of the Study:
- To enhance surgical scene representations by integrating 3D acoustic information.
- To enable temporally and spatially aware multimodal understanding of surgical environments.
Main Methods:
- Developed a framework for 4D audio-visual representations using phased microphone array and RGB-D camera data.
- Implemented a transformer-based module for acoustic event detection and spatial localization of tool-tissue interactions.
Main Results:
- Successfully localized surgical acoustic events in 3D space and associated them with visual scene elements.
- Demonstrated accurate spatial sound localization and robust fusion of multimodal data.
- Achieved a comprehensive, dynamic representation of surgical activity.
Conclusions:
- Introduced the first approach for spatial sound localization in dynamic surgical scenes.
- Advanced multimodal surgical scene representations by integrating acoustic and visual data.
- Provided a foundation for future intelligent and autonomous surgical systems.
