MUTUAL: Towards Holistic Sensing and Inference in the Operating Room

Area of Science:

• Surgical Robotics and Intelligent Systems
• Data Infrastructure for AI Development
• Medical Informatics

Background:

• Embodied AI (E-AI), including surgical robotics, requires specialized data platforms for development and deployment.
• Existing data solutions may not adequately support the complex needs of real-time AI processing in clinical settings.

Purpose of the Study:

• To present and evaluate MUTUAL, a cross-platform multimodal data recording and streaming software.
• To introduce and assess MUTUAL-ROS 2, a distributed adaptation for E-AI systems.
• To compare the performance and suitability of both systems for surgical data management.

Main Methods:

• Deployment of MUTUAL and MUTUAL-ROS 2 on two clinical studies.
• Comparative analysis of recording performance across different settings and modalities (video, audio, expert commentary).
• Evaluation of real-time streaming capabilities and scalability for AI inference.

Main Results:

• MUTUAL demonstrates robust recording performance suitable for data curation and offline analysis.
• MUTUAL-ROS 2 shows advantages in real-time streaming and horizontal scalability for E-AI in the operating room.
• MUTUAL-ROS 2's recording reliability needs further enhancement for distributed, edge-processed modalities.

Conclusions:

• MUTUAL is ideal for comprehensive data collection and post-operative analysis.
• MUTUAL-ROS 2 is better suited for real-time AI inference and scalable E-AI applications in surgery.
• Optimizing MUTUAL-ROS 2 for recording reliability in distributed environments is crucial for advancing surgical E-AI.