Mixed reality based teleoperation and visualization of surgical robotics

Area of Science:

• Robotics
• Surgical Technology
• Human-Computer Interaction

Background:

• Current surgical robot teleoperation relies on consoles, limiting surgeon mobility and awareness.
• This limitation restricts surgical scenarios and surgeon's connection to the patient during procedures.

Purpose of the Study:

• To develop and evaluate a mixed reality (MR)-based teleoperation system for surgical robots.
• To overcome the mobility and awareness limitations of traditional teleoperation systems.

Main Methods:

• Implemented a MR teleoperation system using Microsoft HoloLens 2 and the da Vinci Research Kit (dVRK).
• Utilized hand gestures, head motion tracking, and speech commands for robot control.
• Evaluated system performance through camera navigation and peg transfer tasks.

Main Results:

• The MR system showed comparable viability to manipulator-based teleoperation for endoscope control.
• Instrument teleoperation performance was inferior to traditional methods, indicating areas for improvement.

Conclusions:

• MR-based teleoperation offers a potential alternative to console-based systems, enhancing surgeon interaction.
• Further advancements in hand gesture recognition and video display quality are necessary for optimal instrument control.