Autonomous RCM-less endoscope control: integrating force-based pivoting with deep learning visual servoing

Robotic endoscope holders improve visual stability, yet enforcing a fixed remote center of motion (RCM) often induces lateral interaction forces due to fulcrum drift. This study proposes an autonomous guidance framework combining deep learning-based visual servoing with force-based pivoting to track surgical instruments while minimizing interaction forces, without a fixed geometric RCM. To test this, a UR3e manipulator with an integrated force/torque sensor controlled a laparoscope in a pelvitrainer. Instruments were localized using a fine-tuned YOLOv11n on 3,695 annotated frames. The detected tools' centroids drove a hysteresis-based visual servoing law with jerk-limited planning, while an admittance-based controller generated angular motion from measured forces. Performance was assessed in continuous tracking and step-response tasks, comparing fixed-RCM with the proposed pivoting approach. The localization model achieved 0.91-0.93 precision, 0.83-0.90 mAP, and 28 FPS on a CPU. Visual servoing re-centered targets in 98.8% of transitions (median recovery: 2.60 s). Force-based pivoting significantly outperformed fixed-RCM control, reducing median interaction forces from 2.35 to 0.19 N. Integrating CNN-driven servoing with force-based pivoting enables autonomous guidance that preserves visual stability while substantially reducing interaction forces. By eliminating fixed RCM assumptions, this framework offers a safer, more adaptive alternative for robotic camera assistance.