TwinOR: photorealistic digital twins of dynamic operating rooms for embodied AI research

Developing embodied AI for intelligent surgical systems requires safe, controllable environments for continual learning and evaluation. However, safety regulations and operational constraints in operating rooms (ORs) limit agents from freely perceiving and interacting in realistic settings. Digital twins provide high-fidelity, risk-free environments for exploration and training. How we may create photorealistic and dynamic digital representations of ORs that capture relevant spatial, visual, and behavioral complexity remains an open challenge.

We introduce TWINOR, a real-to-sim infrastructure for constructing photorealistic and dynamic digital twins of ORs for embodied AI research. The system reconstructs static geometry from pre-scan videos and continuously models human and equipment motion through multi-view perception of OR activities. The static and dynamic components are fused into an immersive 3D environment that supports controllable simulation and facilitates future embodied exploration.

The proposed framework reconstructs complete OR geometry with centimeter-level accuracy while preserving dynamic interaction across surgical workflows, enabling realistic renderings and a virtual playground for embodied perception benchmarks. In our experiments, TWINOR synthesizes stereo and monocular RGB streams as well as depth observations for geometry understanding and visual localization tasks. Models such as FoundationStereo and ORB-SLAM3 evaluated on TWINOR-synthesized data achieve performance within their reported accuracy ranges on real-world indoor datasets, demonstrating that TWINOR provides sensor-level realism sufficient for emulating real-world perception and localization challenges in dynamic OR scenes.

By establishing a perception-grounded real-to-sim pipeline, TWINOR enables the automatic construction of dynamic, photorealistic digital twins of ORs. As a safe and scalable environment for experimentation and benchmarking, TWINOR opens new opportunities for translating embodied intelligence from simulation to real-world clinical environments, and sets the stage for future research on interaction, autonomy, and human-robot collaboration in the OR.