Design of a mixed reality-based digital twin human-machine interaction system for special working conditions

Operational robots have demonstrated significant potential in complex scenarios such as live-line maintenance and medical surgery. Existing research on Mixed Reality (MR) and Digital Twin (DT) systems has primarily focused on unidirectional data visualization and passive state monitoring. Existing research on Mixed Reality (MR) and Digital Twin (DT) systems has primarily focused on unidirectional data visualization and passive state monitoring, acting as "open-loop" observation tools that fail to address low operational precision and inefficient human-robot synergy in dynamic, high-risk environments. For the first time, we integrate an MR-based closed-loop digital twin operating system for human-robot collaborative operation into the task execution of live-line operation equipment to address the above challenges. Moving beyond simple visualization, the proposed framework establishes an integrated operational paradigm that bridges the gap between immersive perception and real-time interventional control. This framework comprises three integral components: (1) the construction of a high-fidelity virtual digital twin; (2) the development of a human-computer interaction paradigm based on MR technology; and (3) the establishment of an MR-based human-machine collaborative operation mode. Building upon this framework, a system was implemented for live-line working robots. Experimental results indicate that, compared with traditional control methods, the proposed system reduces the task completion time of live-line equipment tasks by 14.3% on average, verifying the feasibility and effectiveness of the pioneering application of the closed-loop digital twin operating system in live-line operation equipment.