Interaction with Industrial Digital Twin Using Neuro-Symbolic Reasoning

Area of Science:

• Artificial Intelligence
• Manufacturing Technology
• Human-Computer Interaction

Background:

• Digital twins offer advanced simulations for manufacturing and maintenance.
• Current digital twins lack natural language communication capabilities, limiting user interaction.
• Conventional natural language processing methods have limitations in complex task execution.

Purpose of the Study:

• To introduce neuro-symbolic reasoning (NSR) for natural language interaction with 3D digital twins.
• To enable autonomous execution of maintenance procedures and information retrieval from manuals.
• To enhance the versatility and robustness of digital twin interactions.

Main Methods:

• Developed a neuro-symbolic reasoning (NSR) approach for natural language interaction with 3D digital twins.
• Created a neuro-symbolic dataset comprising machine-understandable manuals, 3D models, and user queries.
• Trained the NSR interaction mechanism on the collected dataset.

Main Results:

• NSR accurately interprets user requests and context to manipulate 3D digital twin components.
• The system autonomously performs installations and removal procedures based on manual instructions.
• Achieved 96.2% accuracy in executing user commands on test data.

Conclusions:

• NSR provides a robust method for natural language interaction with 3D digital twins.
• The technology facilitates autonomous maintenance procedures and information access from manuals.
• NSR significantly enhances interaction with complex virtual machinery in industrial settings.