A multimodal digital twin for autonomous micro-drilling in scientific exploration

Area of Science:

• Robotics
• Artificial Intelligence
• Biomedical Engineering

Background:

• Autonomous robotic surgery requires high-fidelity simulation environments for training AI models.
• Creating realistic synthetic data for training is challenging due to the complexity of physical processes like drilling.

Purpose of the Study:

• To develop a multimodal digital twin (DT) for simulating autonomous robotic micro-drilling.
• To generate realistic synthetic images and drilling sounds for training AI models.
• To evaluate the realism and accuracy of the developed DT.

Main Methods:

• Integrated the asynchronous multi-body framework (AMBF) simulator with Isaac Sim for photorealistic rendering.
• Developed a deep audio generator (DAG) model for realistic drilling sound synthesis.
• Utilized a convolutional neural network (CNN) for visual realism assessment and conducted eggshell drilling experiments for accuracy evaluation.

Main Results:

• The DAG model achieved superior realism in sound synthesis compared to pitch modulation methods (lower FAD and FID scores).
• The CNN, trained on synthetic data, achieved a 70.2% mAP for detecting drilling areas in real images.
• The digital twin demonstrated submillimeter accuracy with an alignment error of 0.22 ± 0.03 mm.

Conclusions:

• A multimodal digital twin was successfully developed and validated for simulating cranial window creation via micro-drilling.
• The DT generates highly realistic synthetic audio and visual data.
• The validated DT offers submillimeter accuracy, suitable for training AI models in autonomous robotic surgery.