Virtual interaction and visualisation of 3D medical imaging data with VTK and Unity

Area of Science:

• Computer Graphics
• Medical Imaging
• Virtual Reality

Background:

• Virtual reality (VR), augmented reality (AR), and mixed reality (MR) offer immersive platforms for medical applications.
• Integrating powerful visualization libraries with versatile VR/AR development platforms is crucial for advancing medical imaging applications.

Purpose of the Study:

• To present a method for interconnecting the Visualization Toolkit (VTK) and Unity.
• To leverage VTK's visualization capabilities within Unity's immersive display and interaction framework for medical image applications.

Main Methods:

• Utilizes OpenGL context sharing between Unity and VTK.
• Employs a Unity native plugin to render VTK objects directly into the Unity scene.
• Demonstrates the method with VTK volume rendering of thoracic computed tomography and cardiac magnetic resonance images.

Main Results:

• Achieves over 90 frames per second (fps) on standard hardware.
• The integration is suitable for current augmented reality/virtual reality display devices.
• Successfully renders complex medical imaging data within a virtual environment.

Conclusions:

• The proposed method effectively integrates VTK and Unity for developing medical image applications in virtual environments.
• This approach facilitates the creation of high-performance, immersive medical visualization tools.
• The high frame rates achieved support real-time interaction with medical data in VR/AR settings.