Novel Multimodal, Multiscale Imaging System with Augmented Reality

Area of Science:

• Biomedical Engineering
• Medical Imaging
• Optical Imaging

Background:

• Current medical imaging struggles with multimodal integration and real-time visualization.
• Accurate registration between different imaging modalities is crucial for clinical diagnosis.

Purpose of the Study:

• To develop and characterize a novel multimodal, multiscale imaging system with augmented reality.
• To enable real-time 3D color reflectance, 3D fluorescence, and augmented reality visualization.
• To integrate multiscale fluorescence imaging and tomographical data with ultrasound and MRI.

Main Methods:

• Developed a system integrating 3D color reflectance, 3D fluorescence imaging, and augmented reality.
• Incorporated an in vivo fiber-optic microscope for multiscale fluorescence imaging (~17.5 µm resolution).
• Utilized optically tracked fiducial markers for real-time registration of ultrasound-fluorescence and tomographical data.

Main Results:

• Achieved real-time augmented reality visualization combining multiple imaging modalities.
• Demonstrated multiscale fluorescence imaging with enhanced resolution.
• Obtained a mean Target Registration Error < 2 mm for fluorescence-to-ultrasound and MRI registration, within clinical limits.

Conclusions:

• The developed system successfully integrates multimodal and multiscale imaging with augmented reality.
• The system's accuracy and real-time capabilities show significant promise for future clinical applications.
• This technology advances the potential for improved diagnostic accuracy and surgical guidance.