Colonoscopy 3D video dataset with paired depth from 2D-3D registration.

Area of Science:

• Medical Imaging
• Computer Vision
• Gastroenterology

Background:

• 3D computer vision techniques are crucial for colonoscopy but lack quantitative evaluation due to scarce ground truth data.
• Existing methods for analyzing colonoscopy videos are largely qualitative, hindering objective benchmarking.

Purpose of the Study:

• To introduce a high-fidelity Colonoscopy 3D Video Dataset (C3VD) for benchmarking 3D computer vision methods.
• To present a novel multimodal 2D-3D registration technique for accurate alignment of colonoscopy videos with ground truth 3D models.

Main Methods:

• Developed a multimodal 2D-3D registration technique using Generative Adversarial Networks for image-to-depth transformation and evolutionary optimization for edge feature alignment.
• Registered 22 video sequences, generating 10,015 frames with comprehensive ground truth data including depth, surface normals, and 6-DoF pose.
• Validated registration accuracy in simulations, achieving low translation and rotation errors, significantly improved by leveraging video information.

Main Results:

• The multimodal registration method achieved an average translation error of 0.321 mm and rotation error of 0.159 degrees in simulations.
• Utilizing video information enhanced registration accuracy by over 55% for translation and 60% for rotation compared to single-frame methods.
• The C3VD dataset comprises 10,015 registered frames with extensive ground truth, plus clinical screening videos with pose and surface models.

Conclusions:

• The C3VD dataset and novel registration technique provide a robust platform for quantitative evaluation and development of 3D computer vision in colonoscopy.
• This resource facilitates advancements in AI-powered tools for more accurate and objective colon cancer screening.