Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translation

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Gastroenterology

Background:

• Colorectal cancer screening relies on optical colonoscopy (OC) and virtual colonoscopy (VC).
• VC provides 3D colon structure, while OC offers detailed textures but can obscure polyps.
• Existing image-to-image translation methods struggle with the lossy nature of OC to VC transformations.

Purpose of the Study:

• To develop a deep learning framework for lossy, unpaired image-to-image translation between OC and VC.
• To augment OC videos with depth information from VC and VC with OC textures for realistic polyp synthesis.
• To improve scale-consistent depth inference and polyp visualization in colorectal cancer screening.

Main Methods:

• Introduced Extended and Directional CycleGAN for unpaired image translation.
• Developed an extended cycle consistency loss to preserve geometric structures.
• Incorporated a Directional Discriminator to handle texture and lighting differences.

Main Results:

• Achieved state-of-the-art results in scale-consistent depth inference for phantoms and real colon data.
• Demonstrated realistic synthesis of pedunculated and flat polyps from 3D VC models.
• Successfully augmented OC and VC images with complementary information.

Conclusions:

• The Extended and Directional CycleGAN framework effectively addresses lossy image translation challenges in colonoscopy.
• This approach enhances visualization and diagnostic capabilities for colorectal cancer screening.
• The method shows promise for improving polyp detection and surgical planning.