Leveraging large language and vision models for knowledge extraction from large-scale image-text colonoscopy records

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Gastroenterology

Background:

• AI for colonoscopy analysis requires large, diverse, expert-annotated datasets, which are often limited.
• Routine clinical colonoscopy records offer vast image-text data but are labor-intensive to annotate.
• Existing methods struggle with dataset limitations, hindering AI model performance and generalization.

Purpose of the Study:

• To develop an automated data mining paradigm, EndoKED, for deep knowledge extraction from colonoscopy records.
• To transform raw colonoscopy image-text data into pixel-level annotated datasets for AI training.
• To enhance polyp detection, segmentation, and optical biopsy capabilities using AI.

Main Methods:

• Leveraging advancements in large language and vision models.
• Proposing EndoKED, a data mining paradigm for knowledge extraction and distillation.
• Automating the creation of pixel-level annotated image datasets from raw colonoscopy records.

Main Results:

• EndoKED demonstrated superior performance in detecting polyps at both report and image levels.
• Pixel-level polyp annotation was achieved, significantly improving segmentation models.
• Pretraining with EndoKED led to state-of-the-art performance and generalization in polyp segmentation.
• The EndoKED vision backbone enabled expert-level performance in optical biopsy with data-efficient learning.

Conclusions:

• EndoKED effectively automates the creation of annotated datasets from clinical colonoscopy records.
• The proposed method significantly advances AI capabilities in polyp detection, segmentation, and optical biopsy.
• EndoKED facilitates the development of more robust and generalizable AI tools for colonoscopy analysis.