Related Experiment Video
Updated: Jun 9, 2025
08:25
Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment
Published on: May 7, 2019
8.9K
Fine-Grained Temporal Site Monitoring in EGD Streams via Visual Time-Aware Embedding and Vision-Text Asymmetric
IEEE Journal of Biomedical and Health Informatics
|October 30, 2024
Summary
This study introduces a new AI model for real-time monitoring during esophagogastroduodenoscopy (EGD) cancer screenings. The Visual Time-aware Embedding plus Vision-text Asymmetric Coworking (VTE+VAC) model improves accuracy in identifying upper gastrointestinal sites.
Area of Science:
- Medical Imaging
- Artificial Intelligence
- Gastroenterology
Background:
- Esophagogastroduodenoscopy (EGD) requires thorough inspection of upper gastrointestinal (UGI) sites for accurate cancer screening.
- Existing automated methods for EGD site monitoring struggle with global camera motion and the fine-grained, similar appearances of UGI sites.
Purpose of the Study:
- To develop a novel, customized AI model for real-time, accurate, and fine-grained monitoring of UGI sites during EGD.
- To address the challenges of camera motion and visual homogeneity in EGD video analysis.
Main Methods:
- Proposed a novel EGD-customized model featuring Visual Time-aware Embedding (VTE) and Vision-text Asymmetric Coworking (VAC).
- VTE captures temporal relationships using classification and ranking losses on time-agnostic frames, generating time-aware embeddings.
- VAC utilizes a sliding window, VTE embeddings, and a frozen BERT model, incorporating vision-text multimodal knowledge while enhancing error tolerance through random prediction dropping/replacement.
Main Results:
- The VTE+VAC model demonstrated superior performance in real-time UGI site monitoring compared to state-of-the-art methods.
- Achieved an average F1-score improvement of at least 7.66% in qualitative and quantitative experiments.
Conclusions:
- The proposed VTE+VAC model effectively overcomes limitations of existing methods for automated EGD assistance.
- This approach enables more accurate and reliable fine-grained site monitoring during endoscopic procedures, potentially improving cancer detection rates.
