Large Language Model-Driven Analysis and Report Generation of Endoscopy Videos-A Pilot Study

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Gastroenterology

Background:

• Multimodal large language models (MLLMs) offer potential for automated clinical video analysis.
• The efficacy of MLLMs in full esophagogastroduodenoscopy (EGD) analysis and the influence of computer-aided detection/diagnosis (CAD) overlays are not well-established.

Purpose of the Study:

• To evaluate the ability of an MLLM (Gemini 2.5 Pro) to generate clinically adequate EGD reports.
• To determine the impact of on-screen CAD overlays on MLLM performance in EGD analysis.

Main Methods:

• Five full EGD videos were analyzed by Gemini 2.5 Pro in both clean and CAD-overlay versions.
• Five blinded endoscopists assessed report adequacy across completeness, visualization, and lesion characteristics.
• Two expert endoscopists evaluated MLLM accuracy for landmarks and lesions using the time-window rule.

Main Results:

• MLLM completeness was rated adequate in 56.0% of clean videos versus 48.0% with CAD overlays (p=0.500).
• Visualization and lesion characteristics ratings were identical between clean and overlay videos (p=1.000).
• Overall landmark agreement accuracy was significantly lower with CAD overlays (0.33) compared to clean videos (0.55) (p=0.029).

Conclusions:

• Gemini 2.5 Pro demonstrated inadequate performance for clinical EGD reporting in this pilot study.
• The presence of CAD overlays negatively impacted MLLM landmark detection accuracy.
• Substantial optimization and large-scale validation are required before MLLM deployment for EGD analysis.