Hallucination-Aware Multimodal Benchmark for Gastrointestinal Image Analysis with Large Vision-Language Models

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Natural Language Processing

Background:

• Vision-Language Models (VLMs) show promise in generating medical reports from images.
• Hallucination, or generating content inconsistent with visual data, is a critical issue in medical VLMs.
• Existing datasets lack specific annotations for hallucination detection and correction.

Purpose of the Study:

• To introduce Gut-VLM, a novel multimodal dataset for gastrointestinal (GI) image analysis.
• To develop and evaluate a hallucination-aware finetuning strategy for medical VLMs.
• To establish a benchmark for evaluating state-of-the-art VLMs in GI image analysis.

Main Methods:

• Curated the Gut-VLM dataset using a two-stage pipeline with AI-generated reports and expert corrections.
• Developed a hallucination-aware finetuning approach to train VLMs to detect and correct hallucinations.
• Conducted extensive evaluations of various VLMs on the Gut-VLM dataset.

Main Results:

• The Gut-VLM dataset includes annotations for hallucinated sentences and their corrections.
• Hallucination-aware finetuning demonstrated superior performance compared to standard finetuning for report generation.
• Established a comprehensive benchmark for VLM performance in GI image analysis.

Conclusions:

• The Gut-VLM dataset and hallucination-aware finetuning are effective in mitigating VLM hallucinations in medical imaging.
• This work provides a valuable resource and methodology for advancing reliable VLM applications in healthcare.
• Future research can leverage this dataset to further improve VLM robustness and clinical utility.