Med-VCD: Mitigating hallucination for medical large vision language models through visual contrastive decoding.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Large vision-language models (LVLMs) are increasingly used in healthcare for tasks like medical visual question answering and report generation.
• A significant challenge with LVLMs is their tendency to produce hallucinations—outputs that are plausible but factually incorrect.
• Existing methods to reduce hallucinations often involve slow secondary decoding or domain-specific approaches that can cause misalignment.

Purpose of the Study:

• To introduce Med-VCD, a novel sparse visual-contrastive decoding method designed to mitigate hallucinations in medical LVLMs.
• To address the limitations of existing hallucination mitigation strategies, particularly their impact on inference speed and domain specificity.
• To improve the reliability and factual accuracy of medical LVLMs without compromising efficiency.

Main Methods:

• Med-VCD employs a sparse visual-contrastive decoding approach.
• It utilizes a novel token-sparsification strategy to select visually relevant tokens dynamically.
• This method trims redundancy while preserving essential visual context, enhancing efficiency and reliability.

Main Results:

• Med-VCD significantly reduces hallucinations in medical LVLMs.
• Evaluations across eight diverse medical datasets (ophthalmology, radiology, pathology) demonstrated improvements.
• Factual accuracy increased by an average of 13%, and hallucination accuracy improved by 6% compared to baseline models.

Conclusions:

• Med-VCD offers an effective and efficient solution for mitigating hallucinations in medical LVLMs.
• The method enhances factual accuracy and reliability without the computational overhead of traditional secondary decoding techniques.
• Med-VCD represents a promising advancement for trustworthy AI applications in healthcare imaging.