Related Experiment Video
Updated: Jun 20, 2026
Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography
Published on: November 30, 2022
SurgCheck: Do vision-language models really look at images in surgical VQA?
Jongmin Shin1, Ka Young Kim2, Eunki Cho3
1Department of Surgery, Samsung Medical Center, 81 Irwon-ro, Gangnam-gu, Seoul, 06351, Republic of Korea.
Purpose:
Vision-language models (VLMs) have shown promising performance in surgical visual question answering (VQA). However, existing surgical VQA datasets often contain linguistic shortcuts, where question phrasing implicitly constrains the answer space. In safety-critical surgical settings, it remains unclear whether reported performance reflects visual understanding or reliance on such linguistic shortcuts.
Methods:
We introduce SurgCheck, a diagnostic benchmark for quantifying linguistic shortcut reliance in surgical VQA. SurgCheck employs a paired-question design in which each surgical frame is associated with an original question containing entity names and a less-biased counterpart that removes these names while preserving identical visual content and ground-truth answers. The resulting performance gap provides a diagnostic signal of shortcut reliance. To ensure that the less-biased question remains well defined even without entity names, four grounding cues are incorporated: bounding box, arrow, spatial position, and periphrasis. We evaluate both general-purpose and surgical-specific VLMs under zero-shot and fine-tuned settings on SurgCheck. To evaluate open-ended zero-shot responses, we introduce an LLM-as-a-judge evaluation protocol.
Results:
Using SurgCheck, we observe consistent performance degradation on less-biased questions across five VLMs, despite identical visual inputs. Text-only ablation reveals minimal performance drops for action and target prediction, indicating that action and target prediction are largely driven by linguistic shortcuts rather than visual reasoning.
Conclusion:
SurgCheck provides a controlled diagnostic framework that exposes failure modes masked by linguistic bias in existing surgical VQA benchmarks. Our findings demonstrate that strong benchmark performance does not necessarily imply faithful visual understanding, underscoring the need for bias-aware evaluation in surgical VQA. SurgCheck is publicly available at https://github.com/ailab-kyunghee/SurgCheck .