When to trust the answer: question-aligned semantic nearest neighbor entropy for safer surgical VQA

Future human-in-the-loop surgical visual question answering (VQA) requires detecting unreliable answers before use. In this preclinical methodological study, we distinguish answer consistency from question validity: A model may generate mutually consistent answers that still fail to address the clinical question.

We propose question-aligned semantic nearest neighbor entropy (QA-SNNE), a black-box failure-detection score that adds question-answer alignment to semantic entropy through bilateral gating. QA-SNNE modulates pairwise similarities among sampled answers by their question relevance, using embedding, entailment, or cross-encoder alignment. To assess robustness under language variation, we construct an out-of-template rephrased version of a benchmark surgical VQA dataset, in which only the question wording is modified while images and ground-truth answers remain unchanged.

We evaluate QA-SNNE on five VQA models across two benchmark surgical VQA datasets, considering both zero-shot and parameter-efficient fine-tuned settings, including out-of-template questions. QA-SNNE improves AUROC on EndoVis18-VQA for two of three zero-shot models in-template (e.g., +15% for Llama3.2 and +21% for Qwen2.5) and achieves up to +8% AUROC relative improvement among zero-shot models under out-of-template rephrasing, with mixed results on external validation.

By making uncertainty estimation explicitly question-aware, QA-SNNE provides a practical and model-agnostic safeguard for preclinical surgical VQA evaluation. Linking semantic uncertainty to question relevance enables more reliable failure detection under variation in question phrasing.