Multimodal visual question answering in neuro-oncology and gastroenterology: A domain-specific feasibility study

Medical Visual Question Answering (Med-VQA) requires models to combine image content with the clinical intent expressed by a question, yet domain-specific resources and controlled analyses of modality use remain limited. This study introduces two expert-validated Med-VQA datasets for brain tumor MRI and gastrointestinal endoscopy and evaluates a dual-stream architecture that combines BERT question representations and ViT image features through Modular Co-Attention on Attention (MCAoA). The model formulates answer prediction as closed-vocabulary classification and is assessed through overall, question-type, disease-class, baseline, ablation, and usability analyses. The complete fine-tuned model with product fusion achieved accuracies of 80.86% (WUPS 81.32%) on the brain tumor dataset and 75.95% (WUPS 76.77%) on the gastrointestinal dataset. It exceeded the question-only and image-only baselines by 23.86 and 43.16 percentage points, respectively, on brain tumor MRI, and by 18.54 and 35.92 percentage points on gastrointestinal endoscopy. Product fusion provided a small but consistent advantage over concatenation, while end-to-end encoder fine-tuning improved accuracy over frozen encoders on both datasets. A complementary study with clinical experts indicated positive perceived usability in a controlled setting. These findings show that performance cannot be reproduced from either modality alone and support the complementary use of image and question information. However, the ablations establish modality-level dependence rather than spatially correct anatomical grounding, which requires targeted counterfactual and attribution-based evaluation.