Using deep vision-language models improves multi-task performance in assistance applications for endoscopic ENT...

Deep learning models for endoscopic assistance applications predominantly focus on image-based tasks, such as tool detection, anatomical classification, and workflow segmentation. However, these approaches often neglect the integration of natural language, limiting their assistance capabilities. This work adopts a proven architecture for vision-language models (VLM) to perform multi-task learning for image classification, text prediction, and surgical report generation, specifically for endoscopic ENT surgeries.

We adopted a VLM architecture utilizing encoders biased for the endoscopy domain for image and text embedding and combine them via cross-attention. The model was trained on a newly created multi-task dataset derived from 30 annotated endoscopic procedures, comprising 130,000 multi-label images, anatomical descriptions, and synchronized surgical reports. Two variations of the model, a lightweight 61M parameter and a 176M parameter model, were evaluated both against an existing baseline from previous mono-task studies as well as the EndoVit and SurgicalGPT models as external references. Ablation studies investigate the influence of removing image or text embeddings, and cross-attention on the task performance. Performance was measured for landmark classification, structured text prediction, and report generation using precision, recall, F1-score, BLEU-2, ROUGE-L, and cosine similarity metrics.

The VLM base model improves the baseline f1 score for image classification by up to 12% and natural language text generation by up to 14% across image classification and report generation tasks. The text generation of structured language tasks, however, showed minimal gains, indicating limitations in structured sentence learning from combined image-text embeddings. EndoViT and SurgicalGPT slightly trail our domain-specific VLM. The image-only and text-only ablations confirm that the vision component benefits language tasks, whereas text has limited impact on landmark detection.

We developed a vision-language model capable of integrating image and text data for endoscopic ENT assistance tasks, that is able to replace three isolated models, delivering multi-task assistance while outperforming prior and general‑purpose baselines. Remaining challenges include the handling of imbalanced class distributions and limited gains on templated structured text.