Vision-language models for automated video analysis and documentation in laparoscopic surgery: a proof-of-concept study

Esther Helene Stueker ¹, Fiona R Kolbinger ^2,3,4, Oliver Lester Saldanha ¹

¹Else Kröner Fresenius Center for Digital Health, Dresden University of Technology, Dresden, Germany.
²Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany.
³Weldon School of Biomedical Engineering, Purdue University, West Lafayette, USA.
⁴Department of Biostatistics and Health Data Science, Richard M. Fairbanks School of Public Health, Indiana University, Indianapolis, USA.
⁵National Center for Tumor Diseases Dresden (NCT/UCC), a partnership between DKFZ, Faculty of Medicine and University Hospital Carl Gustav Carus, TU Dresden University of Technology and Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, German.
⁶German Cancer Consortium (DKTK), Partner Site Dresden, German Cancer Research Center (DKFZ), Heidelberg, Germany.
⁷Asklepios-ASB Krankenhaus Radeberg, Radeberg, Germany.
⁸Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany.
⁹Department of Medicine I, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany.

⁰Else Kröner Fresenius Center for Digital Health, Dresden University of Technology, Dresden, Germany.

International Journal of Surgery (london, England) +

|

July 18, 2025

View abstract on PubMed

Summary

Vision-Language Models (VLMs) show promise for surgical documentation. GPT-4o and Gemini-1.5-pro reliably detected surgical tools and classified procedures, though grading pathology requires further development.

Area Of Science

Artificial Intelligence
Medical Informatics
Computer Vision

Background

The healthcare industry faces a critical shortage of medical personnel, necessitating automation in clinical documentation.
Large Vision-Language Models (VLMs) present a significant opportunity to streamline surgical documentation and enhance intraoperative analysis.

Purpose Of The Study

To comparatively evaluate the performance of two general-purpose VLMs, GPT-4o and Gemini-1.5-pro, in surgical video analysis.
To assess the efficacy of in-context learning (ICL) in improving VLM performance for surgical tasks.

Main Methods

An observational study compared GPT-4o and Gemini-1.5-pro using 30 surgical videos (15 cholecystectomy, 15 appendectomy).
Tasks included object detection, surgery classification, appendicitis grading, and surgical report generation.
Performance was evaluated using descriptive accuracy metrics, with and without in-context learning.

Main Results

Both VLMs achieved 100% accuracy in identifying vessel clips. GPT-4o excelled in retrieval bag and gauze detection, while Gemini-1.5-pro demonstrated superior bleeding detection.
Gemini-1.5-pro was more accurate in classifying cholecystectomies, whereas both models showed moderate performance in appendectomy classification.
Appendicitis grading yielded limited accuracy for both models. In-context learning improved tool recognition but had inconsistent effects on other tasks.

Conclusions

Domain-agnostic VLMs demonstrate reliable performance in surgical object detection and procedure classification.
Limitations exist in pathology grading and detailed procedural step description, areas where in-context learning shows potential for enhancement.
Future development of domain-specific VLMs could further revolutionize operating room efficiency and support.

Keywords:

appendectomy cholecystectomy minimally invasive surgery surgical video analysis vision-language models

Related Concept Videos

Endoscopic Procedures III: Video Capsule Endoscopy

Capsule endoscopy, or wireless or video capsule endoscopy, is a diagnostic procedure for examining the entire gastrointestinal tract. Patients swallow a capsule about the size of a vitamin tablet. The capsule is equipped with a transmitter, a battery, an LED light source, and a color video camera to capture images throughout the gastrointestinal tract. This procedure is particularly useful for diagnosing conditions such as Crohn's disease, ulcerative colitis, tumors, polyps, ulcers,...