通过观看数百个手术视频讲座来学习多模式表示
Kun Yuan1, Vinkle Srivastav2, Tong Yu3
1University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France; CAMP, Technische Universität München, Munich, Germany.
Medical image analysis
|June 13, 2025
概括
这项研究介绍了SurgVLP,这是一种用于外科视频分析的新方法,它使用讲座成绩单进行多模式学习. SurgVLP提高了概括性,并减少了手术计算机视觉任务的手动注释需求.
科学领域:
- 计算机视觉 计算机视觉
- 医疗信息学 医疗信息学
- 人工智能的人工智能
背景情况:
- 目前的外科计算机视觉模型由于依赖仅视觉数据和手动注释而缺乏通用性.
- 整合语言语义可以提高外科视频分析系统的性能和适应性.
研究的目的:
- 利用开源电子学习讲座数据,为外科视频开发一种多模式表示学习方法.
- 减少在手术视频分析中对手册注释的依赖.
主要方法:
- 利用多个自动语音识别系统来转录外科视频讲座.
- 开发了SurgVLP (外科视觉语言预训),一种采用对比学习目标的新方法.
- 视频剪辑嵌入线与文本嵌入线在一个联合的潜伏空间.
主要成果:
- SurgVLP在各种不同的手术程序和任务中表现出强大的可转移性和适应性.
- 评估视觉和语言和仅视觉手术任务的表现,包括工具,阶段和三重识别.
- 零射击评估证实了SurgVLP作为手术工作流分析的基础模型的潜力.
结论:
- SurgVLP有效地利用外科讲座的视觉和语言信号进行多模式学习.
- 拟议的方法显著减少了手动注释的需要,为可扩展和数据效率高的外科AI铺平了道路.
- SurgVLP显示出作为一个通用基础模型的前景,用于推进外科计算机视觉和工作流分析.
相关概念视频
Endoscopic Procedures III: Video Capsule Endoscopy
Capsule endoscopy, or wireless or video capsule endoscopy, is a diagnostic procedure for examining the entire gastrointestinal tract. Patients swallow a capsule about the size of a vitamin tablet. The capsule is equipped with a transmitter, a battery, an LED light source, and a color video camera to capture images throughout the gastrointestinal tract. This procedure is particularly useful for diagnosing conditions such as Crohn's disease, ulcerative colitis, tumors, polyps, ulcers, unexplained...
Introduction to Learning
Learning is the process of acquiring knowledge or skills through practice or experience, leading to long-lasting behavioral changes. This acquisition occurs through interaction with the environment and requires practice or experience. For instance, mastering a skill such as surfing requires considerable practice and experience, highlighting the essential role of repeated interactions with the environment in learning.
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
Cognitive Learning
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
Observational Learning
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning because...


