解释视觉和语言生成模型与语义视觉先验
Michele Cafagna1, Lina M Rojas-Barahona2, Kees van Deemter3
1Institute of Linguistics and Language Technology, University of Malta, Msida, Malta.
Frontiers in artificial intelligence
|October 11, 2023
概括
这项研究引入了一个新的框架来解释图像到文本模型,提供更有意义和更有效的视觉解释. 该方法通过专注于语义意义而不是代币对代币分析来提高解释性.
科学领域:
- 人工智能的人工智能
- 计算机视觉 计算机视觉
- 自然语言处理自然语言处理.
背景情况:
- 图像到文本模型的可解释性方法面临着代币对代币解释的挑战,这些解释在计算上昂贵,缺乏全面的洞察力.
- 当前的方法经常使用超级像素作为特征,由于它们在视觉解释中缺乏语义意义,因此妨碍了解释性.
研究的目的:
- 开发一个新的框架,用于生成图像转化为文本模型的全面和语义上有意义的解释.
- 在大规模视觉语言模型中提高视觉解释的效率和可解释性.
主要方法:
- 开发了一个基于SHAP (夏普利增量解释) 的框架,以利用整个输出序列的含义表示.
- 利用视觉骨干中的语义先验来提取任意数量的语义上有意义的特征.
- 实现了对大规模模型的沙普利值的高效计算,产生了高度可解释的视觉解释.
主要成果:
- 与传统方法相比,提出的方法在语义上产生了比传统方法更具表现力的解释.
- 为生成解释实现了较低的计算成本.
- 在广泛的视觉语言模型中展示了概括能力.
结论:
- 开发的框架为图像转化为文本模型的解释性提供了显著的进步.
- 为理解模型行为提供了一种更有效,更易于解释的方法.
- 该方法具有适应性,广泛适用于各种视觉语言架构.
更多相关视频
05:38Interaction between Phonological and Semantic Processes in Visual Word Recognition using Electrophysiology
Published on: June 29, 2021
2.4K
08:17A Semantic Priming Event-related Potential ERP Task to Study Lexico-semantic and Visuo-semantic Processing in Autism Spectrum Disorder
Published on: April 12, 2018
10.6K
相关概念视频
Visual System
607
Light enters the eye through the cornea, a transparent, dome-shaped surface covering the surface of the eyeball that helps to direct and focus incoming light. This light is then channeled toward the pupil, an adjustable opening whose size is controlled by the iris. The iris, a pigmented muscle, regulates the amount of light entering the eye by contracting or dilating the pupil, thereby ensuring optimal light levels for clear vision.
Once through the pupil, the light passes through the lens, a...
Once through the pupil, the light passes through the lens, a...
607
Vision
53.5K
Vision is the result of light being detected and transduced into neural signals by the retina of the eye. This information is then further analyzed and interpreted by the brain. First, light enters the front of the eye and is focused by the cornea and lens onto the retina—a thin sheet of neural tissue lining the back of the eye. Because of refraction through the convex lens of the eye, images are projected onto the retina upside-down and reversed.
53.5K
Higher Mental Functions of the Brain: Language
890
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
890
Depth Perception and Spatial Vision
689
Depth perception is the ability to perceive objects three-dimensionally. It relies on two types of cues: binocular and monocular. Binocular cues depend on the combination of images from both eyes and how the eyes work together. Since the eyes are in slightly different positions, each eye captures a slightly different image. This disparity between images, known as binocular disparity, helps the brain interpret depth. When the brain compares these images, it determines the distance to an object.
689
Color Vision
606
Color perception begins in the retina, the light-sensitive layer at the back of the eye. Two main theories explain how colors are seen: the trichromatic theory and the opponent-process theory. The trichromatic theory, proposed by Thomas Young in 1802 and extended by Hermann von Helmholtz in 1852, suggests that color vision is based on three types of cone receptors in the retina. These cones are sensitive to different but overlapping ranges of wavelengths corresponding to red, blue, and green.
606
Gestalt Principles of Perception
322
Gestalt principles provide a framework for understanding how humans perceive objects as unified wholes within their context. These principles are essential in explaining the cognitive processes that make sense of complex visual stimuli by organizing them into coherent groups. One fundamental principle is proximity, which posits that objects located close to each other are perceived as a collective group. For instance, when dots are positioned near one another, the visual system interprets them...
322
