探测视觉语言模型的可视化素养:好,坏和丑
IEEE transactions on visualization and computer graphics
|November 19, 2025
概括
视觉语言模型 (VLMs) 显示出强烈的图表理解. 我们的研究使用注意引导类激活地图 (AG-CAM) 来揭示VLM推理,发现ChartGemma在开源模型中脱而出.
科学领域:
- 人工智能的人工智能
- 计算机视觉 计算机视觉
- 自然语言处理自然语言处理.
背景情况:
- 视觉语言模型 (VLMs) 在图表理解方面显示出潜力.
- 以前的研究主要评估了VLM响应的准确性,忽视了内部推理过程.
- 了解VLM可视化素养需要探索他们的决策机制.
研究的目的:
- 适应和应用注意引导类激活地图 (AG-CAM) 以可视化图表理解中的VLM内部推理.
- 在基于图表的任务上比较开源和闭源VLM的性能和内部推理.
- 在图表问答过程中调查VLM的空间和语义推理能力.
主要方法:
- 适应的注意引导类激活地图 (AG-CAM) 用于早期融合VLM架构.
- 评估了四个开源的 (ChartGemma,Janus 1B/7B,LLaVA) 和两个封闭的 (GPT-4o,Gemini) VLM.
- 分析了AG-CAM结果,以了解特征的重要性和模型推理.
主要成果:
- ChartGemma是一个3B参数VLM,其性能优于其他开源模型,并且在性能上与封闭源模型相匹配.
- VLM通过定位图表特征来展示空间推理,通过将视觉元素与数据联系起来来展示语义推理.
- AG-CAM可视化显示了对图表QA的VLM决策过程的洞察力.
结论:
- 适应的AG-CAM方法为分析图表理解中的VLM内部推理提供了一种新的方法.
- 开源的VLM,特别是ChartGemma,表现出竞争力的性能和可解释的推理能力.
- 这项工作促进了人工智能可视化素养和图表QA方面的透明和可重现的研究.
相关概念视频
Vision
59.2K
Vision is the result of light being detected and transduced into neural signals by the retina of the eye. This information is then further analyzed and interpreted by the brain. First, light enters the front of the eye and is focused by the cornea and lens onto the retina—a thin sheet of neural tissue lining the back of the eye. Because of refraction through the convex lens of the eye, images are projected onto the retina upside-down and reversed.
59.2K
Depth Perception and Spatial Vision
1.8K
Depth perception is the ability to perceive objects three-dimensionally. It relies on two types of cues: binocular and monocular. Binocular cues depend on the combination of images from both eyes and how the eyes work together. Since the eyes are in slightly different positions, each eye captures a slightly different image. This disparity between images, known as binocular disparity, helps the brain interpret depth. When the brain compares these images, it determines the distance to an object.
1.8K
Visual Agnosia
912
Visual agnosia is a condition characterized by the inability to recognize visually presented objects despite having normal vision. For instance, a person with visual agnosia can describe the shape and color of an object but cannot identify or name it. This impairment does not affect their visual field, acuity, color vision, brightness discrimination, language, or memory. An example of this condition in a social setting is someone at a dinner party asking for "that silver thing with a round...
912
Visual System
1.6K
Light enters the eye through the cornea, a transparent, dome-shaped surface covering the surface of the eyeball that helps to direct and focus incoming light. This light is then channeled toward the pupil, an adjustable opening whose size is controlled by the iris. The iris, a pigmented muscle, regulates the amount of light entering the eye by contracting or dilating the pupil, thereby ensuring optimal light levels for clear vision.
Once through the pupil, the light passes through the lens, a...
Once through the pupil, the light passes through the lens, a...
1.6K


