自然语言驱动的多模式表示学习,用于视听场景意识对话系统
Yoonseok Heo1, Sangwoo Kang2, Jungyun Seo1
1Department of Computer Science and Engineering, Sogang University, Seoul 04107, Republic of Korea.
Sensors (Basel, Switzerland)
|September 28, 2023
概括
本研究介绍了一种用于人工智能的视听场景感知对话系统. 它通过整合音频,视觉和文本数据来增强类似人类的沟通,以更好地理解和生成响应.
科学领域:
- 人工智能的人工智能
- 人与计算机的交互
- 多媒体系统 多媒体系统
背景情况:
- 多媒体系统需要人工智能来进行类似人类的沟通.
- 现有的系统难以整合听觉信息,缺乏可解释性.
- 多模式表示学习已经进步,但有局限性.
研究的目的:
- 开发一个音视觉场景意识的对话系统.
- 提高AI对视听场景的全面理解.
- 在对话系统中提高AI推理的可解释性.
主要方法:
- 提出了一种新的视听场景感知对话系统.
- 利用每个模式的明确信息合并到语言模型中.
- 在多任务学习环境中使用基于变压器的解码器来生成响应.
- 实现了响应驱动的时间时刻定位以实现可解释性.
主要成果:
- 拟议的模型在定量和定性评估中表现出优于基线模型的优势.
- 使用所有三种模式 (音频,视觉,文本) 实现了强大的性能.
- 在系统响应推理任务中达到最先进的性能.
结论:
- 这种新系统有效地整合了多式联运信息,以加强对话.
- 可解释性方法为系统响应生成提供了证据.
- 该模型显示在视听场景理解和AI通信方面取得了重大进展.
相关概念视频
Multi-input and Multi-variable systems
122
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence...
In the absence...
122
Perceiving Loudness, Pitch, and Location
233
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
233
Sensory Modalities
1.4K
Sensation typically is the process by which the sensory receptors and sense organs detect stimuli from the internal and external environment and transmit this information to the central nervous system for processing.
General senses refer to the broad category of sensory information detected by receptors in the body and can be further grouped into somatic and visceral senses. Somatic sensations include touch, pressure, temperature, and pain and are essential for navigating our environment and...
General senses refer to the broad category of sensory information detected by receptors in the body and can be further grouped into somatic and visceral senses. Somatic sensations include touch, pressure, temperature, and pain and are essential for navigating our environment and...
1.4K
Auditory Pathway
5.5K
Auditory pathways constitute the complex neural circuits responsible for transmitting and interpreting auditory information from the peripheral auditory system to the brain. Sound waves are initially captured by the outer ear, funneled through the ear canal, and reach the tympanic membrane (eardrum). These vibrations are transmitted via the middle ear's ossicles to the inner ear's cochlea.
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
5.5K
Control Volume and System Representations
1.2K
Two key frameworks are employed to analyze mass, energy, and momentum transfer: the control volume approach and the system approach. These frameworks offer different perspectives, depending on whether the focus is on a specific region in space (control volume approach) or a defined mass of fluid (system approach).
The control volume approach considers a stationary region in space through which fluid flows. This region is bounded by a control surface. For instance, in the case of water...
The control volume approach considers a stationary region in space through which fluid flows. This region is bounded by a control surface. For instance, in the case of water...
1.2K
Chunking and Rehearsal in Sensory Memory
236
Improving short-term memory can be achieved through techniques like chunking and rehearsal. Chunking involves organizing information into larger, more manageable units. This technique is particularly useful for information that exceeds the typical memory span of between five and nine items. For instance, logging into an online account with a password like "ta89vq0179gz" involves grouping letters and numbers into three chunks—ta89, vq01, and 79gz. It makes large amounts of...
236


