MVIB-Lip:通过时间序列建模实现视觉语音识别的多视图信息瓶
Yuzhe Li1, Haocheng Sun1, Jiayi Cai1
1School of Electronic Engineering, Xi'an University of Posts and Telecommunications, Xi'an 710121, China.
Entropy (Basel, Switzerland)
|November 26, 2025
概括
本研究介绍了MVIB-Lip,这是一个新的多视图学习框架,用于唇读 (视觉语音识别). 它通过结合里程碑轨迹和复发图表来提高准确性和概括性,特别是在资源较少的场景中.
科学领域:
- 计算机科学 计算机科学
- 人工智能的人工智能
- 信号处理 信号处理
背景情况:
- 唇读,或视觉语音识别,通过唇部运动来解释语音.
- 深度学习方法需要大量的数据,并且难以进行概括.
- 现有的方法在资源较少或独立于扬声器的环境中经常失败.
研究的目的:
- 开发一个更强大,数据效率更高的唇读框架.
- 提高视觉语音识别系统的概括能力.
- 探索多视图学习,以整合互补的唇部运动表示.
主要方法:
- MVIB-Lip框架集成原始地标轨迹 (多变量时间序列) 和重复图 (RP) 图像.
- 时间序列的变压器编码器和RP图像特征的ResNet-18 .
- 两个视图的融合使用产品的专家后部和多视图信息瓶.
主要成果:
- MVIB-Lip在OuluVS和自行收集的数据集上始终超过手工制作的基线.
- 演示了语音独立视觉语音识别的改进概括.
- 验证了将复发图形与深度多视图学习相结合的有效性.
结论:
- 再现图,结合深度多视图学习,提供了一个原则和数据效率的方法,用于强大的视觉语音识别.
- MVIB-Lip为克服当前唇读技术的局限性提供了一个有希望的方向.
- 该框架显示了现实世界应用的潜力,这些应用需要准确和可概括的唇读.
更多相关视频
相关概念视频
Chunking and Rehearsal in Sensory Memory
546
Improving short-term memory can be achieved through techniques like chunking and rehearsal. Chunking involves organizing information into larger, more manageable units. This technique is particularly useful for information that exceeds the typical memory span of between five and nine items. For instance, logging into an online account with a password like "ta89vq0179gz" involves grouping letters and numbers into three chunks—ta89, vq01, and 79gz. It makes large amounts of...
546
Classification of Signals
1.3K
In signal processing, signals are classified based on various characteristics: continuous-time versus discrete-time, periodic versus aperiodic, analog versus digital, and causal versus noncausal. Each category highlights distinct properties crucial for understanding and manipulating signals.
A continuous-time signal holds a value at every instant in time, representing information seamlessly. In contrast, a discrete-time signal holds values only at specific moments, often denoted as x(n), where...
A continuous-time signal holds a value at every instant in time, representing information seamlessly. In contrast, a discrete-time signal holds values only at specific moments, often denoted as x(n), where...
1.3K
Multi-input and Multi-variable systems
376
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence of...
In the absence of...
376


