一个基于多流注意力的神经网络,用于视觉语音识别和手语理解
Fatma M Talaat1,2, Basma M Hassan3
1Faculty of Artificial Intelligence, Kafrelsheikh University, Kafrelsheikh, 33516, Egypt.
Scientific reports
|December 24, 2025
概括
新的神经网络SignKeyNet通过分析手,脸和身体运动来改善唇读. 这种以关键点为基础的方法显示出高准确性,有助于视觉语音识别和聋人和听力障碍者访问.
科学领域:
- 计算机视觉 计算机视觉
- 人工智能的人工智能
- 人与计算机的交互
背景情况:
- 唇读识别对于视觉语音理解至关重要.
- 现有的方法经常与人类运动的复杂性作斗争.
- 手语翻译需要强大的多模式分析.
研究的目的:
- 介绍SignKeyNet,一个基于关键点的多流神经网络.
- 提高唇读识别能力,支持手语理解.
- 模拟交叉模式的时空依赖关系以提高准确性.
主要方法:
- 使用姿势估计来提取手,脸和身体的133个关键点.
- 使用专门的注意力模块独立处理每个流.
- 采用基于注意力的融合机制来实现多式联运一体化.
主要成果:
- 在MIRACL-VC1数据集上实现了0.85的准确性,0.12的文字错误率 (WER) 和0.06的字符错误率 (CER).
- SignKeyNet的表现优于包括HMM,DTW,CNN,LSTM和双流ConvNets在内的基线模型.
- 证明了以注意力驱动的多模式架构的有效性.
结论:
- 在唇读识别方面,SignKeyNet提供了卓越的性能.
- 该架构可扩展到完整的手语翻译 (SLT) 系统.
- 显示了对聋人和听力障碍者实时可访问性技术的潜力.
相关概念视频
01:25Neural Circuits
Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
01:22Multi-input and Multi-variable systems
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence of...
In the absence of...