SignFormer-GCN:使用时空图卷积网络进行连续的手语翻译.
Safaeid Hossain Arib1, Rabeya Akter1, Sejuti Rahman1
1Department of Robotics and Mechatronics Engineering, University of Dhaka, Dhaka, Bangladesh.
PloS one
|February 14, 2025
概括
手语翻译 (SLT) 研究往往忽略了关键的时空细节. 本研究介绍了SignFormer-GCN,这是一种结合变压器和图形网络的新方法,以提高手语识别和翻译精度.
科学领域:
- 计算机视觉 计算机视觉
- 自然语言处理自然语言处理.
- 人工智能的人工智能
背景情况:
- 手语是聋人和听力障碍者社区的重要沟通系统.
- 现有的手语翻译 (SLT) 方法主要依赖于RGB特征,这些特征受到背景和手语者变化的限制.
- 当前的SLT研究往往忽视了手语内在的图形结构,无法捕捉低层次的细节.
研究的目的:
- 为了提高手语翻译 (SLT) 系统的准确性和稳定性.
- 为了解决在SLT中仅使用RGB特征的局限性.
- 将手语的时空依赖性纳入翻译模型中.
主要方法:
- 使用了关键点和RGB特征的组合,以更好地捕捉机身部分配置.
- 采用了整合变压器和空间时间图形卷积网络 (STGCN) 架构的联合编码技术.
- 开发了SignFormer-GCN来处理高层上下文和低层时空图形结构.
主要成果:
- 在RWTH-PHOENIX-2014T,How2Sign和BornilDB v1.0.0.0 的基准数据集上,SignFormer-GCN实现了竞争性表现.
- 提出的方法在提高不同手语数据集的翻译准确性方面表现出有效性.
- 实验结果验证了结合变压器和基于图形的方法对SLT的好处.
结论:
- SignFormer-GCN模型通过有效捕获复杂的视觉和结构信息,在手语翻译方面取得了重大进展.
- 将时空图形信息与基于变压器的上下文处理相结合,可以提高翻译的准确性.
- 这项研究提供了对SLT更全面的方法,为为聋人社区改善沟通工具铺平了道路.
相关概念视频
Signal Flow Graphs
158
Signal-flow graphs offer a streamlined and intuitive approach to representing control systems, providing an alternative to traditional block diagrams. These graphs use branches to symbolize systems and nodes to represent signals, effectively illustrating the relationships and interactions within the system.
In a signal-flow graph, branches denote the system's transfer functions, while nodes represent the signals. The direction of signal flow is indicated by arrows, with the corresponding...
In a signal-flow graph, branches denote the system's transfer functions, while nodes represent the signals. The direction of signal flow is indicated by arrows, with the corresponding...
158
Convolution: Math, Graphics, and Discrete Signals
223
In any LTI (Linear Time-Invariant) system, the convolution of two signals is denoted using a convolution operator, assuming all initial conditions are zero. The convolution integral can be divided into two parts: the zero-input or natural response and the zero-state or forced response, with t0 indicating the initial time.
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...
223
SFG Algebra
102
In Signal Flow Graph (SFG) algebra, the value a node represents is determined by the sum of all signals entering that node. This summed value is then transmitted through every branch leaving the node, making the SFG a powerful tool for visualizing and analyzing control systems.
Each node in an SFG corresponds to a variable, and the interactions between nodes are represented by branches with associated gains. When multiple branches lead into a node, the value at that node is the sum of the...
Each node in an SFG corresponds to a variable, and the interactions between nodes are represented by branches with associated gains. When multiple branches lead into a node, the value at that node is the sum of the...
102


