One Model is Not Enough: Ensembles for Isolated Sign Language Recognition.

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Sign language recognition is crucial for bridging communication gaps.
• Isolated sign recognition is a key subfield, treating sequences of frames as distinct linguistic units.
• Existing methods often rely on appearance-based or pose-based approaches.

Purpose of the Study:

• To evaluate and compare appearance-based (I3D, TimeSformer) and pose-based (SPOTER) methods for isolated sign language recognition.
• To explore the impact of different data modalities and preprocessing techniques on recognition accuracy.
• To achieve state-of-the-art performance through ensemble methods.

Main Methods:

• Analysis of appearance-based models (I3D, TimeSformer) trained on diverse data modalities.
• Evaluation of a pose-based model (SPOTER) with varying preprocessing strategies.
• Experimentation with ensemble techniques, including CMA-ES optimization and a novel Transformer-based Neural Ensembler.

Main Results:

• The study tested methods on AUTSL and WLASL300 datasets.
• Ensemble techniques, optimized with CMA-ES, yielded a new state-of-the-art accuracy of 73.84% on the WLASL300 dataset.
• A novel Transformer-based ensembling method, the Neural Ensembler, was introduced.

Conclusions:

• Ensemble methods significantly improve isolated sign language recognition accuracy.
• The proposed Neural Ensembler offers a promising direction for future research in sign language recognition.
• The findings contribute to advancing accessible communication technologies for the deaf and hard-of-hearing community.