Processing Real-Life Recordings of Facial Expressions of Polish Sign Language Using Action Units

Area of Science:

• Computer Science
• Linguistics
• Human-Computer Interaction

Background:

• Automatic sign language translation is complex, requiring accurate representation of both manual gestures and non-manual features like facial expressions.
• Facial expressions are crucial for conveying grammatical and semantic information in sign languages, including Polish Sign Language (PJM).

Purpose of the Study:

• To address the challenge of incorporating facial expressions into automated Polish-to-Polish Sign Language (PJM) translation.
• To explore the use of action unit (AU) recognition for automatic annotation of PJM translation recordings.
• To evaluate data entropy and analyze action unit data in real-life PJM translation scenarios.

Main Methods:

• Utilizing action unit (AU) recognition to analyze facial expressions in recorded Polish Sign Language (PJM) translations.
• Applying entropy evaluation to assess the complexity and information content of the analyzed data.
• Developing and training machine learning models using automatically annotated data.

Main Results:

• The study successfully explored the integration of facial expression analysis using AU recognition for PJM translation.
• Analysis of data entropy and associated action units provided insights into the nuances of PJM expression.
• The developed approach was validated through expert evaluation by Polish Sign Language specialists.

Conclusions:

• The findings demonstrate the feasibility of using AU recognition to enhance automatic PJM translation systems.
• This research contributes to the ongoing project of creating comprehensive PJM animation solutions.
• The results pave the way for improved machine learning models in sign language translation.