Context-Aware Automatic Sign Language Video Transcription in Psychiatric Interviews

Area of Science:

• Computational linguistics
• Artificial intelligence
• Clinical psychology

Background:

• Sign language translation is challenging, especially with limited data for specialized domains like mental health.
• Deep learning models require extensive datasets, which are often unavailable for specific applications.
• Prior information can significantly improve translation by narrowing down possibilities.

Purpose of the Study:

• To improve sign language translation for psychiatric interviews involving deaf and hard of hearing patients.
• To address the challenge of limited training data in specialized domains.
• To leverage domain knowledge for enhanced translation accuracy.

Main Methods:

• Developed a domain-specific approach for sign language translation in psychiatric settings.
• Combined data-driven feature extraction with prior information from domain knowledge.
• Utilized a hierarchical ontology to model interview context and classify interview states.
• Treated video transcription as a sentence retrieval problem, predicting patient responses based on context.

Main Results:

• Experimental evaluation demonstrated significant performance gains by incorporating context awareness.
• The system successfully predicted signed patient sentences within simulated psychiatric interviews.
• The domain-specific approach proved effective in overcoming data limitations.

Conclusions:

• Context-aware sign language translation is crucial for improving accuracy in specialized domains like mental health.
• Leveraging domain knowledge and hierarchical ontologies can effectively address data scarcity.
• This approach offers a promising solution for improving communication in clinical settings for deaf and hard of hearing individuals.