Modeling Word Importance in Conversational Transcripts: Toward improved live captioning for Deaf and hard of hearing...

Area of Science:

• Speech and Language Processing
• Human-Computer Interaction
• Accessibility Technology

Background:

• Automatic speech recognition (ASR) systems have limitations in live conversational accuracy.
• Current evaluation metrics may not fully capture the caption quality experienced by Deaf and Hard of Hearing (DHH) readers.
• Word importance classification is a promising approach to improve caption quality assessment.

Purpose of the Study:

• To analyze a human-annotated word importance dataset for conversational transcripts.
• To explore the relationship between Part-of-Speech (POS) tags and word importance.
• To develop and evaluate supervised models for predicting word importance in ASR captions.

Main Methods:

• Conducted word-token level analysis on a human-annotated dataset.
• Explored Part-of-Speech (POS) distribution in relation to word importance.
• Augmented the dataset with POS tags and addressed class imbalance using text generation.
• Investigated various supervised learning models for word importance prediction.

Main Results:

• The best performing model, trained on the augmented dataset, achieved superior performance compared to prior models.
• Analysis revealed insights into how supervised models learn word importance.
• The findings support the use of POS tags and data augmentation for improved word importance classification.

Conclusions:

• The developed approach can inform the design of novel evaluation metrics for live caption quality.
• This work enhances the perspective of DHH users in assessing caption quality.
• Improved word importance classification contributes to more effective ASR systems for DHH individuals.