How does a deep neural network look at lexical stress in English words?

Area of Science:

• Speech processing
• Computational linguistics
• Machine learning

Background:

• Neural networks are successful in speech processing but often function as black boxes.
• Interpreting neural network decisions is crucial for understanding their mechanisms.
• Lexical stress prediction is a key challenge in speech and language research.

Purpose of the Study:

• To investigate the decision-making process of neural networks in predicting lexical stress.
• To apply interpretability techniques to understand how models identify stress patterns.
• To explore the features that deep learning models utilize for stress prediction in English disyllabic words.

Main Methods:

• Constructed a dataset of English disyllabic words from read and spontaneous speech.
• Trained several convolutional neural network (CNN) architectures to predict stress position.
• Utilized layerwise relevance propagation (LRP) for neural network interpretability analysis.
• Performed feature-specific relevance analysis to identify influential acoustic cues.

Main Results:

• CNNs achieved up to 92% accuracy in predicting stress position.
• Layerwise relevance propagation indicated that stressed versus unstressed syllables, particularly spectral properties of stressed vowels, most influenced predictions.
• The best-performing classifier was significantly influenced by the stressed vowel's first and second formants, with contributions from pitch and third formant.
• Models attended to information throughout the word, not just localized cues.

Conclusions:

• Deep learning models can effectively learn distributed cues for lexical stress from natural speech data.
• Interpretability analysis reveals that CNNs leverage specific acoustic features, like formants, for stress prediction.
• This work extends traditional phonetic studies by using naturally occurring data and deep learning interpretability.