Related Experiment Video
Updated: Oct 18, 2025
13:51
Cross-Modal Multivariate Pattern Analysis
Published on: November 9, 2011
20.1K
Localizing category-related information in speech with multi-scale analyses
Sam Tilsen1, Seung-Eun Kim1, Claire Wang1
1Department of Linguistics, Cornell University, Ithaca, New York, United States of America.
Plos One
|October 1, 2021
Summary
This study introduces a novel machine learning method to pinpoint linguistic information within speech signals. It reveals that phonemic and syntactic categories appear earlier and later in speech than previously assumed.
Area of Science:
- Speech processing
- Computational linguistics
- Machine learning
Background:
- Speech production involves high-dimensional physical signals (vocal tract geometry, acoustic energy).
- Linguistic theories propose low-dimensional categories (phonemes, phrase types).
- Quantifying information about categories in speech signals is challenging.
Purpose of the Study:
- Develop a method to localize category-related information in speech signals.
- Investigate the temporal distribution of phonemic/gestural and syntactic information.
- Compare the effectiveness of different machine learning algorithms for this task.
Main Methods:
- A multi-scale analysis approach using machine learning algorithms.
- Systematically restricting the temporal extent of training input to assess classification accuracy.
- Examining linear discriminant analysis (LDA) and long short-term memory (LSTM) neural networks.
- Analyzing phonemic/gestural and syntactic relative clause categories.
Main Results:
- Both LDA and LSTM detected category-related information earlier and later than standard linguistic theories predict.
- LSTM neural networks identified category-related information more effectively than LDA.
- The method successfully localized information related to both phonemic and syntactic categories.
Conclusions:
- Linguistic information is distributed differently in speech signals than traditionally assumed.
- Machine learning, particularly LSTMs, offers powerful tools for analyzing speech information.
- This approach advances our understanding of the relationship between linguistic theory and acoustic speech signals.
