Similarity structure in visual speech perception and optical phonetic signals

Area of Science:

• Auditory and Speech Science
• Perception Psychology
• Biomechanical Engineering

Background:

• Understanding visual phonetic perception, or lipreading, requires connecting perceptual outcomes to physical stimulus characteristics.
• The talking face is a complex stimulus, making it challenging to identify key physical properties influencing perception.
• Previous research has explored the relationship between visual speech and auditory perception, but a robust link to physical attributes remains an area of investigation.

Purpose of the Study:

• To investigate the second-order isomorphism between perceptual similarities of speech syllables and physical similarities of the stimuli.
• To determine if three-dimensional (3-D) facial motion capture data can predict perceptual similarities in visual phonetic perception.
• To establish a foundation for controlled studies examining first-order relationships between visual phonetic perception and physical stimulus attributes.

Main Methods:

• Four talkers produced speech syllables with 23 initial consonants and 3 vowels.
• Six normal-hearing participants identified syllables in a visual-only condition.
• Perceptual dissimilarity was measured using multidimensional scaling; physical dissimilarity used 3-D optical motion capture data of facial points.

Main Results:

• A robust second-order relationship was found between the 3-D facial point representation of visible speech and perceptual effects.
• The variance accounted for in the relationship between perceptual and physical dissimilarities ranged from 46% to 66% across talkers.
• The variance accounted for ranged from 49% to 64% across vowels when using weighted dissimilarities and full 3-D optical data.

Conclusions:

• The sparse 3-D point representation of visible speech is a viable method for studying visual phonetic perception.
• This approach provides a strong basis for future controlled studies on the relationship between visual phonetic perception and physical stimulus attributes.
• The findings support the use of 3-D facial motion capture in understanding the mechanics of lipreading and speech perception.