Dataset of audiovisual unscripted monologues with speech annotations

Area of Science:

• Linguistics
• Speech Science
• Computational Linguistics

Background:

• High-quality, annotated speech data is crucial for advancing speech processing and linguistic research.
• Existing datasets often lack the multimodal synchronization and fine-grained temporal resolution needed for detailed analysis of spontaneous speech.

Purpose of the Study:

• To present a novel, publicly accessible multimodal dataset of unscripted German monologues.
• To provide millisecond-level time-resolved linguistic annotations for audiovisual speech data.
• To facilitate research in acoustic-phonetic analysis, neurolinguistics, and speech technology development.

Main Methods:

• Audiovisual recordings of spontaneous German monologues from six speakers (over 300 minutes).
• Data collection involved controlled laboratory conditions with variations in background noise and visual cues.
• An annotation pipeline combining OCTRA, G2P, MAUS, PHO2SYL, and RFTagger generated orthographic, phonological, phonetic, syllabic, and part-of-speech annotations.

Main Results:

• A comprehensive multimodal dataset with synchronized audio, video, and detailed linguistic annotations at millisecond resolution.
• Inclusion of metadata files (JSON, HED-formatted events.tsv) compatible with neuroimaging standards like BIDS.
• Dataset structure ensures reliable linkage between audiovisual recordings and speech annotations.

Conclusions:

• The presented dataset offers a valuable resource for diverse research applications, including acoustic-phonetic analysis and neurolinguistic studies.
• Its naturalistic content and high temporal precision support detailed examination of spontaneous speech.
• The resource promotes replication-oriented research and the evaluation of speech-processing tools.