Best practices in evaluating forced-alignment accuracy: The case of Mandarin varieties

Area of Science:

• Phonetics
• Computational Linguistics
• Speech Technology

Background:

• Forced alignment is crucial for phonetic research, but evaluation standards are inconsistent.
• Existing reliability studies for forced aligners predominantly focus on English, leaving other languages underrepresented.

Purpose of the Study:

• To assess the accuracy of the Montreal Forced Aligner (MFA) for Mandarin speech.
• To compare machine-generated alignments with human expert judgments across different Mandarin varieties.
• To provide methodological recommendations for evaluating forced alignment tools.

Main Methods:

• Machine-generated alignments from MFA were compared against two independent human baseline datasets.
• A Bayesian hierarchical multivariate regression model was employed for statistical analysis.
• Alignment accuracy was evaluated across different sequence types, considering speech rate and speaker variation.

Main Results:

• Human annotators showed higher agreement among themselves than with MFA.
• Significant variations in alignment accuracy were observed across different sequence types.
• Error patterns differed between human annotators and MFA, with some influence from speech rate and speaker characteristics.
• Robustness of alignment showed minimal variation across different Mandarin varieties.

Conclusions:

• Forced alignment demonstrates robustness across different varieties of the same language.
• Methodological improvements are needed for evaluating forced alignment accuracy, particularly for non-English languages.
• The study highlights the importance of human baselines in assessing the reliability of automated speech processing tools.