Automated Scoring of the Speech Intelligibility Test Using Autoscore

Area of Science:

• Speech-Language Pathology
• Computational Linguistics
• Biomedical Engineering

Background:

• Speech intelligibility testing is crucial for individuals with dysarthria.
• Automated scoring methods can improve efficiency and consistency in analyzing speech.
• Existing automated tools may require refinement for specific clinical tests.

Purpose of the Study:

• To develop and evaluate extensions for Autoscore, an automated transcription scoring tool.
• To adapt Autoscore for scoring the Speech Intelligibility Test (SIT).
• To enhance the accuracy of automated scoring for dysarthric speech.

Main Methods:

• Developed three new rules for Autoscore: compound, contractions, and numbers.
• Utilized 4,642 listener-scored SIT transcripts from dysarthric speakers.
• Compared SIT-extended Autoscore performance against human scorers using intraclass correlations (ICCs).

Main Results:

• SIT-extended Autoscore demonstrated near-identical performance to human scorers (ICCs > .97).
• Disagreements between automated and human scoring were typically minor.
• The extended Autoscore showed a reduction in disagreements compared to the original version.

Conclusions:

• SIT-extended Autoscore offers comparable accuracy to human scoring for dysarthric speech intelligibility.
• The developed extensions significantly improve the utility of Autoscore for SIT.
• This automated tool enhances efficiency and accuracy in speech intelligibility research.