Online crowdsourcing for efficient rating of speech: a validation study

Area of Science:

• Speech-language pathology
• Human-computer interaction
• Crowdsourcing methodology

Background:

• Blinded listener ratings are crucial for assessing speech disorder interventions but are often resource-intensive.
• Online crowdsourcing platforms offer a potential solution for efficient data collection in speech research.

Purpose of the Study:

• To evaluate the validity of speech ratings collected via Amazon Mechanical Turk (AMT).
• To compare the efficiency and reliability of crowdsourced ratings against traditional methods.

Main Methods:

• 100 child speech samples with /r/ misarticulation were rated by trained listeners and AMT participants.
• Bootstrapping analysis compared varying sample sizes of AMT listeners to established standards.
• Listener agreement was assessed between trained and crowdsourced groups.

Main Results:

• Strong agreement was observed between phonetically trained listeners and naive listeners from AMT.
• As few as 9 AMT listeners matched the performance of the 'industry standard' for rating speech.
• Crowdsourcing via AMT demonstrated a valid and efficient approach to speech data rating.

Conclusions:

• Amazon Mechanical Turk (AMT) is a viable tool for obtaining valid speech ratings in communication disorders research.
• Researchers can leverage crowdsourcing to improve the efficiency and reduce the cost of speech data assessment.
• Increased awareness and adoption of AMT can benefit the field of speech-language pathology.