Does the Use of Crowdsourced Listeners Yield Different Speech Intelligibility Results Than In-Person Listeners for...

We examined the performance of crowdsourced listeners compared with in-person listeners on the measurement of speech intelligibility for typically developing children. We used three different in-task quality check criteria to screen listeners and examined between-listener intelligibility differences and interrater reliability under each criterion. We also examined how crowdsourced intelligibility results compared with in-person results.

Sixty neurotypical children between ages 2;6 and 9;11 (years;months), drawn from Hustad et al. (2021), contributed speech samples. We used the online platform, Prolific, to collect intelligibility data from five crowdsourced listeners per child (N = 300 total) and compared scores with in-person results from two listeners per child. We used intraclass correlation coefficients (ICCs) and computed pairwise differences among listener groups for each of three in-task quality criteria groups and the in-person group. We modeled intelligibility as a function of listener source (in-person vs. crowdsourced) and child age using mixed-effects regression with smoothing splines.

Lower ICCs and larger between-listener differences were observed for crowdsourced compared to in-person listeners, regardless of in-task quality check criteria, but in-task quality check criteria reduced the disparity. Crowdsourced listeners produced intelligibility scores that were up to 7 percentage points lower than in-person listeners, even under the most stringent in-task quality check criterion. Results showed the same patterns of change with children's age as in-person listener findings. Children with midrange (65%-83%) intelligibilities were the most negatively impacted by the use of crowdsourced listeners.

Rigorous in-task quality check criteria improved crowdsourced listener data. Speakers with midrange intelligibility were the most negatively impacted by the use of crowdsourced listeners, with an intelligibility difference of about 7 percentage points. Intelligibility data obtained with crowdsourced listeners should be interpreted with caution, and future studies should evaluate how crowdsourced intelligibility data differs from in-person data for disordered populations of speakers.

https://doi.org/10.23641/asha.31101451.