Factors affecting judgment accuracy when scoring children's responses to non-word repetition stimuli in real time

Non-word repetition (NWR) tests are an important way speech and language therapists (SaLTs) assess language development. NWR tests are often scored whilst participants make their responses (i.e., in real time) in clinical and research reports (documented here via a secondary analysis of a published systematic review).

The main aim was to determine the extent to which real-time coding of NWR stimuli at the whole-item level (as correct/incorrect) was predicted by models that had varying levels of detail provided from phonemic transcriptions using several linear mixed method (LMM) models.

Live scores and recordings of responses on the universal non-word repetition (UNWR) test were available for 146 children aged between 3 and 6 years where the sample included all children starting in five UK schools in one year or two consecutive years. Transcriptions were made of responses to two-syllable NWR stimuli for all children and these were checked for reliability within and between transcribers. Signal detection analysis showed that consonants were missed when judgments were made live. Statistical comparisons of the discrepancies between target stimuli and transcriptions of children's responses were then made and these were regressed against live score accuracy. Six LMM models (three normalized: 1a, 2a, 3a; and three non-normalized: 1b, 2b, 3b) were examined to identify which model(s) best captured the data variance. Errors on consonants for live scores were determined by comparison with the transcriptions in the following ways (the dependent variables for each pair of models): (1) consonants alone; (2) substitutions, deletions and insertions of consonants identified after automatic alignment of live and transcribed materials; and (3) as with (2) but where substitutions were coded further as place, manner and voicing errors.

The normalized model that coded consonants in non-words as 'incorrect' at the level of substitutions, deletions and insertions (2b) provided the best fit to the real-time coding responses in terms of marginal R², Akaike's information criterion (AIC) and Bayesian information criterion (BIC) statistics.

Errors that occur on consonants when non-word stimuli are scored in real time are characterized solely by the substitution, deletion and insertion measure. It is important to know that such errors arise when real-time judgments are made because NWR tasks are used to assess and diagnose several cognitive-linguistic impairments. One broader implication of the results is that future work could automate the analysis procedures to provide the required information objectively and quickly without having to transcribe data.

What is already known on this subject Children and patients with a wide range of cognitive and language difficulties are less accurate relative to controls when they attempt to repeat non-words. Responses to non-words are often scored as correct or incorrect at the time the test is conducted. Limited assessments of this scoring procedure have been conducted to date. What this study adds to the existing knowledge Live NWR scores made by 146 children were available and the accuracy of these judgements was assessed here against ones based on phonemic transcriptions. Signal detection analyses showed that live scoring missed consonant errors in children's responses. Further analyses, using linear mixed effect models, showed that live judgments led to consonant substitution, deletion and insertion errors. What are the practical and clinical implications of this work? Improved and practicable NWR scoring procedures are required to provide SaLTs with better indications about children's language development (typical and atypical) and for clinical assessments of older people. The procedures currently used miss substitutions, deletions and insertions. Hence, procedures are required that provide the information currently only available when materials are transcribed manually. The possibility of training automatic speech recognizers to provide this level of detail is raised.