A Tutorial on Reliability Testing in AAC Language Sample Transcription and Analysis

Establishing reliability is an essential step in language sample transcription and analysis. This tutorial provides an illustration of replicable procedures for reliability testing during transcription and analysis of language samples generated by people who use augmentative and alternative communication (AAC) systems. Statistical measures used for testing agreement between raters coding categorical data are summarized. Detailed procedures for reliability testing in AAC language sample transcription and analysis are provided, beginning with the collection of raw language sample data. Procedures include guidelines for (a) establishing inter-judge agreement during the transcription process, and (b) using Cohen's kappa to establish inter-rater reliability during deeper analysis of transcribed utterances. All procedures are demonstrated in a case example using language samples from children who use AAC.