Reliability of brain-computer interface language sample transcription procedures

We tested the reliability of transcribing language samples of daily brain-computer interface (BCI) communication recorded as language activity monitoring (LAM) logfiles. This study determined interrater reliability and interjudge agreement for transcription of communication of veterans with amyotrophic lateral sclerosis using a P300-based BCI as an augmentative and alternative communication (AAC) system. KeyLAM software recorded logfiles in a universal logfile format during use of BCI-controlled email and word processing applications. These logfiles were encrypted and sent to our laboratory for decryption, transcription, and analysis. The study reports reliability results on transcription of 345 daily logfile samples. The procedure was found to be accurate across transcribers/raters. Frequency of agreement ratios of 97.6% for total number of words and 93.5% for total utterances were found as measures of interrater reliability. Interjudge agreement was 100% for both measures. The results indicated that transcribing language samples using LAM data is highly reliable and the fidelity of the process can be maintained. LAM data supported the transcription of a large number of samples that could not have been completed using audio and video recordings of AAC speakers. This demonstrated efficiency of LAM tools to measure language performance benefits to BCI research and clinical communities.