Treatment fidelity of technology-enhanced reading therapy (CommuniCATE) for people with aphasia

Treatment fidelity (TF), that is, the degree to which the treatment delivery has adhered to protocol, is an important aspect of establishing treatment validity and reliability. Research has shown that establishing TF is only done in a small percentage of aphasia treatment studies.

This project supports the work of the CommuniCATE study, which explored the benefits of technology-enhanced aphasia therapy on participants' reading, writing, speech and conversation skills. It examines the TF of the Reading strand of the CommuniCATE project by assessing whether the therapy adhered to the protocol. The following research questions were asked: Does treatment delivery adhere to treatment protocol? Does the degree of TF vary according to the person delivering the therapy (i.e. student therapist or qualified therapist)? Does the degree of TF vary over time (early treatment sessions compared with later treatment sessions)? Was the checklist tool reliable?

This study assessed the fidelity of 38 retrospective video recordings of therapy. It used a checklist measure of criteria to which the delivery of the sessions should adhere, and against which the sessions were rated. Participants were the people with aphasia receiving therapy, the students and qualified speech and language therapists delivering therapy, and the independent raters assessing the sessions. A sample of sessions was randomly chosen, including sessions delivered by qualified therapists and by students, and sessions from different time points in the treatment process. The fidelity was rated by the first author, and the fidelity rating calculated as a percentage. Comparisons in fidelity scores for the different variables were drawn using Mann-Whitney tests. The reliability of the checklist was assessed through inter and intra-rater reliability testing, and the results were analysed using Kappa statistics.

High fidelity was found across all therapy conditions with a mean score of 98.2%. Fidelity scores were not affected by the administrator of therapy; sessions delivered by qualified and student therapists were rated equally highly. There was a small but significant effect of time, with later treatment sessions scoring more highly than earlier sessions. However, scores across both periods > 90%. Inter-rater reliability found a high percentage agreement of 93.3% and a Poor Kappa agreement level. Intra-rater agreement found a high percentage agreement of 97.3% and a Fair Kappa agreement level.

The CommuniCATE reading therapy was implemented as per the protocol across time points, and withstood delegation to students. The high fidelity and good reliability scores have positive implications for the study's validity and reliability, and for the study's replication.

What is already known on the subject TF refers to the degree to which the delivery of core components of a treatment matches the implementation guidelines, that is, the adherence to protocol. Despite the acknowledged importance of TF reporting, this is often neglected in the literature. What this paper adds to existing knowledge This paper shows that the TF assessment of the CommuniCATE study (reading strand) found a 98.2% fidelity score, and that high fidelity was not compromised across treatment conditions. This paper outlines the principles of TF and highlights the need for measures to be in place to establish TF, for example, manuals, training and supervision; and to monitor TF, for example, via the use of checklists. This paper also underlines the scarcity of TF measures and checks in aphasia research. This paper therefore serves as a model of TF practice in aphasia therapy research. What are the potential or actual clinical implications of this work? This study contributes to the findings of the CommuniCATE project (reading strand), and the high fidelity findings enhance the validity of the project and indicate that the therapy manual and training enable accurate implementation of delivery. This paper also contributes to the literature on TF evaluation in aphasia studies, which is presently lacking, and highlights the need for increased focus on the optimum strategies of TF reporting.