Psychometric properties of discourse measures in aphasia: acceptability, reliability, and validity

Discourse in adults with aphasia is increasingly the focus of assessment and therapy research. A broad range of measures is available to describe discourse, but very limited information is available on their psychometric properties. As a result, the quality of these measures is unknown, and there is very little evidence to motivate the choice of one measure over another.

To explore the quality of a range of discourse measures, targeting sentence structure, coherence, story structure and cohesion. Quality was evaluated in terms of the psychometric properties of acceptability (data completeness and skewness), reliability (inter- and intra-rater), and validity (content, convergent, discriminant and known groups).

Participants with chronic mild-to-moderate aphasia were recruited from community groups. They produced a range of discourses which were grouped into Cinderella and everyday discourses. Discourses were then transcribed orthographically and analyzed using macro- and microlinguistic measures (Story Grammar, Topic Coherence, Local Coherence, Reference Chains and Predicate Argument Structure-PAS). Data were evaluated against standard predetermined criteria to ascertain the psychometric quality of the measures.

A total of 17 participants took part in the study. All measures had high levels of acceptability, inter- and intra-rater reliability, and had good content validity, as they could be related to a level of the theoretical model of discourse production. For convergent validity, as expected, 8/10 measures correlated with the Western Aphasia Battery-Revised (WAB-R) spontaneous speech scores, and 7/10 measures correlated with the Kissing and Dancing Test (KDT) scores (r ≥ 0.3), giving an overall positive rating for construct validity. For discriminant validity, as predicted, all measures had low correlations with Raven's Coloured Progressive Matrices (RCPM) and WAB-R Auditory Verbal Comprehension scores (r < 0.3), giving an overall positive rating for construct validity. Finally, for known groups validity, all measures indicated a difference between speakers with mild and moderate aphasia except for the Local Coherence measures. Overall, Story Grammar, Topic Coherence, Reference Chains and PAS emerged as the strongest measures in the current study because they achieved the predetermined thresholds for quality in terms of each of the psychometric parameters profiled.

The current study is the first to psychometrically profile measures of discourse in aphasia. It contributes to the field by identifying Story Grammar, Topic Coherence, Reference Chains and PAS as the most psychometrically robust discourse measures yet profiled with speakers with aphasia. Until further data are available indicating the strength of other discourse measures, caution should be applied when using them.