Consistency of evaluators assessing inhaler technique

Since correct inhaler technique is vital for maximum efficacy, the evaluation of technique in patients or health care professionals involved in education programs or research projects is essential. Inconsistency could lead to confused patients and inaccurate research findings. This study is the first to determine interrater consistency with assessing recitation and demonstration of inhaler technique and, if needed, was designed to improve consistency until 95% agreement was achieved. A checklist for correct inhaler technique was constructed by consolidating information from the literature and package inserts. Initially three evaluators were used, however, a fourth evaluator was added for the last two trials to create teams to improve consistency. Percent agreement, probability and kappa scores were calculated. Six trials evaluating inhaler technique of 56 health care professionals at five different sites, revisions of the checklist, and evaluator teams were required to increase overall percent agreement from 81% to 98%. Agreement increased substantially on the verbal component because of specific changes made in the checklist and the objective nature. Demonstration steps consistently had lower agreement than verbal steps. Agreement on the demonstration component was more difficult because of subjectivity involved in observing a subject actuate the inhaler at the correct time, inhale or exhale slowly, and hold one's breath. The relative rankings of percent agreement and kappa scores were the same, however, kappa scores allowed differentiation of agreement due to chance and ability. To achieve interrater consistency, training and practice are required. Interrater consistency should be achieved to ensure accurate reporting of research results on inhaler technique and education of patients.