Development, intra- and inter-rater reliability of the New Zealand Secretion Scale (NZSS)

Purpose: The New Zealand Secretion Scale (NZSS) has been developed for comprehensive assessment of accumulated secretions during endoscopy. The scale rates secretion severity under the subcategories location, amount and response. This study describes the scale's development and reliability when used by experts and untrained raters. Method: One expert scored 254 endoscopy videos using the NZSS and performed repeat measures on 100 randomly selected videos one month later. These 100 videos were scored by a second expert in a randomised order. In a second arm of the study, 28 raters scored 10 endoscopy videos, without training on the NZSS. Seventeen had experience in endoscopy (mean 1.8 years, SD = 1.0). Reliability was calculated across the entire cohort and as a function of experience interpreting endoscopy. Result: Strong internal consistency (Cronbach's α = 0.88), and high inter-item (>0.60) and corrected item-total correlations (>0.70) were established. Inter-rater (ICC = 0.99) and intra-rater reliability (ICC = 0.95) of the experts was excellent. Inter-rater reliability of the untrained raters ranged from ICC = 0.65-0.79, with no significant influence of experience on reliability. Conclusion: The NZSS is a reliable assessment of secretion severity during endoscopy and can be used without training. This comprehensive scale will support research evaluating the predictive value of accumulated secretions.