Product analysis and initial reliability testing of the total mesorectal excision-quality assessment instrument

Product analysis of rectal cancer resection specimens before specimen fixation may provide an immediate and relevant evaluation of surgical performance. We tested the interrater reliability (IRR) of a product analysis tool called the Total Mesorectal Excision-Quality Assessment Instrument (TME-QA).

Participants included two gold standard raters, five pathology assistants, and eight pathologists. Domains of the TME-QA reflect total mesorectal excision principles including: (1) completeness of mesorectal margin; (2) completeness of mesorectum; (3) coning of distal mesorectum; (4) physical defects; and (5) overall specimen quality. Specimens were scored independently. We used the generalizability theory to assess the tool's internal consistency and IRR.

There were 39 specimens and 120 ratings. Mean overall specimen quality scores for the gold standard raters, pathologists, and assistants were 4.43, 4.43, and 4.50, respectively (p > 0.85). IRR for the first nine items was 0.68 for the full sample, 0.62 for assistants alone, 0.63 for pathologists alone, and 0.74 for gold standard raters alone. IRR for the item overall specimen quality was 0.67 for the full sample, 0.45 for assistants, 0.80 for pathologists, and 0.86 for gold standard raters. IRR increased for all groups when scores were averaged across two raters.

Assessment of surgical specimens using the TME-QA may provide rapid and relevant feedback to surgeons about their technical performance. Our results show good internal consistency and IRR when the TME-QA is used by pathologists. However, for pathology assistants, multiple ratings with the averaging of scores may be needed.