Inter-observer agreement in interpreting intraoperative ultrasonography during brain tumour surgery

Intraoperative ultrasonography (iUS) is a powerful technology that is being increasingly utilized for brain tumour surgery. However, ultrasonography has been documented to be observer dependent in a range of healthcare settings. Here we objectively assess the degree of inter-observer variability in iUS for brain tumour surgery.

Nine images taken from routinely collected iUS videos from brain tumour surgery were presented to 18 neurosurgeons (5 consultants, 7 senior fellows, 6 residents). This included three tumour types [metastasis, high-grade (HGG) and low-grade glioma (LGG)] at three operative stages (before, during and near resection completion). Using 3D Slicer, observers segmented what they deemed to be tumour. Digital Image Correlation Engine Similarity Coefficients (DSC) were calculated to examine inter-observer variability.

A total of 1,377 DSCs were calculated between 18 observers across 9 images. Metastasis had the highest DSC (0.72 ± 0.32), followed by HGG (0.64 ± 0.33) and LGG (0.58 ± 0.25; p < 0.00001). As the resection progressed, the degree of inter-observer agreement broke down. Before resection the DSC was 0.87 ± 0.11; during resection (0.74 ± 0.17) and at completion (0.32 ± 0.27; p < 0.00001). The trend of decreasing agreement as the resection progressed held across tumour types. Observers reported increasing difficulty with iUS interpretation as the resection proceeded and there was statistically significant (p = 0.014) negative correlation (-0.775) between DSC and difficulty rating of the segmentation.

Here we demonstrate significant inter-observer variability in iUS for brain tumour surgery. The degree of variability is tumour-type and operative stage dependent. This work adds weight to the value of building machine learning augmented iUS for brain tumour surgery.