Related Experiment Video
Updated: Jun 10, 2026
Quantifying Microglia Morphology from Photomicrographs of Immunohistochemistry Prepared Tissue Using ImageJ
Published on: June 5, 2018
Interobserver Variability Across Whole-Slide Imaging Systems
Context.—:
Digital pathology whole-slide images (WSIs) must be of adequate quality for use in diagnostics and the development of artificial intelligence algorithms. Inadequate slides should be re-scanned. There is insufficient data on what qualifies as WSI adequacy.
Objective.—:
To report on the degree of inter-cohort agreement when evaluating WSI quality and adequacy for use in primary diagnostics.
Design.—:
The 60 validation slides were scanned by 3 scanners (Leica GT450, Hamamatsu NanoZoomer S360, and Huron TissueScope iQ). The resulting 180 WSIs were independently evaluated on a quality pass or fail basis by 12 observers representing 4 staff members of the following: in-house attending physicians, fellows, and imaging technicians. The Cohen's kappa was calculated overall and within each cohort, and the acceptance rate was compared between cohorts.
Results.—:
For almost every cohort on each scanner, the Cohen's kappa was weak (0.21-0.4) or very poor (<0.2). Based on WSI pass rates, the data yielded different scanner rankings across cohorts, with the GT450, TissueScope iQ, and NanoZoomer S360 exhibiting the highest pass rates for technicians, fellows, and attendings, respectively.
Conclusions.—:
There is significant interobserver and inter-cohort variability in the rating of WSIs for suitability in diagnostic pathology. Setting uniform, reproducible standards is important for patient safety, creating a uniform level of care, efficiently re-scanning subpar slides, developing artificial intelligence models on images characteristic of those used in clinical practice, and accurately representing the quality of a scanner.
Insights
Evaluating whole-slide images (WSIs) for diagnostic quality shows significant variability among observers and scanners. Establishing uniform standards is crucial for patient safety and AI development in digital pathology.
Area of Science:
- Digital Pathology
- Medical Imaging
- Artificial Intelligence in Healthcare
Background:
- Whole-slide images (WSIs) are critical for diagnostics and AI development.
- Adequate WSI quality is essential, but standards for adequacy are not well-defined.
- Inadequate slides require re-scanning, highlighting the need for clear quality benchmarks.
Purpose of the Study:
- To assess inter-cohort agreement in evaluating WSI quality and diagnostic adequacy.
- To identify variability in WSI quality assessment across different observer groups and scanners.
Main Methods:
- 180 WSIs from 60 slides scanned on three different platforms (Leica GT450, Hamamatsu NanoZoomer S360, Huron TissueScope iQ) were evaluated.
- 12 observers from four groups (attending physicians, fellows, technicians) assessed WSIs for quality (pass/fail).
- Cohen's kappa was calculated to measure interobserver agreement; WSI pass rates were compared across cohorts and scanners.
Main Results:
- Interobserver agreement was generally weak to very poor (Cohen's kappa 0.21-0.4 or <0.2) across most cohorts and scanners.
- Scanner performance rankings varied significantly depending on the observer cohort.
- Technicians favored the GT450, fellows the TissueScope iQ, and attending physicians the NanoZoomer S360.
Conclusions:
- Significant variability exists in WSI quality ratings among observers and cohorts.
- Establishing uniform, reproducible WSI quality standards is vital for patient safety, consistent care, and reliable AI model development.
- Standardization will improve the efficiency of re-scanning subpar slides and ensure AI models are trained on clinically relevant images.
Related Concept Videos
01:26Phase Contrast and Differential Interference Contrast Microscopy
In-phase-contrast microscopes, interference between light directly passing through a cell and light refracted by cellular components is used to create high-contrast, high-resolution images without staining. It is the oldest and simplest type of microscope that creates an image by altering the wavelengths of light rays passing through the specimen. Altered wavelength paths are created using an annular stop in the condenser. The annular stop produces a hollow cone of...
01:18Imaging Biological Samples with Optical Microscopy
In optical microscopy, the specimen to be viewed is placed on a glass slide and clipped on the stage...
01:16Confocal Fluorescence Microscopy