Computerized delineation of nuclei in liquid-based pap smears stained with immunohistochemical biomarkers

Infection with high-risk human papillomaviruses (hrHPVs) is a frequent cause of cervical intraepithelial neoplasias and carcinomas. The recently developed p16/Ki67 dual stain of cytologic preparations possesses superior specificity over current HPV testing for detecting moderate- and high-grade dysplasias and can potentially be applied in routine cytology screening. Image analysis can possibly improve the efficiency of evaluating Papanicolaou (Pap) smears, if dual-stained nuclei are accurately localized and reliably distinguished from the background of other cells.

Here, we describe a technique comprising color deconvolution, radial symmetry detector, and a superpixel-based segmentation for computerized delineation of nuclei in Pap smears stained with p16/Ki67.

The performance of the method was determined by the precision and recall in 99 images (n = 19,323 cells) and reached 0.952 and 0.958, respectively. The accuracy of delineation, assessed by the Jaccard index (n = 1,080 cells), was 0.794. In single cells the precision and recall was higher than in clumps (P = 0.005).

In summary, the new technique delineates large and small nuclei irrespectively of coloration with a significantly better performance than a method solely involving the radial symmetry detector. Therefore, it is suited to automatically define nuclear areas for quantification of nuclear biomarkers in smears.