Automated recognition of atypical nuclei in breast cancer cytology specimens by iterative image transformations

In order to develop an objective grading system for nuclear atypia in breast cancer, an image analysis technique has been applied for the automated recognition of enlarged and hyperchromatic nuclei in cytology specimens. The image segmentation algorithm, based on the 'top hat' image transformation developed in mathematical morphology, is implemented on the LEYTAS automated microscope system. The performance of the segmentation algorithm has been evaluated for fifty malignant and eighty-five benign breast lesions by visual inspection of the displayed 'flagged' objects. The mean number of flagged objects per 1600 image fields for breast cancers was 887 (range 0-7920) of which 87% consisted of single, atypical nuclei. For benign lesions the mean number was 30 (range 0-307) of which 20% were single nuclei. By adaptation of the 'top hat' parameter values, a more extreme subpopulation of atypical nuclei could be discriminated. The large interspecimen variation in the breast cancer results was related to differences in DNA content distribution and mean nuclear area, determined independently with scanning cytophotometry, and to some extent with the histological type.