Overlapping cell nuclei segmentation using a spatially adaptive active physical model

A method for the segmentation of overlapping nuclei is presented, which combines local characteristics of the nuclei boundary and a priori knowledge about the expected shape of the nuclei. A deformable model whose behavior is driven by physical principles is trained on images containing a single nuclei, and attributes of the shapes of the nuclei are expressed in terms of modal analysis. Based on the estimated modal distribution and driven by the image characteristics, we develop a framework to detect and describe the unknown nuclei boundaries in images containing two overlapping nuclei. The problem of the estimation of an accurate nucleus boundary in the overlapping areas is successfully addressed with the use of appropriate weight parameters that control the contribution of the image force in the total energy of the deformable model. The proposed method was evaluated using 152 images of conventional Pap smears, each containing two overlapping nuclei. Comparisons with other segmentation methods indicate that our method produces more accurate nuclei boundaries which are closer to the ground truth.