Hierarchical, model-based merging of multiple fragments for improved three-dimensional segmentation of nuclei

Automated segmentation of fluorescently labeled cell nuclei in three-dimensional confocal images is essential for numerous studies, e.g., spatiotemporal fluorescence in situ hybridization quantification of immediate early gene transcription. High accuracy and automation levels are required in high-throughput and large-scale studies. Common sources of segmentation error include tight clustering and fragmentation of nuclei. Previous region-based methods are limited because they perform merging of two nuclear fragments at a time. To achieve higher accuracy without sacrificing scale, more sophisticated yet computationally efficient algorithms are needed.

A recursive tree-based algorithm that can consider multiple object fragments simultaneously is described. Starting with oversegmented data, it searches efficiently for the optimal merging pattern guided by a quantitative scoring criterion based on object modeling. Computation is bounded by limiting the depth of the merging tree.

The proposed method was found to perform consistently better, achieving merging accuracy in the range of 92% to 100% compared with our previous algorithm, which varied in the range of 75% to 97%, even with a modest merging tree depth of 3. The overall average accuracy improved from 90% to 96%, with roughly the same computational cost for a set of representative images drawn from the CA1, CA3, and parietal cortex regions of the rat hippocampus.

Hierarchical tree model-based algorithms significantly improve the accuracy of automated nuclear segmentation without sacrificing speed.