HistoNeXt: dual-mechanism feature pyramid network for cell nuclear segmentation and classification

To develop an end-to-end convolutional neural network model for analyzing hematoxylin and eosin(H&E)-stained histological images, enhancing the performance and efficiency of nuclear segmentation and classification within the digital pathology workflow.

We propose a dual-mechanism feature pyramid fusion technique that integrates nuclear segmentation and classification tasks to construct the HistoNeXt network model. HistoNeXt utilizes an encoder-decoder architecture, where the encoder, based on the advanced ConvNeXt convolutional framework, efficiently and accurately extracts multi-level abstract features from tissue images. These features are subsequently shared with the decoder. The decoder employs a dual-mechanism architecture: The first branch of the mechanism splits into two parallel paths for nuclear segmentation, producing nuclear pixel (NP) and horizontal and vertical distance (HV) predictions, while the second mechanism branch focuses on type prediction (TP). The NP and HV branches leverage densely connected blocks to facilitate layer-by-layer feature transmission and reuse, while the TP branch employs channel attention to adaptively focus on critical features. Comprehensive data augmentation including morphology-preserving geometric transformations and adaptive H&E channel adjustments was applied. To address class imbalance, type-aware sampling was applied. The model was evaluated on public tissue image datasets including CONSEP, PanNuke, CPM17, and KUMAR. The performance in nuclear segmentation was evaluated using the Dice Similarity Coefficient (DICE), the Aggregated Jaccard Index (AJI) and Panoptic Quality (PQ), and the classification performance was evaluated using F1 scores and category-specific F1 scores. In addition, computational complexity, measured in Giga Floating Point Operations Per Second (GFLOPS), was used as an indicator of resource consumption.

HistoNeXt demonstrated competitive performance across multiple datasets: achieving a DICE score of 0.874, an AJI of 0.722, and a PQ of 0.689 on the CPM17 dataset; a DICE score of 0.826, an AJI of 0.625, and a PQ of 0.565 on KUMAR; and performance comparable to Transformer-based models, such as CellViT-SAM-H, on PanNuke, with a binary PQ of 0.6794, a multi-class PQ of 0.4940, and an overall F1 score of 0.82. On the CONSEP dataset, it achieved a DICE score of 0.843, an AJI of 0.592, a PQ of 0.532, and an overall classification F1 score of 0.773. Specific F1 scores for various cell types were as follows: 0.653 for malignant or dysplastic epithelial cells, 0.516 for normal epithelial cells, 0.659 for inflammatory cells, and 0.587 for spindle cells. The tiny model's complexity was 33.7 GFLOPS.

By integrating novel convolutional technology and employing a pyramid fusion of dual-mechanism characteristics, HistoNeXt enhances both the precision and efficiency of nuclear segmentation and classification. Its low computational complexity makes the model well suited for local deployment in resource-constrained environments, thereby supporting a broad spectrum of clinical and research applications. This represents a significant advance in the application of convolutional neural networks in digital pathology analysis.