A scalable deep attention mechanism of instance segmentation for the investigation of chromosome

Chromosome segmentation in metaphase images is a critical yet challenging task in cytogenetics and genomics due to the inherent complexity, variability in chromosome shapes, and the scarcity of high-quality annotated datasets. This study proposes a robust instance segmentation framework that integrates an automated annotation pipeline with an enhanced deep learning architecture to address these challenges. A novel dataset is introduced, comprising metaphase images and corresponding karyograms, annotated with precise instance segmentation information across 24 chromosome classes in COCO format. To overcome the labor-intensive manual annotation process, a feature-based image registration technique leveraging SIFT and homography is employed, enabling the accurate mapping of chromosomes from karyograms to metaphase images and significantly improving annotation quality and segmentation performance. The proposed framework includes a custom Mask R-CNN model enhanced with an Attention-based Feature Pyramid Network (AttFPN), spatial attention mechanisms, and a LastLevelMaxPool block for superior multi-scale feature extraction and focused attention on critical regions of the image. Experimental evaluations demonstrate the model's efficacy, achieving a mean average precision (mAP) of 0.579 at IoU = 0.50:0.95, surpassing the baseline Mask R-CNN and Mask R-CNN with AttFPN by 3.94% and 5.97% improvements in mAP and AP50, respectively. Notably, the proposed architecture excels in segmenting small and medium-sized chromosomes, addressing key limitations of existing methods. This research not only introduces a state-of-the-art segmentation framework but also provides a benchmark dataset, setting a new standard for chromosome instance segmentation in biomedical imaging. The integration of automated dataset creation with advanced model design offers a scalable and transferable solution, paving the way for tackling similar challenges in other domains of biomedical and cytogenetic imaging.