Accurate full segmentation of organs-at-risk in head and neck cancer based on multimodal point cloud fusion

Accurate segmentation of multiple organs is essential for the diagnosis and treatment of head and neck cancer. However, the intricate anatomical structure and dense organ distribution in the head and neck region pose significant challenges for existing automated segmentation models, which predominantly target single organs and rely on single-modality imaging. Achieving comprehensive, one-step segmentation of organs-at-risk (OARs) remains challenging. To this end, we propose a Point-cloud Matrix Fusion-based Segmentation Model (PMFM) that leverages an improved multi-modal data fusion strategy for the automated full segmentation of OARs in head and neck cancer. The proposed PMFM involves three core modules: 1) a camera model-based 3D feature mapping and point-cloud extraction module (PEM) that enables vertical decoupling of modalities and objects; (2) a Point Cloud Matrix Module (PMM) utilizing PointNet and a virtual point cloud-based attention mechanism to facilitate horizontal association and global feature learning across modalities; and (3) a Cross Fusion Module (CFM) based on virtual point clouds to achieve deep intermodal object fusion and enhance inter-organ correlation. PMFM effectively integrates multi-modal image information, transforming them into a unified virtual point cloud matrix, and enables precise, comprehensive segmentation of OARs in head and neck cancer. Extensive validation and comparative experiments on the HaNSeg dataset demonstrate that PMFM significantly outperforms state-of-the-art methods, achieving an average Dice coefficient of 79.8% and an average Hausdorff distance of 2.47 mm. The source code for this study will be publicly available on GitHub at https://github.com/zhouxinyu1028/PMFM.