Hierarchical attention fusion of EUS-doppler features for GISTs risk assessment

Assessing the preoperative malignancy risk of gastrointestinal stromal tumors (GISTs) is crucial for determining the appropriate treatment plan and prognosis. The current automated diagnosis of GISTs based on endoscopic ultrasound (EUS) pose challenge in stable GISTs classification due to the similarity in structure among different risk levels, thus incorporating blood flow density information from Doppler images is essential to assist in the diagnosis. Meanwhile, the variability of tumor size leads to limitations in feature extraction, as a single receptive field is unable to capture both global and local features, which in turn affects classification accuracy. In this paper, we propose a Hierarchical Attention-based Multimodal Feature Fusion Network (HAMNet) for stable GISTs diagnosis by fusing both structural and blood flow information. First, both EUS and Doppler image features are extracted through specific branches to preserve intra-modal information and the masks are added for location supplementary. Second, they are integrated through an iterative multimodal attention integrator (IMAI) which is designed to utilize supervised blood flow information from Doppler images and selectively enhance structural information from EUS images. This is achieved by emphasizing cross-modal complementary features through an attention mechanism and facilitating further refinement of multimodal information through an iteration strategy. Third, we devise a Hierarchical Multi-Scale Tumor Classification (HMTC) module, which enables the model to accommodate the varying sizes of GISTs by capturing features across different receptive fields. Afterwards, we construct a first GISTs dataset called pEUS-Doppler-GISTs comprising 179 cases with 555 paired endoscopic ultrasound (EUS) and Doppler images and conduct experiments to validate the performance of our HAMNet in preoperative malignancy risk assessment. HAMNet has been demonstrated to outperform other state-of-the-art (SOTA) algorithms, achieving an ACC and AUC of 0.875 and 0.856, respectively. It is noteworthy that the model sensitivity is enhanced by a maximum of 0.196 in comparison to other multimodal methods, indicating its effectiveness in identifying high risk tumors and potential application in GISTs CAD systems.