Bilateral network with text guided aggregation architecture for lung infection image segmentation

Objective.Lung image segmentation is a crucial problem for autonomous understanding of the potential illness. However, existing approaches lead to a considerable decrease in accuracy for lung infection areas with varied shapes and sizes. Recently, researchers aimed to improve segmentation accuracy by combining diagnostic reports based on text prompts and image vision information. However, limited by the network structure, these methods are inefficient and ineffective.Method.To address this issue, this paper proposes a Bilateral Network with Text Guided Aggregation Architecture (BNTGAA) to fully fuse local and global information for text and image vision. This proposed architecture involves (i) a global fusion branch with a Hadamard product to align text and vision feature representation and (ii) a multi-scale cross-fusion branch with positional coding and skip connection, performing text-guided segmentation in different resolutions. (iii) The global fusion and multi-scale cross-fusion branches are combined to feed a mamba module for efficient segmentation.Results.Extensive quantitative and qualitative evaluations demonstrate that the proposed architecture performs better both in accuracy and efficiency. Our architecture outperforms the current best methods on the QaTa-COVID19 dataset, improving mIoU and Dice scores by 3.08% and 2.35%, respectively. Meanwhile, our architecture surpasses the computational speed of existing multimodal networks. Finally, the architecture has a quick convergence and generality. It can exceed the performance of the current best methods even if it is trained with only 50% of the dataset.Conclusion.With the backbone Mamba, the proposed fusion architecture, which performs text-guided aggregation under different scales, can greatly improve segmentation performance both in accuracy and efficiency. Codes are available at https://github.com/Meihanxiao/BNTGAA.