From diagnostic labels to radiology reports: a unified multi-modal framework for lesion detection and segmentation

Deep learning-based vision models are playing an increasingly pivotal role in clinical diagnosis and treatment. However, existing approaches predominantly rely on visual information, often neglecting the accompanying radiology reports. Even when textual data is considered, current methods typically restrict their input to discrete text labels, failing to exploit the comprehensive, fine-grained semantic information embedded in full-text real-world radiology reports. This limitation represents a significant underutilization of available training resources. Furthermore, effectively leveraging complementary information from multi-modal imaging and exploiting the synergy between detection and segmentation remain open challenges. To address these limitations, we propose TextDSNet, a unified multi-task framework for joint medical detection and segmentation, incorporating radiology reports. Specifically, TextDSNet leverages reports as cross-modal guidance to orchestrate multi-modal image analysis. We formulate this process as a unified framework termed 'Lesion-Aware Segmentation,' which enables the model to identify specific lesion targets, facilitating the joint optimization of coarse-grained localization and fine-grained segmentation. To overcome the scarcity of public datasets containing paired full-text reports, we curated radiology-style report annotations for the ISLES 2022 dataset under strict clinical supervision. Experimental results show that TextDSNet surpasses existing state-of-the-art methods on this dataset. Our findings demonstrate that integrating full-text radiology reports is a viable path toward building more intelligent and clinically aligned medical AI systems. Code and dataset are available athttps://github.com/wenxuan163/TextDSNet.git.