Medical Image Segmentation Assisted with Clinical Inputs via Language Encoder in A Deep Learning Framework

Auto-segmentation of tumor volumes and organs at risk (OARs) is a critical step in cancer radiotherapy treatment planning, where rapid, precise adjustments to treatment plans are required to match the patient anatomy. Although auto-segmentation has been clinically accepted for most OARs, auto-segmentation of tumor volumes, particularly clinical target volumes (CTVs), remains a challenge. This difficulty arises because images alone are often insufficient to capture the necessary information for accurate delineation of microscopic tumor invasion invisible on the image itself.

We propose a deep learning-based medical image segmentation framework designed to mimic the clinical process of delineating CTVs and OARs. At its core, the model performs precise segmentation of medical images while enhancing accuracy by integrating clinical information in text format. A transformer-based text encoder converts textual clinical data into vectors, which are incorporated into the segmentation process with image features. This integration bridges the gap between traditional automated segmentation methods and clinician-guided, context-rich delineations. The framework's effectiveness is demonstrated through a prostate segmentation example in the context of radiation therapy for localized prostate cancer, where incorporating clinical context significantly impacts the delineation process.

In our experiments, we included additional clinical information potentially influencing clinicians' prostate segmentation. The results show that our proposed method not only outperforms the baseline model, but also surpasses current state-of-the-art methods, with or without clinical contexts. Furthermore, our method demonstrates high performance even with limited data.

This proposed segmentation framework has shown to significantly improve auto-segmentation, particularly for CTVs, in cancer radiotherapy.