TGMT: A Terminology-Guided Multi-Task Approach for Lung Cancer CT Report Generation

Computed tomography (CT) is an indispensable examination of lung cancer screening, diagnosis, and staging. Writing accurate and comprehensive CT reports takes years of diagnostic experience and extensive medical knowledge and is usually tedious, time-consuming, and error-prone for radiologists. In this study, we proposed a Terminology-Guided Multi-Task (TGMT) approach to automatically generate CT reports for lung cancer patients. The TGMT method first applied a multitask learning strategy to detect the tumor in the image and predict its characteristics to learn better visual feature. Then, a terminological label guidance strategy was employed to explicitly utilize the predicted tumor characteristics to guide the report generation. We conducted experiments using 694 patients' data collected from Peking University Cancer Hospital. The experimental results indicate that the TGMT outperformed the state- of-the-art baselines with a BLEU1 value of 58.75, a BLEU4 value of 39.84, a METEOR value of 30.01, a ROUGE-L value of 52.61 and a CIDER value of 16.47. Moreover, the accuracy of the description of tumor characteristics in the generated reports was also improved by 10%. Based on the experimental results, we can conclude that the TGMT can learn more informative visual feature using the multi-task learning strategy and generate more accurate reports under the terminological label guidance.