Medical semantic knowledge-integrated multitask learning network for report generation and neoadjuvant chemotherapy...

Area of Science:

• Medical imaging analysis
• Artificial intelligence in healthcare
• Natural language processing for clinical reports

Background:

• Automated reports and predictions of neoadjuvant chemotherapy (NAC) responses aid clinical decisions.
• Previous studies have not fully explored the link between report details and NAC response prediction.

Purpose of the Study:

• Develop a medical semantic knowledge-integrated multitask learning (MSKMTL) network.
• Enhance NAC response prediction accuracy.
• Improve the quality of automatically generated medical reports.

Main Methods:

• Proposed an MSKMTL network leveraging semantic knowledge from medical reports.
• Trained a knowledge base with DCE-MRI and reports to link visual-text representations.
• Used a pretrained BERT model for semantic knowledge encoding and contrastive learning for image-report feature alignment.

Main Results:

• Achieved strong performance in report generation (BLEU-1: 0.408, BLEU-4: 0.186, ROUGE: 0.426, METEOR: 0.402).
• Demonstrated high accuracy in NAC prediction (AUC: 0.849, Recall: 0.750, Precision: 0.795, F1: 0.794).
• Showcased robust results on both breast DCE-MRI and public IU X-ray datasets.

Conclusions:

• Integrating medical semantic knowledge significantly improves NAC prediction accuracy.
• The proposed method generates superior medical reports compared to existing methodologies.