MCFL: Multimodal Collaborative Fusion Learning for Hashimoto's Thyroiditis Recognition

Ultrasound imaging and biochemical examinations are the primary methods for diagnosing Hashimoto's thyroiditis (HT). However, neither of them is sufficient to accurately diagnose HT alone. Most existing multimodal models for HT diagnosis focus primarily on extracting and concatenating features from different modalities, which are ineffective due to the dimensional imbalance of the features between the textual and image data. To address this issue, we propose a novel Multimodal Collaborative Fusion Learning (MCFL) approach, which can enhance and recalibrate the biochemical indicators using ultrasound images, effectively improving the significance and specificity of biochemical indicators for the diagnosis of HT. Specifically, MCFL first constructs a novel INNet to convert the image-level characteristics of the HT ultrasound image into two numerical indicators, i.e., the Local prominent inflammatory (Lpi) and the Global diffuse lesion (Gdl), unifying image data and textual data into a single representation space. Then, a decision tree-based optimization strategy is employed to supervise the training of INNet, interactively recalibrating biochemical indicators with the guidance of the two numerical indicators mentioned above and obtaining a more accurate feature representation of HT. Finally, based on the deep Q-learning framework, a reward mechanism is established to guide the HT diagnostic process, in which the experience replay mechanism and the $\epsilon $ -greedy strategy are utilized collaboratively to improve the accuracy and robustness of the model. Extensive experiments are conducted on a multimodal dataset from multiple medical centers, and the results demonstrate that MCFL achieves state-of-the-art performance, setting a new benchmark.