MMFVS-Net: A triple-symmetric cross-attention network for multimodal optical image fusion and high-accuracy virtual...

Virtual staining is based on label-free imaging technology and deep learning, which enables the efficient and non-destructive generation of virtual stained images with effects similar to those of traditional pathological staining, thus holding great application potential in future clinical medical practice. However, the information provided by a single imaging technology is limited, which is insufficient to support high-accuracy virtual staining.

This study took breast cancer tissue as the research object. Bright-field, auto-fluorescence and phase contrast imaging technology were introduced to simultaneously extract the multimodal complementary information. A multi-branch input network was established to fuse the three types of optical images and generate high-accuracy virtual stained images. To enhance the model's performance, three strategies were employed, including designing a Triple-symmetric cross-attention module to handle the interactive information of multimodal image features, introducing multi-scale discriminator to ensure the model's ability to capture both global features and local details, and applying transfer-learning to address the gradient dilution issue of multi-branches.

Through the evaluation of various image similarity metrics and nuclear statistical metrics, the virtual stained images based on multimodal fusion achieved high accuracy and were significantly superior to any single-modal virtual staining approach. Further ablation experiments explained the reason of the aforementioned results and quantitatively demonstrated the improvements brought by the three optimization strategies.

This work advances the application of multimodal fusion in virtual staining, address the key challenges of cross-modal interactive information mining and multi-branch model training, and further enhance the accuracy and reliability of virtual staining.