Deep Learning-Based Annotation Transfer between Molecular Imaging Modalities: An Automated Workflow for Multimodal...

Area of Science:

• Multimodal biological imaging
• Computational pathology
• Biomolecular analysis

Background:

• Diverse imaging technologies offer complementary data for complex biological samples.
• Integrating information across modalities is crucial for a holistic understanding.
• Manual or machine learning annotations can bridge data from different imaging techniques.

Purpose of the Study:

• To present a generic method for transferring annotations between microscopy modalities.
• To introduce a fast, general multimodal registration workflow for automatic data alignment.
• To demonstrate the combined power of annotation transfer and registration for investigating biological complexity.

Main Methods:

• Developed a generic annotation transfer method for complementary microscopy modalities.
• Implemented a fast, general multimodal registration workflow evaluated on mass spectrometry imaging (MSI) techniques (MALDI, DESI, REMPI).
• Combined MSI, histological staining (H&E), and deep learning for automatic histology image annotation.

Main Results:

• Achieved an order of magnitude speed-up in multimodal registration compared to previous methods.
• Successfully applied the combined workflows to a pancreatic cancer mouse model.
• Identified metabolically distinct neoplastic pancreatic tissue regions that were histologically indistinguishable.

Conclusions:

• The proposed annotation transfer and multimodal registration workflows enhance the understanding of tumor heterogeneity and the tumor microenvironment.
• Machine learning results can be effectively transferred between modalities, facilitating deeper biological insights.
• This integrated approach provides a powerful tool for complex biological sample analysis.