MIPHEI-ViT: Multiplex immunofluorescence prediction from H&E images using ViT foundation models

Area of Science:

• Computational pathology
• Artificial intelligence in oncology
• Biomedical image analysis

Background:

• Hematoxylin and Eosin (H&E) staining is standard for cancer diagnosis, visualizing morphology and architecture.
• Multiplex immunofluorescence (mIF) offers precise cell identification but faces clinical adoption barriers (cost, logistics).
• A gap exists in leveraging routine H&E for detailed cell-type analysis.

Purpose of the Study:

• To develop an AI model (MIPHEI) that predicts mIF signals from H&E images.
• To enable cell-type identification and spatial analysis using only H&E data.
• To bridge the gap between routine H&E and advanced mIF analysis in clinical settings.

Main Methods:

• Developed MIPHEI, a U-Net-inspired architecture with a Vision Transformer (ViT) pathology foundation model encoder.
• Trained MIPHEI on the OrionCRC dataset (colorectal cancer H&E and mIF images).
• Validated MIPHEI on five independent datasets (HEMIT, PathoCell, IMMUcan, Lizard, PanNuke).

Main Results:

• MIPHEI accurately classified cell types from H&E images, achieving high F1 scores for Pan-CK (0.93) and α-SMA (0.83).
• The model demonstrated strong performance for CD3e (0.68) and outperformed baselines for most tested markers.
• Results indicate MIPHEI captures complex relationships between nuclear morphology and cell-type-defining molecular markers.

Conclusions:

• MIPHEI successfully predicts multiplex immunofluorescence signals from H&E images.
• The model facilitates cell-type-aware analysis of large-scale H&E datasets.
• This approach offers a promising pathway to integrate advanced spatial biology insights into routine cancer diagnostics.