HONeYBEE: enabling scalable multimodal AI in oncology through foundation model-driven embeddings

Area of Science:

• Oncology
• Biomedical Informatics
• Artificial Intelligence in Medicine

Background:

• Integrating multimodal biomedical data is crucial for advancing oncology research.
• Existing frameworks often struggle to unify diverse data types like clinical, imaging, and molecular profiles.
• Developing robust patient-level representations is key for predictive modeling in cancer.

Purpose of the Study:

• To introduce the Harmonized ONcologY Biomedical Embedding Encoder (HONeYBEE), an open-source framework for multimodal oncology data integration.
• To generate unified patient-level embeddings for diverse oncology applications.
• To evaluate the performance of clinical and multimodal embeddings in tasks such as survival prediction and patient similarity retrieval.

Main Methods:

• Utilized domain-specific foundation models and fusion strategies to process clinical data (structured/unstructured), whole-slide images, radiology scans, and molecular profiles.
• Generated unified patient-level embeddings.
• Evaluated embeddings on The Cancer Genome Atlas (TCGA) dataset (>11,400 patients, 33 cancer types).
• Compared performance of general-purpose and specialized large language models for clinical text representation.

Main Results:

• Clinical embeddings demonstrated superior single-modality performance: 98.5% cancer-type classification accuracy and 96.4% precision@10 for patient retrieval.
• Clinical embeddings achieved the highest survival prediction concordance indices across most cancer types.
• Multimodal fusion offered complementary benefits, improving overall survival prediction beyond clinical features alone for specific cancers.
• General-purpose large language models (e.g., Qwen3) outperformed specialized medical models for clinical text representation, with fine-tuning enhancing performance on heterogeneous data.

Conclusions:

• HONeYBEE effectively integrates multimodal biomedical data for oncology, generating powerful patient-level embeddings.
• Clinical data embeddings are highly effective for classification and retrieval tasks, serving as a strong baseline.
• Multimodal data integration and advanced language models offer significant potential to enhance predictive accuracy and uncover novel insights in cancer research.