STonKGs: a sophisticated transformer trained on biomedical text and knowledge graphs

Area of Science:

• Biomedical informatics
• Machine learning
• Bioinformatics

Background:

• Biomedical knowledge is predominantly stored in structured databases and unstructured scientific text.
• Existing machine learning applications often rely on single data modalities (text or structured data), limiting representation power.
• Natural Language Processing (NLP) and knowledge graph embedding models are common but have inherent limitations.

Purpose of the Study:

• To develop a sophisticated Transformer model, STonKGs, for enhanced representation of biomedical knowledge.
• To leverage multimodal learning by combining biomedical text and knowledge graphs (KGs).
• To improve biological applications by learning joint representations in a shared embedding space.

Main Methods:

• Developed STonKGs, a multimodal Transformer integrating structured KG data and unstructured biomedical literature.
• Pre-trained STonKGs on a large knowledge base of millions of text-triple pairs.
• Benchmarked STonKGs against unimodal baselines across eight diverse biological classification tasks.

Main Results:

• STonKGs significantly outperformed baseline models trained on single modalities.
• The performance improvement was particularly notable on more challenging classification tasks.
• Achieved an F1-score improvement of up to 0.084 (from 0.881 to 0.965) over the best baseline.

Conclusions:

• Multimodal learning with STonKGs offers a powerful approach for representing complex biomedical knowledge.
• The STonKGs architecture and pre-trained models are adaptable for various transfer learning applications in biology.
• The study demonstrates the advantage of integrating diverse data sources for advancing biological data analysis.