Contrastive learning and mixture of experts enables precise vector embeddings in biological databases.

Area of Science:

• Biomedical Informatics
• Natural Language Processing
• Machine Learning

Background:

• Transformer neural networks excel at sentence similarity but struggle with complex scientific documents.
• Effective vector representations are crucial for retrieval augmentation and searching diverse research literature.
• Existing models generate sub-optimal embeddings for structurally and thematically varied scientific texts.

Purpose of the Study:

• To improve vector embeddings for scientific text, particularly in biomedical domains.
• To develop a novel Mixture of Experts (MoE) extension pipeline for pretrained BERT models.
• To enhance the representation learning of heterogeneous biomedical inputs for efficient search and compilation.

Main Methods:

• Assembled domain-specific datasets using co-citations as a similarity metric in biomedical domains.
• Introduced a novel Mixture of Experts (MoE) extension pipeline applied to pretrained BERT models.
• Trained MoE variants to classify co-cited publications based on scientific abstracts, utilizing a unique routing scheme.

Main Results:

• The Mixture of Experts (MoE) variants demonstrated improved vector embeddings for scientific text.
• The unique routing scheme ensured the MoE system maintained the same throughput as regular transformers.
• The methodology shows promise for encoding heterogeneous biomedical inputs efficiently.

Conclusions:

• The proposed MoE extension pipeline offers a versatile and efficient solution for encoding diverse biomedical documents.
• Advancements in representation learning can significantly enhance vector database search and compilation for scientific literature.
• This approach paves the way for One-Size-Fits-All transformer networks in scientific information retrieval.