Small, open-source text-embedding models as substitutes to OpenAI models for gene analysis.

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• Foundation transformer models for gene expression analysis are computationally expensive.
• GenePT uses OpenAI's text-embedding function for gene information encoding.
• OpenAI's service raises data privacy and accessibility concerns due to its closed-source, online nature.

Purpose of the Study:

• To investigate the feasibility of replacing OpenAI's text-embedding models with open-source alternatives.
• To evaluate the performance of lightweight, open-source transformer models for gene expression data analysis.
• To determine if fine-tuning is necessary for optimal performance of these open-source models.

Main Methods:

• Identified ten small, computationally light transformer-based text-embedding models from Hugging Face.
• Evaluated model performance across four distinct gene classification tasks.
• Compared the performance of open-source models against OpenAI's text-embedding function.

Main Results:

• Several identified open-source models matched or outperformed OpenAI's text-embedding function on gene classification tasks.
• The selected open-source models are smaller, easier to install, and require less computational resources.
• Fine-tuning the open-source models did not consistently yield significant performance improvements.

Conclusions:

• Open-source text-embedding models present a viable and potentially superior alternative to proprietary solutions like OpenAI for gene expression analysis.
• These models offer improved data privacy, cost-efficiency, and accessibility.
• The study suggests that pre-trained open-source models can be effectively used without extensive fine-tuning for gene classification tasks.