Small, Open-Source Text-Embedding Models as Substitutes to OpenAI Models for Gene Analysis.

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• Foundation transformer models for gene expression analysis are computationally expensive.
• GenePT uses OpenAI's text-embedding for gene information encoding.
• OpenAI's closed-source model raises data privacy concerns.

Purpose of the Study:

• To investigate open-source transformer-based text-embedding models as alternatives to OpenAI's service.
• To evaluate the performance of lightweight, open-source models for gene expression data analysis.

Main Methods:

• Identified ten small, computationally light transformer models from Hugging Face.
• Evaluated models across four distinct gene classification tasks.
• Assessed the impact of fine-tuning on model performance.

Main Results:

• Several open-source models matched or exceeded OpenAI's performance.
• Model size and computational lightness were key selection criteria.
• Fine-tuning did not consistently yield significant performance improvements.

Conclusions:

• Open-source text-embedding models are viable, efficient, and potentially superior alternatives for gene expression analysis.
• These models mitigate data privacy concerns associated with closed-source solutions.
• Further fine-tuning is often unnecessary for achieving high performance.