Reaping the Fruits of LLM Pruning: Towards Small Language Models for Efficient Non-Coding Variant Effect Prediction

Area of Science:

• Genomics
• Computational Biology
• Artificial Intelligence

Background:

• Interpreting genetic variants is crucial for precision medicine.
• Large genomic language models (LLMs) struggle with non-coding variant prediction due to computational scaling.
• Layer pruning, successful in natural language processing, can optimize LLMs.

Purpose of the Study:

• To systematically evaluate the contribution of each Transformer layer in genomic LLMs (DNABERT 2, Nucleotide Transformer) for variant prediction.
• To develop pruned, more computationally efficient LLMs by removing non-critical layers.
• To assess the performance of pruned LLMs on a non-coding variant effect prediction benchmark.

Main Methods:

• Systematic layer ablation of DNABERT 2 and Nucleotide Transformer models.
• Building layer importance profiles based on performance changes.
• Fine-tuning pruned and full models on the Enformer eQTL causal variant dataset.
• Comparing performance metrics (accuracy, AUC) and resource usage (training time, memory).

Main Results:

• Layer importance varied significantly across models, with some layers being removable with minimal performance loss.
• Pruned models achieved accuracy and AUC comparable to full models after fine-tuning.
• Pruned models demonstrated substantial reductions in training time and memory requirements.

Conclusions:

• Layer-wise pruning is an effective strategy for creating compact and efficient genomic LLMs.
• Pruned LLMs maintain predictive power while significantly lowering computational demands.
• This approach enhances the accessibility of large-scale non-coding variant analysis for research and clinical applications.