Accelerating inference in genomic and proteomic foundation models via speculative decoding

Area of Science:

• Computational Biology
• Bioinformatics
• Machine Learning

Background:

• Genomic and protein foundation models (GFMs and PFMs) excel at learning biological sequence languages.
• Autoregressive decoding limits large-scale sequence generation due to high latency and computational cost.
• Efficient generation of long DNA and protein sequences is crucial for various biological applications.

Purpose of the Study:

• To adapt and evaluate speculative decoding for accelerating GFMs and PFMs.
• To assess the impact of speculative decoding on sequence generation speed and prediction quality.
• To demonstrate the model-agnostic applicability of speculative decoding in biological sequence modeling.

Main Methods:

• Adapted a probabilistic variant of speculative decoding for GFMs and PFMs.
• Utilized a lightweight draft model to propose token spans for parallel verification by a larger target model.
• Systematically studied parameters like speculation window length, temperature, draft architecture, and prompt length.
• Benchmarked generation speed (tokens per second) across multiple configurations.

Main Results:

• Speculative decoding achieved consistent speedups over standard decoding methods.
• Observed maximum speedups of 100% and average gains of 20-40% (1.2x-1.4x) across tested models.
• Demonstrated that speculative decoding preserves the target model's sampling distribution and prediction accuracy.
• Confirmed practical and model-agnostic acceleration for genomic and proteomic sequence generation.

Conclusions:

• Speculative decoding is an effective strategy for accelerating large genomic and protein foundation models.
• The method provides significant speed improvements without compromising the quality of generated sequences.
• This approach enhances the feasibility of large-scale sequence generation in bioinformatics and computational biology.