Optimizing speculative decoding via dynamic multi-path and dual-stream networks.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Large language models (LLMs) based on the Transformer architecture face inference efficiency challenges due to sequential token generation.
• Speculative decoding (SD) accelerates LLM inference using small speculative models (SSMs) but is limited by low token acceptance rates.

Purpose of the Study:

• To enhance the inference efficiency and generation accuracy of large language models.
• To overcome the limitations of existing speculative decoding methods, particularly the low acceptance rate of predicted tokens.

Main Methods:

• Proposes a Dual-Stream Network Architecture (DSNA) with parallel streams for word and feature sequences.
• Introduces a dynamic multi-path decoding (DMPD) mechanism for simultaneous evaluation of multiple candidate token paths.
• Progressively fuses outputs from dual streams to improve candidate prediction quality.

Main Results:

• The proposed DSNA method significantly outperforms state-of-the-art speculative decoding approaches.
• Achieves substantial improvements in inference throughput and generation accuracy across multiple benchmarks.
• Demonstrates enhanced quality of candidate predictions through dual-stream modeling and fusion.

Conclusions:

• The DSNA with DMPD effectively addresses the low token acceptance rate challenge in speculative decoding.
• The novel architecture significantly boosts LLM inference speed and output quality.
• Represents a significant advancement in efficient large language model deployment.