动态微批量和代币预算调度用于物联网规模管道并行LLM推理.
Juncheol Ahn1, Yubin Son1, Daemin Kim1
1System Software Laboratory, Department of Computer Engineering, Keimyung University, Daegu 42601, Republic of Korea.
Sensors (Basel, Switzerland)
|February 27, 2026
概括
我们为物联网边缘云设置中的大型语言模型 (LLM) 开发了一个运行时适应性调度器. 这种动态调度显著减少了GPU置时间,并提高了LLM推断的吞吐量.
科学领域:
- 计算机科学 计算机科学
- 人工智能的人工智能
- 分布式系统 分布式系统
背景情况:
- 物联网边缘云环境中的大型语言模型 (LLM) 处理各种各样的,不可预测的请求.
- 管道平行LLM推断容易导致微批量失衡和通信延迟,导致GPU置和服务水平目标 (SLO) 违规.
研究的目的:
- 提出一种新的运行时适应性调度器,以优化资源有限的物联网边缘云设置中的LLM推理.
- 为了解决管道平行LLM推断中的微批量失衡和通信摊位.
主要方法:
- 开发了一个调度器,可以动态调整代币预算和微批量大小.
- 优化了预填充和解码工作负载之间的平衡.
- 在不同的网络和计算条件下,尽量减少管道泡.
主要成果:
- 实现了高达55%的GPU置时间的减少.
- 与vLLM和SGLang.com等现有方法相比,通过率提高了1.61倍.
- 增强的时间到第一个令牌 (TTFT) 和代延迟 (ITL) SLO满意度.
结论:
- 动态调度对于物联网边缘云系统中高效稳定的LLM推断至关重要.
- 拟议的自适应调度器有效地减轻了异质请求环境中的性能瓶.
相关概念视频
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
372
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
372
Parallel Processing
819
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
819
Rapidly Varying Flow
561
Rapidly varying flow (RVF) in open channels is characterized by abrupt changes in flow depth over a short distance, with the rate of depth change relative to distance often approaching unity. These flows are inherently complex due to their transient and multi-dimensional nature, making exact analysis difficult. However, approximate solutions using simplified models provide valuable insights into their behavior.Key Features of Rapidly Varying FlowRVF is commonly observed in scenarios involving...
561
