Dynamic Micro-Batch and Token-Budget Scheduling for IoT-Scale Pipeline-Parallel LLM Inference

Area of Science:

• Computer Science
• Artificial Intelligence
• Distributed Systems

Background:

• Large language models (LLMs) in IoT-edge-cloud environments handle diverse, unpredictable requests.
• Pipeline-parallel LLM inference is susceptible to micro-batch imbalance and communication delays, leading to GPU idleness and Service Level Objective (SLO) violations.

Purpose of the Study:

• To propose a novel runtime-adaptive scheduler for optimizing LLM inference in resource-constrained IoT-edge-cloud settings.
• To address micro-batch imbalance and communication stalls in pipeline-parallel LLM inference.

Main Methods:

• Developed a scheduler that dynamically adjusts token budgets and micro-batch sizes.
• Optimized the balance between prefill and decoding workloads.
• Minimized pipeline bubbles under varying network and compute conditions.

Main Results:

• Achieved up to 55% reduction in GPU idle time.
• Improved throughput by up to 1.61 times compared to existing methods like vLLM and SGLang.
• Enhanced Time-To-First-Token (TTFT) and Iteration Latency (ITL) SLO satisfaction.

Conclusions:

• Dynamic scheduling is crucial for efficient and stable LLM inference in IoT-edge-cloud systems.
• The proposed adaptive scheduler effectively mitigates performance bottlenecks in heterogeneous request environments.