Efficient State Management for Scaling Out Stateful Operators in Stream Processing Systems.

Area of Science:

• Computer Science
• Data Engineering
• Distributed Systems

Background:

• Big data applications increasingly rely on real-time analysis of continuous data streams.
• Stream Processing Systems (SPSs) use interconnected operators for continuous queries on streaming data.
• Dynamic data stream characteristics can lead to bottleneck operators, hindering scalability.

Purpose of the Study:

• To address the challenge of detecting and scaling bottleneck operators in SPSs, particularly stateful ones, without significant performance overhead.
• To achieve near-zero latency during the scaling process for stateful operators.
• To present efficient and resource-aware techniques for handling stateful operator scaling.

Main Methods:

• Runtime bottleneck detection using defined alarming_threshold and scale_out_threshold parameters.
• Two scaling techniques: active backup (Secondary Execution with state/stream partitioning) and checkpointing (State Manager with external state storage).
• Implementation and evaluation of both techniques to assess latency reduction and resource utilization.

Main Results:

• Both active backup and checkpointing techniques effectively reduce overall latency during scale-out operations.
• The active backup method achieves near-zero latency goals.
• The checkpointing method offers a resource-efficient alternative while improving utilization.

Conclusions:

• Runtime detection and novel scaling strategies enable efficient handling of bottleneck stateful operators in SPSs.
• The proposed techniques significantly improve scalability and performance of real-time data stream processing.
• Near-zero latency and resource efficiency are achievable through tailored stateful operator scaling approaches.