Efficient State Management for Scaling Out Stateful Operators in Stream Processing Systems

Many big data applications require real-time analysis of continuous data streams. Stream Processing Systems (SPSs) are designed to act on real-time streaming data using continuous queries consisting of interconnected operators. The dynamic nature of data streams, for example, fluctuation in data arrival rates and uneven data distribution, can cause an operator to be a bottleneck one. Scalability is an important factor in SPS, but detecting bottleneck operator correctly and scaling it without affecting application execution are challenging. A stateful operator such as aggregation or join makes scaling operation more difficult as it involves state management. Current research does not address the issue of scaling stateful operators efficiently as mostly stop application for handling state, which results in significant overheads to the performance. In this article, the key idea is to detect bottleneck operator correctly using the runtime bottleneck detection approach and then scale out this operator and manage its internal state in a way that we can achieve almost zero latency. During the bottleneck detection process, we have defined alarming_threshold, a parameter for the operators that can be bottleneck operators in the future and scale_out_threshold, when the operator is bottleneck. To scale out, we have presented two techniques, active backup and checkpointing, the former one will start a Secondary Execution (SE) in back end by partitioning state and input streams to multiple nodes at alarming_threshold; this SE will replace primary node at scale_out_threshold. In the latter technique, a State Manager (SM) module will start state checkpointing at alarming_threshold to external store and perform scale out by managing state and input stream at scale_out_threshold. The first approach will help us to achieve almost zero latency goal, while the latter one is a resource efficient technique. Our results show that both techniques are working while providing desired goals of reducing overall latency during scale out and improving resource utilization.