Predictive topology refinements in distributed stream processing system

Area of Science:

• Computer Science
• Data Science
• Cloud Computing

Background:

• Cloud computing has led to a surge in data volume, driving interest in big data analytics and streaming applications.
• Maintaining Quality of Service (QoS) to meet Service Level Agreements (SLAs) cost-effectively is challenging due to fluctuating workloads.
• Workload prediction is crucial for optimizing cloud-based streaming systems.

Purpose of the Study:

• To present a novel topology-refining scheme for streaming systems that incorporates workload prediction.
• To enhance the performance and robustness of streaming systems against dynamic workloads.
• To ensure cost-effective achievement of QoS goals within SLA constraints.

Main Methods:

• Developed a workload prediction model combining Support Vector Regression (SVR), autoregressive, and moving average models with a feedback mechanism.
• Implemented a topology-refining scheme that dynamically adapts to predicted workloads.
• Utilized Apache Flink as a testbed for evaluating the proposed system.

Main Results:

• The proposed prediction scheme effectively handles both synthetic and real-world workload traces.
• The topology-refining scheme demonstrated robustness in adapting to incoming workloads.
• The system successfully met QoS goals and SLA constraints.

Conclusions:

• Workload prediction is a viable strategy for improving the performance of cloud-based streaming systems.
• The novel topology-refining scheme offers a robust solution for dynamic workload management.
• The approach ensures cost-effective QoS adherence in competitive cloud environments.