An innovative parameter optimization of Spark Streaming based on D3QN with Gaussian process regression

Area of Science:

• Computer Science
• Artificial Intelligence
• Big Data Analytics

Background:

• Spark Streaming is crucial for processing real-time data from sources like social media and IoT devices.
• Optimizing Spark Streaming performance is vital due to its widespread use in data analysis.
• Manual parameter tuning for Spark Streaming is complex and inefficient, involving over 200 parameters.

Purpose of the Study:

• To develop an automated and efficient method for optimizing Spark Streaming performance.
• To address the challenges of manual parameter configuration in Spark Streaming.
• To significantly enhance the performance of Spark Streaming through intelligent parameter tuning.

Main Methods:

• Proposed an improved dueling double deep Q-network (DQN) technique for automated parameter tuning.
• Integrated reinforcement learning with Gaussian process regression to accelerate convergence.
• Focused on optimizing task scheduling, resource allocation, and data skew in Spark Streaming.

Main Results:

• The proposed dueling double DQN method with Gaussian process regression demonstrated significant performance improvements.
• Achieved up to a 30.24% enhancement in Spark Streaming performance.
• Reduced the number of iterations required for parameter tuning and sped up convergence.

Conclusions:

• The improved dueling double DQN technique offers a highly effective solution for Spark Streaming performance optimization.
• Automated parameter tuning using reinforcement learning and Gaussian process regression is more efficient than manual methods.
• This approach provides a scalable and powerful tool for big data stream processing.