A Novel Reinforcement Learning Approach for Spark Configuration Parameter Optimization.

Area of Science:

• Computer Science
• Artificial Intelligence
• Data Engineering

Background:

• Apache Spark is a widely used distributed data processing framework.
• Manual tuning of Spark's 180+ configuration parameters is complex and time-consuming due to interdependencies.
• Optimizing Spark performance is crucial for efficient big data processing.

Purpose of the Study:

• To develop an automated system for optimizing Apache Spark configuration parameters.
• To overcome the limitations of manual tuning and experience-based adjustments.
• To enhance the performance of diverse Spark applications through intelligent configuration.

Main Methods:

• Developed a deep neural network model to predict Spark application performance.
• Implemented a reinforcement learning approach, specifically an improved Q-learning algorithm, for parameter optimization.
• Integrated automatic start and end state setting within the Q-learning iterations to enhance exploration efficiency.

Main Results:

• The deep neural network model demonstrated high accuracy and effectiveness in predicting Spark performance.
• The reinforcement learning optimizer successfully identified optimal configurations.
• Experimental results showed an average performance improvement of 47%, 43%, 31%, and 45% for four different Spark application types compared to default settings.

Conclusions:

• The proposed reinforcement-learning-based optimizer effectively tunes Apache Spark configurations.
• The system significantly enhances the performance of various Spark applications.
• Automated optimization offers a viable alternative to manual tuning for complex distributed systems.