Configuring Parallelism for Hybrid Layouts Using Multi-Objective Optimization.

Area of Science:

• Data Engineering
• Distributed Systems
• Database Management

Background:

• Organizations store data in data lakes, often processed into hybrid layouts for efficient querying.
• Distributed frameworks like Hadoop and Spark partition data based on file size, not read size, leading to inefficient task allocation.
• This inefficiency increases query times and wastes computing resources.

Purpose of the Study:

• To propose a novel method for optimizing task and resource allocation in distributed data processing.
• To reduce query execution time and minimize computational waste in data-intensive organizations.

Main Methods:

• Developed a cost-based model to estimate data read size in hybrid layouts.
• Utilized a multi-objective optimization method, incorporating estimated read size, to determine optimal task and resource allocation.
• Prototyped the solution for Apache Parquet and Spark.

Main Results:

• Achieved a high correlation (0.96) between estimated and actual data read sizes.
• Recommended configurations were close to the Pareto front (5.6% deviation).
• Demonstrated a 2.1x speedup compared to default configurations using TPC-H benchmarks.

Conclusions:

• The proposed method effectively optimizes task and resource allocation for analytical queries on hybrid data layouts.
• This approach significantly enhances the efficiency of distributed data processing frameworks.
• The findings suggest a practical solution for improving performance and resource utilization in big data systems.