Exploiting Reused-Based Sharing Work Opportunities in Big Data Multiquery Optimization with Flink.

Area of Science:

• Computer Science
• Data Engineering
• Database Systems

Background:

• Multiquery optimization is crucial for efficient big data retrieval.
• Exploiting shared operations (join, aggregation, sort) minimizes I/O costs.
• In-memory platforms like Flink enhance query performance.

Purpose of the Study:

• To extend the Multi-Query Optimization using Tuple Size and Histogram (MOTH) system.
• To develop the Join-Aggregation-Sort (JAS)-MOTH system for enhanced multiquery optimization.
• To minimize shuffle time by exploiting shared work among join, aggregation, and sort queries.

Main Methods:

• Developed query explorer and JAS-MOTH optimizer modules.
• Implemented sort exploiter for shared explicit and implicit sorts.
• Refined pipelined multiway join execution with coarse-grained data sharing and join ordering.
• Introduced an end-to-end multiway join optimizer over Flink.

Main Results:

• Improved query execution time by 47% over naive methods and 30% over state-of-art.
• Reduced intermediate data size by 50% compared to naive methods and 31% over state-of-art.
• Demonstrated significant performance gains on Flink and Hadoop-like infrastructures.

Conclusions:

• The JAS-MOTH system effectively exploits shared operations for multiquery optimization.
• The system significantly reduces query execution time and intermediate data shuffle.
• JAS-MOTH offers a robust solution for optimizing complex queries on big data platforms.