Abstract Cost Models for Distributed Data-Intensive Computations.

Area of Science:

• Computer Science
• Distributed Systems
• Performance Optimization

Background:

• Data analytics workloads are increasingly deployed on distributed architectures, particularly clusters of commodity machines.
• Minimizing job running time requires an accurate cost model, referred to as a makespan model, for effective job partitioning.

Purpose of the Study:

• To develop a simplified yet accurate makespan model for distributed data analytics workloads.
• To reduce the computational cost of finding optimal job partitioning by exploiting the model's functional structure.

Main Methods:

• Exploration of piecewise linear functions of input, output, and computational complexity to model job performance.
• Development of abstract models capturing fundamental algorithm properties without system-specific details.
• Identification of a lower bound for search-space pruning in general cases.
• Direct integration of the model into makespan optimization for homogeneous tasks to reduce search-space dimensionality.

Main Results:

• The proposed abstract makespan model demonstrates good prediction quality across various operators and cluster architectures.
• Significant reduction in search-space dimensionality and complexity (orders of magnitude) for homogeneous tasks.
• Effective makespan optimization achieved through the simplified modeling approach.

Conclusions:

• Simplified, abstract makespan models can accurately represent distributed data analytics workloads.
• Exploiting the functional structure of these models leads to substantial computational cost reductions in optimization.
• The approach is effective for optimizing job partitioning and resource allocation in distributed computing environments.