Fault-Tolerant and Data-Intensive Resource Scheduling and Management for Scientific Applications in Cloud Computing

Area of Science:

• Cloud Computing
• Scientific Workflow Management
• Distributed Systems

Background:

• Scientific applications, like Montage and CyberShake, are complex workflows with intensive data and computation demands.
• These workflows involve tasks requiring integration, disintegration, pipeline, and parallelism, necessitating specialized management for task execution and resource scheduling.
• Pipeline tasks are critical bottlenecks, and their failure can lead to complete execution failure, demanding fault-tolerant approaches.

Purpose of the Study:

• To introduce a novel scheduling strategy for scientific applications in cloud environments.
• To address the challenges of data intensiveness, fault tolerance, and efficient resource management in scientific workflows.
• To enhance the performance of scientific applications by optimizing task scheduling and resource allocation.

Main Methods:

• Development of a cluster-based, fault-tolerant, and data-intensive (CFD) scheduling strategy.
• Implementation of CFD mechanisms to manage data-intensive tasks and provide fault tolerance.
• Simulation of the Montage scientific workflow to evaluate the CFD strategy against existing policies (MCT, Max-min, Min-min).

Main Results:

• The CFD strategy reduced the make-span (total execution time) by 14.28% compared to MCT, 20.37% to Max-min, and 11.77% to Min-min.
• Execution costs were reduced by 1.27% (MCT), 5.3% (Max-min), and 2.21% (Min-min) using the CFD strategy.
• The CFD strategy consistently met Service Level Agreements (SLAs) for time and cost constraints, unlike existing policies that frequently violated them.

Conclusions:

• The proposed CFD scheduling strategy offers significant improvements in performance and cost-efficiency for scientific applications in cloud environments.
• CFD effectively handles data-intensive tasks and provides robust fault tolerance, crucial for complex scientific workflows.
• This research demonstrates the superiority of CFD over traditional heuristic scheduling policies in meeting performance and reliability requirements for scientific cloud computing.