iQCAR: inter-Query Contention Analyzer for Data Analytics Frameworks
Prajakta Kalmegh1, Shivnath Babu2, Sudeepa Roy1
1Duke University, Durham, North Carolina.
Proceedings. ACM-SIGMOD International Conference on Management of Data
|April 12, 2021
Summary
Resource interferences in cluster computing cause unpredictable performance. Our inter-Query Contention Analyzer (iQCAR) accurately identifies which concurrent queries cause slowdowns, aiding database administrators.
Area of Science:
- Computer Science
- Database Systems
- Distributed Computing
Background:
- Concurrent queries in cluster computing systems lead to resource interferences, causing unpredictable performance and missed Service Level Agreements (SLAs).
- Diagnosing query performance degradation requires understanding resource contention and attributing slowdowns to specific concurrent queries or external factors.
Purpose of the Study:
- To introduce an inter-Query Contention Analyzer (iQCAR) for attributing query slowdowns to specific concurrent queries.
- To provide database administrators with tools to identify the root causes of resource conflicts and performance bottlenecks.
Main Methods:
- Developed iQCAR, a system that models resource conflicts using a multi-level directed acyclic graph.
- Implemented iQCAR to analyze inter-query resource interactions in online executions within a selected time window.
Main Results:
- iQCAR accurately attributes blame for query slowdowns to concurrent queries.
- The system enables administrators to compare impacts from concurrent queries and identify contentious queries, resources, and hosts.
- Experiments on Apache Spark using TPC-DS queries demonstrated iQCAR's superior accuracy over overlap-time methods.
Conclusions:
- iQCAR effectively identifies and quantifies inter-query resource contention, significantly improving the diagnosis of query performance issues.
- The proposed method offers a more accurate approach to understanding and mitigating performance degradation in cluster computing environments.
Related Concept Videos
01:15Quartile
7.7K
Quartiles are numbers that separate the data into quarters. Quartiles may or may not be part of the data. To find the quartiles, first, find the median or second quartile. The first quartile, Q1, is the middle value of the lower half of the data, and the third quartile, Q3, is the middle value, or median, of the upper half of the data. To get the idea, consider the same data set:
1; 1; 2; 2; 4; 6; 6.8; 7.2; 8; 8.3; 9; 10; 10; 11.5
The median or second quartile is seven. The lower half of the...
1; 1; 2; 2; 4; 6; 6.8; 7.2; 8; 8.3; 9; 10; 10; 11.5
The median or second quartile is seven. The lower half of the...
7.7K
01:12Maximum Size of Aggregate
314
The maximum size of aggregate is defined as the aperture of the sieve retaining 15 percent or more of the particles present in the aggregate sample. The aggregate's maximum size impacts the concrete's water requirement, workability, and strength. Larger aggregates reduce the surface area needing cement paste coverage, which can lower water needs, thereby allowing a decrease in the water-to-cement ratio when the desired workability and richness of the mix are to be maintained, which can...
314
01:00Detection of Gross Error: The Q Test
6.6K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.6K
01:12Quantitative Analysis
882
Quantitative analysis is a technique for measuring the amount of specific constituents in a sample. When the sample's composition is unknown, qualitative analysis is performed first to identify its components, which ensures that the correct substances are measured during the quantitative phase.
In quantitative analysis, two key measurements are made: the sample quantity and a property proportional to the amount of the analyte (the substance being analyzed). This forms the basis of the...
In quantitative analysis, two key measurements are made: the sample quantity and a property proportional to the amount of the analyte (the substance being analyzed). This forms the basis of the...
882
02:46Chi-square Analysis
41.3K
The chi-square test is a statistical hypothesis test. It is used to check whether there is a significant difference between an expected value and an observed value. In the context of genetics, it enables us to either accept or reject a hypothesis, based on how much the observed values deviate from the expected values.
The chi-square test was developed by Pearson in 1990.
The first step of performing a Chi-square analysis is to establish a null hypothesis, which assumes that there is no real...
The chi-square test was developed by Pearson in 1990.
The first step of performing a Chi-square analysis is to establish a null hypothesis, which assumes that there is no real...
41.3K
01:12Column Efficiency: Rate Theory
678
The rate theory of chromatography provides quantitative insight into the shapes and widths of elution bands. These bands are based on the random-walk mechanism governing molecular migration within a column. The Gaussian profile of chromatographic bands arises from the cumulative effect of random molecular motions as they progress through the column.
During elution, a solute molecule experiences numerous transitions between stationary and mobile phases, exhibiting irregular residence times in...
During elution, a solute molecule experiences numerous transitions between stationary and mobile phases, exhibiting irregular residence times in...
678