Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suites

Area of Science:

• Computer Science
• Software Engineering

Background:

• Performance issues in applications necessitate early detection upon code modification.
• Application benchmarks and microbenchmarks are used in build pipelines to ensure performance goals, but have limitations.

Purpose of the Study:

• To improve the practical relevance and verification of microbenchmark suites using application flow data.
• To reduce the risk of undetected performance problems by optimizing microbenchmark suites.

Main Methods:

• Determining overlap of common function calls between application and microbenchmarks.
• Identifying redundant microbenchmarks.
• Recommending relevant functions not yet covered by microbenchmarks.

Main Results:

• Microbenchmark suites covered 35.62% to 66.29% of functions called during application benchmarks.
• Redundancy removal reduced microbenchmark suites to ~10% and ~23% of their original size.
• Recommendations identified up to 26 and 14 new functions for benchmarking.

Conclusions:

• Optimized microbenchmark suites can effectively test relevant functions after code changes.
• The approach enhances software performance assurance by integrating microbenchmarks and application benchmarks.
• Utilizing synergies between different performance test granularities improves overall software quality.