Static test flakiness prediction: How Far Can We Go?

Area of Science:

• Software Engineering
• Machine Learning
• Software Testing

Background:

• Test flakiness, where tests non-deterministically pass or fail, is a significant challenge in software development.
• Existing detection methods often rely on computationally expensive dynamic analysis, limiting scalability.
• Machine learning has been explored for predicting test flakiness using mixed static and dynamic metrics.

Purpose of the Study:

• To investigate the prediction of test flakiness using exclusively static metrics.
• To assess the performance of a static-only approach against state-of-the-art methods.
• To analyze the impact of production code characteristics on flaky test prediction.

Main Methods:

• A large-scale experiment was conducted on 70 Java projects from the iDFlakies and FlakeFlagger datasets.
• Statistical analysis of 25 code metrics and smells to differentiate between flaky and non-flaky tests.
• Development and evaluation of a machine learning model for predicting test flakiness using only static features.

Main Results:

• The static-only approach demonstrated performance comparable to existing baseline methods.
• Analysis revealed significant differences between flaky and non-flaky tests based on static metrics.
• Production code characteristics were found to influence the effectiveness of flaky test prediction models.

Conclusions:

• Predicting test flakiness using solely static metrics is a viable and scalable approach.
• Static metrics alone can offer competitive performance in identifying flaky tests.
• Future research should consider production code attributes for enhanced flaky test prediction accuracy.