Related Experiment Videos
A taxonomy for detecting and preventing temporal data leakage in machine learning-based build prediction: A
Lalit Narayan Mishra1, Amit Rangari2, Sandesh Nagrare3
1Lowe's Companies, Inc., Charlotte, North Carolina, United States of America.
Plos One
|May 27, 2026
Summary
Predicting software build success or failure accurately requires avoiding temporal data leakage. This study introduces a taxonomy to identify and prevent leakage, revealing realistic build prediction accuracy is substantially lower than previously reported.
Area of Science:
- Software Engineering
- Machine Learning
- Data Science
Background:
- Automated build systems are crucial for modern software development, but predicting build success/failure before execution is challenging.
- Many machine learning models for build prediction suffer from temporal data leakage, leading to inflated accuracy and poor real-world performance.
Purpose of the Study:
- To develop a taxonomy for systematically identifying and preventing temporal data leakage in build prediction models.
- To re-evaluate the accuracy of build prediction models by removing leaky features and assess realistic performance.
Main Methods:
- A three-type taxonomy was developed: Direct Outcome Encoding, Execution-Dependent Metrics, and Future Information Leakage.
- The methodology was validated on 175,706 builds from TravisTorrent (2013-2017) and GHALogs (2023).
- Random Forest classifiers were used with legitimately available pre-build features.
Main Results:
- Prior studies reporting 95-99% accuracy likely used contaminated features; realistic accuracy is substantially lower.
- Removing leaky features reduced accuracy by 15.07% on TravisTorrent but only 0.48% on GHALogs.
- Random Forest models achieved 82.73% (TravisTorrent) and 83.30% (GHALogs) accuracy using only pre-build features.
- Project maturity and build history were more predictive than code complexity.
- Models generalized across multiple programming languages.
Conclusions:
- Modern CI/CD platforms like GitHub Actions enable accurate build prediction using static project metadata alone, mitigating temporal leakage.
- Organizational factors like project maturity and build history are key predictors of build success.
- Open-source tools for detecting temporal leakage in software prediction tasks are provided.
Related Concept Videos
01:19Survival Tree
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a survival tree begins...
Building a Survival Tree
Constructing a survival tree begins...
01:03Prediction Intervals
The interval estimate of any variable is known as the prediction interval. It helps decide if a point estimate is dependable.
However, the point estimate is most likely not the exact value of the population parameter, but close to it. After calculating point estimates, we construct interval estimates, called confidence intervals or prediction intervals. This prediction interval comprises a range of values unlike the point estimate and is a better predictor of the observed sample value, y.
The...
However, the point estimate is most likely not the exact value of the population parameter, but close to it. After calculating point estimates, we construct interval estimates, called confidence intervals or prediction intervals. This prediction interval comprises a range of values unlike the point estimate and is a better predictor of the observed sample value, y.
The...