Studying the impact of CI on pull request delivery time in open source projects-a conceptual replication

Area of Science:

• Software Engineering
• Empirical Software Engineering
• Open Source Software Development

Background:

• Continuous Integration (CI) is widely adopted to accelerate software delivery.
• A prior study suggested CI adoption has limited impact on pull request (PR) delivery times.
• Concerns exist regarding the methodology of the prior study, potentially affecting its conclusions.

Purpose of the Study:

• To conduct a conceptual replication of a study on CI's impact on PR delivery speed.
• To address potential methodological limitations of the original study.
• To evaluate the generalizability of findings to other open source projects.

Main Methods:

• Replication of original study's methodology and subject pool.
• Application of Regression Discontinuity Design (RDD) to control for time-dependent project changes.
• Inclusion of a control group of projects that never adopted CI.
• Mining of a new set of open source projects for generalizability analysis.

Main Results:

• Replication confirmed the original study's findings regarding CI's limited impact on PR delivery speed.
• RDD analysis showed no robust evidence of natural speed-up in PR delivery over time.
• The control group also did not exhibit significant speed-up in PR delivery.
• Analysis of newly mined projects yielded comparable results to the original study.

Conclusions:

• The original study's findings on CI's impact on PR delivery speed are largely supported by this replication.
• Methodological improvements did not reveal a significant acceleration in PR delivery due to CI adoption.
• The study suggests that the benefits of CI in accelerating software delivery may be less pronounced than commonly assumed.