Performance evaluation of SFP models using ML/DL and feature selection via cost evaluation framework.

Area of Science:

• Computer Science
• Software Engineering
• Artificial Intelligence

Background:

• Software systems complexity challenges fault prevention.
• Existing Software Fault Prediction (SFP) research lacks extensive comparative analysis of Deep Learning (DL) and Machine Learning (ML) with cost evaluation.
• The combined impact of feature selection (FS) and DL algorithms in SFP is underexplored.

Purpose of the Study:

• To conduct a large-scale comparative analysis of DL and ML-based SFP models.
• To evaluate the effectiveness of various FS techniques and the Synthetic Minority Oversampling Technique (SMOTE) in SFP.
• To assess the cost-effectiveness of DL-based SFP models using a Cost Evaluation Framework (CEF) and their potential to reduce real-world testing costs.

Main Methods:

• Trained 32 SFP models (8 state-of-the-art, 16 ML, 6 DL) on 54 open-source projects.
• Employed 11 FS techniques and SMOTE to address feature and class redundancy.
• Utilized Accuracy, Area Under the ROC Curve (AUC), and a novel CEF for performance evaluation.

Main Results:

• Without SMOTE, ML models like AdaBoost, Random Forest, and Extremely Randomized Trees showed high median AUC (0.72-0.73).
• With SMOTE, Extremely Randomized Trees, Random Forest, and Bagging Decision Tree achieved superior median AUC (0.93-0.94).
• DL models, particularly DL5 with SMOTE, demonstrated competitive performance (AUC 0.82) and feature selection reduced dataset size by 70-75% without performance loss.
• Cost analysis indicated DL-based SFP models are most beneficial for projects with below-average faulty class percentages (27.17%-49.48%).

Conclusions:

• DL-based SFP models are competitive with ML approaches, offering comparable predictive capabilities.
• Effective feature selection and class balancing techniques like SMOTE are crucial for optimizing SFP model performance.
• The proposed CEF demonstrates the practical utility of SFP models in reducing software testing costs, especially for projects with lower fault densities.