Performance evaluation of SFP models using ML/DL and feature selection via cost evaluation framework

The increasing complexity of software systems (SW) makes fault prevention challenging. Although significant amount of research has explored Software Fault Prediction (SFP). However the extensive comparative analysis of Deep Learning (DL), state-of-the-art, and Machine Learning (ML) techniques, with the use of a Cost Evaluation Framework (CEF) remains largely unexplored. Furthermore, the combined impact of different feature selection (FS) techniques and DL algorithms has not been comprehensively investigated. This research aims to perform a large-scale comparative analysis DL and ML-based SFP models, incorporating SMOTE and ten feature selection (FS) strategies to address class and feature redundancy. Additionally the study employs a Cost Evaluation Framework (CEF), in addition to traditional metrics such as Accuracy, and AUC to assess the effectiveness of DL algorithms in SFP. Furthermore, the study investigates whether DL-based SFP models can reduce testing costs in real-world projects compared to state-of-the-art and ML approaches. This study employs 32 methods to train SFP models, comprising 8 state-of-the-art techniques, 16 ML methods, and 6 DL approaches, evaluated against 54 open-source projects to assess performance. To address feature redundancy and develop an efficient SFP model, the study analysed 11 FS techniques, including one that utilised the original dataset with complete feature measurements. The research additionally examined the application of the Synthetic Minority Oversampling Technique (SMOTE) to rectify class imbalance. The study introduced a cost evaluation methodology to assess the cost-effectiveness of the developed fault prediction model relative to actual testing efforts and to ascertain whether the implementation of the fault prediction model yields profit. Accuracy, Area under the Receiver Operating Characteristic (ROC) Curve (AUC) and CEF performance metrics are used to analyse the performance of developed SFP models. The experimental outcome indicates AdaBoost (ABC), Random Forest (RFC), and Extremely Randomized Trees (ETC), excluding SMOTE, exhibit the highest median AUC values of 0.73, 0.72, and 0.72 compared to other methods. ETC, RFC, and BAGD (Bagging Decision Tree) exhibited superior prediction, attaining median AUC values of 0.94, 0.93, and 0.93, respectively, following the adoption of SMOTE, across1188 datasets. Without SMOTE, the DL2 algorithm attained optimal performance, evidenced by the lowest mean-rank value of 11.58. The DL5 model utilising SMOTE exhibited superior prediction, achieving a mean rank of 9.37 and an AUC of 0.82, indicating a significantly enhanced performance. The completed comparative evaluation of various feature selection methodologies indicated that the SMOTE-based Consistency Feature Subset Selection (FS3) achieved a marginally superior median AUC value of 0.70 and the lowest mean rank of 5.84, aligning more closely with the AUC achieved by all metrics, thus exhibiting improved performance compared to other techniques. The experimental analysis, evaluated using AUC, Accuracy, Statical description and Friedman test indicates that while SMOTE based ML models achieved slightly higher performance than SMOTE-based DL models, the differences were marginal, indicating that DL performed competitively and attained comparable predictive capability. FS techniques eliminated 70-75% of irrelevant features, enabling DL-based SFP models to maintain or improve performance using reduced datasets. The cost-analysis framework results demonstrate that according to the engineer's low, medium and high testing efficiency, the fault prediction model is better suited for projects with faulty classes below the average threshold value of 49.48%, 38.86%, and 27.17% respectively.