The effect of data complexity on classifier performance.

Area of Science:

• Computer Science
• Software Engineering

Background:

• Software Defect Prediction (SDP) is a popular research area, typically framed as a classification problem.
• Despite advancements in classification, pre-processing, and tuning, SDP models often hit a performance ceiling.
• This suggests limitations beyond standard model optimization techniques.

Purpose of the Study:

• To analyze classifier performance in SDP from a data complexity perspective.
• To investigate the correlation between data complexity metrics and the performance of various machine learning classifiers.
• To identify specific strengths and weaknesses of different classifiers across diverse datasets.

Main Methods:

• Calculated data complexity metrics using the Unified Bug Dataset, a compilation of well-known SDP datasets.
• Evaluated the performance of machine learning classifiers including C5.0, Naive Bayes, Artificial Neural Networks, Random Forests, and Support Vector Machines.
• Correlated data complexity metrics with classifier performance to understand their relationships.

Main Results:

• Identified distinct domains of competence and incompetence for different classifiers.
• Found similarities and differences in classifier performance and their relationship with performance metrics.
• Demonstrated that data complexity is a critical factor influencing SDP model performance.
• Observed that certain classifiers perform optimally under specific data complexity conditions.

Conclusions:

• Classifier performance in Software Defect Prediction is highly dependent on data complexity.
• No single classifier is universally superior; optimal choice depends on the specific dataset characteristics.
• Data complexity metrics offer valuable insights into understanding and potentially improving SDP model performance.