Is common sense all you need? Using expert defined rules to identify vulnerability patches instead of machine...

Area of Science:

• Software Engineering
• Cybersecurity
• Machine Learning

Background:

• Machine learning (ML) models have shown mixed success in identifying security fixing commits.
• An alternative approach uses expert-defined common sense rules with power-law weights.

Purpose of the Study:

• Evaluate the performance of expert-defined rules against ML models for identifying security fixing commits.
• Analyze the consistency of rules with expert usage.
• Assess the benefit of rule-based explanations for less experienced users.

Main Methods:

• Compared expert-defined rules with ML models on the ProjectKB dataset of Java security commits.
• Conducted a think-aloud protocol with senior analysts.
• Ran experiments with Master students to evaluate user support.

Main Results:

• Expert-defined rules performed similarly to classical ML approaches.
• ML models learned similar top features as identified by experts.
• Junior analysts' performance was comparable to experts', with no significant benefit from feature explanations.

Conclusions:

• Common sense rules defined by security experts are as effective as ML approaches for identifying security fixing commits.
• ML can be used to analyze data and distill effective common sense rules.
• Further research is needed to enhance the utility of features for final decision-making.