BlackboxBench: A Comprehensive Benchmark of Black-Box Adversarial Attacks

Area of Science:

• Artificial Intelligence
• Machine Learning Security
• Deep Neural Networks

Background:

• Deep neural networks (DNNs) are vulnerable to adversarial examples, necessitating robust evaluation methods.
• Black-box adversarial attacks, where model details are unknown, are crucial for real-world security assessments.
• Existing research lacks comprehensive comparisons of black-box attack algorithms, hindering progress analysis.

Purpose of the Study:

• To establish a comprehensive benchmark, BlackboxBench, for evaluating black-box adversarial attacks.
• To provide a unified, extensible, and modular codebase for diverse attack algorithms.
• To conduct thorough evaluations and analyses of current black-box attack strategies.

Main Methods:

• Developed a unified codebase implementing 29 query-based and 30 transfer-based adversarial attack algorithms.
• Conducted extensive evaluations of these algorithms on mainstream DNN architectures using CIFAR-10 and ImageNet datasets.
• Performed comprehensive analysis of algorithm performance, success rates, and query efficiency.

Main Results:

• BlackboxBench facilitates the evaluation of 59 distinct adversarial attack algorithms.
• Over 14,950 evaluations were performed across different models and datasets, generating extensive performance data.
• The benchmark provides new insights into the advantages and disadvantages of various attack methodologies.

Conclusions:

• BlackboxBench serves as a critical resource for understanding and advancing black-box adversarial attack research.
• The benchmark enables systematic comparison and analysis, guiding the development of more robust DNN security.
• This work offers analytical tools and insights crucial for the future roadmap of adversarial machine learning.