DeepCompare: Visual and Interactive Comparison of Deep Learning Model Performance.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Visualization

Background:

• Deep learning models achieve state-of-the-art performance but understanding their internal workings remains challenging.
• Traditional performance metrics (accuracy, precision, recall) lack qualitative insights into model behavior and decision-making processes.
• Identifying reasons for performance differences between models is crucial for enhancing effectiveness, accuracy, and fairness.

Purpose of the Study:

• To develop a visual analytic approach for systematically comparing deep learning models.
• To provide insights into model behavior and enable interactive assessment of tradeoffs between models.
• To aid machine learning practitioners in understanding and evaluating model performance beyond quantitative metrics.

Main Methods:

• Interviews with machine learning practitioners to understand current evaluation workflows.
• Iterative design of a visual analytic tool, DeepCompare, based on practitioner feedback.
• Case studies applying DeepCompare to real-world deep learning models.

Main Results:

• DeepCompare enables users to evaluate model results and compare activation patterns for misclassifications.
• The tool facilitates linking test results to specific neurons, offering granular insights into model learning.
• Preliminary evaluations demonstrate that experts can make more informed decisions regarding model strengths, weaknesses, and overall behavior.

Conclusions:

• DeepCompare provides a novel approach to qualitatively analyze and compare deep learning models.
• The visual analytic tool enhances understanding of model behavior, aiding in the development of more effective and fair AI systems.
• Experts using DeepCompare can achieve a more holistic evaluation of model performance and decision-making processes.