Compress and Compare: Interactively Evaluating Efficiency and Behavior Across ML Model Compression Experiments

Area of Science:

• Computer Science
• Machine Learning
• Data Visualization

Background:

• Deploying machine learning models on-device requires compression algorithms to reduce model size and improve inference speed.
• Effective comparison of compression experiments is crucial for analyzing accuracy-efficiency trade-offs and model behavior changes.
• Existing tools often fragment the analysis process, leading to inefficiencies and incomplete insights.

Purpose of the Study:

• To develop an interactive visual system, COMPRESS AND COMPARE, that supports real-world comparative workflows for machine learning model compression.
• To provide a unified interface for visualizing model provenance and comparing predictions, weights, and activations.

Main Methods:

• Development of the COMPRESS AND COMPARE interactive visual system.
• Demonstration of the system's utility through case studies on generative language models and image classification models.
• Evaluation of the system via a user study with eight machine learning compression experts.

Main Results:

• COMPRESS AND COMPARE effectively visualizes provenance relationships between compressed models.
• The system reveals compression-induced behavior changes by comparing model predictions, weights, and activations.
• User study indicated the system structures compression workflows, builds practitioner intuition, and encourages thorough analysis.

Conclusions:

• COMPRESS AND COMPARE enhances the analysis of machine learning model compression by providing a unified visual interface.
• The system aids in debugging compression failures and identifying compression artifacts.
• Identified compression-specific challenges and generalizable visualizations for future visual analytics tools in model comparison.