Mitigating carbon footprint for knowledge distillation based deep learning model compression.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Environmental Science

Background:

• Deep learning models achieve high performance but incur substantial energy costs and carbon footprints.
• Massive CO2 emissions from deep learning pose an ethical concern, largely overlooked in research.
• Knowledge Distillation (KD) creates lightweight models for edge devices but can still have high environmental costs.

Purpose of the Study:

• To focus on the environmental costs of deep learning models, particularly those using KD.
• To propose and evaluate a method for mitigating carbon footprints in KD models.
• To introduce a metric for measuring the environmental costs of deep learning.

Main Methods:

• Proposed a stochastic approach for selecting the hyperparameter Temperature (τ) during KD model training.
• Measured environmental costs in terms of GFLOPS, energy consumption (kWh), and CO2 equivalent (grams).
• Applied and evaluated the method on image classification and object detection tasks using various datasets and models.

Main Results:

• The stochastic hyperparameter selection significantly reduced energy consumption and CO2 emissions (by a factor of 19 for CIFAR 10).
• The proposed method achieved comparable accuracy to traditional hyperparameter tuning (e.g., 91.67% vs. 91.78% accuracy on CIFAR 10).
• Similar reductions in environmental costs and maintained performance were observed across different datasets and object detection tasks.

Conclusions:

• The stochastic approach effectively reduces the environmental costs associated with KD models.
• This method eliminates the need for expensive hyperparameter tuning, making deep learning more sustainable.
• The findings highlight a practical way to develop lightweight, deployable, and environmentally friendly deep learning models.