Comparative analysis of model compression techniques for achieving carbon efficient AI.

Area of Science:

• Artificial Intelligence
• Computer Science
• Environmental Science

Background:

• Growing computational demands of large language models (LLMs) like BERT raise environmental concerns.
• Need for sustainable AI practices to mitigate the carbon footprint of AI.
• Transformer-based models are increasingly prevalent but energy-intensive.

Purpose of the Study:

• Investigate the efficiency of model compression techniques (pruning, knowledge distillation, quantization) for reducing energy consumption and carbon emissions.
• Evaluate the performance impact of these techniques on transformer models.
• Compare compressed models with inherently carbon-efficient architectures.

Main Methods:

• Applied pruning, knowledge distillation, and quantization to BERT, DistilBERT, ALBERT, and ELECTRA.
• Utilized the Amazon Polarity Dataset for sentiment analysis.
• Measured energy consumption and carbon emissions using the CodeCarbon tool.
• Compared performance metrics (accuracy, precision, recall, F1, ROC AUC) of compressed models.

Main Results:

• Significant energy reduction achieved: BERT (32.1% with pruning/distillation), DistilBERT (specific % missing, but reduced), ALBERT (7.12% with quantization), ELECTRA (23.9% with pruning/distillation).
• Performance metrics maintained within 95.87-99.06% accuracy, precision, recall, F1, and ROC AUC for most models.
• ALBERT with quantization showed significant performance degradation, highlighting sensitivity.

Conclusions:

• Model compression techniques are effective in reducing the environmental impact of transformer-based AI models.
• Sustainable AI practices can be achieved through efficient model design and optimization.
• Further research needed to address quantization sensitivity in certain architectures.