EmbBERT: Attention under 2 MB memory

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Edge Computing

Background:

• Transformer architectures, while powerful for Natural Language Processing (NLP), have significant memory and computational demands.
• Deployment of advanced NLP models on ultra-constrained devices like wearables and IoT units is challenging due to limited memory (megabytes).

Purpose of the Study:

• To introduce EmbBERT, a tiny language model (TLM) architecturally optimized for extreme efficiency on edge devices.
• To demonstrate that simplified transformer architectures can maintain high performance under strict memory constraints.

Main Methods:

• Designed EmbBERT with a compact embedding layer, streamlined feed-forward blocks, and an efficient attention mechanism.
• Evaluated EmbBERT on the TinyNLP benchmark and GLUE suite, comparing its performance against larger state-of-the-art (SotA) models and similarly sized BERT and MAMBA variants.
• Assessed the model's resilience to 8-bit quantization and its scalability across different memory ranges (sub-megabyte to tens-of-megabytes).

Main Results:

• EmbBERT requires only 2 MB of memory, achieving accuracy comparable to SotA models with 10x more memory.
• Outperformed downsized BERT and MAMBA models of similar size on NLP tasks.
• Demonstrated resilience to 8-bit quantization, reducing memory footprint to 781 kB.
• Showcased scalability of the EmbBERT architecture across various memory constraints.

Conclusions:

• Highly simplified transformer architectures are effective for edge NLP tasks under tight resource constraints.
• EmbBERT offers a viable solution for deploying advanced NLP capabilities on memory-limited edge devices.
• The proposed architecture and pre-training strategy contribute to efficient and accurate edge AI.