Meta-learning for real-world class incremental learning: a transformer-based approach.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Deep learning (DL) models in Natural Language Processing (NLP) are highly complex with millions of parameters, requiring large datasets for training.
• While pretraining reduces data needs, fine-tuning still necessitates human-labeled datasets, incurring significant costs.
• Few-shot learning (FSL) techniques, like meta-learning, aim to train models effectively on smaller datasets.

Purpose of the Study:

• To apply meta-learning to class incremental learning (IL), a more relevant real-world problem than standard FSL evaluation tasks.
• To enable models to classify newly introduced classes after initial training without complete retraining.
• To leverage the generalization capabilities of meta-learners for effective class IL.

Main Methods:

• Emulation of class IL using proxy new classes to allow meta-learners to adapt without retraining.
• Development of a transformer-based aggregation function within a meta-learner to modify data across all classes for prediction.
• Concurrent consideration of entire support and query sets, prioritizing attention to crucial samples to enhance inference impact.

Main Results:

• The proposed meta-learning approach surpasses current industry benchmarks in class incremental learning.
• Meta-learners demonstrate significant generalization capabilities in class IL, even without task-specific training.
• The study establishes a high-performing baseline for transformer-based aggregation techniques in IL.

Conclusions:

• Meta-learning offers a practical and effective solution for class incremental learning challenges.
• The proposed transformer-based meta-learner provides a robust framework for future advancements in IL.
• The findings highlight the significant potential of meta-learning for real-world NLP applications requiring continuous learning.