Information-theoretic complementary prompts for improved continual text classification.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Continual Text Classification (CTC) addresses the challenge of classifying evolving text data over time.
• Existing CTC methods often neglect the crucial role of shared, task-agnostic knowledge.
• Catastrophic forgetting remains a significant hurdle in sequential learning tasks.

Purpose of the Study:

• To introduce a novel approach, Information-Theoretic Complementary Prompts (InfoComp), for continual text classification.
• To address the limitations of existing methods by explicitly learning both task-specific and task-agnostic knowledge.
• To enable sequential learning without data replay by leveraging complementary learning systems theory.

Main Methods:

• InfoComp learns two distinct prompt spaces: P(rivate)-Prompt for task-specific knowledge and S(hared)-Prompt for task-invariant knowledge.
• An information-theoretic framework maximizes mutual information between parameters to guide prompt learning.
• Two novel loss functions are designed to strengthen task-specific knowledge accumulation and enhance task-invariant knowledge retention.

Main Results:

• InfoComp effectively mitigates catastrophic forgetting of previously acquired knowledge.
• The approach demonstrates improved forward knowledge transfer by retaining task-invariant knowledge.
• Experiments on diverse CTC benchmarks show InfoComp outperforming state-of-the-art methods.

Conclusions:

• InfoComp offers a promising solution for continual text classification by balancing task-specific and task-agnostic knowledge learning.
• The method enables efficient sequential learning without the need for data replay.
• InfoComp advances the field of continual learning by providing a more robust and transferable knowledge acquisition framework.