Rethinking Topic Modeling With Information Bottleneck Principle.

Area of Science:

• Natural Language Processing
• Machine Learning
• Computational Linguistics

Background:

• Topic models analyze word co-occurrence for semantic structure but struggle with short texts due to data sparsity.
• Word embedding (e.g., skip-gram) and masked language models (e.g., BERT) excel in short texts by using local context.
• Existing explanations for performance differences focus on exploited information types.

Purpose of the Study:

• To propose a unified perspective for topic models, skip-gram, and BERT using the information bottleneck (IB) principle.
• To analyze performance differences of these models in short-text scenarios through their adherence to the IB principle.
• To demonstrate that topic models can be improved for short texts via information-theoretic regularization.

Main Methods:

• Formulating topic models, skip-gram, and BERT as text autoencoders (AEs).
• Analyzing model performance based on their adherence to the IB principle.
• Constraining mutual information (MI) between observed data and latent variables in empirical studies.

Main Results:

• Theoretical analysis shows varying degrees of IB compliance explain distinct short-text behaviors.
• Empirical results demonstrate topic models achieve performance comparable to skip-gram and BERT in short-text settings under the IB perspective.
• The IB principle provides a unified framework for understanding and improving these language models.

Conclusions:

• The information bottleneck principle offers a novel theoretical foundation for understanding language model behavior, especially in short texts.
• Adherence to the IB principle is key to overcoming data sparsity limitations in topic models for short texts.
• Information-theoretic regularization presents a principled approach to enhance topic model performance on short texts.