Rethinking Topic Modeling With Information Bottleneck Principle

By modeling global word co-occurrence patterns, topic models aim to uncover the underlying semantic structure of a corpus. However, their effectiveness is often undermined in short texts due to severe data sparsity, as limited co-occurrence information is available. In contrast, word embedding models (e.g., skip-gram) and masked language models (e.g., BERT) rely on local contextual signals, which makes them more effective in short-text scenarios. The prevailing explanation attributes the performance differences between these approaches to the type of information they exploit. In this article, we revisit this view and propose a unified perspective for understanding topic models, skip-gram, and BERT through the lens of the information bottleneck (IB) principle. By formulating all three as instances of text autoencoders (AEs), we analyze their performance differences in terms of how tightly they adhere to the IB principle. Our theoretical analysis reveals that the varying degrees of IB compliance explain their distinct behaviors on short texts. We further illustrate this insight by examining cases where the mutual information (MI) between observed data and latent variables is explicitly constrained. Empirically, we demonstrate that, under the IB perspective, topic models can achieve performance comparable to that of skip-gram and BERT in short-text settings. These findings provide a new theoretical foundation for understanding the strengths and weaknesses of various language models and suggest a principled approach for improving topic models through information-theoretic regularization.