Neural topic modeling on hyperspheres: Spherical representation learning with von Mises-Fisher mixtures

Area of Science:

• Natural Language Processing
• Machine Learning
• Computational Linguistics

Background:

• Neural topic models (NTMs) offer scalable text analysis but often ignore embedding geometry or use unsuitable priors.
• Existing variational autoencoder (VAE)-based NTMs struggle with KL divergence collapse and lack directional semantics for coherent topics.

Purpose of the Study:

• To propose a hyperspherical neural topic modeling framework, the von Mises-Fisher Mixture Neural Variational Topic Model (vMNVTM).
• To address limitations of current NTMs by incorporating the von Mises-Fisher (vMF) distribution for geometric and directional semantics.

Main Methods:

• Developed vMNVTM using the vMF distribution to model document representations and topic-word relationships on a hypersphere.
• Introduced a vMF-aware embedding clustering (vEC) loss and a temperature-controlled concentration mechanism for enhanced topic alignment and separability.
• Leveraged hyperspherical geometry for directional alignment and angular dispersion in latent spaces.

Main Results:

• vMNVTM demonstrated superior performance over state-of-the-art NTMs on benchmark datasets.
• Achieved improvements in topic coherence, diversity, and interpretability.
• Showcased the effectiveness of directional modeling in neural topic inference.

Conclusions:

• The vMF distribution is crucial for capturing directional semantics in neural topic modeling.
• vMNVTM provides a more coherent and interpretable approach to uncovering thematic structures in text.
• Hyperspherical modeling offers significant advantages for advanced natural language understanding tasks.