A topical VAEGAN-IHMM approach for automatic story segmentation.

Area of Science:

• Natural Language Processing
• Machine Learning
• Artificial Intelligence

Background:

• Effective story segmentation relies on rich topic information in feature representations.
• Traditional Hidden Markov Models (HMM) for story segmentation require manual setting of the number of topics, which is often unknown.
• Variational Autoencoder Generative Adversarial Network (VAEGAN) offers advanced feature learning by combining VAE and GAN for intricate and diverse data representations.

Purpose of the Study:

• To develop an advanced story segmentation method using improved feature representations.
• To address the limitation of manually setting the number of states in HMM for story segmentation.
• To enhance topical domain representation learning for more accurate document splitting.

Main Methods:

• Utilized a topical classifier to supervise the VAEGAN training for topical domain representation.
• Implemented an Infinite Hidden Markov Model (IHMM) with an HDP prior for automatic state number inference.
• Employed a Blocked Gibbs sampler to label states with topic classes and identify story boundaries where topics change.

Main Results:

• The proposed topical VAEGAN-IHMM approach significantly improved story segmentation performance compared to traditional HMM.
• Achieved state-of-the-art results on the TDT2 corpus for story segmentation tasks.
• Demonstrated the effectiveness of combining VAEGAN for feature learning and IHMM for topic modeling.

Conclusions:

• The VAEGAN-IHMM framework provides a robust and automated solution for story segmentation.
• This method enhances the accuracy and efficiency of identifying topic shifts within documents.
• The findings suggest a promising direction for advancing natural language processing tasks requiring topic-aware document analysis.