Generative Text Convolutional Neural Network for Hierarchical Document Representation Learning.

Area of Science:

• Natural Language Processing
• Machine Learning
• Probabilistic Modeling

Background:

• Existing document analysis methods often lose word order or project words into low-dimensional embeddings.
• Data sparsity and high dimensionality limit exploration of semantic structures in sequence-based document representations.
• Probabilistic modeling literature has had limited success with sequence-based document representations.

Purpose of the Study:

• To develop a probabilistic generative model for sequence-based document representations.
• To capture richer semantic information and preserve word order in documents.
• To enable efficient and scalable model inference and out-of-sample prediction.

Main Methods:

• Developed convolutional Poisson factor analysis (CPFA) for sparse data and model parallelism.
• Extended CPFA to a generative text convolutional neural network (GTCNN) using probabilistic pooling layers.
• Implemented parallel Gibbs sampler, SG-MCMC, and a hierarchical Weibull convolutional inference network.

Main Results:

• CPFA effectively utilizes data sparsity and enables model parallelism.
• GTCNN captures richer semantic information and alleviates word order loss in topic models.
• Proposed inference methods provide efficient and scalable training and prediction.

Conclusions:

• The proposed GTCNN and associated inference methods are effective for document representation learning.
• This work advances probabilistic generative models for sequence-based document analysis.
• The methods offer improved semantic understanding and word order preservation in text data.