Estimating Predictive Rate-Distortion Curves via Neural Variational Inference.

Area of Science:

• Information Theory
• Machine Learning
• Computational Linguistics

Background:

• The Predictive Rate-Distortion (PRD) curve measures the trade-off between past information compression and future prediction accuracy.
• Current PRD estimation methods struggle with complex processes like natural language due to large alphabets and unknown causal states.
• Existing methods based on clustering or known causal states lack scalability.

Purpose of the Study:

• To develop a scalable method for estimating the PRD curve for complex stochastic processes.
• To leverage neural networks for estimating PRD curves where analytical solutions are intractable.
• To provide improved PRD bounds for natural language processing.

Main Methods:

• Introduced Neural Predictive Rate-Distortion (NPRD), a novel estimation technique.
• Utilized the universal approximation capabilities of neural networks.
• Computed a variational bound on the PRD curve using only time series data.

Main Results:

• NPRD demonstrates scalability for processes with large alphabets and long dependencies.
• The method was validated on processes with analytically known PRD curves.
• NPRD provided improved PRD bounds for natural language compared to sequence clustering.

Conclusions:

• NPRD offers a scalable approach to estimating the PRD curve for complex systems.
• The PRD curve is a more effective characterization tool than statistical complexity for highly complex processes like natural language.
• This work advances the application of information theory to natural language and other complex data.