Online Density Estimation of Nonstationary Sources Using Exponential Family of Distributions

Area of Science:

• Machine Learning
• Statistical Inference
• Information Theory

Background:

• Online learning algorithms are crucial for adapting to changing data distributions.
• Nonstationary sources pose significant challenges for traditional probability density estimation methods.
• Existing methods often require prior knowledge of data characteristics, limiting their applicability.

Purpose of the Study:

• To introduce a novel sequential algorithm for online probability density estimation.
• To achieve Hannan-consistent log-loss regret performance without prior information on observation sequences.
• To provide efficient and robust density estimation for nonstationary sources.

Main Methods:

• Utilizing the exponential family of distributions for density modeling.
• Designing a truly sequential algorithm with exponentially quantized learning rates.
• Employing a mixture-of-experts approach to combine estimators.
• Achieving logarithmic computational complexity with respect to time horizon.

Main Results:

• The algorithm achieves Hannan-consistent log-loss regret bounds of O((CT)^1/2).
• Performance guarantees hold in an individual sequence manner.
• Demonstrated substantial performance gains over state-of-the-art methods in experiments.
• Efficient computation with logarithmic complexity in time horizon.

Conclusions:

• The proposed algorithm offers a theoretically sound and practically efficient solution for online probability density estimation.
• It effectively handles nonstationary data without requiring prior knowledge of sequence parameters.
• The method shows significant advantages for big data applications requiring adaptive density estimation.