A Sequential Monte Carlo Method for Bayesian Analysis of Massive Datasets.

Area of Science:

• Statistical computing
• Bayesian inference
• Data mining

Background:

• Markov chain Monte Carlo (MCMC) methods are computationally intensive for large datasets.
• Current MCMC techniques require full dataset scans per iteration, limiting their use in data mining.
• Massive datasets necessitate scalable and statistically sound analytical methods.

Purpose of the Study:

• To develop a computationally feasible method for Bayesian analysis of massive datasets.
• To adapt MCMC techniques for large-scale data mining applications.
• To reduce the computational burden of Bayesian inference on large data.

Main Methods:

• Simulating posterior distributions from a subset of data.
• Incorporating remaining data via importance sampling with reweighting.
• Utilizing a rejuvenation step from particle filters to maintain estimation efficiency.

Main Results:

• Demonstrated proof-of-concept on mixture transition models and Bayesian logistic regression.
• Achieved a 99% reduction in data accesses for mixture models without loss of efficiency.
• Achieved a 98% reduction in data accesses for Bayesian logistic regression.

Conclusions:

• The proposed method makes Bayesian analysis computationally feasible for massive datasets.
• The algorithm significantly reduces data access requirements for large-scale statistical modeling.
• This approach offers a scalable solution for applying Bayesian methods in data mining.