Predictive Modeling With Big Data: Is Bigger Really Better?

Area of Science:

• Computer Science
• Data Science
• Machine Learning

Background:

• Big data analytics is increasingly used for decision-making.
• Predictive analytics is a key method for leveraging data.
• The impact of data scale on predictive model performance remains an open question.

Purpose of the Study:

• To empirically investigate the relationship between data scale and predictive model performance.
• To determine if larger datasets lead to significant improvements in predictive accuracy.
• To assess the value of increased data instances and features for predictive tasks.

Main Methods:

• Empirical analysis across nine diverse predictive modeling applications.
• Utilizing sparse, fine-grained data, including human behavior and transaction data.
• Development and application of a scalable multivariate Bernoulli Naïve Bayes algorithm for big data.

Main Results:

• Predictive performance shows marginal but continuous increases with larger datasets, even at massive scales.
• Sparse, fine-grained data benefits significantly from increased data volume and features.
• The study confirms that larger data assets enhance predictive analytics capabilities.

Conclusions:

• Institutions with extensive data assets and analytical skills gain a competitive advantage.
• Companies should prioritize gathering more data instances and features for predictive tasks.
• Scalable algorithms are crucial for effectively utilizing big data in predictive modeling.