PEnBayes: A Multi-Layered Ensemble Approach for Learning Bayesian Network Structure from Big Data | JoVE Visualize

Area of Science:

Computer Science
Data Science
Artificial Intelligence

Background:

Discovering Bayesian network (BN) structures from big datasets with causal relationships is crucial for modeling and reasoning under uncertainty.
Current BN structure learning algorithms struggle with big data due to memory constraints, difficulty in selecting accurate learners, and challenges in merging learned structures.
Existing methods often fail to efficiently process large volumes of sensor data common in fields like patient monitoring and transportation.

Purpose of the Study:

To introduce a novel parallel learning approach, PEnBayes (Parallel Ensemble-based Bayesian network learning), to address the shortcomings of current BN structure learning algorithms for big data.
To develop an intelligent method for preprocessing big datasets, enabling fast distributed local structure learning.
To create an effective ensemble method for merging local BN structures into a robust global network.

Main Methods:

PEnBayes employs adaptive data preprocessing to determine appropriate learning size and intelligently partition big datasets.
It utilizes a two-layered weighted adjacent matrix-based structure ensemble method for parallel learning of local BN structures.
Local BN structures are merged into a global network using a structure ensemble method at the global layer.

Main Results:

PEnBayes demonstrates significantly improved execution performance compared to three baseline learning algorithms.
The approach yields more consistent and stable results in learning BN structures from simulated big datasets.
Experiments were conducted using simulated sensor data from patient monitoring, transportation, and disease diagnosis domains.

Conclusions:

PEnBayes effectively overcomes the limitations of traditional BN structure learning algorithms when applied to big data.
The parallel ensemble-based approach provides a scalable and accurate solution for discovering causal relationships in complex, high-volume datasets.
This method offers improved efficiency, consistency, and stability for Bayesian network discovery in various data-intensive applications.