Distributed Simultaneous Inference in Generalized Linear Models via Confidence Distribution.

Area of Science:

• Statistics
• Machine Learning
• Data Science

Background:

• Analyzing large-scale datasets (N >> p) in generalized linear models presents computational challenges for single machines.
• Existing distributed 'divide-and-combine' strategies face issues with uneven data partitions and require robust regularization.
• Lack of clear theoretical guidance for combining regularized estimates hinders simultaneous inference in distributed settings.

Purpose of the Study:

• To develop a statistically sound and scalable distributed method for simultaneous inference in generalized linear models.
• To address the challenges of combining regularized estimates from partitioned datasets.
• To provide a practical approach for analyzing big data where centralized computation is infeasible.

Main Methods:

• Proposed a novel 'divide-and-combine' strategy for distributed data analysis.
• Employed bias-corrected lasso-type estimators to handle potential numerical instability from data partitioning.
• Utilized confidence distributions to effectively combine results from distributed sub-analyses for simultaneous inference.

Main Results:

• The developed distributed method achieves estimation efficiency equivalent to centralized maximum likelihood estimation.
• The combined estimator demonstrates robust performance even with unevenly sized data partitions.
• Simulated and real-world data analyses confirm that the proposed method yields inference nearly identical to centralized benchmarks.

Conclusions:

• The proposed distributed method offers a scalable and theoretically justified solution for simultaneous inference in large-scale generalized linear models.
• This approach effectively overcomes the limitations of traditional centralized analysis for big data.
• The method provides a reliable alternative for researchers and practitioners dealing with massive datasets stored in distributed systems.