Real-Time Regression Analysis of Streaming Clustered Data With Possible Abnormal Data Batches

Area of Science:

• Statistics
• Machine Learning
• Data Science

Background:

• Analyzing streaming datasets with correlated outcomes, like longitudinal and clustered data, presents significant computational and statistical challenges.
• Existing methods often require reprocessing all historical data for updates, leading to inefficiency.
• The need for efficient, incremental learning algorithms for dynamic datasets is crucial.

Purpose of the Study:

• To develop an incremental learning algorithm for analyzing streaming datasets with correlated outcomes.
• To propose a novel renewable quadratic inference function (RenewQIF) method for efficient parameter estimation.
• To introduce a sequential goodness-of-fit test for diagnosing regression coefficient homogeneity in streaming data.

Main Methods:

• Developed a renewable quadratic inference function (RenewQIF) for incremental learning.
• Employed a renewable estimation and incremental inference paradigm, updating parameters with current data and historical summary statistics.
• Compared RenewQIF with offline quadratic inference function (QIF) and generalized estimating equations (GEE) methods.
• Proposed a sequential goodness-of-fit test for homogeneity assumption diagnosis.
• Implemented the methodology using an expanded Spark's Lambda architecture.

Main Results:

• Theoretically and numerically demonstrated that the renewable procedure offers statistical and computational efficiency compared to offline methods.
• The proposed sequential goodness-of-fit test effectively screens for abnormal data batches and diagnoses homogeneity assumption violations.
• The methodology was successfully illustrated through extensive simulation studies and real-world analysis of car crash data.

Conclusions:

• The proposed RenewQIF method provides an efficient and statistically sound approach for analyzing streaming data with correlated outcomes.
• The integrated data quality diagnosis tool enhances the reliability of statistical inference in dynamic environments.
• This work offers a scalable solution for real-time statistical analysis in big data applications.