Related Experiment Video
Updated: Mar 9, 2026
10:56
A User-friendly and Powerful R Analysis of Large-scale Datasets
Published on: November 4, 2025
434
Online Updating of Statistical Inference in the Big Data Setting
Elizabeth D Schifano1, Jing Wu1, Chun Wang1
1Department of Statistics, University of Connecticut.
Summary
We developed efficient online statistical methods for analyzing streaming big data without historical storage. Our iterative algorithms provide fast, accurate analysis for linear models and estimating equations, outperforming existing approaches.
Area of Science:
- Statistics
- Data Science
- Computational Statistics
Background:
- Big data analytics face challenges with streaming data and limited historical access.
- Online analytical processing (OLAP) requires rapid analysis of continuous data streams.
- Traditional methods struggle with computational efficiency and storage demands for high-velocity data.
Purpose of the Study:
- To develop novel statistical methods for big data from online analytical processing.
- To create iterative, computationally efficient algorithms for streaming data analysis.
- To enable fast statistical inference without requiring historical data storage.
Main Methods:
- Iterative estimating algorithms for linear models and estimating equations.
- Online-updating framework for continuous data streams.
- Development of predictive residual tests for model goodness-of-fit.
- Theoretical analysis of estimators and goodness-of-fit tests.
Main Results:
- Proposed algorithms are computationally efficient and storage-minimal.
- Methods accommodate rank deficiencies in design matrices from rare-event covariates.
- Online-updating framework enables effective goodness-of-fit assessment.
- New online estimator shows favorable comparison in simulations and real data.
Conclusions:
- The developed online statistical methods are effective for big data stream analysis.
- The proposed algorithms offer significant advantages in efficiency and storage.
- The methods provide robust statistical inference and model assessment for streaming data environments.
Related Concept Videos
01:16Statistical Inference Techniques in Hypothesis Testing: Parametric Versus Nonparametric Data
560
Statistical inference techniques, paramount in hypothesis testing, differentiate into two broad categories: parametric and nonparametric statistics.
Parametric statistics, as the name suggests, assumes that data follow a specific distribution, often a normal distribution. This assumption enables robust hypothesis testing and estimation. Parametric methods, like the Student's t-test or Goodness-of-fit test, are frequently employed in biostatistics due to their robustness. For instance,...
Parametric statistics, as the name suggests, assumes that data follow a specific distribution, often a normal distribution. This assumption enables robust hypothesis testing and estimation. Parametric methods, like the Student's t-test or Goodness-of-fit test, are frequently employed in biostatistics due to their robustness. For instance,...
560
01:25Statistical Methods for Analyzing Epidemiological Data
1.1K
Epidemiological data primarily involves information on specific populations' occurrence, distribution, and determinants of health and diseases. This data is crucial for understanding disease patterns and impacts, aiding public health decision-making and disease prevention strategies. The analysis of epidemiological data employs various statistical methods to interpret health-related data effectively. Here are some commonly used methods:
1.1K
01:11Statistical Analysis: Overview
16.8K
When we take repeated measurements on the same or replicated samples, we will observe inconsistencies in the magnitude. These inconsistencies are called errors. To categorize and characterize these results and their errors, the researcher can use statistical analysis to determine the quality of the measurements and/or suitability of the methods.
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
16.8K
01:20Biostatistics: Overview
993
Biostatistics plays a crucial role in understanding and analyzing data in healthcare and biology. Biostatisticians conduct experiments, gather evidence, and draw meaningful conclusions using statistical methods and techniques. Different variables form the foundation of biostatistical analysis, allowing researchers to understand and interpret data effectively. These variables are classified into different types, each serving a specific purpose in statistical analysis.
Discrete variables are...
Discrete variables are...
993
01:16Statistical Hypothesis Testing
7.0K
Hypothesis testing is a critical statistical procedure facilitating informed, evidence-based decisions. It begins with a hypothesis, which is a tentative explanation, or a prediction about a population parameter. This hypothesis can be either a null hypothesis (H0), indicating no effect or difference, or an alternative hypothesis (Ha), suggesting an effect or difference.
Statistical significance measures the probability that an observed result occurred by chance. If this probability, known as...
Statistical significance measures the probability that an observed result occurred by chance. If this probability, known as...
7.0K
01:52Regression Toward the Mean
7.3K
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
7.3K