Neural Estimator of Information for Time-Series Data with Dependency
Sina Molavipour1, Hamid Ghourchian1, Germán Bassi2
1School of Electrical Engineering and Computer Science (EECS), KTH Royal Institute of Technology, 100 44 Stockholm, Sweden.
Entropy (Basel, Switzerland)
|June 2, 2021
Summary
Neural networks can now consistently estimate information measures, even with dependent data. This breakthrough in machine learning and information theory is crucial for analyzing complex, time-dependent variables and causality.
Area of Science:
- Information Theory
- Machine Learning
- Causality Analysis
Background:
- Neural network estimators for information measures are popular but their consistency with dependent samples is unclear.
- Directed information is vital for time-dependent variables and causality but its estimation is complex.
- Existing convergence proofs for neural estimators often assume independent samples.
Purpose of the Study:
- To investigate the consistency of neural estimators for information measures with dependent data.
- To extend convergence proofs for neural estimators to stationary and ergodic data sources.
- To apply and validate the approach for estimating directed information.
Main Methods:
- Utilizing neural networks for information measure estimation.
- Applying Birkhoff's ergodic theorem to prove estimator convergence.
- Simulating directed information estimation with the proposed method.
Main Results:
- Demonstrated the consistency of a neural estimator for conditional mutual information with stationary and ergodic data.
- Proved asymptotic convergence to the true value with probability one.
- Successfully applied the technique to estimate directed information in simulations.
Conclusions:
- Neural estimators for information measures can be consistent even with dependent, stationary, and ergodic data.
- Birkhoff's ergodic theorem is key to establishing convergence guarantees.
- The proposed method offers a robust approach for estimating directed information and analyzing causality.
Related Concept Videos
01:03Prediction Intervals
2.5K
The interval estimate of any variable is known as the prediction interval. It helps decide if a point estimate is dependable.
However, the point estimate is most likely not the exact value of the population parameter, but close to it. After calculating point estimates, we construct interval estimates, called confidence intervals or prediction intervals. This prediction interval comprises a range of values unlike the point estimate and is a better predictor of the observed sample value, y.
However, the point estimate is most likely not the exact value of the population parameter, but close to it. After calculating point estimates, we construct interval estimates, called confidence intervals or prediction intervals. This prediction interval comprises a range of values unlike the point estimate and is a better predictor of the observed sample value, y.
2.5K
01:29Econometric Views (EViews)
331
Econometric Views, often stylized as EViews, is a package that merges statistical analysis with econometric studies. It is designed to provide tools for time series analysis, forecasting, and econometric model simulation. The software originated from MicroTSP software and has evolved significantly since its inception in 1981. The history of EViews is marked by a continuous effort to enhance its computational speed and user interface. It was initially developed for large computing systems but...
331
01:24One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for ka Estimation
852
This lesson introduces two critical methods in pharmacokinetics, the Wagner-Nelson and Loo-Riegelman methods, used for estimating the absorption rate constant (ka) for drugs administered via non-intravenous routes. The Wagner-Nelson method relates ka to the plasma concentration derived from the slope of a semilog percent unabsorbed time plot. However, it is limited to drugs with one-compartment kinetics and can be impacted by factors like gastrointestinal motility or enzymatic degradation.
On...
On...
852
01:06What are Estimates?
6.8K
It isn't easy to measure a parameter such as the mean height or the mean weight of a population. So, we draw samples from the population and calculate the mean height or mean weight of the individuals in the sample. This sample data acts as a representative measure of the population parameter. These sample statistics are known as estimates.
The estimate for the mean of a sample is denoted by ͞x, whereas the mean of the population is designated as μ. Further, parameters such...
The estimate for the mean of a sample is denoted by ͞x, whereas the mean of the population is designated as μ. Further, parameters such...
6.8K
01:22Estimating Population Mean with Unknown Standard Deviation
8.4K
In practice, we rarely know the population standard deviation. In the past, when the sample size was large, this did not present a problem to statisticians. They used the sample standard deviation s as an estimate for σ and proceeded as before to calculate a confidence interval with close enough results. However, statisticians ran into problems when the sample size was small. A small sample size caused inaccuracies in the confidence interval.
William S. Gosset (1876–1937) of the...
William S. Gosset (1876–1937) of the...
8.4K
01:26Estimating Population Standard Deviation
3.1K
When the population standard deviation is unknown and the sample size is large, the sample standard deviation s is commonly used as a point estimate of σ. However, it can sometimes under or overestimate the population standard deviation. To overcome this drawback, confidence intervals are determined to estimate population parameters and eliminate any calculation bias accurately. However, this only applies to random samples from normally distributed populations. Knowing the sample mean and...
3.1K