Related Experiment Video
Updated: Jul 5, 2025
11:15
Applications of EEG Neuroimaging Data: Event-related Potentials, Spectral Power, and Multiscale Entropy
Published on: June 27, 2013
33.7K
Entropy Estimators for Markovian Sequences: A Comparative Analysis
Juan De Gregorio1, David Sánchez1, Raúl Toral1
1Institute for Cross-Disciplinary Physics and Complex Systems IFISC (UIB-CSIC), Campus Universitat de les Illes Balears, E-07122 Palma de Mallorca, Spain.
Entropy (Basel, Switzerland)
|January 22, 2024
Summary
This study compares entropy estimators for sequences with memory. Results show performance varies with system properties and data size, offering insights for information theory applications.
Area of Science:
- Information Theory
- Statistical Modeling
- Computational Science
Background:
- Entropy estimation is crucial across diverse scientific fields.
- Accurate entropy estimation for sequences with memory (Markovian systems) is challenging due to data limitations and estimator biases.
- Existing estimators often assume independent events, limiting their applicability to complex systems.
Purpose of the Study:
- To systematically compare the performance of various entropy estimators when applied to Markovian sequences.
- To analyze the impact of system properties, such as transition probabilities and sample size, on estimator accuracy.
- To identify limitations and provide guidance for entropy estimation in systems with memory.
Main Methods:
- Evaluation of common entropy estimators using binary Markovian sequences.
- Analysis of Markovian systems, particularly in the undersampled regime.
- Calculation of bias, standard deviation, and mean squared error for selected estimators.
Main Results:
- Performance of entropy estimators is significantly influenced by the transition probabilities of the Markov process.
- Estimator accuracy degrades in undersampled regimes, highlighting the impact of sample size.
- Different estimators exhibit varying degrees of bias and variance, affecting their suitability for specific Markovian systems.
Conclusions:
- No single entropy estimator is universally optimal for all Markovian sequences.
- Understanding the interplay between estimator properties, system memory, and data availability is key for accurate entropy estimation.
- This comparative analysis provides valuable insights for researchers applying entropy estimation to systems with temporal dependencies.
Related Concept Videos
02:39Entropy
30.2K
Salt particles that have dissolved in water never spontaneously come back together in solution to reform solid particles. Moreover, a gas that has expanded in a vacuum remains dispersed and never spontaneously reassembles. The unidirectional nature of these phenomena is the result of a thermodynamic state function called entropy (S). Entropy is the measure of the extent to which the energy is dispersed throughout a system, or in other words, it is proportional to the degree of disorder of a...
30.2K
01:26Per-Unit Sequence Models
74
An ideal Y-Y transformer, grounded through neutral impedances, displays per-unit sequence networks akin to those of a single-phase ideal transformer when subjected to balanced positive- or negative-sequence currents. These currents do not produce neutral currents, and their associated voltage drops.
Zero-sequence currents, which are identical in magnitude and phase, generate a neutral current, resulting in voltage drops across the neutral impedance and the low-voltage winding. If the...
Zero-sequence currents, which are identical in magnitude and phase, generate a neutral current, resulting in voltage drops across the neutral impedance and the low-voltage winding. If the...
74
01:20Entropy and the Second Law of Thermodynamics
2.8K
The second law of thermodynamics can be stated quantitatively using the concept of entropy. Entropy is the measure of disorder of the system.
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
2.8K
00:59Propagation of Uncertainty from Random Error
691
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
691
01:10Entropy Change in Reversible Processes
2.5K
In the Carnot engine, which achieves the maximum efficiency between two reservoirs of fixed temperatures, the total change in entropy is zero. The observation can be generalized by considering any reversible cyclic process consisting of many Carnot cycles. Thus, it can be stated that the total entropy change of any ideal reversible cycle is zero.
The statement can be further generalized to prove that entropy is a state function. Take a cyclic process between any two points on a p-V diagram.
The statement can be further generalized to prove that entropy is a state function. Take a cyclic process between any two points on a p-V diagram.
2.5K
01:10Propagation of Uncertainty from Systematic Error
521
The atomic mass of an element varies due to the relative ratio of its isotopes. A sample's relative proportion of oxygen isotopes influences its average atomic mass. For instance, if we were to measure the atomic mass of oxygen from a sample, the mass would be a weighted average of the isotopic masses of oxygen in that sample. Since a single sample is not likely to perfectly reflect the true atomic mass of oxygen for all the molecules of oxygen on Earth, the mass we obtain from this...
521