Related Experiment Video
Updated: Nov 27, 2025
11:00
Experimental Investigation of Secondary Flow Structures Downstream of a Model Type IV Stent Failure in a 180° Curved Artery Test Section
Published on: July 19, 2016
11.8K
Shannon Entropy Estimation in ∞-Alphabets from Convergence Results: Studying Plug-In Estimators
1Information and Decision System Group, Department of Electrical Engineering, Universidad de Chile, Av. Tupper 2007, Santiago 7591538, Chile.
Entropy (Basel, Switzerland)
|December 3, 2020
Summary
This study introduces novel methods for estimating Shannon entropy in infinite alphabets, developing new consistent estimators. These data-driven approaches optimize accuracy by balancing estimation and approximation errors for improved entropy calculation.
Area of Science:
- Information Theory
- Statistical Inference
- Probability Theory
Background:
- Estimating Shannon entropy in countably infinite alphabets is challenging due to the discontinuity of the entropy functional.
- Existing methods often struggle with infinite support distributions and require prior knowledge of the distribution's properties.
- Recent convergence results offer potential avenues for developing more robust entropy estimators.
Purpose of the Study:
- To develop new, strongly consistent estimators for Shannon entropy in countably infinite alphabets.
- To leverage recent convergence results and deviation inequalities for improved entropy estimation.
- To introduce a data-driven, plug-in estimator that adapts to the underlying data distribution.
Main Methods:
- Utilized recent convergence results for entropy functionals in infinite alphabets.
- Applied deviation inequalities under both finite and infinite support assumptions.
- Developed and analyzed four plug-in histogram-based estimators, focusing on a novel data-driven partition estimator.
Main Results:
- Established conditions for the convergence of entropy functionals in infinite alphabets.
- Derived new strongly consistent estimators for Shannon entropy.
- The proposed data-driven partition estimator provides a consistent, distribution-free entropy estimate in infinite alphabets with optimal convergence rates under specific conditions.
Conclusions:
- The developed methodology provides a robust framework for Shannon entropy estimation in infinite alphabets.
- The data-driven partition estimator effectively balances estimation and approximation errors, offering improved performance.
- This work advances the field of information theory by providing practical and theoretically sound tools for entropy estimation.
Related Concept Videos
01:21Convergence of Fourier Series
285
The Fourier series is a powerful mathematical tool for representing periodic signals as an infinite sum of complex exponentials. In practice, this infinite series is truncated to a finite number of terms, yielding a partial sum. This truncation makes the approximation of the signal feasible but introduces certain challenges, particularly near discontinuities, known as the Gibbs phenomenon.
The Gibbs phenomenon refers to the persistent oscillations and overshoots that occur near discontinuities...
The Gibbs phenomenon refers to the persistent oscillations and overshoots that occur near discontinuities...
285
02:39Entropy
33.7K
Salt particles that have dissolved in water never spontaneously come back together in solution to reform solid particles. Moreover, a gas that has expanded in a vacuum remains dispersed and never spontaneously reassembles. The unidirectional nature of these phenomena is the result of a thermodynamic state function called entropy (S). Entropy is the measure of the extent to which the energy is dispersed throughout a system, or in other words, it is proportional to the degree of disorder of a...
33.7K
01:18Entropy
3.3K
The first law of thermodynamics is quantitatively formulated via an equation relating the internal energy of a system, the heat exchanged by it, and the work done on it. A quantitative formulation of the second law of thermodynamics leads to defining a state function, the entropy.
When an ideal gas expands isothermally, the disorder in the gas increases. From the molecular perspective, the gas molecules have more volume to move around in.
Consider an infinitesimal step in the expansion, which...
When an ideal gas expands isothermally, the disorder in the gas increases. From the molecular perspective, the gas molecules have more volume to move around in.
Consider an infinitesimal step in the expansion, which...
3.3K
01:30The Squeeze Theorem
69
Certain mathematical functions exhibit unpredictable or highly variable behavior near specific input values, making direct evaluation of their limits challenging. This complexity may arise from rapid oscillations or irregular patterns that obscure the function’s trend. In such cases, the Squeeze Theorem offers a reliable method for determining limits.According to the Squeeze Theorem, if a function is confined between two other functions near a particular point, and both outer functions...
69
01:24One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for ka Estimation
940
This lesson introduces two critical methods in pharmacokinetics, the Wagner-Nelson and Loo-Riegelman methods, used for estimating the absorption rate constant (ka) for drugs administered via non-intravenous routes. The Wagner-Nelson method relates ka to the plasma concentration derived from the slope of a semilog percent unabsorbed time plot. However, it is limited to drugs with one-compartment kinetics and can be impacted by factors like gastrointestinal motility or enzymatic degradation.
On...
On...
940
01:20Entropy and the Second Law of Thermodynamics
4.0K
The second law of thermodynamics can be stated quantitatively using the concept of entropy. Entropy is the measure of disorder of the system.
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
4.0K