Related Experiment Video
Updated: Mar 29, 2026
11:00
Experimental Investigation of Secondary Flow Structures Downstream of a Model Type IV Stent Failure in a 180° Curved Artery Test Section
Published on: July 19, 2016
12.0K
An Invariant Measure for Differential Entropy: From Kullback-Leibler Divergence to Scale-Invariant Information Theory
Félix Truong1, Alexandre Giuliani1,2
1Synchrotron SOLEIL, L'Orme des Merisiers, 91190 Saint-Aubin, France.
Entropy (Basel, Switzerland)
|March 28, 2026
Summary
This study introduces a scale-invariant differential entropy measure, overcoming limitations of traditional methods. The new approach ensures entropy is independent of measurement units, crucial for reliable data analysis.
Area of Science:
- Information Theory
- Statistical Mechanics
- Machine Learning
Background:
- Shannon's differential entropy lacks scale invariance, complicating interpretation.
- Existing methods struggle with variables across different scales.
- Jaynes's limiting density of discrete points (LDDP) lacked a practical computation method.
Purpose of the Study:
- Develop a scale-invariant differential entropy measure.
- Establish a rigorous connection between Kullback-Leibler divergence and invariant measures.
- Provide a practical computational method for invariant entropy.
Main Methods:
- Proved entropy normalized by median k-nearest neighbor distances is invariant under affine transformations.
- Established theoretical proofs of invariance under affine transformations.
- Connected Kullback-Leibler divergence to the invariant measure.
Main Results:
- Developed a novel, scale-invariant differential entropy computation method.
- Empirically validated non-negativity of the new entropy measure.
- Extended the approach to multivariate settings for scale-invariant mutual information.
Conclusions:
- The proposed method provides a robust, scale-invariant entropy measure.
- This overcomes fundamental limitations of traditional differential entropy.
- Open-source implementations facilitate practical application in diverse fields.
Keywords:
Kullback–Leibler divergencedifferential entropyinvariant measurek-nearest neighborlimiting density of discrete pointsmutual informationscale invarianceMore Related Videos
09:23Quantification of Information Encoded by Gene Expression Levels During Lifespan Modulation Under Broad-range Dietary Restriction in C. elegans
Published on: August 16, 2017
8.7K
10:05Measurement & Analysis of the Temporal Discrimination Threshold Applied to Cervical Dystonia
Published on: January 27, 2018
10.3K
Related Concept Videos
01:14The Entropy as a State Function
93
Consider an arbitrary process that moves between two specific states (A and B) in a cyclic manner. This process is reversible and broken down into smaller parts that each follow a Carnot cycle. A Carnot cycle has two isothermal (constant temperature) processes. During these processes, the ratio of the amount of heat transferred to their respective temperature remains constant. The other two processes in the Carnot cycle are also reversible but adiabatic, which means they occur without any heat...
93
01:20Entropy and the Second Law of Thermodynamics
5.3K
The second law of thermodynamics can be stated quantitatively using the concept of entropy. Entropy is the measure of disorder of the system.
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
5.3K
01:26Entropy and the Second Law of Thermodynamics
262
Consider an isolated system in which a hot object is placed in contact with a cold one. This is an irreversible process that eventually leads both objects to reach the same equilibrium temperature. It is crucial to note that the constituents of any substance exhibit increased disorder at higher temperatures. As a cold substance absorbs heat, its constituents become more disordered. The energy transfer from a hotter object to a cooler one increases the system's disorder or randomness. This...
262
02:39Entropy
37.8K
Salt particles that have dissolved in water never spontaneously come back together in solution to reform solid particles. Moreover, a gas that has expanded in a vacuum remains dispersed and never spontaneously reassembles. The unidirectional nature of these phenomena is the result of a thermodynamic state function called entropy (S). Entropy is the measure of the extent to which the energy is dispersed throughout a system, or in other words, it is proportional to the degree of disorder of a...
37.8K
01:18Entropy
3.8K
The first law of thermodynamics is quantitatively formulated via an equation relating the internal energy of a system, the heat exchanged by it, and the work done on it. A quantitative formulation of the second law of thermodynamics leads to defining a state function, the entropy.
When an ideal gas expands isothermally, the disorder in the gas increases. From the molecular perspective, the gas molecules have more volume to move around in.
Consider an infinitesimal step in the expansion, which...
When an ideal gas expands isothermally, the disorder in the gas increases. From the molecular perspective, the gas molecules have more volume to move around in.
Consider an infinitesimal step in the expansion, which...
3.8K
01:14The Second Law of Thermodynamics
7.2K
In the quest to identify a property that may reliably predict the spontaneity of a process, a promising candidate has been identified: entropy. Scientists refer to the measure of randomness or disorder within a system as entropy. High entropy means high disorder and low energy. To better understand entropy, think of a student’s bedroom. If no energy or work were put into it, the room would quickly become messy. It would exist in a very disordered state, one of high entropy. Energy must be...
7.2K