Causation and information flow with respect to relative entropy

Area of Science:

• Dynamical Systems Theory
• Information Theory
• Statistical Mechanics

Background:

• Formalisms for information flow and causality in dynamical systems typically use Shannon entropy.
• Previous methods often yield inconsistent results for higher-dimensional systems.
• Relative entropy (Kullback-Leiber divergence) offers desirable properties like invariance and thermodynamic consistency.

Purpose of the Study:

• To re-establish the formalism for information flow and causality using relative entropy.
• To extend the formalism to systems of arbitrary dimensionality.
• To investigate the properties of the resulting information flow measure.

Main Methods:

• Utilizing relative entropy (Kullback-Leiber divergence) as the measure of predictability.
• Developing a formalism for information flow and causality applicable to arbitrary dimensions.
• Validating the formalism with a stochastic gradient system.

Main Results:

• The information flow (T) derived from relative entropy is consistent with that from Shannon entropy across all dimensions, differing only by a sign.
• The formalism demonstrates the principle of nil causality, a property often missed by classical methods.
• The information flow is invariant under nonlinear transformations, suggesting it's an intrinsic physical property.

Conclusions:

• The relative entropy-based formalism provides a consistent and robust framework for quantifying information flow and causality in dynamical systems.
• This approach offers a more generalizable and physically meaningful understanding of information dynamics.
• The invariance property highlights the fundamental nature of this information flow measure.