Regularities unseen, randomness observed: levels of entropy convergence

Area of Science:

• Information Theory
• Computational Complexity
• Statistical Mechanics

Background:

• Understanding information source behavior is crucial for data analysis and prediction.
• Existing methods often struggle with short sequences and complex system dynamics.
• The relationship between observed randomness and underlying process structure requires further investigation.

Purpose of the Study:

• To analyze the convergence of Shannon entropy to an information source's entropy rate.
• To develop information-theoretic measures for apparent memory and synchronization difficulty.
• To investigate the impact of process structure on entropy convergence and apparent randomness.

Main Methods:

• Synthesized phenomenological approaches using successive derivatives of the Shannon entropy growth curve.
• Defined and analyzed 'transient information' for measuring synchronization difficulty in Markov processes.
• Performed numerical and analytical determinations of entropy growth curves for various stochastic and deterministic processes.

Main Results:

• Developed natural measures for apparent memory and information required for optimal prediction and synchronization.
• Proved a relationship between transient information and total uncertainty during synchronization for Markov processes.
• Demonstrated that ignoring structural properties can convert missed regularities into apparent randomness, especially with short sequences.

Conclusions:

• The study provides a framework for understanding information source dynamics through entropy convergence.
• Apparent memory and synchronization difficulty can be quantified using information-theoretic measures.
• Entropy convergence behavior is intrinsically linked to a process's underlying computational structure.