Related Experiment Video
Updated: Nov 27, 2025
08:00
Decoding Natural Behavior from Neuroethological Embedding
Published on: October 3, 2025
339
Cross Entropy of Neural Language Models at Infinity-A New Bound of the Entropy Rate
Shuntaro Takahashi1, Kumiko Tanaka-Ishii2
1Graduate School of Frontier Sciences, The University of Tokyo, Chiba 277-8561, Japan.
Entropy (Basel, Switzerland)
|December 3, 2020
Summary
Researchers estimated natural language entropy rate using neural language models. They found the entropy rate for English is 1.12 bits per character, potentially lower than previously reported values.
Area of Science:
- Computational Linguistics
- Information Theory
Background:
- Neural language models (NLMs) excel at predicting text.
- Estimating the entropy rate of natural language is crucial for understanding language complexity.
Purpose of the Study:
- To estimate the entropy rate of natural language using state-of-the-art neural language models.
- To investigate the impact of training data size and context length on cross-entropy predictions.
Main Methods:
- Utilized cross-entropy as a measure of NLM prediction accuracy.
- Theoretically idealized conditions: infinite training data and infinite context length.
- Empirically verified power-law decay for data size and context length effects on cross-entropy.
Main Results:
- Observed consistent power-law decay for training data size and context length effects across different NLMs and datasets.
- Extrapolated to infinite parameters, yielding an English entropy rate of 1.12 bits per character.
Conclusions:
- The estimated entropy rate of 1.12 bits/character for English suggests a potentially lower upper bound than previously reported.
- This finding has implications for language modeling, compression, and understanding linguistic structure.
Related Concept Videos
01:18Entropy
3.3K
The first law of thermodynamics is quantitatively formulated via an equation relating the internal energy of a system, the heat exchanged by it, and the work done on it. A quantitative formulation of the second law of thermodynamics leads to defining a state function, the entropy.
When an ideal gas expands isothermally, the disorder in the gas increases. From the molecular perspective, the gas molecules have more volume to move around in.
Consider an infinitesimal step in the expansion, which...
When an ideal gas expands isothermally, the disorder in the gas increases. From the molecular perspective, the gas molecules have more volume to move around in.
Consider an infinitesimal step in the expansion, which...
3.3K
02:39Entropy
33.7K
Salt particles that have dissolved in water never spontaneously come back together in solution to reform solid particles. Moreover, a gas that has expanded in a vacuum remains dispersed and never spontaneously reassembles. The unidirectional nature of these phenomena is the result of a thermodynamic state function called entropy (S). Entropy is the measure of the extent to which the energy is dispersed throughout a system, or in other words, it is proportional to the degree of disorder of a...
33.7K
01:20Entropy and the Second Law of Thermodynamics
4.0K
The second law of thermodynamics can be stated quantitatively using the concept of entropy. Entropy is the measure of disorder of the system.
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
4.0K
01:10Entropy Change in Reversible Processes
3.0K
In the Carnot engine, which achieves the maximum efficiency between two reservoirs of fixed temperatures, the total change in entropy is zero. The observation can be generalized by considering any reversible cyclic process consisting of many Carnot cycles. Thus, it can be stated that the total entropy change of any ideal reversible cycle is zero.
The statement can be further generalized to prove that entropy is a state function. Take a cyclic process between any two points on a p-V diagram.
The statement can be further generalized to prove that entropy is a state function. Take a cyclic process between any two points on a p-V diagram.
3.0K
01:14The Second Law of Thermodynamics
6.3K
In the quest to identify a property that may reliably predict the spontaneity of a process, a promising candidate has been identified: entropy. Scientists refer to the measure of randomness or disorder within a system as entropy. High entropy means high disorder and low energy. To better understand entropy, think of a student’s bedroom. If no energy or work were put into it, the room would quickly become messy. It would exist in a very disordered state, one of high entropy. Energy must be...
6.3K
02:38Third Law of Thermodynamics
21.0K
A pure, perfectly crystalline solid possessing no kinetic energy (that is, at a temperature of absolute zero, 0 K) may be described by a single microstate, as its purity, perfect crystallinity,and complete lack of motion means there is but one possible location for each identical atom or molecule comprising the crystal (W = 1). According to the Boltzmann equation, the entropy of this system is zero.
21.0K