Related Experiment Video
Updated: Nov 27, 2025
07:35
Selecting Multiple Biomarker Subsets with Similarly Effective Binary Classification Performances
Published on: October 11, 2018
7.8K
On the Use of Entropy to Improve Model Selection Criteria
Andrea Murari1, Emmanuele Peluso2, Francesco Cianfrani2
1Consorzio RFX (CNR, ENEA, INFN, Universita' di Padova, Acciaierie Venete SpA), 35127 Padova, Italy.
Entropy (Basel, Switzerland)
|December 3, 2020
Summary
This study enhances model selection criteria like Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC) by incorporating Shannon entropy for better residual distribution analysis. This improves model performance, especially with Gaussian noise and outliers.
Area of Science:
- Statistics
- Data Science
- Machine Learning
Background:
- Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC) are standard model selection tools.
- These criteria rely on variance and mean-squared error of residuals.
- Gaussian noise is common in scientific data.
Purpose of the Study:
- To improve existing model selection criteria (BIC and AIC).
- To incorporate residual distribution uniformity into model selection.
- To enhance model performance in the presence of Gaussian noise and outliers.
Main Methods:
- Quantified residual uniformity using Shannon entropy.
- Integrated Shannon entropy into BIC and AIC formulations.
- Employed simulations with various functions and noise levels.
- Utilized Geodesic Distance for outlier treatment.
Main Results:
- Enhanced BIC and AIC incorporating Shannon entropy showed significant performance improvements.
- Simulations confirmed better model selection across different function classes and noise conditions.
- Geodesic Distance provided superior error handling for datasets with outliers.
Conclusions:
- Shannon entropy integration offers a more robust approach to model selection.
- The enhanced criteria are particularly effective for data with Gaussian noise.
- Geodesic Distance is crucial for reliable model selection in the presence of outliers.
Related Concept Videos
02:39Entropy
33.7K
Salt particles that have dissolved in water never spontaneously come back together in solution to reform solid particles. Moreover, a gas that has expanded in a vacuum remains dispersed and never spontaneously reassembles. The unidirectional nature of these phenomena is the result of a thermodynamic state function called entropy (S). Entropy is the measure of the extent to which the energy is dispersed throughout a system, or in other words, it is proportional to the degree of disorder of a...
33.7K
01:18Entropy
3.3K
The first law of thermodynamics is quantitatively formulated via an equation relating the internal energy of a system, the heat exchanged by it, and the work done on it. A quantitative formulation of the second law of thermodynamics leads to defining a state function, the entropy.
When an ideal gas expands isothermally, the disorder in the gas increases. From the molecular perspective, the gas molecules have more volume to move around in.
Consider an infinitesimal step in the expansion, which...
When an ideal gas expands isothermally, the disorder in the gas increases. From the molecular perspective, the gas molecules have more volume to move around in.
Consider an infinitesimal step in the expansion, which...
3.3K
01:20Entropy and the Second Law of Thermodynamics
4.0K
The second law of thermodynamics can be stated quantitatively using the concept of entropy. Entropy is the measure of disorder of the system.
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
4.0K
01:14The Second Law of Thermodynamics
6.3K
In the quest to identify a property that may reliably predict the spontaneity of a process, a promising candidate has been identified: entropy. Scientists refer to the measure of randomness or disorder within a system as entropy. High entropy means high disorder and low energy. To better understand entropy, think of a student’s bedroom. If no energy or work were put into it, the room would quickly become messy. It would exist in a very disordered state, one of high entropy. Energy must be...
6.3K
01:19Survival Tree
255
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
255
01:29Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
185
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
185