Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Estimation of the Physical Quantities01:05

Estimation of the Physical Quantities

4.2K
On many occasions, physicists, other scientists, and engineers need to make estimates of a particular quantity. These are sometimes referred to as guesstimates, order-of-magnitude approximations, back-of-the-envelope calculations, or Fermi calculations. The physicist Enrico Fermi was famous for his ability to estimate various kinds of data with surprising precision. Estimating does not mean guessing a number or a formula at random. Instead, estimation means using prior experience and sound...
4.2K
What are Estimates?01:06

What are Estimates?

5.0K
It isn't easy to measure a parameter such as the mean height or the mean weight of a population. So, we draw samples from the population and calculate the mean height or mean weight of the individuals in the sample. This sample data acts as a representative measure of the population parameter. These sample statistics are known as estimates. 
The estimate for the mean of a sample is denoted by ͞x, whereas the mean of the population is designated as μ. Further, parameters such...
5.0K
Distributions to Estimate Population Parameter01:26

Distributions to Estimate Population Parameter

4.1K
The accurate values of population parameters such as population proportion, population mean, and population standard deviation (or variance) are usually unknown. These are fixed values that can only be estimated from the data collected from the samples. The estimates of each of these parameters are sample proportion, the sample mean, and sample standard deviation (or variance). To obtain the values of these sample statistics, data are required that have particular distribution and central...
4.1K
One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for ka Estimation01:24

One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for ka Estimation

481
This lesson introduces two critical methods in pharmacokinetics, the Wagner-Nelson and Loo-Riegelman methods, used for estimating the absorption rate constant (ka) for drugs administered via non-intravenous routes. The Wagner-Nelson method relates ka to the plasma concentration derived from the slope of a semilog percent unabsorbed time plot. However, it is limited to drugs with one-compartment kinetics and can be impacted by factors like gastrointestinal motility or enzymatic degradation.
On...
481
Estimating Population Mean with Unknown Standard Deviation01:22

Estimating Population Mean with Unknown Standard Deviation

7.7K
In practice, we rarely know the population standard deviation. In the past, when the sample size was large, this did not present a problem to statisticians. They used the sample standard deviation s as an estimate for σ and proceeded as before to calculate a confidence interval with close enough results. However, statisticians ran into problems when the sample size was small. A small sample size caused inaccuracies in the confidence interval.
William S. Gosset (1876–1937) of the...
7.7K
Sample Size Calculation01:19

Sample Size Calculation

3.3K
Knowledge of the sample size is the first requirement to conduct random sampling or an experiment. The sample size is the total number of units, observations, or groups (in some cases) used to get the data to estimate a population parameter. As the name suggests, the sample size is that of the sample drawn from the population and differs from the population size.
The sample size for the given experiment or sampling effort is fundamental to any study design. Sample size decides the number of...
3.3K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Computing Accurate Probabilistic Estimates of One-D Entropy from Equiprobable Random Samples.

Entropy (Basel, Switzerland)·2021
Same author

A Maximum-Entropy Method to Estimate Discrete Distributions from Samples Ensuring Nonzero Probabilities.

Entropy (Basel, Switzerland)·2020
Same author

An Information Theory Approach to Identifying a Representative Subset of Hydro-Climatic Simulations for Impact Modeling Studies.

Water resources research·2018
See all related articles

Related Experiment Video

Updated: Jun 25, 2025

Quantification of Information Encoded by Gene Expression Levels During Lifespan Modulation Under Broad-range Dietary Restriction in C. elegans
09:23

Quantification of Information Encoded by Gene Expression Levels During Lifespan Modulation Under Broad-range Dietary Restriction in C. elegans

Published on: August 16, 2017

8.1K

On the Accurate Estimation of Information-Theoretic Quantities from Multi-Dimensional Sample Data.

Manuel Álvarez Chaves1, Hoshin V Gupta2, Uwe Ehret3

  • 1Stuttgart Center for Simulation Science, Cluster of Excellence EXC 2075, University of Stuttgart, 70569 Stuttgart, Germany.

Entropy (Basel, Switzerland)
|May 24, 2024
PubMed
Summary

Estimating information-theoretic quantities from continuous data is challenging. The k-nearest neighbors (k-NN) method generally outperforms kernel density estimation and histograms, especially with sufficient data, offering better accuracy and efficiency.

Keywords:
Kullback–Leibler divergencebinningdataentropyinformation theoryk-NNk-nearest neighborskernel density estimationmutual informationnon-parametric estimationrelative entropy

More Related Videos

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments
08:12

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

Published on: March 1, 2022

2.5K
Using Informational Connectivity to Measure the Synchronous Emergence of fMRI Multi-voxel Information Across Time
07:12

Using Informational Connectivity to Measure the Synchronous Emergence of fMRI Multi-voxel Information Across Time

Published on: July 1, 2014

12.3K

Related Experiment Videos

Last Updated: Jun 25, 2025

Quantification of Information Encoded by Gene Expression Levels During Lifespan Modulation Under Broad-range Dietary Restriction in C. elegans
09:23

Quantification of Information Encoded by Gene Expression Levels During Lifespan Modulation Under Broad-range Dietary Restriction in C. elegans

Published on: August 16, 2017

8.1K
A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments
08:12

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

Published on: March 1, 2022

2.5K
Using Informational Connectivity to Measure the Synchronous Emergence of fMRI Multi-voxel Information Across Time
07:12

Using Informational Connectivity to Measure the Synchronous Emergence of fMRI Multi-voxel Information Across Time

Published on: July 1, 2014

12.3K

Area of Science:

  • Data Science
  • Information Theory
  • Computational Statistics

Background:

  • Estimating information-theoretic quantities (e.g., entropy, mutual information) from continuous data is crucial but often computationally challenging due to the need for accurate probability density function estimation in high dimensions.
  • Existing methods like kernel density estimation (KDE) can be unreliable or infeasible for high-dimensional data, necessitating exploration of alternative approaches.

Purpose of the Study:

  • To systematically compare the performance of kernel density estimation (KDE), binned frequencies (histograms), and k-nearest neighbors (k-NN) for estimating information-theoretic quantities from continuous data.
  • To evaluate these methods across various data distributions, dimensions (1-10), sample sizes, and hyperparameters, assessing accuracy, computational efficiency, and implementation challenges.

Main Methods:

  • Generated synthetic data samples from distributions of varying shapes and dimensions.
  • Estimated entropy, Kullback-Leibler divergence, and mutual information using KDE, histograms, and k-NN.
  • Compared estimation results against closed-form solutions or numerical integrals as a reference.
  • Evaluated performance based on estimation accuracy, computation time, and implementation complexity.

Main Results:

  • The k-nearest neighbors (k-NN) estimation method demonstrated superior performance across most evaluated metrics, including algorithmic implementation, computational efficiency, and estimation accuracy, particularly when sufficient data was available.
  • Kernel density estimation (KDE) and histogram-based methods showed limitations, especially in higher dimensions or with sparse data.
  • Performance varied based on data characteristics, sample size, and specific information-theoretic quantity being estimated.

Conclusions:

  • The k-NN method is recommended as a robust and efficient technique for estimating information-theoretic quantities from continuous data, offering a practical solution for high-dimensional applications.
  • The choice of estimation method should consider the specific data characteristics, the targeted information-theoretic quantity, and the available sample size.
  • An open-source Python 3 toolbox has been developed to facilitate the application of these methods, promoting wider use of information-theoretic quantities in data analysis across disciplines.