用渐变相似度进行数据估值
Nathaniel J Evans1, Gordon B Mills2,3, Guanming Wu1
1Division of Bioinformatics and Computational Biomedicine, Department of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon, United States of America.
ArXiv
|May 27, 2024
概括
识别低质量的数据对于可靠的机器学习至关重要. 用梯度相似度 (DVGS) 数据估值提供了一个可扩展,有效的方法来确定和过数据错误,提高分析准确性.
科学领域:
- 机器学习 机器学习
- 数据科学数据科学数据科学
- 人工智能的人工智能
背景情况:
- 高质量的数据是准确的机器学习 (ML) 和可靠的分析的基础.
- 错误标记或杂的数据在各个领域都构成了重大挑战,通常需要专家知识和手动数据清理.
- 数据估值算法量化个人数据样本对预测任务的重要性,有助于识别错误标记的数据并提高ML性能.
结论:
- DVGS为识别低质量的数据提供了快速而准确的解决方案.
- 这种方法有可能大大简化数据清理流程.
- 通过自动化数据质量评估,DVGS可以提高ML模型开发的效率和有效性.
相关概念视频
Gradient and Del Operator
2.5K
In mathematics and physics, the gradient and del operator are fundamental concepts used to describe the behavior of functions and fields in space. The gradient is a mathematical operator that gives both the magnitude and direction of the maximum spatial rate of change. Consider a person standing on a mountain. The slope of the mountain at any given point is not defined unless it is quantified in a particular direction. For this reason, a "directional derivative" is defined, which is a vector...
2.5K
Sieve Analysis and Grading Curves
342
Sieve analysis is a method used to determine the particle size distribution of aggregate materials. This process involves the following steps:
342
How Data are Classified: Numerical Data
28.4K
Data that are countable or measurable in specific units are called numerical or quantitative data. Quantitative data are always numbers. Quantitative data are the result of counting or measuring the attributes of a population. Amount of money, pulse rate, weight, number of people living in a town, and number of students who opt for statistics are examples of quantitative data.
Quantitative data may be either discrete or continuous. All quantitative data that take on only specific numerical...
Quantitative data may be either discrete or continuous. All quantitative data that take on only specific numerical...
28.4K
Mean Absolute Deviation
2.6K
The mean absolute deviation is also a measure of the variability of data in a sample. It is the absolute value of the average difference between the data values and the mean.
Let us consider a dataset containing the number of unsold cupcakes in five shops: 10, 15, 8, 7, and 10. Initially, calculate the sample mean. Then calculate the deviation, or the difference, between each data value and the mean. Next, the absolute values of these deviations are added and divided by the sample size to...
Let us consider a dataset containing the number of unsold cupcakes in five shops: 10, 15, 8, 7, and 10. Initially, calculate the sample mean. Then calculate the deviation, or the difference, between each data value and the mean. Next, the absolute values of these deviations are added and divided by the sample size to...
2.6K
Design Example: Aggregate Gradation
93
The right type and quality of aggregates are crucial for concrete as they significantly influence its properties, mix proportions, and cost-effectiveness. If different sources are available for sand, the commonly used fine aggregate in concrete, the selection of sand is primarily based on its gradation.
The grading, or particle-size distribution, of sand is determined using sieve analysis, with standard sizes ranging from 150 μm to 10 mm (ASTM No. 100 sieve to 3⁄8 in. sieve). Sand is...
The grading, or particle-size distribution, of sand is determined using sieve analysis, with standard sizes ranging from 150 μm to 10 mm (ASTM No. 100 sieve to 3⁄8 in. sieve). Sand is...
93
Weighted Mean
5.1K
While taking the arithmetic, geometric, or harmonic mean of a sample data set, equal importance is assigned to all the data points. However, all the values may not always be equally important in some data sets. An intrinsic bias might make it more important to give more weightage to specific values over others.
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
5.1K


