通过强化学习检测异常值,用于昂贵的数据验证
Michiel Nijhuis1, Iman van Lelyveld1,2
1De Nederlandsche Bank, 1000 AB Amsterdam, The Netherlands.
Entropy (Basel, Switzerland)
|June 28, 2023
概括
本研究介绍了一种强化学习异常检测方法,该方法适应新数据. 它通过优化异常检测系数以获得更好的准确性来改进现有的合并方法.
科学领域:
- 数据科学数据科学数据科学
- 机器学习 机器学习
- 统计建模 统计建模
背景情况:
- 异常值在数据集中很常见,需要验证,这需要大量的时间.
- 数据错误及其原因可能会演变,需要自适应异常值检测.
- 现有的方法可能无法最佳地利用经过验证的异常值信息.
研究的目的:
- 开发一种使用强化学习的自适应异常值检测方法.
- 通过动态调整组合系数来增强统计异常值的检测.
- 提高识别和管理数据错误的效率和准确性.
主要方法:
- 使用了一组统计异常值检测方法.
- 强化学习被应用到调整组合的系数适应性.
- 该方法使用荷兰保险公司和养老基金在Solvency II和FTK框架下的细分数据进行了验证.
主要成果:
- 整体学习者成功地确定了金融数据集中的异常值.
- 整合强化学习进一步提高了异常值检测性能.
- 强化学习者的自适应系数优化提高了整体准确性.
结论:
- 强化学习提供了一种有效的方法来创建自适应异常检测系统.
- 拟议的方法优化了集体学习,以更好地识别数据错误.
- 这种方法为动态异常值管理提供了更有效,更准确的解决方案.
相关概念视频
Quantifying and Rejecting Outliers: The Grubbs Test
1.7K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
1.7K
What Are Outliers?
3.9K
Outliers are observed data points that are far from the least squares line. They have unusual values and need to be examined carefully. Though an outlier may result from erroneous data, at other times, it may hold valuable information about the population under study and should be included in the data. Hence, it is crucial to examine what causes a data point to be an outlier.
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
3.9K
Outliers and Influential Points
4.1K
An outlier is an observation of data that does not fit the rest of the data. It is sometimes called an extreme value. When you graph an outlier, it will appear not to fit the pattern of the graph. Some outliers are due to mistakes (for example, writing down 50 instead of 500), while others may indicate that something unusual is happening. Outliers are present far from the least squares line in the vertical direction. They have large "errors," where the "error" or residual is the...
4.1K
Detection of Gross Error: The Q Test
6.3K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.3K
Reinforcement
289
Positive and negative reinforcement are key concepts in operant conditioning, a learning process where the consequences of a behavior affect the likelihood of that behavior being repeated.
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
289
Observational Learning
222
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
222


