在机器学习模型中解决数据不平衡问题,以解决罕见和破坏性中断事件的机器学习模型
Morteza Azizi1,2, Xinxuan Zhang2,3, Tala Yasenpoor4
1School of Civil and Environmental Engineering, University of Connecticut, Storrs, CT, 06268, USA.
Scientific reports
|March 5, 2026
概括
生成具有稳定扩散的合成风暴数据可以改善用于预测自然灾害影响的机器学习模型. 这种数据丰富提高了预测恶劣天气事件及其后果的准确性.
科学领域:
- 环境科学环境科学
- 计算机科学 计算机科学
- 数据科学是数据科学.
背景情况:
- 由于观测数据有限,模拟罕见的自然灾害具有挑战性,导致数据集中高影响事件的代表性不足.
- 这种数据不平衡阻碍了机器学习 (ML) 模型在预测恶劣天气后果方面的有效性.
研究的目的:
- 开发一个框架来生成物理一致的合成风暴事件以丰富数据集.
- 通过数据丰富,提高ML模型用于预测自然灾害影响的准确性.
主要方法:
- 利用稳定扩散和变化自编码器来生成配对合成风暴事件数据.
- 将气象场压缩到潜伏空间中,并采用集群条件扩散模型来将事件与中断严重程度对齐.
- 使用分布相似性和物理一致性的指标过合成事件.
主要成果:
- 用选的合成数据进行丰富,显著改善了基于ML的中断预测.
- 实现了CRMSE减少39%,R2增加11%,NSE增加200%以上,以提高预测准确度.
- 证明了拟议框架的有效性,提高了用于罕见事件预测的ML模型性能.
结论:
- 拟议的框架成功地产生了物理一致的合成风暴事件以丰富数据.
- 合成数据生成显著提高了ML模型在预测自然灾害影响方面的准确性和有效性.
- 这种方法为模拟罕见的自然灾害的数据稀缺问题提供了可行的解决方案.
相关概念视频
Outliers and Influential Points
6.5K
An outlier is an observation of data that does not fit the rest of the data. It is sometimes called an extreme value. When you graph an outlier, it will appear not to fit the pattern of the graph. Some outliers are due to mistakes (for example, writing down 50 instead of 500), while others may indicate that something unusual is happening. Outliers are present far from the least squares line in the vertical direction. They have large "errors," where the "error" or residual is the...
6.5K
Detection of Gross Error: The Q Test
7.2K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
7.2K
Quantifying and Rejecting Outliers: The Grubbs Test
4.3K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
4.3K
Steps in Outbreak Investigation
656
In the ever-evolving field of public health, statistical analysis serves as a cornerstone for understanding and managing disease outbreaks. By leveraging various statistical tools, health professionals can predict potential outbreaks, analyze ongoing situations, and devise effective responses to mitigate impact. For that to happen, there are a few possible stages of the analysis:
656
Censoring Survival Data
612
Survival analysis is a statistical method used to analyze time-to-event data, often employed in fields such as medicine, engineering, and social sciences. One of the key challenges in survival analysis is dealing with incomplete data, a phenomenon known as "censoring." Censoring occurs when the event of interest (such as death, relapse, or system failure) has not occurred for some individuals by the end of the study period or is otherwise unobservable, and it might have many different...
612
Survival Tree
453
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
453


