我们可以使用观察医疗保健数据开发现实世界的预后模型吗? 大规模实验调查模型对数据库和表型的敏感性
Jenna M Reps1,2, Peter R Rijnbeek3, Patrick B Ryan4
1, Johnson & Johnson, Raritan, NJ, USA. jreps@its.jnj.com.
Diagnostic and prognostic research
|April 17, 2025
概括
模型在临床环境中的性能可以根据用于开发的数据源和患者特征而有很大差异. 研究人员必须在现实数据中验证预测模型,以确保准确的临床决策.
科学领域:
- 医疗信息学 医疗信息学
- 临床流行病学临床流行病学
- 预测建模预测建模
背景情况:
- 大型观察性医疗保健数据库对于开发临床预测模型至关重要.
- 数据库和临床环境之间的患者群体 (病例组合) 和患者表型的差异带来了挑战.
- 之前的模型,就像COVID-19严重程度的模型一样,强调了需要可靠的现实世界的适用性.
研究的目的:
- 评估预测模型性能对开发数据库,人口特征和结果定义变化的敏感性.
- 在不同数据场景中评估模型歧视和校准的稳定性.
主要方法:
- 开发了450多个物流回归模型,用于7个不同的数据库中的9个预测任务.
- 通过将模型应用于交替数据 (数据库,人口,结果表型) 来评估性能稳定性.
- 使用接收器运行特征曲线 (AUROC) 下面的区域,缩放的Brier分数和整体校准来评估模型性能.
主要成果:
- 当结果定义或人口表型发生变化时,模型歧视保持相对稳定.
- 当开发数据库被改变时,验证歧视被证明是不稳定的.
- 校准和屏障性能在人口变化,结果定义和数据库中表现出不稳定性.
结论:
- 使用观察数据开发的模型在实施之前需要在目标临床环境中进行严格的验证.
- 评估校准和布里尔得分对于防止在临床决策中使用错误校准的风险估计至关重要.
相关概念视频
Observational Studies
8.1K
Observational studies are a type of analytical study where researchers observe events without any interventions. In other words, the researcher does not influence the response variable or the experiment's outcome.
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One...
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One...
8.1K
Introduction to Epidemiology
543
Epidemiology, known as the cornerstone of public health, involves studying the distribution and determinants of health-related events in defined populations and applying these insights to control health issues. This is essential for understanding how diseases spread, identifying populations at greater risk, and implementing measures to control or prevent outbreaks. Epidemiology addresses not only infectious diseases but also non-communicable conditions like cancer and cardiovascular disease,...
543
Data Collection by Observations
11.6K
Data collection refers to a systematic way of obtaining, observing, measuring, and analyzing accurate information. Observational studies are one of the most widely used methods of data collection. It involves collecting data by observing the behavior and physical characteristics of a sample without making any modifications to the sample.
An astronomer viewing the motion and brightness of stars in the sky and recording the data is an example of observational data collection. A botanist recording...
An astronomer viewing the motion and brightness of stars in the sky and recording the data is an example of observational data collection. A botanist recording...
11.6K
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
106
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
106
Data Collection by Experiments
23.6K
Data collection is a systematic method of obtaining, observing, measuring, and analyzing accurate information. An experimental study is a standard method of data collection that involves the manipulation of the samples by applying some form of treatment prior to data collection. It refers to manipulating one variable to determine its changes on another variable. The sample subjected to treatment is known as “experimental units.”
An example of the experimental method is a public...
An example of the experimental method is a public...
23.6K
Study Designs in Epidemiology
134
Epidemiological study designs are fundamental tools for investigating the distribution, determinants, and control of health conditions in populations. They help researchers understand the relationships between exposures and outcomes, and they broadly fall into two categories: "observational" and "experimental" studies.
Observational studies are those where the researcher does not intervene but rather observes natural variations. They include cross-sectional, cohort, and...
Observational studies are those where the researcher does not intervene but rather observes natural variations. They include cross-sectional, cohort, and...
134


