评估大型语言模型在评估健康经济研究质量方面的表现
Chen Dun1,2, Cody J Couperus3, Seohu Lee1
1Biomedical Informatics and Data Science Johns Hopkins University School of Medicine, Baltimore, Maryland.
Journal of health economics and outcomes research
|October 28, 2025
概括
大型语言模型 (LLM) 在评估健康经济研究质量方面表现有前途,与人类评估密切结合,但有一些局限性. 这项研究量化了LLM对质量评估的可靠性,推进了自动化方法.
科学领域:
- 卫生经济学 卫生经济学
- 在研究中的人工智能.
- 质量评估方法 质量评估方法
背景情况:
- 经济评估对于医疗保健决策至关重要,但由于不一致的报告和复杂性而受到影响.
- 大型语言模型 (LLM) 为评估遵守报告标准提供了一个可扩展的解决方案.
- 像Hileas这样的现有工具为自动化质量评估提供了基础.
研究的目的:
- 评估LLM产生的经济研究质量评估的准确性和可靠性.
- 使用CHEERS检查表来比较LLM绩效与人类审稿人.
- 为了确定局限性,并推进健康经济学中的自动化,辅助性质量评估.
主要方法:
- 通过LLM提示,使用CHEERS检查清单评估了110篇经济评估论文.
- 我们将LLM的结果与两个人类审稿人的共识评级进行了比较.
- 性能指标包括科恩的kappa,灵敏度,特异性,AUC和系统可用性量表.
主要成果:
- 人类评审者对25.3%的项目级评价持不同意见,这表明评审者之间的可靠性很低 (kappa=-0.07到0.43).
- 该LLM实现了72.3%94.7%的人类共识的协议,AUC高达0.96.
- 授予LLM的纸质水平CHEERS分数始终低于人类分数.
结论:
- 在健康经济学中,LLM提供了一个可行的,尽管不完美的,概念验证,用于评估健康经济学研究质量.
- 法学士评估通常与人类评估保持一致,提供了充分推理的判断.
- 需要进一步发展,以解决LLM在充分支持其质量评估方面的局限性.
相关概念视频
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
397
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
397
Models of Health Promotion and Illness Prevention I
2.7K
A model is a theoretical way to understand a concept or an idea. Models can overcome barriers to health regardless of diverse economic and cultural backgrounds. In addition, models make the task easier by providing different ways to approach complex issues. There are two major health promotion models: the health belief model and the health promotion model.
The health belief model (HBM) attempts to predict health-related behavior in specific belief patterns. According to the HBM, a person's...
The health belief model (HBM) attempts to predict health-related behavior in specific belief patterns. According to the HBM, a person's...
2.7K
Pharmacokinetic Models: Comparison and Selection Criterion
328
Physiological and compartmental models are valuable tools used in studying biological systems. These models rely on differential equations to maintain mass balance within the system, ensuring an accurate representation of the dynamic processes at play.
Physiological models take a detailed approach by considering specific molecular processes. They can predict drug distribution, metabolism, and elimination changes, providing a comprehensive understanding of how drugs interact with the body.
Physiological models take a detailed approach by considering specific molecular processes. They can predict drug distribution, metabolism, and elimination changes, providing a comprehensive understanding of how drugs interact with the body.
328
Bias in Epidemiological Studies
1.3K
Biases can arise at various stages of research, from study design and data collection to analysis and interpretation. Recognizing and addressing these biases is essential to ensure the validity and reliability of epidemiological findings.Broadly speaking, biases in epidemiology fall into three main categories: selection bias, information bias, and confounding. A more detailed description of possible biases is:
1.3K
Models of Health Promotion and Illness Prevention II
2.0K
The person's health status fluctuates continually, varying from being in good health to becoming ill and returning to being healthy. To understand the concept of illness prevention, there are two models. First, the health-illness continuum model is a graphic representation of an individual's wellness. It states that a person is considered healthy in the absence of physical disease and the presence of good emotional health.
The agent-host-environment model states that disease results...
The agent-host-environment model states that disease results...
2.0K
Analysis of Population Pharmacokinetic Data
671
Analysis of population pharmacokinetic data involves studying the behavior of drugs within diverse populations to understand their pharmacokinetic parameters. Traditional pharmacokinetic methods typically involve collecting samples from a few individuals and estimating these parameters. While these methods are commonly used, they have limitations in capturing the variability in drug response among individuals or heterogeneous populations. Population pharmacokinetics is employed to address these...
671


