在医疗保健中流行开源大型语言模型的性能评估
Saif Khairat1, Tianyi Niu1, John Geracitano1
1University of North Carolina at Chapel Hill.
Studies in health technology and informatics
|July 1, 2025
概括
开源大型语言模型 (LLM) 显示出作为医疗保健助理的希望,提供透明度和定制性. 虽然GPT-4o更受欢迎,但开源模型在负载下表现出效率和可靠性.
科学领域:
- 人工智能的人工智能
- 自然语言处理自然语言处理.
- 医疗保健信息学 医疗保健信息学
背景情况:
- 大型语言模型 (LLM) 越来越多地被用于医疗保健应用.
- 开源的LLM提供了诸如透明度,成本效益和定制潜力等优势.
- 评估用户偏好和计算性能对于实际的LLM部署至关重要.
研究的目的:
- 为了评估用户对开源LLM (Llama 3.1 8B,Mistral 3 Small 24B) 和专有模型 (GPT-4o) 的偏好,作为医疗保健助理.
- 在不同的用户负载下对这些LLM的计算性能指标进行基准测试.
- 了解开源LLM对于面向用户的医疗保健应用程序的适用性.
主要方法:
- 一种双重方法的方法,结合了主观的用户评估和客观的性能基准测试.
- 十名参与者对Llama 3.1 8B,Mistral 3 Small 24B (AWQ) 和GPT-4o对与医疗保健相关的查询的回答进行了排名.
- 使用响应时间,吞吐量和模拟用户负载下的时间到第一个令牌来测量计算性能.
主要成果:
- 大多数参与者更喜欢GPT-4o产生的反应.
- 开源的LLMs,Llama 3.1 8B和Mistral 3 Small 24B (AWQ),获得了类似的用户评分.
- 基准测试显示,开源模型的性能高效,可靠,即使在显著的用户负载下.
结论:
- 开源的LLM证明了医疗保健助理角色的可行性和效率.
- 虽然专有模式目前可能导致用户偏好,但开源替代方案提供了令人信服的优势.
- 需要对各种领域的开源LLM采用进行进一步的研究.
相关概念视频
02:07Improving Translational Accuracy
2.7K
2.7K
01:12Statistical Software for Data Analysis and Clinical Trials
801
Statistical software is pivotal in data analysis and clinical trials by providing tools to analyze data, draw conclusions, and make predictions. These software packages range from simple data management applications to complex analytical platforms, supporting various statistical tests, models, and simulation techniques. Their significance lies in their ability to handle vast amounts of data with precision and efficiency, enabling researchers to validate hypotheses, identify trends, and make...
801
01:15Nursing Evaluation
3.6K
The evaluation stage signals the end of the nursing process. The nurse gathers evaluative data to assess whether or not the patient has attained the expected results. Whereas the nurse collects data in the nursing assessment to identify the patient's health concerns, the evaluation stage data determines if the indicated health issues are resolved. Evaluative data collection includes two sections: the data acquired to evaluate patient outcomes and the time criteria for data collection.
3.6K
01:26Pharmacokinetic Models: Comparison and Selection Criterion
153
Physiological and compartmental models are valuable tools used in studying biological systems. These models rely on differential equations to maintain mass balance within the system, ensuring an accurate representation of the dynamic processes at play.
Physiological models take a detailed approach by considering specific molecular processes. They can predict drug distribution, metabolism, and elimination changes, providing a comprehensive understanding of how drugs interact with the body.
Physiological models take a detailed approach by considering specific molecular processes. They can predict drug distribution, metabolism, and elimination changes, providing a comprehensive understanding of how drugs interact with the body.
153
01:30Health Information Technology and Healthcare Information System
936
Health Information Technology (HIT)
Health Information Technology, commonly called HIT, integrates advanced information systems and technology in healthcare settings. Its primary functions include:
Health Information Technology, commonly called HIT, integrates advanced information systems and technology in healthcare settings. Its primary functions include:
936
01:17Classification of Illness
8.0K
The meaning of illness is individualized to each person who experiences an alteration in health. In contrast, disease is a medical term indicating a pathological change in the structure and function of the body or mind. It is a condition that has specific symptoms and boundaries.
An illness is a response to a disease in which the person's level of functioning is changed compared with a previous level. The general classification of illness includes acute and chronic.
Acute illness is severe...
An illness is a response to a disease in which the person's level of functioning is changed compared with a previous level. The general classification of illness includes acute and chronic.
Acute illness is severe...
8.0K