评估一个大型语言模型对自由文本医学学生临床笔记的评分能力:定量研究
Harry B Burke1, Albert Hoang1, Joseph O Lopreiato1
1Uniformed Services University of the Health Sciences, Bethesda, MD, 20814, United States, 1 301-938-2212.
JMIR medical education
|August 9, 2024
概括
像ChatGPT这样的大型语言模型可以准确地对医学学生的临床笔记进行评分. 这种人工智能工具的错误率明显低于人类评估器,进步了医学教育.
科学领域:
- 医学教育 医学教育
- 医疗保健中的人工智能
- 自然语言处理自然语言处理.
背景情况:
- 医学教育强调教学临床信息技能.
- 对学生生成的临床笔记进行有效的反至关重要.
- 自由文本临床笔记是医学学生评估的关键组成部分.
研究的目的:
- 评估ChatGPT 3.5在评分医学学生的自由文本历史和物理笔记方面的能力.
- 将基于AI的得分与人类 (标准化患者) 的得分进行比较.
- 在医学教育中确定生成预训练变压器模型的准确性.
主要方法:
- 追溯性,单一机构研究,涉及168名一年级医学学生.
- 学生的自由文本笔记与85个案例元素的标题对比得分.
- 评分是由标准化患者和ChatGPT 3.5.5独立进行的.
主要成果:
- 聊天GPT实现了1.0%的不正确得分率,明显低于标准化患者 (7.2%).
- 聊天GPT的整体错误率比标准化患者低86%.
- 聊天GPT的平均不正确得分率 (12) 显著低于标准化患者 (85;P=.002).
结论:
- 与标准化患者相比,ChatGPT在评分医学学生的临床笔记方面表现出卓越的准确性.
- 这项研究是第一个评估GPT程序以评分标准化基于患者的笔记的研究.
- 像GPT这样的AI工具准备通过提供实时反来增强医学教育和临床实践.
相关概念视频
Nursing Clinical Information System
763
Nursing Clinical Information System (NCIS)
A Nursing Clinical Information System (NCIS) is a specialized type of healthcare information system tailored to meet the unique needs of nursing practice. It incorporates the principles of nursing informatics to streamline information management and improve the quality of care delivery.
Critical attributes of NCIS include:
A Nursing Clinical Information System (NCIS) is a specialized type of healthcare information system tailored to meet the unique needs of nursing practice. It incorporates the principles of nursing informatics to streamline information management and improve the quality of care delivery.
Critical attributes of NCIS include:
763
Methods of Documentation VI: Case Management Model
564
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic...
For example, a patient with a chronic...
564
Clinical Trials: Overview
2.9K
Clinical development focuses on how the drug will interact with the human body and encompasses four key phases of clinical trials, each serving a specific purpose in assessing the safety and effectiveness of new drugs. These phases overlap and build upon one another. Phase I involves a small group of healthy volunteers (typically 20-80 individuals) or, in cases where significant toxicity is expected, patients with the targeted disease, such as cancer or AIDS. The volunteers are tested for...
2.9K
Statistical Software for Data Analysis and Clinical Trials
526
Statistical software is pivotal in data analysis and clinical trials by providing tools to analyze data, draw conclusions, and make predictions. These software packages range from simple data management applications to complex analytical platforms, supporting various statistical tests, models, and simulation techniques. Their significance lies in their ability to handle vast amounts of data with precision and efficiency, enabling researchers to validate hypotheses, identify trends, and make...
526
Clinical Trials
6.6K
Clinical trials are prospective experimental studies conducted on humans to determine the safety and efficacy of treatments, drugs, diet methods, and medical devices. Using statistics in clinical trials enables researchers to derive reasonable and accurate conclusions from the collected data, allowing them to make wise decisions in uncertain situations. In medical research, statistical methods are crucial for preventing errors and bias.
There are four phases in a clinical trial. A phase one...
There are four phases in a clinical trial. A phase one...
6.6K
Nursing Assessment
7.5K
The two sources for collecting information are primary and secondary. After gathering information, interpretation and validation help to complete the data. The purpose of assessment is to establish data with the initial information, to interpret data about the patient's perceived needs and health problems, and to respond to these problems identified.
The nurse collects all aspects of the patient's health in the initial assessment, establishing priorities for ongoing focused assessments...
The nurse collects all aspects of the patient's health in the initial assessment, establishing priorities for ongoing focused assessments...
7.5K


