大型语言模型在中国全国护士执照考试中的表现:横断评估研究
Longhui Xu1, Xiao Cong1, Renxiu Wang2
1School of Nursing, Shandong University of Traditional Chinese Medicine, Jinan, China.
JMIR medical informatics
|November 3, 2025
概括
大型语言模型 (LLM) 对于中国全国护士执照考试 (CNNLE) 是有前途的,但可靠性问题如不良的信心校准和不一致的重复性限制了它们在护理教育中的安全使用.
科学领域:
- 医疗保健中的人工智能
- 护理教育 技术 技术
- 医疗信息学 医疗信息学
背景情况:
- 大型语言模型 (LLM) 越来越多地被考虑用于护理教育.
- 他们在中国全国护士执照考试 (CNNLE) 等高风险的专业考试中的表现基本上没有得到评估.
- 在将LLM纳入护理培训和实践之前,严格的评估至关重要.
研究的目的:
- 评估四个领先的LLM在2024年CNNLE的性能,准确性,可重复性,信心和稳定性.
- 确定当前LLM在高风险的护理评估中的优点和局限性.
- 在专业医学教育中为LLMs的未来发展和应用提供信息.
主要方法:
- 四个LLM (Sider Fusion,GPT-4o,Gemini 2.0 Pro,DeepSeek V3) 在2024年CNNLE的237个选择题中进行了测试.
- 使用两个提示策略来评估准确性和可重复性.
- 通过自我评级来评估信心,并使用对抗提示来测试强度.
主要成果:
- 与GPT-4o和Sider Fusion (<71%) 相比,DeepSeek V3和Gemini 2.0 Pro的准确性显著提高了 (>83%).与GPT-4o和Sider Fusion相比,DeepSeek V3和Gemini 2.0 Pro的准确性明显提高了 (>83%).与GPT-4o和Sider Fusion相比,DeepSeek V3和Gemini 2.0 Pro的准确性显著提高了 (>83%).
- 所有的LLM都表现出低于最佳的重复性 (<87%的一致性).
- 一个关键的发现是信心校准不佳,高信心通常与模型中的实际准确性不一致.
结论:
- 虽然一些LLM在CNNLE上表现出了显著的准确性,但仍然存在严重的可靠性问题.
- 信心校准和可重复性的局限性目前阻碍了在护理教育和实践中的安全应用.
- 未来的LLM开发必须专注于提高可信度和校准可靠性,而不仅仅是表面准确性.
相关概念视频
01:15Nursing Evaluation
4.1K
The evaluation stage signals the end of the nursing process. The nurse gathers evaluative data to assess whether or not the patient has attained the expected results. Whereas the nurse collects data in the nursing assessment to identify the patient's health concerns, the evaluation stage data determines if the indicated health issues are resolved. Evaluative data collection includes two sections: the data acquired to evaluate patient outcomes and the time criteria for data collection.
4.1K
01:22The Professional Nurse
5.7K
Professional nurses are not limited to bedside care and are taking roles of greater responsibility. A nurse should have a knowledge-based practice, including personal, theoretical, procedural, cultural, and reflexive knowledge. Additionally, nurses must be competent in cognitive, technical, interpersonal, and ethical/legal skills. Some of the best attributes of successful nurses include the following:
Communication skills: These are critical characteristics, especially speaking and listening.
Communication skills: These are critical characteristics, especially speaking and listening.
5.7K
01:30National Nursing Organizations II
1.9K
Nursing organizations play a vital role in representing nurses working in specialized clinical settings, such as the American Association of Critical-Care Nurses (AACN).
The AACN emphasizes a healthy work environment through six standards to achieve an optimal patient outcome. The standards are appropriate staffing, meaningful recognition, collaboration, authentic leadership, effective communication, and decision-making. In addition, AACN provides certification programs, webinars, journals, and...
The AACN emphasizes a healthy work environment through six standards to achieve an optimal patient outcome. The standards are appropriate staffing, meaningful recognition, collaboration, authentic leadership, effective communication, and decision-making. In addition, AACN provides certification programs, webinars, journals, and...
1.9K