答案可能有所不同:大型语言模型响应模式挑战了它们在测试项目分析中的使用
1Department of Anesthesiology, Dartmouth Hitchcock Medical Center, Lebanon, NH, USA.
Medical teacher
|May 4, 2025
概括
大型语言模型 (LLM) 显示出预测多选题 (MCQ) 性能指标的能力有限,例如难度和点二序列索引. 对LLM答案的一致性是评估开发的关键,而不是对项目特征的预测.
科学领域:
- 医学教育 医学教育
- 人工智能在评估中的作用
背景情况:
- 验证多选题 (MCQ) 需要进行广泛的测试.
- 大型语言模型 (LLM) 提供了简化评估开发的潜力.
- 预测MCQ的心理特征是一个关键的挑战.
研究的目的:
- 调查LLM预测MCQ难度和点 biserial指数的能力.
- 评估LLM是否可以减少对初步测试群体分析的需求.
- 将LLM绩效与人类专家评估进行比较.
主要方法:
- 60个麻醉学MCQ给了5名LLM和临床研究员.
- 分析了LLM响应模式,难度指数和点二序列指数.
- 斯皮尔曼相关系数比较了LLM和其他绩效指标.
主要成果:
- 在LLM的回应一致性不同,克劳德3.5索内特和拉玛3.2是最一致的.
- 总体而言,LLM的得分高于研究员的得分 (58-85%vs57%).
- 实际上,LLMs与其他难度指数的相关性很弱或不存在,并且未能预测点 biserial 指数.
结论:
- 在预测特定的MCQ心理测量属性方面,LLM的实用性有限.
- 性能更高的LLM与人类的表现相关性较低,这表明潜在的反向关系.
- 未来的研究应该专注于LLM以更广泛的评估优化,而不是项目级预测.
更多相关视频
03:14Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
461
09:00Author Spotlight: Validation of SICOLE-R for Assessing Cognitive and Reading Skills in Spanish-Speaking Children and Its Role in Personalized Education
Published on: August 16, 2024
614
相关概念视频
Response Surface Methodology
904
Response Surface Methodology (RSM) is a collection of statistical and mathematical techniques used to develop, improve, and optimize processes. It is particularly valuable when many input variables or factors potentially influence a response variable.
The process of RSM involves several key steps:
The process of RSM involves several key steps:
904
Typical Model Studies
842
Fluid mechanics model studies often utilize scaled-down systems to predict fluid behavior in full-scale environments, such as river flows, dam spillways, and structures interacting with open surfaces. Maintaining Froude number similarity in river models is crucial, as it replicates surface flow features like wave patterns and velocities.
842
Components of Language
840
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
840
Survival Tree
499
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
499
