测量协议的重复调整方法用于开发高有效性文本分类器
Alex Goddard1, Alex Gillespie1
1Department of Psychological and Behavioural Science, London School of Economics and Political Science.
Psychological methods
|October 6, 2025
概括
在心理学中开发高有效性文本分类器需要整合手动编码和计算算法. 重复调整测量协议 (RAMP) 方法通过反复改进概念和构造以提高准确性来解决这一问题.
科学领域:
- 心理学 心理学 心理学
- 计算语言学 计算语言学
- 数据科学数据科学数据科学
背景情况:
- 在心理学中开发文本分类器依赖于手动编码,但其评估通常与算法评估分开.
- 这种分离阻碍了识别和解决对高有效性分类器至关重要的概念和测量问题的代过程.
研究的目的:
- 引入重复调整测量协议 (RAMP) 方法,用于开发心理学中的高有效性文本分类器.
- 整合内容分析,数据科学和心理学的最佳实践,以创建一个强大的分类器开发框架.
主要方法:
- RAMP方法包括三个阶段:手动编码,分类器开发和整合性评估.
- 它利用一个推理循环来代地改进基于实证数据的概念和构造.
- 一个案例研究涉及手动编码21815个句子,并开发基于规则的,监督的机器学习和大型语言模型分类器.
主要成果:
- 手动编码实现了高度的跨主体协议 (克里多夫α = .79).
- 监督机器学习 (来自变压器的双向编码器表示) 实现了最高的分类器性能 (马修斯相关系数[MCC] = 0.69).
- 通过RAMP方法,成功地发现并解决了与误解有关的概念有效性问题.
结论:
- RAMP将有效性作为一种动态的,反复调整的过程,以实现主体间的协议.
- 整合手动编码和分类器开发对于解决概念有效性问题至关重要.
- 该RAMP方法提供了一个结构化的方法,以提高文本分类器在心理学研究中的有效性.
相关概念视频
Data Validation
542
Method validation is a crucial process in analytical chemistry designed to confirm that a given method consistently produces reliable and high-quality results. This process is essential when a method is applied to different sample matrices or when procedural modifications are made, ensuring that the results meet acceptable standards across various applications.
Key parameters for method validation include:
Key parameters for method validation include:
542
Uncertainty in Measurement: Accuracy and Precision
99.2K
Scientists typically make repeated measurements of a quantity to ensure the quality of their findings and to evaluate both the precision and the accuracy of their results. Measurements are said to be precise if they yield very similar results when repeated in the same manner. A measurement is considered accurate if it yields a result that is very close to the true or the accepted value. Precise values agree with each other; accurate values agree with a true value.
99.2K
Systematic Error: Methodological and Sampling Errors
8.4K
In the case of systematic errors, the sources can be identified, and the errors can be subsequently minimized by addressing these sources. According to the source, systematic errors can be divided into sampling, instrumental, methodological, and personal errors.
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
8.4K
Random and Systematic Errors
14.3K
Scientists always try their best to record measurements with the utmost accuracy and precision. However, sometimes errors do occur. These errors can be random or systematic. Random errors are observed due to the inconsistency or fluctuation in the measurement process, or variations in the quantity itself that is being measured. Such errors fluctuate from being greater than or less than the true value in repeated measurements. Consider a scientist measuring the length of an earthworm using a...
14.3K
Reliability and Validity
13.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.7K
Accuracy and Errors in Hypothesis Testing
545
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
545


