分析测试机构在中文短语中的能力描述,使用半监督K-Means集群和BERT的联合方法进行分析
Gaoqing Xu1, Qun Chen2, Shuhang Jiang3
1Zhejiang Institute of Standardization, Zhijiang Standardization Think Tank, Hangzhou, 310000, China.
Scientific reports
|April 2, 2025
概括
本研究引入了一种新的BERT和半监督的K-Means集群模型来分类第三方测试机构的能力. 该方法有效地分析了中国能力参数,提高了分类准确性,克服了传统技术的局限性.
科学领域:
- 信息科学 信息科学 信息科学
- 数据科学数据科学数据科学
- 计算语言学 计算语言学
背景情况:
- 第三方测试机构的能力参数对于评估技术和质量管理能力至关重要.
- 不一致的基于中文短语的描述阻碍了测试能力的标准化分类,影响了客户和监管机构.
- 传统的文本特征提取方法与稀疏,高维度和语义差的数据作斗争.
研究的目的:
- 为测试能力开发一个科学和合理的分类系统.
- 解决传统文本分析在分类测试机构能力方面的局限性.
- 提出一个共同的模型,用于特征提取和集群的中国测试能力描述.
主要方法:
- 使用BERT从中文描述性短语中提取高级文本特征.
- 实施半监督的K-Means集群,使用一小组标记样本.
- 训练多输出Softmax分类器对能力分类的聚类结果进行训练.
主要成果:
- 拟议的BERT和半监督K-Means模型的性能优于TF-IDF和一热编码等传统方法.
- 在降低特征维度和提高对测试机构数据的聚类性能方面证明了优势.
- 在10%的标记样本中实现了最佳集群,平均分类器准确率为89.8%.
结论:
- 联合BERT和半监督集群模型为分类第三方测试机构能力提供了强大的解决方案.
- 这种方法有效地处理能力参数中中文描述文本的复杂性.
- 这些发现为更一致,更可靠地评估测试机构绩效提供了基础.
相关概念视频
01:20Cluster Sampling Method
11.5K
Appropriate sampling methods ensure that samples are drawn without bias and accurately represent the population. Because measuring the entire population in a study is not practical, researchers use samples to represent the population of interest.
To choose a cluster sample, divide the population into clusters (groups) and then randomly select some of the clusters. All the members from these clusters are in the cluster sample. For example, if you randomly sample four departments from your...
To choose a cluster sample, divide the population into clusters (groups) and then randomly select some of the clusters. All the members from these clusters are in the cluster sample. For example, if you randomly sample four departments from your...
11.5K
01:23Test for Homogeneity
1.9K
The goodness–of–fit test can be used to decide whether a population fits a given distribution, but it will not suffice to decide whether two populations follow the same unknown distribution. A different test, called the test for homogeneity, can be used to conclude whether two populations have the same distribution. To calculate the test statistic for a test for homogeneity, follow the same procedure as with the test of independence. The hypotheses for the test for homogeneity can...
1.9K
02:46Chi-square Analysis
35.8K
The chi-square test is a statistical hypothesis test. It is used to check whether there is a significant difference between an expected value and an observed value. In the context of genetics, it enables us to either accept or reject a hypothesis, based on how much the observed values deviate from the expected values.
The chi-square test was developed by Pearson in 1990.
The first step of performing a Chi-square analysis is to establish a null hypothesis, which assumes that there is no real...
The chi-square test was developed by Pearson in 1990.
The first step of performing a Chi-square analysis is to establish a null hypothesis, which assumes that there is no real...
35.8K
01:16Goodness-of-Fit Test
3.2K
The goodness-of-fit test is a type of hypothesis test which determines whether the data "fits" a particular distribution. For example, one may suspect that some anonymous data may fit a binomial distribution. A chi-square test (meaning the distribution for the hypothesis test is chi-square) can be used to determine if there is a fit. The null and alternative hypotheses may be written in sentences or stated as equations or inequalities. The test statistic for a goodness-of-fit test is...
3.2K