通过微调的大型语言模型增强语义文本理解:关于Quora问题对重复识别的案例研究
Sifei Han1, Lingyun Shi1, Fuchiang Rich Tsui1,2,3,4
1Department of Biomedical and Health Informatics, Tsui Laboratory, Children's Hospital of Philadelphia, Philadelphia, PA, United States of America.
PloS one
|January 10, 2025
概括
在Quora问题对数据集上微调7B LLaMA模型 (qLLaMA_LoRA-7B) 在文本相似性任务中取得了最佳表现. 这种方法显著改善了以前的模型,并证明了大型语言模型的监督微调的有效性.
科学领域:
- 自然语言处理 (NLP) 是一种自然语言处理.
- 机器学习 机器学习
- 人工智能的人工智能
背景情况:
- 在NLP中,语义文本理解至关重要,像Quora问题对 (QQP) 等数据集用于评估.
- 之前的一项研究开发了一个语卷积神经网络 (S-CNN),在文本相似性方面获得了82.02%的F1分数.
研究的目的:
- 探索大型语言模型 (LLM) 在文本相似性任务中的有效性.
- 将LLM的各种微调方法进行比较,包括快速工程,n-shot学习和低级适应 (LoRA) 的监督学习.
主要方法:
- 杆化了5名预训练的LLM,并使用F1分数比较了他们的表现.
- 采用10倍交叉验证监督模型培训和评估,反映先前研究的设计和数据集.
- 进行了一项二次研究,比较7B和70B参数LLM,使用GLUE基准,在微调后使用语料库.
主要成果:
- 微调的LLaMA 7B模型 (qLLaMA_LoRA-7B) 在QQP数据集上获得了最高的F1得分84.9%,超过了Alpaca_LoRA-65B和S-CNN模型.
- 微调的LLaMA3.1-70B型号 (qLLaMA3.1_LoRA-70B) 在GLUE基准上获得了F1得分74.4%,优于7B型号 (71.9%).
- 与没有微调或使用n-shot学习的大型预训练模型相比,特定任务的监督微调提高了LLM性能.
结论:
- 展示了有效的LLM微调框架,突出了微调对于提高性能的重要性.
- 在LLMs的监督微调下,比使用没有微调或使用n-shot学习的大型模型获得更好的结果.
- 微调较大的LLM与较小的LLM相比,进一步提高了性能,在各种文档相似性任务中具有潜在的应用.
相关概念视频
Improving Translational Accuracy
2.5K
2.5K
Detection of Gross Error: The Q Test
5.6K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
5.6K
Mismatch Repair
4.8K
Organisms are capable of detecting and fixing nucleotide mismatches that occur during DNA replication. This sophisticated process requires identifying the new strand and replacing the erroneous bases with correct nucleotides. Mismatch repair is coordinated by many proteins in both prokaryotes and eukaryotes.
The Mutator Protein Family Plays a Key Role in DNA Mismatch Repair
The human genome has more than 3 billion base pairs of DNA per cell. Prior to cell division, that vast amount of genetic...
The Mutator Protein Family Plays a Key Role in DNA Mismatch Repair
The human genome has more than 3 billion base pairs of DNA per cell. Prior to cell division, that vast amount of genetic...
4.8K
Gene Duplication and Divergence
6.0K
The seminal work of Ohno in 1970 popularized the idea of gene duplication and divergence. DNA sequence comparison studies reveal that a large portion of the genes in bacteria, archaebacteria, and eukaryotes was generated by gene duplication and divergence, indicating its critical role in evolution.
The duplicated copies of the gene are called Paralogs. Paralogs with similar sequences and functions form a gene family. Across several species, a large number of gene families are...
The duplicated copies of the gene are called Paralogs. Paralogs with similar sequences and functions form a gene family. Across several species, a large number of gene families are...
6.0K
Second Uniqueness Theorem
972
Consider a region consisting of several individual conductors with a definite charge density in the region between these conductors. The second uniqueness theorem states that if the total charge on each conductor and the charge density in the in-between region are known, then the electric field can be uniquely determined.
In contrast, consider that the electric field is non-unique and apply Gauss's law in divergence form in the region between the conductors and the integral form to the...
In contrast, consider that the electric field is non-unique and apply Gauss's law in divergence form in the region between the conductors and the integral form to the...
972
Comparing Copy Number Variations and SNPs
17.1K
Sequencing of the human genome has opened up several best-kept secrets of the genome. Scientists have identified thousands of genome variations that exist within a population. These variations can be a single nucleotide or a larger chromosomal variation.
Copy number variations or CNVs are the structural variations that cover more than 1kb of DNA sequence. The single nucleotide polymorphism (SNP), on the other hand, is a single nucleotide change or a point mutation that is found in more than 1%...
Copy number variations or CNVs are the structural variations that cover more than 1kb of DNA sequence. The single nucleotide polymorphism (SNP), on the other hand, is a single nucleotide change or a point mutation that is found in more than 1%...
17.1K


