一个基于语言建模的搜索工具,为"中英语散文索引"开发
Alpo Honkapohja1, Jacob Thaisen1, Anders Nøklestad2
1Department of Literature, Area Studies and European Languages, University of Oslo, Oslo, Oslo, 0315, Norway.
Open research Europe
|January 26, 2024
概括
这项研究引入了早期的本地语言文本的新搜索工具,通过处理拼写和措辞的变化来改善发现. 该系统优先考虑寻找潜在匹配,而不是确定单一的正确结果.
科学领域:
- 数字人文学科 数字人文学科
- 计算语言学 计算语言学
- 信息检索 信息检索
背景情况:
- 非标准化的早期民间语言在手稿副本中表现出显著的拼写,语法和词汇变化.
- 传统的搜索方法 (精确的字符串匹配,正则表达式) 不足以解决这些语言和文本的不一致性.
- 中英散文索引 (IMEP) 在搜索其收藏中面临挑战,原因是固有的文本变化.
研究的目的:
- 开发和介绍一个新的基于网络的搜索工具,能够处理早期本土手稿中的语言和文本变化.
- 通过适应标题,首字和明确字的变化,提高中英语散文索引 (IMEP) 中文本的可发现性.
- 为了优化搜索回忆,用户寻找潜在的相关的手稿条目尽管非标准化的语言.
主要方法:
- 开发一个与中英语散文索引 (IMEP) 集成的基于网络的搜索工具.
- 利用一个数据库的incipits和显式,基于字符的n-gram语言模型 (LMs) 来自斯坦福研究院语言建模 (SRILM) 工具包,以及基于Python的模糊搜索脚本 (IMEP: FSS).
- 实施两步搜索过程:直接查找数据库,结合使用语言模型进行模糊匹配,优化速度和回忆.
主要成果:
- 开发的搜索工具有效地检索搜索字符串的多个潜在匹配,并优先考虑回忆.
- 与传统的以精度为重点的方法相比,双步搜索方法大大减少了处理时间.
- 这款用Django和Docker构建的Web应用程序呈现了结合结果 (精确和模糊匹配),按模型困惑度排序.
结论:
- 新的搜索工具成功地解决了早期本地语言文本中语言和文本变化的挑战.
- 该系统提供了一个开放式访问解决方案,增强数字人文项目搜索能力.
- 该方法可适应用于其他具有类似文本变异挑战的数据集.
更多相关视频
06:48Lexical Decision Task for Studying Written Word Recognition in Adults with and without Dementia or Mild Cognitive Impairment
Published on: June 25, 2019
9.2K
08:32Examining Online Syntactic Processing of Spoken Complex Sentences in Chinese Using Dual-Modal Interference Tasks
Published on: September 5, 2019
5.6K
相关概念视频
Midrange
3.7K
A somewhat easy to compute quantitative estimate of a data set’s central tendency is its midrange, which is defined as the mean of the minimum and maximum values of an ordered data set.
Simply put, the midrange is half of the data set’s range. Similar to the mean, the midrange is sensitive to the extreme values and hence the prospective outliers. However, unlike the mean, the midrange is not sensitive to all the values of the data set that lie in the middle. Thus, it is prone to...
Simply put, the midrange is half of the data set’s range. Similar to the mean, the midrange is sensitive to the extreme values and hence the prospective outliers. However, unlike the mean, the midrange is not sensitive to all the values of the data set that lie in the middle. Thus, it is prone to...
3.7K
Higher Mental Functions of the Brain: Language
855
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
855
Median
18.5K
Besides mean, the median is a widely used measure of central tendency. Typically, median is defined as the central or middle value of a data set, measured by arranging the data elements in an increasing or decreasing order. Since this middle value is not affected by the precise numerical values of the outliers or fluctuations, it is insensitive to them. Hence, in cases where a data set may have outliers or the extreme values are not known, the median is a better measure of the central tendency...
18.5K
Proofreading
6.3K
Synthesis of new DNA molecules is carried out by the enzyme DNA polymerase, which adds nucleotides on the daughter strand complementary to the template DNA strand. DNA polymerase has a higher affinity to add the correct base and ensures fidelity during DNA replication. Furthermore, it exhibits proofreading activity during replication, using an exonuclease domain that cuts off incorrect nucleotides from the nascent DNA strand.
Errors During Replication are Corrected by the DNA Polymerase...
Errors During Replication are Corrected by the DNA Polymerase...
6.3K
Models, Theories, and Laws
5.4K
Scientists frequently use models to help them comprehend a specific collection of phenomena. In physics, a model is a condensed version of a physical system that is too complex to study thoroughly. One such example is the light wave model; unlike water waves, light waves are typically invisible to us. Nonetheless, it is helpful to think of light as being composed of waves, since investigations show that light behaves like water waves. Since it is impossible to visually see what is genuinely...
5.4K
Molecular Models
38.4K
Physical models representing molecular architectures of chemical compounds play essential roles in understanding chemistry. The use of molecular models makes it easier to visualize the structures and shapes of atoms and molecules.
38.4K
