IndicDialogue:用于印度语语言建模的10种印度语语言的字幕数据集
Noor Mairukh Khan Arnob1, A Faiyaz1, Md Mubtasim Fuad1
1Department of Computer Science and Engineering, University of Asia Pacific, Dhaka, Bangladesh.
Data in brief
|August 7, 2024
概括
IndicDialogue数据集解决了印度语言在NLP研究中的不足. 它提供了10种主要印第安语的大量预处理对话,以支持低资源语言模型的开发.
科学领域:
- 自然语言处理自然语言处理.
- 计算语言学 计算语言学
- 低资源的语言技术
背景情况:
- 印度次大陆的语言在当前的自然语言处理 (NLP) 文献中显著不足.
- 现有的NLP资源往往缺乏主要印第安语的足够数据,阻碍了研究和开发.
- 这种差距限制了NLP应用在全球大量人口中的发展.
研究的目的:
- 介绍IndicDialogue数据集,这是一个新的资源,旨在弥合印地语语言NLP数据的差距.
- 为低资源印第安语提供预培训语言模型的基础.
- 为了促进下游的NLP任务,如机器翻译和文本总结.
主要方法:
- 从OpenSubtitles.org收集了涵盖10种主要印度语语言的字幕数据.
- 预处理原始字幕文件 (SRT) 来删除无关标签,时间,括号和链接.
- 将已清理的对话数据组织到JSONL文件中,确保保留相关的对话内容.
主要成果:
- 该IndicDialogue数据集包含7750个原始字幕文件.
- 数据集包括11个JSONL文件,包括6,853,518个对话和42,188,569个单词.
- 数据涵盖了10种主要的印度语:印度语,孟加拉语,马拉提语,特鲁古语,泰米尔语,乌尔都语,奥迪亚语,辛迪语,尼泊尔语和阿萨姆语.
结论:
- IndicDialogue数据集是推动低资源印第安语语言NLP研究的宝贵资源.
- 它可以开发更具包容性和有效性的NLP模型和应用程序.
- 这一数据集作为未来在印第安语语言技术方面的工作的重要基础.
更多相关视频
10:11Portable Intermodal Preferential Looking IPL: Investigating Language Comprehension in Typically Developing Toddlers and Young Children with Autism
Published on: December 14, 2012
18.4K
05:48Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception
Published on: August 9, 2024
1.5K
相关概念视频
Indicators
48.5K
Certain organic substances change color in dilute solution when the hydronium ion concentration reaches a particular value. For example, phenolphthalein is a colorless substance in any aqueous solution with a hydronium ion concentration greater than 5.0 × 10−9 M (pH < 8.3). In more basic solutions where the hydronium ion concentration is less than 5.0 × 10−9 M (pH > 8.3), it is red or pink. Substances such as phenolphthalein, which can be used to determine the pH of a solution, are...
48.5K
Induction
4.0K
An emf is induced when the magnetic field in a coil is changed by pushing a bar magnet into or out of the coil. emfs of opposite signs are produced by motion in opposite directions, and the directions of emfs are also reversed by reversing poles. The same results are produced if the coil is moved rather than the magnet—it is the relative motion that is important. The faster the motion, the greater the emf. Additionally, there is no emf when the magnet is stationary relative to the coil.
A...
A...
4.0K
Inductors
396
An inductor is a passive component built to store energy within its magnetic field. It can be fabricated by coiling a wire around a magnetic core. When current is permitted to flow through this inductor, it is observed that the voltage across the inductor is directly proportional to the time rate of change of the current. Mathematically,
396
Induced-fit Model
80.6K
Most chemical reactions in cells require enzymes—biological catalysts that speed up the reaction without being consumed or permanently changed. They reduce the activation energy needed to convert the reactants into products. Enzymes are proteins, that usually work by binding to a substrate—a reactant molecule that they act upon.
Enzymes exhibit substrate specificity, meaning that they can only bind to certain substrates. This is mainly determined by the shape and chemical...
Enzymes exhibit substrate specificity, meaning that they can only bind to certain substrates. This is mainly determined by the shape and chemical...
80.6K
Translation
14.7K
Translation is the process of synthesizing proteins from the genetic information carried by messenger RNA (mRNA). Following transcription, it constitutes the final step in the expression of genes. This process is carried out by ribosomes, complexes of protein and specialized RNA molecules. Ribosomes, transfer RNA (tRNA), and other proteins produce a chain of amino acids—the polypeptide—as the end product of translation.
Translation Produces the Building Blocks of Life
Proteins are...
Translation Produces the Building Blocks of Life
Proteins are...
14.7K
Inductive Reasoning
60.2K
Inductive reasoning is a form of logical thinking that uses related observations to arrive at a general conclusion. It is uncertain and operates in degrees to which the conclusions are credible. As such, inductive arguments can be weak or strong, rather than valid or invalid, and conclusions can be used to formulate testable, falsifiable hypotheses.
Inductive reasoning is common in descriptive science. A life scientist makes observations and records them. This data can be qualitative or...
Inductive reasoning is common in descriptive science. A life scientist makes observations and records them. This data can be qualitative or...
60.2K
