低资源语言识别英语和Kokborok代码混合句子的语言识别
1Department of Computer Science & Engineering, National Institute of Technology; enjula.phd23@nitap.ac.in.
Journal of visualized experiments : JoVE
|January 19, 2026
概括
本研究引入了一个无监督的模型,用于在低资源的英语-Kokborok文本中进行词级语言检测. 这种新的方法实现了93.15%的准确性,超过了代码混合语言识别的现有方法.
科学领域:
- 计算语言学 计算语言学
- 自然语言处理自然语言处理.
- 低资源的语言技术
背景情况:
- 多语言文本使用需要自动语言检测.
- 代码交换和代码混合在许多语言中普遍存在,包括印度语言.
- 现有的研究往往忽略了低资源语言对.
研究的目的:
- 开发一个无监督的模型,用于单词级别的语言识别.
- 为了应对在英语-Kokborok代码混合文本中检测语言的特殊挑战.
- 开创低资源语言对的语言识别.
主要方法:
- 一种混合方法,将频率词典与字符n-gram模型相结合.
- 使用维特比技术整合一个字符n-gram马尔科夫模型和一个频率词典.
- 设计用于低资源场景的无监督方法.
主要成果:
- 在英语-Kokborok语言识别中达到93.15%的单词级准确度.
- 超过了BiLSTM-CRF模型 (88.5%) 和基于规则的基线 (85.3%).
- 证明了结合词典和统计方法的有效性.
结论:
- 拟议的无监督模型是有效的词级语言检测在低资源设置.
- 结合词典和统计方法是低资源语言的可行策略.
- 这项工作为分析以前未被解决的语言对中的代码混合文本提供了基础方法.
相关概念视频
Language
891
Language is a unique communication system that uses words and systematic rules to organize and transmit information. Unlike other forms of communication, which may involve postures, movements, odors, or vocalizations, language relies on symbols and grammar. This makes human communication distinct from that of other species, who also communicate but do not use language in the same way humans do.
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
891
Short-distance Transport of Resources
17.5K
Short-distance transport refers to transport that occurs over a distance of just 2-3 cells, crossing the plasma membrane in the process. Small uncharged molecules, such as oxygen, carbon dioxide, and water, can diffuse across the plasma membrane on their own. In contrast, ions and larger molecules require the assistance of transport proteins due to their charge or size. Transport across membranes also occurs within individual cells, playing a variety of essential roles for the plant as a whole.
17.5K
lncRNA - Long Non-coding RNAs
9.8K
In humans, more than 80% of the genome gets transcribed. However, only around 2% of the genome codes for proteins. The remaining part produces non-coding RNAs which includes ribosomal RNAs, transfer RNAs, telomerase RNAs, and regulatory RNAs, among other types. A large number of regulatory non-coding RNAs have been classified into two groups depending upon their length – small non-coding RNAs, such as microRNA, which are less than 200 nucleotides in length, and long non-coding RNA...
9.8K
lncRNA - Long Non-coding RNAs
3.5K
3.5K
Components of Language
763
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
763
Language Development
862
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
862


