编码混合印尼语-爪语-英语推特的语料库创建和语言识别
Ahmad Fathan Hidayatullah1,2, Rosyzie Anna Apong1, Daphne T C Lai1
1School of Digital Science, Universiti Brunei Darussalam, Bandar Seri Begawan, Brunei.
PeerJ. Computer science
|July 6, 2023
概括
这项研究引入了社交媒体文本中语言识别的新模型,特别是印尼语,爪语和英语的代码混合. 微调的IndoBERTweet模型在这些混合语言的推文中识别语言方面表现出卓越的表现.
科学领域:
- 计算语言学 计算语言学
- 自然语言处理 (NLP) 是一种自然语言处理.
- 社交媒体分析 社交媒体分析
背景情况:
- 代码混合,语言混合的现象,在社交媒体文本中越来越普遍.
- 这种语言趋势对自然语言处理 (NLP) 任务,特别是语言识别 (LID) 提出了重大挑战.
- 现有的LID模型经常与代码混合数据的复杂性作斗争,需要专门的方法.
研究的目的:
- 开发和评估一个词级语言识别模型,用于印尼语,爪语和英语编码混合的推文.
- 为培训和评估引入一套新的注释体,即印尼语-爪语-英语语言识别 (IJELID) 数据集.
- 为了比较不同建模策略的有效性,包括微调的BERT,BLSTM和CRF,对于这个特定的任务.
主要方法:
- 创建印尼语 - 爪语 - 英语语言识别 (IJELID) 库,详细说明数据收集和注释标准.
- 研究各种机器学习方法用于词级语言识别.
- 微调基于BERT的模型 (IndoBERTweet),双向长期短期内存 (BLSTM) 网络和条件随机场 (CRF).
主要成果:
- 与BLSTM和CRF相比,微调的IndoBERTweet模型在混合编码的印尼语,爪语和英语tweet中识别语言方面表现出了卓越的表现.
- 伯特模型的有效性归因于它们能够捕获文本序列中的每个单词的上下文信息.
- 在BERT模型中的子词语言表示被证明是代码混合文本中语言识别的可靠机制.
结论:
- 微调的IndoBERTweet模型为代码混合的社交媒体文本中语言识别提供了强大的解决方案.
- 开发的IJELID语料库为未来的代码混合语言识别研究提供了宝贵的资源.
- 伯特的上下文理解和子词表示能力是成功解决NLP中代码混合挑战的关键.
更多相关视频
09:27Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language
Published on: October 13, 2018
10.1K
09:09Foreign Accent and Forensic Speaker Identification in Voice Lineups: The Influence of Acoustic Features Based on Prosody
Published on: September 27, 2024
475
相关概念视频
Components of Language
327
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
327
Language Development
401
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
401
Language
249
Language is a unique communication system that uses words and systematic rules to organize and transmit information. Unlike other forms of communication, which may involve postures, movements, odors, or vocalizations, language relies on symbols and grammar. This makes human communication distinct from that of other species, who also communicate but do not use language in the same way humans do.
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
249
Language and Cognition
376
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
376
Higher Mental Functions of the Brain: Language
928
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
928
Genetic Lingo
103.3K
Overview
103.3K
