在乌尔都语中使用语义和嵌入模型识别攻击性语言
Sajid Hussain1, Muhammad Shahid Iqbal Malik1, Nayyer Masood1
1Department of Computer Science, Capital university of Science and Technology, Islamabad, Pakistan.
PeerJ. Computer science
|June 22, 2023
概括
这项研究引入了一套新的数据集,用于检测攻击性乌尔都语,通过结合TF-IDF,词包和word2vec功能的混合模型实现90%的准确性. 这一进步为在线安全和未来研究提供了实际应用.
科学领域:
- 自然语言处理自然语言处理.
- 计算语言学 计算语言学
- 社交媒体分析 社交媒体分析
背景情况:
- 自动检测冒犯性语言对于在线安全至关重要.
- 由于语言的多样性,跨语言的解决方案的泛化是具有挑战性的.
- 关于攻击性语言检测的先前研究主要集中在西方语言上,对乌尔都语等低资源语言的工作有限.
研究的目的:
- 开发和评估一个强大的系统,用于乌尔都语的攻击性语言检测.
- 为乌尔都语攻击性语言的识别创建一个新的,全面的数据集.
- 探索和比较各种功能工程技术来完成这个任务.
主要方法:
- 一个新的数据集由7500个乌尔都语Facebook帖子组成,用于检测攻击性语言.
- 使用了四种特征工程模型:术语频率-反向文档频率 (TF-IDF),字袋,Word n-grams和Word2Vec嵌入.
- 实验使用独立模型和组合方法,包括堆叠和基于包装的特征选择.
主要成果:
- Word2Vec嵌入模型,当与堆叠组合一起使用时,作为独立模型实现了88.27%的准确性.
- 结合TF-IDF,字袋和Word2Vec功能的混合模型达到90%的准确性和97%的曲线下面面积 (AUC).
- 拟议的混合方法在多个绩效指标上显著超过了基线方法.
结论:
- 开发的混合功能模型在乌尔都语攻击性语言检测方面表现出卓越的性能.
- 创建一个专门的乌尔都语数据集解决了低资源语言NLP研究的差距.
- 这些发现对开发内容调节工具和加强乌尔都语社区在线安全具有实际意义.
更多相关视频
05:56Objectification of Tongue Diagnosis in Traditional Medicine, Data Analysis, and Study Application
Published on: April 14, 2023
2.5K
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
332
相关概念视频
02:16Stereotype Content Model
14.8K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
14.8K
01:19Encoding
210
Information enters the brain through encoding, which is the input of information into the memory system. Once sensory information is received from the environment, the brain labels or codes it. The information is then organized with similar information and connected to existing concepts. Encoding occurs through automatic processing and effortful processing.
Automatic processing involves the encoding of details like time, space, frequency, and the meaning of words, usually done without conscious...
Automatic processing involves the encoding of details like time, space, frequency, and the meaning of words, usually done without conscious...
210
01:24Components of Language
329
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
329
01:27Language and Cognition
377
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
377
01:22Language Development
403
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
403
01:16Language
250
Language is a unique communication system that uses words and systematic rules to organize and transmit information. Unlike other forms of communication, which may involve postures, movements, odors, or vocalizations, language relies on symbols and grammar. This makes human communication distinct from that of other species, who also communicate but do not use language in the same way humans do.
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
250