相关实验视频
Updated: Jul 17, 2025

07:30
Optimization for Sequencing and Analysis of Degraded FFPE-RNA Samples
Published on: June 8, 2020
12.1K
对非结构化文本数据的预处理方法进行范围审查,以评估数据质量.
Marcello Nesca1,2, Alan Katz1,2,3, Carson K Leung4
1Department of Community Health Sciences, University of Manitoba, Winnipeg, MB, Canada.
International journal of population data science
|September 6, 2023
概括
自然语言处理 (NLP) 方法预处理非结构化文本数据 (UTD) 用于研究. 本文审查了NLP预处理技术及其对数据质量的影响,特别是电子病历 (EMR) 数据.
科学领域:
- 计算机科学 计算机科学
- 医疗信息学 医疗信息学
- 数据科学数据科学数据科学
背景情况:
- 非结构化文本数据 (UTD) 在研究数据库中越来越普遍,包括电子医疗记录 (EMR).
- UTD的质量显著影响其研究实用性.
- 自然语言处理 (NLP) 技术通常用于预处理UTD进行分析,但方法各不相同,可能会影响数据质量.
研究的目的:
- 系统地记录有关UTDNLP预处理方法的当前研究和实践.
- 确定用于描述或提高UTD质量的方法,重点是EMR数据.
主要方法:
- 2002年12月至2021年1月期间发表的同行评审研究的范围审查.
- 在Scopus,科学网络,ProQuest和EBSCOhost.host上进行了搜索.
- 提取的数据包括文章特征,数据类型,预处理方法和数据质量方面,通过叙事合成进行分析.
主要成果:
- 包括41篇文章,其中超过50%发表于2016-2021年期间;近20%出现在健康科学期刊中.
- 常见的NLP预处理方法包括删除停止词,删除标点/数字,令牌化和部分语音标签.
- 关于EMR数据质量的担忧包括拼写错误,取消识别,单词变化,噪音,注释质量和缩写模糊性.
结论:
- 对于UTD预处理,存在各种NLP技术,对EMR数据进行了特定的调整.
- 对于UTD的数据质量维度与结构化数据有相似之处.
- 对于UTD存在很少的通用数据质量指标,主要集中在噪声测量上.
更多相关视频
相关概念视频
Quality Assurance
155
Quality assurance is the overarching term used to describe the activities employed to ensure the proper performance of a system. These activities can be classified into three categories: quality control, quality assessment, and internal corrective measures. Typically, these activities work cyclically: quality control is performed before and during the analysis, while quality assessment occurs during and after the investigation. Internal corrective measures are implemented based on the findings...
155
Data Validation
183
Method validation is a crucial process in analytical chemistry designed to confirm that a given method consistently produces reliable and high-quality results. This process is essential when a method is applied to different sample matrices or when procedural modifications are made, ensuring that the results meet acceptable standards across various applications.
Key parameters for method validation include:
Key parameters for method validation include:
183
Protein Folding Quality Check in the RER
3.7K
ER is the primary site for the maturation and folding of soluble and transmembrane secretory proteins. The calnexin cycle is a specific chaperone system that folds and assesses the confirmation of N-glycosylated proteins before they can exit the ER lumen. The primary players of this quality check pipeline are the lectins, ER-resident chaperones, and a glucosyl transferase enzyme. In case the calnexin system in the lumen fails to salvage a misfolded protein, it is transported to the cytoplasm...
3.7K
Quantifying and Rejecting Outliers: The Grubbs Test
1.6K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
1.6K
Qualitative Analysis
297
Qualitative analysis is the process of identifying elements, ions, or compounds in an unknown sample. It is the first and most fundamental type of analysis based on the hierarchy of analytical goals. This hierarchy is significant as it provides a structured approach to scientific research, with qualitative analysis serving as the initial step, providing essential information before moving on to quantitative or other forms of analysis.
There are two main approaches to qualitative analysis:...
There are two main approaches to qualitative analysis:...
297
Review and Preview
9.0K
Data are individual items of information obtained from a population or sample. Data may be classified as qualitative (categorical), quantitative continuous, or quantitative discrete. Because it is not practical to measure the entire population in a study, researchers use samples to represent the population. A random sample is a representative group from the population chosen by using a method that gives each individual in the population an equal chance of being included in the sample. Random...
9.0K

