DataCurator.jl:高效,便携和可重复的验证,修复和转换大型异质数据集,使用可读的人类食谱编译成机器可验证的模板
Ben Cardoen1, Hanene Ben Yedder1, Sieun Lee2,3
1Department of Computing Science, Simon Fraser University, 8888 University Dr W, Burnaby, British Columbia V5A1S6, Canada.
Bioinformatics advances
|June 26, 2023
概括
通过自动化数据策划和验证,DataCurator简化了跨学科研究. 这个软件包简化了复杂的数据集管理,提高了可复制性,减少了计算集群的处理延迟.
科学领域:
- 计算生物学 计算生物学
- 数据科学数据科学数据科学
- 科学计算科学计算
背景情况:
- 大规模的跨学科研究涉及复杂的,异构的数据集.
- 手动数据整理是耗时的,容易出现错误,影响可复制性.
- 数据不一致会破坏计算集群上的处理工作.
研究的目的:
- 介绍DataCurator,一个用于数据集验证的便携式软件包.
- 允许用户在没有编码的情况下定义自定义数据验证和转换规则.
- 提高大规模数据处理工作流程的效率和可重复性.
主要方法:
- 数据管理器将人类可读的TOML食谱转换为可执行的机器可验证的模板.
- 配方支持数据转换,验证,子集选择,采样和聚合.
- 多线程执行确保了集群上的可扩展性;与Julia,R和Python库集成.
主要成果:
- 数据管理器验证了局部系统和集群上的混合格式的复杂数据集.
- 自动化数据策划和验证取代了人工工作,减少了错误.
- 通过与Slack,OwnCloud和SCP的集成,可以实现高效的远程工作流.
结论:
- DataCurator显著提高了科学研究中的数据处理效率和可重复性.
- 该软件包简化了跨学科项目的复杂数据管理任务.
- 数据管理器 (DataCurator) 在分布式计算环境中促进了无的数据处理和处理.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
332
08:03Heuristic Mining of Hierarchical Genotypes and Accessory Genome Loci in Bacterial Populations
Published on: December 7, 2021
2.2K
相关概念视频
Data Validation
5.1K
Data validation is an essential part of a comprehensive assessment. Validation is confirming or verifying and opening the door to gathering more assessment data as it clarifies vague or unclear data. The process of checking and verifying the collected information is called data validation. The primary purpose of data validation is to ensure data is as free from error, bias, and misinterpretation as possible.
Nursing assessment guides are generally based on holistic models rather than medical...
Nursing assessment guides are generally based on holistic models rather than medical...
5.1K
Data Collection by Observations
12.1K
Data collection refers to a systematic way of obtaining, observing, measuring, and analyzing accurate information. Observational studies are one of the most widely used methods of data collection. It involves collecting data by observing the behavior and physical characteristics of a sample without making any modifications to the sample.
An astronomer viewing the motion and brightness of stars in the sky and recording the data is an example of observational data collection. A botanist recording...
An astronomer viewing the motion and brightness of stars in the sky and recording the data is an example of observational data collection. A botanist recording...
12.1K
Data Collection II
8.3K
The nursing history captures and records the patient's health status, so that a care plan evolves to meet the patient's individual needs. The nursing health history is a part of the initial assessment. A comprehensive history covers all health dimensions and plays a significant role in the assessment process. A comprehensive history includes the patient's biographical information, reasons for seeking health care, expectations, present and past health history, medications, and...
8.3K
Data Collection I
6.3K
Data collection gathers information needed to make accurate judgments about a patient's present condition. During a health history interview, subjective data is collected from the patient, their caregivers, or family members, and objective data is collected through observations and physical assessment. Patients are the primary source of subjective data. Thus information gathered from patients through interviews, observations, and physical examination is primary data. Secondary sources of...
6.3K
Data Collection III
2.8K
The physical assessment examines the patient for objective data that defines the patient's condition, and aids in formulating the nursing care plan. The purpose of physical assessment is a health status appraisal, which includes identifying health problems, and establishing a database for nursing intervention.
The principles to begin the physical assessment include conducting a comprehensive or problem-related history in a quiet, well-lit room, emphasizing privacy and comfort for the...
The principles to begin the physical assessment include conducting a comprehensive or problem-related history in a quiet, well-lit room, emphasizing privacy and comfort for the...
2.8K
Data Collection by Experiments
24.4K
Data collection is a systematic method of obtaining, observing, measuring, and analyzing accurate information. An experimental study is a standard method of data collection that involves the manipulation of the samples by applying some form of treatment prior to data collection. It refers to manipulating one variable to determine its changes on another variable. The sample subjected to treatment is known as “experimental units.”
An example of the experimental method is a public...
An example of the experimental method is a public...
24.4K
