SparkDWM:使用Apache Spark的数据清洗机的可扩展设计
Nicholas Kofi Akortia Hagan1, John R Talburt1
1Department of Information Sciences, University of Arkansas at Little Rock, Little Rock, AR, United States.
Frontiers in big data
|September 24, 2024
概括
这项研究增强了使用Apache Spark进行可扩展实体解析的数据清洗机,提高了数据质量并超过了以前的版本. 重构系统有效处理大型数据集,确保准确的记录匹配.
科学领域:
- 计算机科学 计算机科学
- 数据科学数据科学数据科学
- 人工智能的人工智能
背景情况:
- 应用程序中的快速数据增长导致数据质量问题增加,如重复和错误.
- 传统的实体解决方案 (ER) 系统由于大数据量而难以扩展.
- 传统的数据洗衣机 (DWM) 概念验证具有单线设计,限制了其性能.
研究的目的:
- 使用Apache Spark重构数据洗衣机以实现高可扩展性.
- 为了解决传统DWM的单线程限制.
- 通过将来自引用的内在元数据纳入,改进DWM的设计.
主要方法:
- 使用Apache Spark的弹性分布式数据集 (RDD) 来进行分布式数据处理.
- 将传统的DWM重构为并行处理架构.
- 从引用中集成的内在元数据来增强ER过程.
- 通过使用18个合成数据集与旧的DWM对比验证了结果.
- 测试了与现实世界基准ER数据集的可扩展性,范围从数千到数百万条记录.
主要成果:
- 重构的DWM在合成数据上实现了与传统系统相同的结果.
- 与基于MapReduce的DWM相比,拟议的系统表现出优越的性能.
- 实验结果证实了重构系统在大型数据集上的可扩展性.
- 该系统在最佳集群参数下识别了比Famer系统更多的集群.
结论:
- 基于Apache Spark的数据清洗机是一种高度可扩展和有效的实体解决方案.
- 重构系统克服了传统DWM的性能限制.
- 这种增强的DWM为大规模应用提供了准确和高效的数据质量改进.
相关概念视频
01:21Design Example: Creating a Hydraulic Model of a Dam Spillway
137
Scaled hydraulic models of dam spillways provide a practical way to replicate and study the intricate flow dynamics of these structures. Often built to a 1:15 ratio, these models allow for observing critical water behavior, such as velocity distribution, flow patterns, and energy dissipation.
137
01:20Parallel Processing
145
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
145
01:30Machines: Problem Solving II
300
Machines are complex structures consisting of movable, pin-connected multi-force members that work together to transmit forces. Consider a lifting tong carrying a 100 kg load. It comprises movable sections DAF and CBG linked together with member AB.
300
01:22Machines: Problem Solving I
306
A toggle clamp is a mechanical device commonly used for holding and clamping objects in various applications, such as woodworking, metalworking, and assembly operations. Consider a toggle clamp subjected to a force of 200 N at the handle. The vertical clamping force can be calculated, provided the dimensions of the toggle clamp are known.
The toggle clamp system is a machine structure consisting of movable, pin-connected multi-force members that form a stabilized system to transmit forces. The...
The toggle clamp system is a machine structure consisting of movable, pin-connected multi-force members that form a stabilized system to transmit forces. The...
306
01:12Maximum Size of Aggregate
94
The maximum size of aggregate is defined as the aperture of the sieve retaining 15 percent or more of the particles present in the aggregate sample. The aggregate's maximum size impacts the concrete's water requirement, workability, and strength. Larger aggregates reduce the surface area needing cement paste coverage, which can lower water needs, thereby allowing a decrease in the water-to-cement ratio when the desired workability and richness of the mix are to be maintained, which can...
94
01:25Distribution Reliability and Automation
107
Distribution reliability in electrical power systems is critical for ensuring an uninterrupted power supply to consumers at minimal cost. According to IEEE Standard Terms, reliability is the probability that a device will function without failure over a specified time period or amount of usage. For electric power distribution, this translates to maintaining continuous power supply and addressing customer concerns over power outages. Several indices, as defined by IEEE Standard 1366-2012, are...
107