Related Experiment Video
Updated: Jun 12, 2025
04:58
Author Spotlight: Investigating the Role of Repetitive DNA Misregulation in Cancer Initiation and Immunotherapy Resistance
Published on: December 13, 2024
2.2K
SparkDWM: a scalable design of a Data Washing Machine using Apache Spark
Nicholas Kofi Akortia Hagan1, John R Talburt1
1Department of Information Sciences, University of Arkansas at Little Rock, Little Rock, AR, United States.
Frontiers in Big Data
|September 24, 2024
Summary
This study enhances the Data Washing Machine for scalable entity resolution using Apache Spark, improving data quality and outperforming previous versions. The refactored system efficiently handles large datasets, ensuring accurate record matching.
Area of Science:
- Computer Science
- Data Science
- Artificial Intelligence
Background:
- Rapid data growth in applications leads to increased data quality issues like duplication and errors.
- Traditional Entity Resolution (ER) systems struggle with scalability due to large data volumes.
- The legacy Data Washing Machine (DWM) proof-of-concept had a single-threaded design, limiting its performance.
Purpose of the Study:
- To refactor the Data Washing Machine for high scalability using Apache Spark.
- To address the single-threaded limitation of the legacy DWM.
- To improve the DWM's design by incorporating intrinsic metadata from references.
Main Methods:
- Utilized Apache Spark's Resilient Distributed Dataset (RDD) for distributed data processing.
- Refactored the legacy DWM to a parallel processing architecture.
- Integrated intrinsic metadata from references to enhance the ER process.
- Validated results against the legacy DWM using 18 synthetic datasets.
- Tested scalability with real-world benchmark ER datasets ranging from thousands to millions of records.
Main Results:
- The refactored DWM achieved identical results to the legacy system on synthetic data.
- The proposed system demonstrated superior performance compared to a MapReduce-based DWM.
- Experimental results confirmed the scalability of the refactored system on large datasets.
- The system identified more clusters than the Famer system under optimal clustering parameters.
Conclusions:
- The Apache Spark-based Data Washing Machine is a highly scalable and effective solution for entity resolution.
- The refactored system overcomes the performance limitations of the legacy DWM.
- This enhanced DWM provides accurate and efficient data quality improvement for large-scale applications.
Related Concept Videos
01:21Design Example: Creating a Hydraulic Model of a Dam Spillway
137
Scaled hydraulic models of dam spillways provide a practical way to replicate and study the intricate flow dynamics of these structures. Often built to a 1:15 ratio, these models allow for observing critical water behavior, such as velocity distribution, flow patterns, and energy dissipation.
137
01:20Parallel Processing
145
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
145
01:30Machines: Problem Solving II
300
Machines are complex structures consisting of movable, pin-connected multi-force members that work together to transmit forces. Consider a lifting tong carrying a 100 kg load. It comprises movable sections DAF and CBG linked together with member AB.
300
01:22Machines: Problem Solving I
306
A toggle clamp is a mechanical device commonly used for holding and clamping objects in various applications, such as woodworking, metalworking, and assembly operations. Consider a toggle clamp subjected to a force of 200 N at the handle. The vertical clamping force can be calculated, provided the dimensions of the toggle clamp are known.
The toggle clamp system is a machine structure consisting of movable, pin-connected multi-force members that form a stabilized system to transmit forces. The...
The toggle clamp system is a machine structure consisting of movable, pin-connected multi-force members that form a stabilized system to transmit forces. The...
306
01:12Maximum Size of Aggregate
94
The maximum size of aggregate is defined as the aperture of the sieve retaining 15 percent or more of the particles present in the aggregate sample. The aggregate's maximum size impacts the concrete's water requirement, workability, and strength. Larger aggregates reduce the surface area needing cement paste coverage, which can lower water needs, thereby allowing a decrease in the water-to-cement ratio when the desired workability and richness of the mix are to be maintained, which can...
94
01:25Distribution Reliability and Automation
107
Distribution reliability in electrical power systems is critical for ensuring an uninterrupted power supply to consumers at minimal cost. According to IEEE Standard Terms, reliability is the probability that a device will function without failure over a specified time period or amount of usage. For electric power distribution, this translates to maintaining continuous power supply and addressing customer concerns over power outages. Several indices, as defined by IEEE Standard 1366-2012, are...
107