Search research articles

ABOUT JoVE

Overview Leadership Blog JoVE Help Center

AUTHORS

Publishing Process Editorial Board Scope & Policies Peer Review FAQ Submit

LIBRARIANS

Testimonials Subscriptions Access Resources Library Advisory Board FAQ

RESEARCH

JoVE Journal Methods Collections JoVE Encyclopedia of Experiments Archive

EDUCATION

JoVE Core JoVE Business JoVE Science Education JoVE Lab Manual Faculty Resource Center Faculty Site

Terms & Conditions of Use

Related Concept Videos

Design Example: Creating a Hydraulic Model of a Dam Spillway

Design Example: Creating a Hydraulic Model of a Dam Spillway

Scaled hydraulic models of dam spillways provide a practical way to replicate and study the intricate flow dynamics of these structures. Often built to a 1:15 ratio, these models allow for observing critical water behavior, such as velocity distribution, flow patterns, and energy dissipation.

Parallel Processing

Parallel Processing

The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...

Machines: Problem Solving II

Machines: Problem Solving II

Machines are complex structures consisting of movable, pin-connected multi-force members that work together to transmit forces. Consider a lifting tong carrying a 100 kg load. It comprises movable sections DAF and CBG linked together with member AB.

Machines: Problem Solving I

Machines: Problem Solving I

A toggle clamp is a mechanical device commonly used for holding and clamping objects in various applications, such as woodworking, metalworking, and assembly operations. Consider a toggle clamp subjected to a force of 200 N at the handle. The vertical clamping force can be calculated, provided the dimensions of the toggle clamp are known.
The toggle clamp system is a machine structure consisting of movable, pin-connected multi-force members that form a stabilized system to transmit forces. The...

Maximum Size of Aggregate

Maximum Size of Aggregate

The maximum size of aggregate is defined as the aperture of the sieve retaining 15 percent or more of the particles present in the aggregate sample. The aggregate's maximum size impacts the concrete's water requirement, workability, and strength. Larger aggregates reduce the surface area needing cement paste coverage, which can lower water needs, thereby allowing a decrease in the water-to-cement ratio when the desired workability and richness of the mix are to be maintained, which can...

Distribution Reliability and Automation

Distribution Reliability and Automation

Distribution reliability in electrical power systems is critical for ensuring an uninterrupted power supply to consumers at minimal cost. According to IEEE Standard Terms, reliability is the probability that a device will function without failure over a specified time period or amount of usage. For electric power distribution, this translates to maintaining continuous power supply and addressing customer concerns over power outages. Several indices, as defined by IEEE Standard 1366-2012, are...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by

Same author

Case count metric for comparative analysis of entity resolution results.

Frontiers in big data·2026

Same author

A scalable MapReduce-based design of an unsupervised entity resolution system.

Frontiers in big data·2024

Same author

Editorial: Automated data curation and data governance automation.

Frontiers in big data·2023

Same author

Preprocessing to Address Bias in Healthcare Data.

Studies in health technology and informatics·2022

Same author

A Rule-Based Data Quality Assessment System for Electronic Health Record Data.

Applied clinical informatics·2020

Search research articles

Related Experiment Video

Updated: Jun 12, 2025

Author Spotlight: Investigating the Role of Repetitive DNA Misregulation in Cancer Initiation and Immunotherapy Resistance

Author Spotlight: Investigating the Role of Repetitive DNA Misregulation in Cancer Initiation and Immunotherapy Resistance

Published on: December 13, 2024

SparkDWM: a scalable design of a Data Washing Machine using Apache Spark.

Nicholas Kofi Akortia Hagan¹, John R Talburt¹

¹Department of Information Sciences, University of Arkansas at Little Rock, Little Rock, AR, United States.

Frontiers in Big Data

|September 24, 2024

Summary

This study enhances the Data Washing Machine for scalable entity resolution using Apache Spark, improving data quality and outperforming previous versions. The refactored system efficiently handles large datasets, ensuring accurate record matching.

Keywords:

Data Washing Machine PySpark SparkDWM data curation distributed DWM entity resolution

More Related Videos

Databases to Efficiently Manage Medium Sized, Low Velocity, Multidimensional Data in Tissue Engineering

Databases to Efficiently Manage Medium Sized, Low Velocity, Multidimensional Data in Tissue Engineering

Published on: November 22, 2019

Design and Analysis for Fall Detection System Simplification

Design and Analysis for Fall Detection System Simplification

Published on: April 6, 2020

Related Experiment Videos

Last Updated: Jun 12, 2025

Author Spotlight: Investigating the Role of Repetitive DNA Misregulation in Cancer Initiation and Immunotherapy Resistance

Author Spotlight: Investigating the Role of Repetitive DNA Misregulation in Cancer Initiation and Immunotherapy Resistance

Published on: December 13, 2024

Databases to Efficiently Manage Medium Sized, Low Velocity, Multidimensional Data in Tissue Engineering

Databases to Efficiently Manage Medium Sized, Low Velocity, Multidimensional Data in Tissue Engineering

Published on: November 22, 2019

Design and Analysis for Fall Detection System Simplification

Design and Analysis for Fall Detection System Simplification

Published on: April 6, 2020

Area of Science:

Computer Science
Data Science
Artificial Intelligence

Background:

Rapid data growth in applications leads to increased data quality issues like duplication and errors.
Traditional Entity Resolution (ER) systems struggle with scalability due to large data volumes.
The legacy Data Washing Machine (DWM) proof-of-concept had a single-threaded design, limiting its performance.

Purpose of the Study:

To refactor the Data Washing Machine for high scalability using Apache Spark.
To address the single-threaded limitation of the legacy DWM.
To improve the DWM's design by incorporating intrinsic metadata from references.

Main Methods:

Utilized Apache Spark's Resilient Distributed Dataset (RDD) for distributed data processing.
Refactored the legacy DWM to a parallel processing architecture.
Integrated intrinsic metadata from references to enhance the ER process.
Validated results against the legacy DWM using 18 synthetic datasets.
Tested scalability with real-world benchmark ER datasets ranging from thousands to millions of records.

Main Results:

The refactored DWM achieved identical results to the legacy system on synthetic data.
The proposed system demonstrated superior performance compared to a MapReduce-based DWM.
Experimental results confirmed the scalability of the refactored system on large datasets.
The system identified more clusters than the Famer system under optimal clustering parameters.

Conclusions:

The Apache Spark-based Data Washing Machine is a highly scalable and effective solution for entity resolution.
The refactored system overcomes the performance limitations of the legacy DWM.
This enhanced DWM provides accurate and efficient data quality improvement for large-scale applications.