Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Extraction: Advanced Methods00:56

Extraction: Advanced Methods

446
Metal ions can be separated from one another by complexation with organic ligands–the chelating agent– to form uncharged chelates. Here, the chelating agent must contain hydrophobic groups and behave as a weak acid, losing a proton to bind with the metal. Since most organic ligands used in this process are insoluble or undergo oxidation in the aqueous phase, the chelating agent is initially added to the organic phase and extracted into the aqueous phase. The metal-ligand complex is...
446
Classification of Systems-II01:31

Classification of Systems-II

145
Continuous-time systems have continuous input and output signals, with time measured continuously. These systems are generally defined by differential or algebraic equations. For instance, in an RC circuit, the relationship between input and output voltage is expressed through a differential equation derived from Ohm's law and the capacitor relation,
145
Woodward–Hoffmann Selection Rules and Microscopic Reversibility01:34

Woodward–Hoffmann Selection Rules and Microscopic Reversibility

3.1K
Electrocyclic reactions, cycloadditions, and sigmatropic rearrangements are concerted pericyclic reactions that proceed via a cyclic transition state. These reactions are stereospecific and regioselective. The stereochemistry of the products depends on the symmetry characteristics of the interacting orbitals and the reaction conditions. Accordingly, pericyclic reactions are classified as either symmetry-allowed or symmetry-forbidden. Woodward and Hoffmann presented the selection criteria for...
3.1K
Classification of Systems-I01:26

Classification of Systems-I

184
Linearity is a system property characterized by a direct input-output relationship, combining homogeneity and additivity.
Homogeneity dictates that if an input x(t) is multiplied by a constant c, the output y(t) is multiplied by the same constant. Mathematically, this is expressed as:
184
Aggregates Classification01:29

Aggregates Classification

320
Aggregate classification is generally based on its size, petrographic characteristics, weight, and source. Size classification ranges from coarse to fine aggregates, defined by the size of the particles. Coarse aggregates are particles that do not pass through ASTM sieve No. 4, and aggregates that pass through the sieve are fine aggregates.
Petrographic classification groups aggregates based on common mineralogical characteristics. Some of the common mineral groups found in aggregates are...
320
Statically Indeterminate Problem Solving01:16

Statically Indeterminate Problem Solving

378
Statically indeterminate problems are those where statics alone can not determine the internal forces or reactions. Consider a structure comprising two cylindrical rods made of steel and brass. These rods are joined at point B and restrained by rigid supports at points A and C. Now, the reactions at points A and C and the deflection at point B are to be determined. This rod structure is classified as statically indeterminate as the structure has more supports than are necessary for maintaining...
378

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Case count metric for comparative analysis of entity resolution results.

Frontiers in big data·2026
Same author

SparkDWM: a scalable design of a Data Washing Machine using Apache Spark.

Frontiers in big data·2024
Same author

Editorial: Automated data curation and data governance automation.

Frontiers in big data·2023
Same author

Preprocessing to Address Bias in Healthcare Data.

Studies in health technology and informatics·2022
Same author

A Rule-Based Data Quality Assessment System for Electronic Health Record Data.

Applied clinical informatics·2020

Related Experiment Video

Updated: Jun 30, 2025

Cloud-Based Phrase Mining and Analysis of User-Defined Phrase-Category Association in Biomedical Publications
09:20

Cloud-Based Phrase Mining and Analysis of User-Defined Phrase-Category Association in Biomedical Publications

Published on: February 23, 2019

8.7K

A scalable MapReduce-based design of an unsupervised entity resolution system.

Nicholas Kofi Akortia Hagan1, John R Talburt1, Kris E Anderson1

  • 1Department of Information Sciences, University of Arkansas at Little Rock, Little Rock, AR, United States.

Frontiers in Big Data
|March 18, 2024
PubMed
Summary

This study presents Hadoop Data Washing Machine (HDWM), a scalable, unsupervised entity resolution (ER) system. HDWM overcomes previous limitations, enabling efficient clustering of millions of records across distributed computing nodes.

Keywords:
Data Washing MachineHadoop Distributed File SystemMapReducedata curationentity resolution

More Related Videos

The Spatial Memory Game: Testing the Relationship Between Spatial Language, Object Knowledge, and Spatial Cognition
05:15

The Spatial Memory Game: Testing the Relationship Between Spatial Language, Object Knowledge, and Spatial Cognition

Published on: February 19, 2018

10.8K
Executing Complexity-Increasing Queries in Relational MySQL and NoSQL MongoDB and EXist Size-Growing ISO/EN 13606 Standardized EHR Databases
07:26

Executing Complexity-Increasing Queries in Relational MySQL and NoSQL MongoDB and EXist Size-Growing ISO/EN 13606 Standardized EHR Databases

Published on: March 19, 2018

9.3K

Related Experiment Videos

Last Updated: Jun 30, 2025

Cloud-Based Phrase Mining and Analysis of User-Defined Phrase-Category Association in Biomedical Publications
09:20

Cloud-Based Phrase Mining and Analysis of User-Defined Phrase-Category Association in Biomedical Publications

Published on: February 23, 2019

8.7K
The Spatial Memory Game: Testing the Relationship Between Spatial Language, Object Knowledge, and Spatial Cognition
05:15

The Spatial Memory Game: Testing the Relationship Between Spatial Language, Object Knowledge, and Spatial Cognition

Published on: February 19, 2018

10.8K
Executing Complexity-Increasing Queries in Relational MySQL and NoSQL MongoDB and EXist Size-Growing ISO/EN 13606 Standardized EHR Databases
07:26

Executing Complexity-Increasing Queries in Relational MySQL and NoSQL MongoDB and EXist Size-Growing ISO/EN 13606 Standardized EHR Databases

Published on: March 19, 2018

9.3K

Area of Science:

  • Data Science
  • Computer Science
  • Information Management

Background:

  • Traditional data curation relies heavily on manual intervention, hindering efficiency with increasing data volumes.
  • Entity Resolution (ER) is crucial for data curation, but scaling existing unsupervised ER systems like the Data Washing Machine (DWM) is challenging due to computational complexity and single-threaded limitations.
  • Current DWM implementations lack scalability beyond a few thousand records and require significant shared memory, limiting their practical application.

Purpose of the Study:

  • To address the scalability and shared memory limitations of the existing unsupervised Data Washing Machine (DWM) for entity resolution.
  • To develop a parallelized version of the DWM using Hadoop MapReduce to handle large datasets efficiently.
  • To demonstrate the effectiveness of the proposed Hadoop Data Washing Machine (HDWM) in clustering a high volume of records.

Main Methods:

  • Parallelization of the unsupervised entity resolution (ER) process using Hadoop MapReduce.
  • Implementation of the Hadoop Data Washing Machine (HDWM) as a MapReduce-based system.
  • Evaluation of HDWM's scalability using publicly available ER datasets.

Main Results:

  • The proposed Hadoop Data Washing Machine (HDWM) effectively overcomes the single-threaded nature and shared memory dependency of the legacy DWM.
  • HDWM demonstrates significant scalability, capable of clustering from thousands to millions of equivalent references.
  • The system leverages multiple computational nodes with independent resources, enabling efficient processing of large-scale data.

Conclusions:

  • The Hadoop Data Washing Machine (HDWM) provides a scalable and efficient solution for unsupervised entity resolution.
  • HDWM successfully addresses the limitations of previous DWM implementations, making it suitable for big data environments.
  • This parallelized approach enhances data curation processes by enabling automated, large-scale record clustering.