A proficient cost reduction framework for de-duplication of records in data integration

Area of Science:

• Data Science
• Database Management
• Information Retrieval

Background:

• Record de-duplication identifies records representing the same entity, crucial for data mining and cleansing.
• Challenges include computational complexity and data representation variations across sources.
• Blocking and windowing methods reduce comparisons but require parameter tuning.

Purpose of the Study:

• To propose a novel framework for record de-duplication.
• To eliminate the need for parameter tuning in blocking and windowing techniques.
• To evaluate the framework's performance on diverse datasets.

Main Methods:

• A framework employing blocking and windowing techniques in succession was developed.
• The framework's impact was evaluated on dirty and massively dirty datasets.
• Experiments were conducted using Febrl (Freely Extensible Biomedical Record Linkage).

Main Results:

• The proposed framework was evaluated using quality and complexity parameters (reduction ratio, precision, recall).
• A significant reduction in the number of record comparisons was observed.
• The framework demonstrated effectiveness across different dataset configurations.

Conclusions:

• The selection of the linkage key is a critical performance factor in record linkage.
• The proposed framework offers an efficient approach to record de-duplication.
• Parameter-free de-duplication enhances data integration processes.