Automatic (near-) duplicate content document detection in a cancer registry.

Area of Science:

• Medical Informatics
• Data Management
• Computational Biology

Background:

• Duplicate and near-duplicate medical documents pose challenges in data management, clinical use, and research.
• Large volumes of multisourced documents in cancer registries complicate processing.
• Pairwise comparison is infeasible for identifying near-duplicates in extensive datasets.

Purpose of the Study:

• To develop and validate an efficient system for identifying full-duplicates and near-duplicates in multisourced medical documents.
• To address the challenges of document processing in population-based cancer registries.
• To reduce the false positive rate in near-duplicate detection without compromising sensitivity.

Main Methods:

• Implemented a system combining Simhash (Locality Sensitive Hashing) and Smith-Waterman text alignment.
• Validated the method on 3042 manually verified document pairs.
• Achieved high performance metrics: AUC 0.96, sensitivity 0.92, specificity 1.00, PPV 1.00, NPV 0.91.

Main Results:

• Applied to 224,398 medical documents, identifying 5.5% duplicates and 0.17-0.24% near-duplicates.
• Most near-duplicates shared patient and transmitter information.
• Differences were primarily administrative (83%) rather than medical (2%).

Conclusions:

• The developed method efficiently identifies full-duplicates and most near-duplicates in large medical document sets.
• The system demonstrates high accuracy and can be applied across various domains.
• Potential improvements and broader applications of the method are discussed.