Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Nursing Clinical Information System01:27

Nursing Clinical Information System

1.3K
Nursing Clinical Information System (NCIS)
A Nursing Clinical Information System (NCIS) is a specialized type of healthcare information system tailored to meet the unique needs of nursing practice. It incorporates the principles of nursing informatics to streamline information management and improve the quality of care delivery.
Critical attributes of NCIS include:
1.3K
Clinical Trials01:16

Clinical Trials

10.9K
Clinical trials are prospective experimental studies conducted on humans to determine the safety and efficacy of treatments, drugs, diet methods, and medical devices. Using statistics in clinical trials enables researchers to derive reasonable and accurate conclusions from the collected data, allowing them to make wise decisions in uncertain situations. In medical research, statistical methods are crucial for preventing errors and bias.
There are four phases in a clinical trial. A phase one...
10.9K
Identifying Statistically Significant Differences: The F-Test01:14

Identifying Statistically Significant Differences: The F-Test

3.9K
The F-test is used to compare two sample variances to each other or compare the sample variance to the population variance. It is used to decide whether an indeterminate error can explain the difference in their values. The underlying assumptions that allow the use of the F-test include the data set or sets are normally distributed, and the data sets are independent of each other. The test statistic F is calculated by dividing one variance by another. In other words, the square of one standard...
3.9K
Clinical Trials: Overview01:11

Clinical Trials: Overview

5.0K
Clinical development focuses on how the drug will interact with the human body and encompasses four key phases of clinical trials, each serving a specific purpose in assessing the safety and effectiveness of new drugs. These phases overlap and build upon one another. Phase I involves a small group of healthy volunteers (typically 20-80 individuals) or, in cases where significant toxicity is expected, patients with the targeted disease, such as cancer or AIDS. The volunteers are tested for...
5.0K
Changes in Skin Color: Clinical Perspectives01:14

Changes in Skin Color: Clinical Perspectives

3.8K
The first thing a clinician sees is the skin, so the examination of the skin should be part of any thorough physical examination. Most skin disorders are relatively benign, but a few, including melanomas, can be fatal if untreated. A couple of the more noticeable disorders, albinism and vitiligo, affect the appearance of the skin and its accessory organs.
Albinism
Albinism is a genetic disorder that affects (completely or partially) the coloring of skin, hair, and eyes. The defect is primarily...
3.8K
COPD: Pathogenesis and Clinical Features01:20

COPD: Pathogenesis and Clinical Features

1.9K
Chronic obstructive pulmonary disease (COPD) is a group of lung conditions that progressively worsen over time, including chronic bronchitis and emphysema. This cluster of diseases collectively leads to a gradual and irreversible decline in lung function over time.
The primary cause for the onset of COPD is cigarette smoking and exposure to air pollution. These hazardous factors initiate a chain reaction within the lungs, resulting in chronic inflammation, damage to the airways, and a...
1.9K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Development of the authentication and authorization processes for the iAgree portal, a platform for patient-controlled data sharing across health systems.

JAMIA open·2026
Same author

Propofol-based total intravenous anesthesia and recurrence-free survival after hepatectomy-does it improve outcomes?

Annals of translational medicine·2026
Same author

Engineering biomarker representations of vital signs data enhances deep learning mortality prediction.

Journal of the American Medical Informatics Association : JAMIA·2026
Same author

Applying Large Language Models in Perioperative Medicine With an Equity-Informed Perspective.

Anesthesia and analgesia·2026
Same author

The Anesthesia Preparedeness Clinic (APC) triage score tool: an integrated electronic health record tool for improving resource allocation for preoperative care clinics.

Journal of clinical monitoring and computing·2026
Same author

Neostigmine versus sugammadex on outpatient recovery among obese patients with obstructive sleep apnea: A randomized controlled trial.

Scientific reports·2026

Related Experiment Video

Updated: Feb 11, 2026

Stereo-Electro-Encephalo-Graphy SEEG With Robotic Assistance in the Presurgical Evaluation of Medical Refractory Epilepsy: A Technical Note
05:54

Stereo-Electro-Encephalo-Graphy SEEG With Robotic Assistance in the Presurgical Evaluation of Medical Refractory Epilepsy: A Technical Note

Published on: June 13, 2016

18.2K

Identifying and characterizing highly similar notes in big clinical note datasets.

Rodney A Gabriel1, Tsung-Ting Kuo2, Julian McAuley3

  • 1UCSD Health Department of Biomedical Informatics, University of California, San Diego, 9500 Gilman Dr, La Jolla, CA 92093, USA; Department of Anesthesiology, University of California, San Diego, 200 West Arbor Dr, San Diego, CA 92103, USA.

Journal of Biomedical Informatics
|April 22, 2018
PubMed
Summary

A scalable algorithm effectively de-duplicated large clinical note datasets from electronic health records (EHR), identifying significant near-to-exact duplicate notes. This process enhances data quality for developing accurate patient outcome prediction models.

Keywords:
De-deduplicationElectronic medical recordNatural language processing

More Related Videos

Optimized Workflow for Iterative Bleaching Extends Multiplexity Imaging of Highly Autofluorescent Clinical Samples
06:52

Optimized Workflow for Iterative Bleaching Extends Multiplexity Imaging of Highly Autofluorescent Clinical Samples

Published on: July 11, 2025

899
Mining Spatial Transcriptomics Datasets using DeepSpaceDB
10:16

Mining Spatial Transcriptomics Datasets using DeepSpaceDB

Published on: September 5, 2025

804

Related Experiment Videos

Last Updated: Feb 11, 2026

Stereo-Electro-Encephalo-Graphy SEEG With Robotic Assistance in the Presurgical Evaluation of Medical Refractory Epilepsy: A Technical Note
05:54

Stereo-Electro-Encephalo-Graphy SEEG With Robotic Assistance in the Presurgical Evaluation of Medical Refractory Epilepsy: A Technical Note

Published on: June 13, 2016

18.2K
Optimized Workflow for Iterative Bleaching Extends Multiplexity Imaging of Highly Autofluorescent Clinical Samples
06:52

Optimized Workflow for Iterative Bleaching Extends Multiplexity Imaging of Highly Autofluorescent Clinical Samples

Published on: July 11, 2025

899
Mining Spatial Transcriptomics Datasets using DeepSpaceDB
10:16

Mining Spatial Transcriptomics Datasets using DeepSpaceDB

Published on: September 5, 2025

804

Area of Science:

  • Biomedical Informatics
  • Data Science
  • Natural Language Processing

Background:

  • Electronic health records (EHR) contain vast clinical note datasets valuable for statistical modeling.
  • Near-to-exact duplication of notes is a prevalent challenge in these datasets.
  • Accurate de-duplication is crucial for reliable analysis of patient diagnoses and outcomes.

Purpose of the Study:

  • To develop and implement a scalable algorithm for de-duplicating clinical notes.
  • To characterize the sources and extent of note duplication in large EHR datasets.
  • To improve the quality of clinical note datasets for downstream machine learning applications.

Main Methods:

  • Employed an approximation algorithm involving Minhashing with Locality Sensitive Hashing (LSH).
  • Utilized a clustering method with tree-structured disjoint sets to group similar notes.
  • Classified near-duplicates using Jaccard Similarity (JS) for pairwise comparisons within clusters.
  • Analyzed institutional EHR data and the MIMIC-III dataset.

Main Results:

  • Processed 1,528,940 institutional notes in 36.3 hours.
  • At a JS threshold of 0.7, 82,371 clusters were formed, containing 304,418 notes.
  • The algorithm demonstrated high accuracy, with no incorrectly clustered pairs across JS thresholds.
  • Achieved 100% capture of duplicate pairs at JS thresholds of 0.9 and 1.0.
  • Similar de-duplication performance was observed on the MIMIC-III dataset.

Conclusions:

  • Confirmed a significant presence of near-to-exact duplicated notes in both institutional EHR and MIMIC-III datasets.
  • The developed algorithm is effective and scalable for de-duplicating large clinical note corpora.
  • Improved data quality through de-duplication is essential for robust clinical data analysis and predictive modeling.