SNER: Semi-Supervised Named Entity Recognition for Large Volume of Diabetes Data

Area of Science:

• Computational linguistics
• Medical informatics
• Artificial intelligence in healthcare

Background:

• Diabetes prevention and treatment rely on medical literature and records.
• Extracting entities from diabetes textual data is crucial yet challenging.
• Existing named entity recognition (NER) methods are insufficient for specialized diabetes data.

Purpose of the Study:

• To propose a novel semi-supervised learning method, SNER, for effective NER in diabetes data.
• To address the challenges of large data volumes, lack of labeled data, and high manual labeling costs in diabetes text processing.

Main Methods:

• Developed SNER, a semi-supervised learning approach for diabetes named entity recognition.
• Utilized large unlabeled datasets to overcome the scarcity of labeled data.
• Filtered predicted labels using confidence and uncertainty scores, creating positive and negative pseudo-labels, with strategic use of negative pseudo-labels.

Main Results:

• SNER effectively processes large volumes of diabetes-related textual data.
• The method demonstrates superior performance compared to existing state-of-the-art models on two public diabetes datasets.
• SNER successfully mitigates issues related to limited labeled data and manual annotation costs.

Conclusions:

• SNER offers an effective solution for named entity recognition in the challenging domain of diabetes data.
• Semi-supervised learning, particularly with the proposed pseudo-labeling strategy, significantly enhances NER performance for specialized medical text.
• The developed method holds promise for improving the extraction and utilization of critical information from diabetes literature and records.