Unsupervised multi-granular Chinese word segmentation and term discovery via graph partition

Area of Science:

• Natural Language Processing
• Medical Informatics
• Computational Linguistics

Background:

• Supervised algorithms struggle with Chinese electronic health records (EHRs) due to scarce annotated medical data.
• Effective term discovery in EHRs requires robust word segmentation.

Purpose of the Study:

• To develop an unsupervised word segmentation and term discovery method for Chinese EHRs.
• To address the limitations of existing supervised methods in low-resource medical text scenarios.

Main Methods:

• Proposed the Graph Partition-based Term Segmentation (GTS) method, converting sentences into graphs.
• Utilized n-gram statistics for edge weights and ratio cut for segmentation, solved via dynamic programming.
• Employed a BERT-based discriminator for boundary verification and retained unlisted words as potential medical terms.

Main Results:

• GTS outperformed mature segmentation systems in word segmentation accuracy.
• In term discovery, GTS achieved a 17 percentage point higher F1-score than the best baseline, a 47% relative increase.
• Manual review by medical students validated the fine and coarse granularity segmentation results.

Conclusions:

• The graph partition technique enables effective unsupervised word segmentation and term discovery in EHRs without annotated data.
• Multi-granular segmentation accurately identifies potential medical terms of varying lengths.
• GTS offers a viable solution for leveraging EHRs in data-scarce environments.