ChemTok: A New Rule Based Tokenizer for Chemical Named Entity Recognition

Area of Science:

• Computational chemistry
• Bioinformatics
• Natural Language Processing

Background:

• Named Entity Recognition (NER) is crucial for text mining.
• Effective tokenization is a vital preprocessing step for NER systems.
• Existing tokenization methods may lead to suboptimal segmentation.

Purpose of the Study:

• To introduce ChemTok, an enhanced rule-based tokenizer for chemical and biomedical text.
• To improve tokenization by merging previously split tokens, creating more discriminative units.
• To evaluate ChemTok's performance against established tokenizers.

Main Methods:

• Development of ChemTok using rules extracted from training data.
• Comparison of ChemTok with ChemSpot and tmChem tokenization methods.
• Utilizing Support Vector Machines and Conditional Random Fields as machine learning classifiers.

Main Results:

• Classifiers trained on ChemTok's output demonstrated superior performance.
• ChemTok-based models achieved better classification accuracy.
• Reduced instances of incorrectly segmented entities were observed with ChemTok.

Conclusions:

• ChemTok offers a significant improvement over existing tokenization techniques for NER.
• The rule-based merging strategy enhances token discriminability and downstream task performance.
• ChemTok is a valuable tool for advancing chemical and biomedical text mining.