Building a biomedical tokenizer using the token lattice design pattern and the adapted Viterbi algorithm.

Area of Science:

• Natural Language Processing
• Computational Biology
• Bioinformatics

Background:

• Tokenization lacks a universal standard for English, particularly in biomedical texts.
• Existing biomedical tokenizers often use classification, leading to varied outputs that hinder reuse.
• There's a need for systematic approaches to create and adapt tokenizers for new biomedical subdomains.

Purpose of the Study:

• To introduce a novel tokenizer design pattern and systematic creation approach for biomedical text.
• To develop a hybrid tokenizer combining regular expressions and machine learning.
• To evaluate the effectiveness of the new approach against existing tokenization methods.

Main Methods:

• Developed a novel tokenizer design pattern and systematic creation guidelines.
• Implemented a tokenizer using a combination of regular expressions and a novel machine learning approach.
• Evaluated the tokenizer's performance on biomedical text tokenization tasks against three other methods.

Main Results:

• The developed tokenizer achieved high accuracy in biomedical text tokenization.
• Medpost and the adapted Viterbi tokenizer showed the best performance, with accuracies of 92.9% and 92.4%, respectively.
• The study confirmed that Part-of-Speech (POS) tagging significantly aids in disambiguating tokenization.

Conclusions:

• The proposed design pattern and guidelines offer a viable method for constructing effective biomedical tokenizers.
• The new approach yields tokenizers comparable to leading custom-built solutions.
• Effective tokenization relies heavily on POS tag sequences and the quality of training data.