Infusing clinical knowledge into language models by subword optimisation and embedding initialisation.

Area of Science:

• Natural Language Processing (NLP)
• Medical Informatics
• Machine Learning

Background:

• Clinical text processing requires sophisticated language models.
• Existing tokenization methods may not fully capture clinical domain knowledge.
• Integrating medical semantics into language models is crucial for accuracy.

Purpose of the Study:

• To introduce K-Tokeniser, a novel tokenization methodology for clinical text.
• To infuse clinical knowledge into language models for improved performance.
• To enhance semantic understanding in clinical natural language processing tasks.

Main Methods:

• K-Tokeniser populates token representations using domain ontologies (e.g., UMLS) or corpus data.
• It utilizes sentence-level context to select optimal global token representations during training/inference.
• An embedding initialization approach supports new tokens without pre-training.

Main Results:

• K-Tokeniser demonstrated consistent improvements across transformer-based models and four real-world clinical datasets.
• Significant gains were observed in automated clinical coding (13% Micro F1 score increase).
• K-Tokeniser facilitated quicker convergence of language models, reducing data requirements.

Conclusions:

• Models utilizing K-Tokeniser exhibit faster convergence and improved performance.
• Achieved baseline performance with significantly less training data (e.g., 20% for automated coding).
• The generalizable approach requires no pre-training, enhancing its applicability.