Does the magic of BERT apply to medical code assignment? A quantitative study.

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Machine Learning

Background:

• Unsupervised pretraining and transfer learning with language models yield significant results in various downstream tasks.
• Clinical code assignment, crucial for healthcare, involves inferring diagnosis and procedure codes from extensive clinical notes.
• The efficacy of pretrained models for medical code prediction without specific architectural modifications remains unclear.

Purpose of the Study:

• To conduct a comprehensive quantitative analysis of various pretrained language models for medical code assignment from clinical notes.
• To evaluate the performance of different models pretrained in diverse domains.
• To propose and assess a novel hierarchical fine-tuning architecture with label-wise attention.

Main Methods:

• Quantitative analysis of multiple contextualized language models.
• Implementation of a hierarchical fine-tuning architecture to capture long-range word interactions.
• Adoption of label-wise attention to leverage label information.
• Evaluation on a subset of the MIMIC-III dataset, focusing on frequently occurring codes.

Main Results:

• Contrary to prevailing trends, a classical Convolutional Neural Network (CNN), when carefully trained, demonstrated superior performance compared to attention-based models for medical code assignment.
• The proposed hierarchical fine-tuning architecture and label-wise attention mechanisms were investigated.
• Performance variations were observed among models pretrained in different domains.

Conclusions:

• Carefully trained classical CNNs can be highly effective for medical code assignment, even outperforming complex attention-based models for frequent codes.
• Empirical findings provide valuable insights for developing more robust and efficient medical code assignment models.
• Further research into model architecture and training strategies is warranted for optimizing clinical NLP tasks.