Automated ICD-10-Anchored Classification of Primary Care Text Data: Development and Evaluation of a Custom Multilabel...

Area of Science:

• Natural Language Processing in Healthcare
• Machine Learning for Medical Informatics
• Clinical Data Analysis

Background:

• Electronic medical records (EMRs) contain valuable clinical data, but much is unstructured free text.
• Manual coding of free-text EMR data for analysis is time-consuming and resource-intensive.
• Automated coding using language models is promising but understudied for German primary care notes.

Purpose of the Study:

• To provide guidance for applied health researchers on automatic classification of free-text notes.
• To demonstrate the effective and accurate use of a fine-tuned language model for International Statistical Classification of Diseases, Tenth Revision (ICD-10) coding.
• To develop and evaluate a multilabel classifier for diagnostic information extraction.

Main Methods:

• Trained a large language model-based multilabel classifier on 38,728 manually categorized free-text notes from the FIRE database.
• Utilized ICD-10 codes and ad hoc labels for categorization across 47 classes.
• Stratified data into training (70%), validation (15%), and testing (15%) sets, and trained using the Transformers Python library over 10 epochs.

Main Results:

• The FIRE classifier achieved strong performance on the held-out test set across 48 classes.
• Reported F1-scores of 0.85 (micro), 0.86 (macro), and 0.84 (weighted).
• Demonstrated robust classification accuracy for diagnostic information.

Conclusions:

• This study outlines steps for training open-source large language models for healthcare applications.
• Highlights the potential to streamline and scale diagnostic information extraction from EMRs.
• The model can be deployed for prescreening and labeling free-text data, reducing manual handling burden.