Combining Token Classification With Large Language Model Revision for Age-Friendly 4M Entity Recognition From Nursing...

Area of Science:

• Natural Language Processing in Healthcare
• Clinical Informatics
• Long-Term Care Technology

Background:

• Nursing home text messages (TMs) contain valuable clinical data aligned with the Age-Friendly Health Systems 4Ms (What Matters, Medication, Mentation, Mobility).
• Current methods fail to capture this unstructured information for systematic monitoring or quality reporting.
• Automated extraction is challenging due to informal language, abbreviations, and fragmented syntax in TMs.

Purpose of the Study:

• To develop and evaluate a multi-stage 4M Entity Recognition (4M-ER) pipeline for improved 4M information extraction from clinical TMs.
• To utilize only locally deployed, open-source models, combining a fine-tuned token classifier with large language model (LLM) revision.
• To enhance the accuracy and efficiency of extracting 4M data from nursing home communications.

Main Methods:

• A 4M-ER pipeline was created using an expert-annotated dataset of 1,169 TMs from 16 nursing homes.
• The pipeline employed a fine-tuned Bio-ClinicalBERT token classifier for initial candidate span identification.
• LLM revision (Gemma, Phi, Qwen, Mistral) refined spans, guided by semantic similarity retrieval, with performance compared against baselines and ablation studies.

Main Results:

• The 4M-ER pipeline achieved superior performance across all 4M domains compared to prior fine-tuned LLMs, with F1 improvements of +2 to +11 percentage points.
• It demonstrated enhanced accuracy over single-stage Bio-ClinicalBERT for Mobility, Mentation, and What Matters.
• LLM revision reduced false positives by 25-35%, while Bio-ClinicalBERT's recall captured subtle entities; silver data augmentation further boosted performance in challenging domains.

Conclusions:

• The 4M-ER pipeline offers an accurate and scalable method for extracting 4M entities from clinical TMs using open-source models.
• The structured 4M data facilitates taxonomy/ontology development and supports downstream applications like clinical surveillance and predictive modeling.
• This approach provides a foundation for compliance with age-friendly quality measures and enhances long-term care quality improvement.