Addressing Generalizability in Clinical Named Entity Recognition: Federated Learning or Large Language Models?: A...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Clinical Named Entity Recognition (NER) is crucial for structuring clinical text data.
• Ensuring NER model generalizability across different healthcare institutions is a significant challenge.
• Federated Learning (FL) and Large Language Models (LLMs) are emerging approaches for clinical NLP tasks.

Purpose of the Study:

• To compare the effectiveness of Federated Learning (FL) and Large Language Models (LLMs) for clinical NER.
• To evaluate the generalizability of these models in extracting Visual Acuity (VA) from ophthalmology notes across two distinct institutions (Stanford, USA and Moorfields Eye Hospital, UK).

Main Methods:

• Utilized BERT-based models for NER.
• Implemented Federated Learning strategies: Federated Averaging (FedAvg) and Stratified Federated Learning with Weighted Training (STWT).
• Employed Large Language Models: Llama-3-70B and Mixtral-8x7B.
• Evaluated model performance on Visual Acuity extraction from ophthalmology notes.

Main Results:

• Federated Learning significantly improved model generalizability across institutions.
• STWT demonstrated superior stability and accuracy compared to FedAvg.
• LLMs performed well on Moorfields Eye Hospital data but showed limitations with structured Stanford notes.
• FL approaches proved more effective for cross-institutional clinical NER tasks.

Conclusions:

• Federated Learning is a highly effective method for enhancing the generalizability of clinical NER models.
• LLMs exhibit domain-specific limitations and may not be universally optimal for all structured clinical data extraction tasks.
• Tailored approaches are necessary to address the challenges of cross-institutional clinical NER.