Clinical-ShiftEval: a framework for simulating and evaluating model adaptation in dynamic clinical NLP tasks

Area of Science:

• Clinical Natural Language Processing (NLP)
• Machine Learning in Healthcare
• Electronic Health Records (EHR) Data Analysis

Background:

• Clinical NLP models are crucial for extracting information from EHRs, but often fail in dynamic clinical environments.
• Evolving data distributions (e.g., new guidelines, diseases) cause significant performance degradation in existing models.
• Current evaluation methods assume static data, failing to capture real-world model adaptation challenges.

Purpose of the Study:

• To introduce Clinical-ShiftEval, a framework for simulating and evaluating NLP model adaptation in dynamic clinical settings.
• To assess model performance under label-set incompatibility (LSI) and task definition evolution (TDE) using real clinical text.
• To compare the effectiveness of continual training, in-context learning, and hybrid approaches for model adaptation.

Main Methods:

• Developed Clinical-ShiftEval to operationalize LSI and TDE through controlled data transitions from real datasets.
• Applied the framework to the Chilean Waiting List Corpus, modeling LSI with referral specialty classification and TDE with disease prioritization.
• Systematically compared continual training, in-context learning with large language models, and a hybrid adaptation strategy.

Main Results:

• Clinical-ShiftEval reliably simulates realistic clinical changes and produces interpretable performance drops, validating its utility for benchmarking.
• Conventional supervised models degraded significantly (up to 82% F1 drop for LSI, 43% for TDE).
• In-context learning reduced performance drops (35% LSI, 10% TDE), while a hybrid method achieved ~8% drop; continual training required >30% new data to surpass these.

Conclusions:

• Clinical-ShiftEval provides a robust method for benchmarking NLP model adaptation in dynamic EHR environments.
• In-context learning and hybrid approaches offer substantial resilience to evolving clinical data without needing new labeled data.
• Continual training with moderate new data is most effective for optimal accuracy recovery, eventually outperforming other methods.