Do Neural Information Extraction Algorithms Generalize Across Institutions?

Area of Science:

• Medical Informatics
• Computational Linguistics
• Artificial Intelligence in Healthcare

Background:

• Natural language processing (NLP) reduces electronic health record (EHR) curation costs.
• Generalizability of NLP models to data from unseen institutions remains a challenge.

Purpose of the Study:

• To evaluate a neural NLP algorithm's performance on data from known and heldout hospitals.
• To assess the impact of training data diversity on NLP model generalization.

Main Methods:

• Collected 24,881 breast pathology reports from seven hospitals.
• Manually annotated reports with nine key attributes.
• Trained convolutional neural networks (CNNs) on data from one (CNN1), two (CNN2), or four (CNN4) hospitals.
• Tested systems on data from five institutions, including heldout ones.

Main Results:

• Cross-institutional accuracy reached 93.87% (CNN1), 95.7% (CNN2), and 96% (CNN4).
• Training diversity improved performance on heldout institutions.
• CNN4 outperformed CNN1 and CNN2 by 2.13% and 0.3% on heldout data, respectively.

Conclusions:

• Neural NLP algorithms can scale to data from previously unseen institutions.
• Diverse training data enhances the generalization ability of NLP models for EHR analysis.