Cross-institution natural language processing for reliable clinical association studies: a methodological exploration

Area of Science:

• Clinical informatics
• Biostatistics
• Health services research

Background:

• Electronic medical records (EMRs) contain valuable patient data.
• Natural language processing (NLP) facilitates extracting this data for research.
• Current NLP methodology in clinical research needs further investigation.

Purpose of the Study:

• To investigate the reliability of NLP models in extracting study variables for exposure-outcome association studies.
• To compare association estimates derived from NLP-extracted versus manually extracted exposures.

Main Methods:

• Conducted association studies using convenience sample from an ICU.
• Compared NLP-extracted vs. manually extracted exposures (employment, living status, substance use).
• Varied NLP models, training paradigms, outcome measures, and statistical methods.

Main Results:

• Substantial differences observed in association estimates between NLP-extracted and manually extracted exposures.
• The error in association showed weak correlation with NLP model performance (F1 score).
• Study included 1,174 participants; NLP models trained on up to 2,528 external reports.

Conclusions:

• Associations derived from NLP-extracted exposures should be interpreted with caution.
• Further research is necessary to establish conditions for reliable NLP use in medical association studies.