A comparative analysis of machine learning models and human expertise for nursing intervention classification

Area of Science:

• Clinical Informatics
• Artificial Intelligence in Healthcare
• Nursing Documentation

Background:

• Automated classification of nursing notes is crucial for standardized data and benchmarking.
• The Nursing Interventions Classification (NIC) system provides a standardized framework for nursing care.

Purpose of the Study:

• To compare the performance of machine learning (ML) models against human experts in mapping nursing notes to the NIC system.
• To identify areas where ML models excel and where they fall short in clinical documentation classification.

Main Methods:

• Developed and evaluated four ML models: TF-IDF, UMLS semantic mapping, GPT-4o mini, and Bio-Clinical BERT.
• Used a dataset of de-identified home healthcare nursing notes.
• Assessed performance using agreement statistics, precision, recall, F1 scores, and Cohen's Kappa, comparing ML models to expert nurse classifications.

Main Results:

• Human experts demonstrated higher agreement and F1 scores than ML models.
• GPT-4o mini achieved the best performance among ML models but still lagged behind human experts.
• ML models performed well on common, clearly defined interventions (e.g., drug management) but struggled with nuanced, context-dependent ones (e.g., information management).

Conclusions:

• Current ML models can assist, but not entirely substitute, human judgment for complex nursing intervention classification.
• Improved ML methods are necessary to accurately capture the nuances of clinical terminology and context-specific documentation.