GPT-4 Underperforms Experts in Detecting IV Fluid Contamination.

Area of Science:

• Clinical Laboratory Science
• Artificial Intelligence in Healthcare
• Diagnostic Accuracy

Background:

• Intravenous (IV) fluid contamination in clinical laboratory specimens is a frequent issue.
• Current detection methods, such as delta checks and manual review, are often insensitive and disrupt laboratory workflows.
• There is a need for more effective and automated methods to identify IV fluid contamination.

Purpose of the Study:

• To evaluate the effectiveness of large language models (LLMs) in detecting IV crystalloid contamination in laboratory specimens.
• To compare the performance of an LLM (GPT-4) against trained healthcare personnel (HCP) in identifying contaminated samples.

Main Methods:

• Simulated contamination of basic metabolic panels using normal saline (NS) and 5% dextrose in normal saline (D5NS) at varying mixture ratios.
• A multimodal LLM (GPT-4) and a panel of 8 HCP were tasked with distinguishing between genuine and contaminated results.
• Performance metrics including classification accuracy, mixture quantification, and confidence levels were compared using statistical analysis.

Main Results:

• Healthcare personnel demonstrated higher accuracy (95% CIs: 0.73-0.80) than GPT-4 (95% CIs: 0.57-0.71 for NS, 0.57-0.57 for D5NS) in detecting IV fluid contamination.
• HCP tended to overestimate contamination severity in lower mixture ratios, while GPT-4 significantly overestimated contamination with D5NS mixtures.
• No correlation was found between the confidence level reported by either GPT-4 or HCP and the accuracy of their classifications.

Conclusions:

• Large language models, specifically GPT-4, are currently less accurate than trained healthcare personnel for detecting IV fluid contamination in basic metabolic panels.
• The imperfect performance of even trained individuals highlights the ongoing need for novel, automated solutions for reliable contamination detection in clinical laboratories.