ChatGPT and reference intervals: a comparative analysis of repeatability in GPT-3.5 Turbo, GPT-4, and GPT-4o

Area of Science:

• Artificial Intelligence in Healthcare
• Laboratory Medicine and Diagnostics
• Clinical Pathology and Informatics

Background:

• Large language models (LLMs) offer potential for rapid clinical consultation in laboratory medicine.
• Uncertainty exists regarding the consistency and clinical reliability of reference intervals generated by LLMs, especially without clinical context.

Purpose of the Study:

• To evaluate the repeatability of reference interval outputs from three ChatGPT versions (GPT-3.5-Turbo, GPT-4, GPT-4o).
• To assess model consistency by using reference interval variability as a stress test when prompts omit interval information.

Main Methods:

• A cross-sectional study involving 726,000 chatbot requests with standardized prompts.
• Analysis of 246,842 reference intervals across 47 laboratory parameters for consistency.
• Statistical analysis using coefficient of variation (CV) and regression models to assess variability.

Main Results:

• Average CVs for reference intervals were 26.50% (lower limit) and 15.82% (upper limit).
• GPT-4 and GPT-4o demonstrated significantly lower CVs than GPT-3.5-Turbo.
• Inconsistent outputs were notable for poorly standardized parameters and varied unit expressions.

Conclusions:

• While newer ChatGPT versions show improved repeatability, diagnostically unacceptable variability remains, particularly for unstandardized analytes.
• Thoughtful prompt design, global standardization of lab practices, model refinement, and regulatory oversight are crucial.
• Current AI chatbots should be limited to professional use and trained to decline interpretation without provided reference intervals.