Human-in-the-Loop Performance of LLM-Assisted Arterial Blood Gas Interpretation: A Single-Center Retrospective Study

Area of Science:

• Medical Diagnostics
• Artificial Intelligence in Medicine

Background:

• Interpreting acid-base disorders from arterial blood gas (ABG) results is complex, especially in mixed cases.
• Large language models (LLMs) show potential for assisting in complex cognitive tasks.

Purpose of the Study:

• To evaluate the performance of LLMs in interpreting ABG results for acid-base disorders.
• To compare LLM interpretation with expert clinical pathologist evaluation.

Main Methods:

• A retrospective study of 200 ABG datasets across five diagnostic categories.
• Three LLMs (ChatGPT GPT-4o, Copilot GPT-4, Gemini 1.5/2.5-flash) were used by medical students for interpretation.
• Two evaluation methods were used: direct interpretation (LLM-I) and interpretation with a supervision model (LLM-S).

Main Results:

• Strong agreement (Cohen's κ ≥ 0.88) was observed for identifying primary acid-base disorders across all methods.
• LLM-S achieved strong agreement (κ = 0.81–0.91) for identifying primary and secondary disorders, while LLM-I showed moderate agreement (κ = 0.61–0.65).

Conclusions:

• LLM-assisted ABG interpretation demonstrates strong concordance with expert evaluation for primary acid-base disorders.
• LLMs show promise in enhancing medical students' understanding of acid-base disorders and minimizing calculation errors.