Performance of 3 Conversational Generative Artificial Intelligence Models for Computing Maximum Safe Doses of Local...

Area of Science:

• Medical Informatics
• Anesthesiology
• Artificial Intelligence in Medicine

Background:

• Generative artificial intelligence (AI) shows potential for optimizing medical decision-making.
• Accurate calculation of maximum safe local anesthetic (LA) doses is critical in anesthesiology to prevent local anesthetic systemic toxicity (LAST).
• Current LA dosing relies on empirical guidelines and clinician experience, leading to variability and errors.

Purpose of the Study:

• To evaluate the efficacy and safety of three generative AI models (ChatGPT, Copilot, Gemini) in calculating maximum safe LA doses.
• To determine the potential clinical utility of these AI models for LA dose calculation.

Main Methods:

• A comparative analysis utilized a 51-item questionnaire across 10 simulated clinical vignettes.
• AI-generated LA dose calculations were compared against scientifically validated reference doses.
• Quantitative dose comparisons and qualitative assessments by independent reviewers were performed.

Main Results:

• All AI models generated responses aligned with LA calculation principles, but safety varied significantly.
• Gemini often deferred to specialists or provided doses exceeding safe limits by 140% (SD 103%) for mixtures.
• ChatGPT (90% unsafe, +198% over limit) and Copilot (67% unsafe, +217% over limit) demonstrated poor performance, with qualitative ratings of 'poor'.

Conclusions:

• Current generative AI models (Gemini, ChatGPT, Copilot) lack the necessary accuracy and reliability for safe LA dose calculation.
• These AI models should not be used as decision-making tools for LA dosing.
• Clinicians must continue to rely on expertise and patient assessment for safe LA administration until validated AI solutions emerge.