Generative Artificial Intelligence Models in Clinical Infectious Disease Consultations: A Cross-Sectional Analysis...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Microbiology and Infectious Diseases

Background:

• Generative artificial intelligence (GenAI) offers potential to enhance clinical consultation in infectious diseases (ID) and microbiology.
• Evaluating the performance of GenAI chatbots in real-world clinical scenarios is crucial.

Purpose of the Study:

• To assess the performance of four GenAI chatbots in clinical microbiology and ID settings.
• To compare the factual consistency, comprehensiveness, coherence, and medical harmfulness of AI-generated responses.

Main Methods:

• A cross-sectional study involving 40 unique clinical scenarios.
• Four GenAI chatbots (GPT-4.0, Custom GPT-4.0, Gemini Pro, Claude 2) were evaluated.
• Six specialists and resident trainees conducted blinded, randomized evaluations.

Main Results:

• GPT-4.0 significantly outperformed Gemini Pro and Claude 2 in composite scores, factual consistency, comprehensiveness, and absence of medical harm.
• Specialists rated AI responses higher than trainees, but were also more susceptible to potentially harmful outputs.
• Fewer than two-fifths of AI-generated responses were deemed harmless, indicating significant risks.

Conclusions:

• Clinical experience influences the interpretation of GenAI outputs, with specialists showing unexpected vulnerabilities.
• Current GenAI models are not safe for unsupervised clinical deployment in microbiology and ID.
• Human oversight is essential to mitigate risks associated with AI in clinical decision-making.