Large Language Models' Clinical Decision-Making on When to Perform a Kidney Biopsy: Comparative Study

Area of Science:

• Nephrology
• Artificial Intelligence
• Medical Decision Support

Background:

• Artificial intelligence (AI) and large language models (LLMs) are increasingly sophisticated and integrated into various disciplines.
• The application of LLMs to augment clinical decision-making is an active area of research.

Purpose of the Study:

• To compare the responses of over 1000 nephrologists with the outputs of commonly used LLMs regarding kidney biopsy indications.
• To assess the alignment and risk tolerance of different LLMs compared to human expert consensus.

Main Methods:

• A large international online questionnaire was administered to over 1000 nephrologists to determine kidney biopsy indications.
• The same questions were posed to eight LLMs (ChatGPT-3.5, Mistral, Perplexity, Copilot, Llama 2, GPT-4, MedLM, Claude 3).
• Responses were scored (0-44) to reflect biopsy propensity, with higher scores indicating greater risk tolerance.

Main Results:

• LLM performance in replicating human expert consensus varied widely.
• OpenAI models (ChatGPT-3.5, GPT-4) showed the highest alignment with human consensus (6/11 questions).
• LLMs exhibited diverse risk tolerance, with MedLM being most risk-averse (score 11) and Claude 3 least risk-averse (score 34).

Conclusions:

• LLMs demonstrated a modest ability to replicate human clinical decision-making in kidney biopsy indications.
• Performance variation among LLMs suggests limitations for real-world clinical practice.
• Alignment was higher for questions with uniform human responses and lower for those with less consensus.