ChatGPT's competence in responding to urological emergencies

Area of Science:

• Artificial Intelligence in Medicine
• Urology
• Medical Information Systems

Background:

• Artificial intelligence (AI) is increasingly utilized as a source of medical information.
• This study is the first to assess ChatGPT's capabilities in addressing urological emergencies (UE).

Purpose of the Study:

• To evaluate the performance of ChatGPT 4 in answering public and guideline-based questions related to urological emergencies.
• To assess the accuracy and quality of AI-generated responses in a specialized medical field.

Main Methods:

• Collected frequently asked questions (FAQs) on UE from social media and websites.
• Formulated UE questions based on European Association of Urology (EAU) guidelines.
• Submitted questions to ChatGPT 4 and recorded responses.
• Evaluated response quality using a 5-point Global Quality Score (GQS) by two urologists.

Main Results:

• ChatGPT achieved a high satisfaction rate for public FAQs, with 72.6% receiving a GQS of 5.
• Responses to urosepsis FAQs were rated highest (82.3% GQS 5), while renal trauma and acute kidney injury questions scored lower.
• For EAU guideline-based questions, only 54.8% received a GQS of 5.
• Mean GQS for FAQs (4.38) was significantly higher than for guideline-based questions (3.88).

Conclusions:

• ChatGPT demonstrates significant accuracy in answering general public inquiries about urological emergencies.
• AI performance notably declines when addressing complex, guideline-specific urological emergency information.
• Further refinement of AI models is needed for reliable clinical guideline interpretation in urology.