Experimenting With the New Frontier: Artificial Intelligence-Powered Chat Bots in Hand Surgery

Area of Science:

• Orthopedics
• Artificial Intelligence in Medicine
• Medical Information Systems

Background:

• Growing use of AI and large language models (LLMs) by patients and medical professionals.
• Need to assess the accuracy of AI responses for common hand surgery inquiries.
• Hypothesis: LLMs provide predominantly correct answers to hand surgery questions.

Purpose of the Study:

• Evaluate the accuracy, completeness, and usefulness of AI-generated responses to hand surgery questions.
• Assess interrater agreement on the quality of AI responses.
• Inform the medical community about the capabilities and limitations of AI in hand surgery.

Main Methods:

• Compiled 56 hand surgery questions based on Green's Operative Hand Surgery, 8th Edition.
• Posed questions to ChatGPT (OpenAI).
• Two independent hand surgeons reviewed responses for accuracy, completeness, and usefulness, using Cohen's kappa for agreement analysis.

Main Results:

• ChatGPT provided correct answers for 80% of questions (45/56).
• Responses were deemed useful for 70% (39/56) and complete for 57% (32/56).
• Fair to moderate interrater agreement (Cohen's kappa) was observed, with disagreements on correctness, usefulness, and completeness.

Conclusions:

• LLMs show potential to influence patient perception and referral patterns based on response quality.
• While generally accurate, AI responses require improvement in completeness and precision.
• Hand surgeons and societies must engage with AI developers to ensure patient safety and accurate medical information.