Evaluation of Online Artificial Intelligence-Generated Information on Common Hand Procedures

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Patient Education

Background:

• Online artificial intelligence (AI) platforms are increasingly used for health information.
• Patient-facing medical information requires accuracy, quality, and readability.
• Hand surgery patient education is crucial for informed decision-making.

Purpose of the Study:

• To evaluate the quality and readability of AI-generated information on common hand surgeries.
• To compare AI responses against established resources from the American Society for Surgery of the Hand (ASSH).

Main Methods:

• An open AI model (ChatGPT) answered patient questions on four common hand surgeries.
• AI-generated responses were assessed for medical accuracy, quality, and readability using JAMA benchmark and DISCERN criteria.
• AI responses were compared to content from the ASSH HandCare website.

Main Results:

• AI responses scored 0/4 on JAMA benchmark and 58 on DISCERN (good), lacking source attribution, reliability, and currency.
• AI-generated content had a Flesch Kinkaid Reading Ease score of 15 and Grade Level of 34 (college level).
• ASSH HandCare materials scored 3/4 on JAMA benchmark, 71 on DISCERN (excellent), with a Grade Level of 9 (eighth/ninth grade).

Conclusions:

• AI platforms offer generally high-quality answers for hand surgery FAQs but lack citations and present information at a high reading level.
• AI consistently advised patients to consult surgeons, emphasizing shared decision-making and individualized care.
• Understanding AI's limitations is essential for hand surgeons as these technologies integrate into patient care.