ChatGPT-4 Responses on Ankle Cartilage Surgery Often Diverge from Expert Consensus: A Comparative Analysis.

Area of Science:

• Orthopedics
• Artificial Intelligence in Medicine
• Surgical Innovation

Background:

• Limited research exists on large language models (LLMs) like ChatGPT for clinical guidance in foot and ankle surgery.
• This study addresses the need to evaluate LLM accuracy in orthopedic decision-making.

Purpose of the Study:

• To assess the accuracy of ChatGPT's guidance on ankle cartilage repair.
• To compare ChatGPT's responses against established expert consensus statements.

Main Methods:

• ChatGPT-4 was queried with 14 questions on ankle cartilage lesion treatments based on the 2017 International Consensus Meeting.
• Responses were evaluated against consensus statements using a Likert scale (1-5) for similarity.
• ChatGPT answers were further scored for Accuracy, Overconclusiveness, Supplementarity, and Incompleteness.

Main Results:

• The mean similarity score between ChatGPT answers and expert consensus was 3.1 ± 0.8.
• ChatGPT achieved 71.4% accuracy, but also showed 35.7% overconclusiveness and 78.6% supplementarity.
• A significant portion of responses (14.3%) were incomplete.

Conclusions:

• ChatGPT-4 frequently provides information that deviates from expert consensus on surgical treatments for ankle cartilage lesions.
• The findings suggest caution is needed when using LLMs for clinical guidance in this specialized surgical field.