Application of generative language models to orthopaedic practice.

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing in Healthcare

Background:

• Large Language Models (LLMs) demonstrate potential in various text-generation tasks.
• The application of LLMs in clinical documentation and management planning requires thorough evaluation.

Purpose of the Study:

• To assess the capability of LLMs, specifically GPT-3 and ChatGPT, in generating clinical letters and management plans for orthopaedic scenarios.
• To evaluate the readability and accuracy of LLM-generated content.

Main Methods:

• Fifteen common orthopaedic scenarios were used to prompt GPT-3 and ChatGPT for clinical letter and management plan generation.
• Readability was assessed using Flesch-Kincade Grade Level, Flesch Readability Ease, and SMOG Index.
• Accuracy of generated letters and plans was evaluated by three independent orthopaedic surgeons.

Main Results:

• Both LLMs generated complete letters for all scenarios with minimal prompting.
• ChatGPT produced more accurate letters (8.7/10) and management plans (7.9/10) compared to GPT-3 (7.3/10 and 6.8/10, respectively).
• Readability scores indicated generally accessible content, though LLMs sometimes included inaccurate information or omissions.

Conclusions:

• LLMs are effective tools for generating clinical letters, offering good readability and accuracy.
• Current LLMs require careful review due to potential inaccuracies and inconsistencies in generated content.
• Future development of healthcare-specific LLMs could enhance clinician efficiency by summarizing complex data into clinical letters.