Enhancing Orthopedic Knowledge Assessments: The Performance of Specialized Generative Language Model Optimization.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Orthopedic Knowledge Management

Background:

• Large language models (LLMs) show potential in specialized fields, but their general nature may limit clinical accuracy.
• Optimizing LLMs with domain-specific knowledge bases is crucial for enhancing their utility in medicine.

Purpose of the Study:

• To evaluate and compare the effectiveness of knowledge base-optimized versus unoptimized LLMs in orthopedics.
• To explore optimization strategies for applying LLMs in specialized medical domains.

Main Methods:

• A specialized orthopedic knowledge base was created using AAOS guidelines and publications.
• Thirty orthopedic questions were posed to optimized and unoptimized versions of GPT-4, ChatGLM, and Spark LLM.
• Responses were evaluated for quality, accuracy, and comprehensiveness by three orthopedic surgeons.

Main Results:

• Knowledge base optimization improved GPT-4's quality (15.3%), accuracy (12.5%), and comprehensiveness (12.8%).
• ChatGLM showed improvements of 24.8% (quality), 16.1% (accuracy), and 19.6% (comprehensiveness).
• Spark LLM demonstrated gains of 6.5% (quality), 14.5% (accuracy), and 24.7% (comprehensiveness).

Conclusions:

• Knowledge base optimization significantly enhances LLM performance in orthopedics.
• This approach improves response quality, accuracy, and comprehensiveness for specialized medical applications.
• Knowledge base optimization is an effective strategy for improving LLM performance in specific fields.