Evaluating and Enhancing Large Language Models' Performance in Domain-Specific Medicine: Development and Usability...

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Clinical Decision Support

Background:

• Efficacy of large language models (LLMs) in specialized medical domains, like osteoarthritis (OA) management, is largely unknown.
• General LLMs may lack the precision required for complex disease management.
• Need for robust evaluation frameworks for medical LLMs.

Purpose of the Study:

• To evaluate and enhance the clinical capabilities and explainability of LLMs in specific medical domains.
• To use osteoarthritis (OA) management as a case study for LLM application.
• To develop and validate a domain-specific benchmark for medical LLMs.

Main Methods:

• Developed a domain-specific benchmark framework for evaluating LLMs from knowledge to clinical application.
• Created DocOA, a specialized LLM for OA management using retrieval-augmented generation and instructional prompts.
• Compared GPT-3.5, GPT-4, and DocOA using objective and human evaluations in real-world clinical scenarios.

Main Results:

• General LLMs (GPT-3.5, GPT-4) showed limited effectiveness in OA management, especially for personalized treatment recommendations.
• The specialized LLM, DocOA, demonstrated significant improvements in performance.
• DocOA's retrieval-augmented generation enabled identification of clinical evidence, enhancing answer explainability.

Conclusions:

• Introduced a novel benchmark for assessing domain-specific LLM abilities comprehensively.
• Highlighted limitations of generalized LLMs in clinical settings.
• Demonstrated the potential of tailored LLM approaches for developing effective domain-specific medical AI tools.