A Novel Evaluation Framework for Medical LLMs: Combining Fuzzy Logic and MCDM for Medical Relation and Clinical...

Area of Science:

• Artificial Intelligence in Healthcare
• Medical Natural Language Processing
• Decision Support Systems

Background:

• Large Language Models (LLMs) are increasingly used in healthcare, necessitating robust evaluation frameworks.
• Current evaluation methods lack comprehensive approaches for assessing medical LLM efficacy, safety, and ethics.
• The integration of AI in medicine requires methods to handle data uncertainty and multifaceted criteria.

Purpose of the Study:

• To develop a comprehensive Multi-Criteria Decision Making (MCDM) approach for evaluating medical LLMs.
• To establish a framework that addresses the uncertainties and complexities of medical data and decision-making.
• To guide the selection and implementation of suitable LLMs in clinical settings.

Main Methods:

• Utilized an extended Fuzzy-Weighted Zero-InConsistency (FWZIC) method with p, q-quasirung orthopair fuzzy sets (p, q-QROFS) for criterion weighting.
• Employed the MultiAtributive Ideal-Real Comparative Analysis (MAIRCA) method for assessing medical LLMs.
• Incorporated fuzzy logic principles to manage imprecise medical data and criteria.

Main Results:

• The 'Medical Relation Extraction' criteria (0.504) were found to be more important than 'Clinical Concept Extraction' (0.495).
• Among six evaluated LLMs, GatorTron S 10B achieved the highest rank.
• GatorTron 90B was ranked last out of the six alternatives.

Conclusions:

• The proposed MCDM framework effectively evaluates medical LLMs, considering efficacy, safety, and ethical compliance.
• The study provides a practical tool for healthcare professionals to make informed decisions about LLM adoption.
• This research contributes to the responsible integration of AI in healthcare, potentially improving patient outcomes.