Taxonomy Portraits: Deciphering the Hierarchical Relationships of Medical Large Language Models

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing
• Machine Learning Performance Evaluation

Background:

• Large language models (LLMs) are increasingly adopted in healthcare, necessitating robust performance evaluation.
• Performance benchmarks are crucial for ranking and guiding the adoption of LLMs in the healthcare sector.

Purpose of the Study:

• To develop taxonomy portraits of medical LLMs (n=33) using multivariate performance benchmarks.
• To identify objective approaches for selecting high-performing LLMs without compromising efficiency in healthcare.

Main Methods:

• Hierarchical clustering of domain-specific and non-specific performance benchmarks from Hugging Face leaderboards.
• Utilizing Wilcoxon rank-sum test and linear correlation to analyze benchmark differences and redundancies.

Main Results:

• Two distinct families of medical LLMs were identified based on statistically significant performance differences.
• Consensus in performance across tasks indicated LLM robustness, while benchmark correlations suggested potential redundancies.

Conclusions:

• Medical LLM taxonomy aids in identifying models with similar performance, aligning with healthcare needs and economics.
• Enhanced transparency in performance benchmarks and economics is crucial for broader, equitable LLM adoption in healthcare.