Explainability of decoder-only clinical large language models: A scoping review

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Computational Linguistics

Background:

• Clinical large language models (LLMs) offer potential in healthcare but face trust and safety challenges due to opaque reasoning.
• Existing explainability reviews often focus on traditional machine learning or general LLMs, not specifically clinical decoder-only models.

Purpose of the Study:

• To systematically map and analyze explainability approaches for large, decoder-only clinical LLMs.
• To identify trends, gaps, and regulatory considerations in the field of clinical LLM explainability.

Main Methods:

• A PRISMA-ScR scoping review of studies on clinical LLM explainability published up to early 2026.
• Searches conducted across major scientific databases (PubMed, Scopus, Web of Science, ACM Digital Library, arXiv).

Main Results:

• LLM-native generative/interactive methods (e.g., chain-of-thought, retrieval-augmented citation) dominated explainability approaches (58.0%).
• Proprietary models were prevalent (75.4%), yet mechanistic analysis relied solely on open-source models, highlighting a transparency gap.
• Quantitative evaluation of explanations was common (59.4%), but metrics lacked standardization and rarely assessed faithfulness; local explanations predominated.

Conclusions:

• Clinical LLM explainability is evolving towards generative rationales, but their alignment with true model reasoning requires further evidence.
• Recommendations for trustworthy deployment include prioritizing verifiable explanations, preferring inspectable models, and prospectively validating in clinical settings.