Logit Fingerprinting: A Novel, Accuracy-Independent Method for Validating Large Language Model Stability in...

Area of Science:

• Artificial Intelligence
• Clinical Informatics
• Medical AI

Background:

• Large Language Models (LLMs) integration into clinical settings necessitates robust quality assurance.
• Standard accuracy metrics are insufficient for detecting behavioral volatility caused by model compression (quantization, distillation) and sparse architectures.

Purpose of the Study:

• To introduce a novel method, the "Single-Token Forced-Choice Logit Probe," for assessing LLM behavioral stability.
• To generate a "behavioral fingerprint" to detect hidden effects of model compression and architectural instability.

Main Methods:

• Developed and validated the "Single-Token Forced-Choice Logit Probe" on 11 local model families using a domain-specific (MedQA) benchmark.
• Conducted a longitudinal audit of commercial APIs to assess decision-making stability.
• Utilized forensic classification to identify compression techniques (e.g., Q8 vs. FP8) and analyze non-determinism sources (e.g., Sparse Mixture-of-Experts routing).

Main Results:

• The proposed method achieved 100% accuracy in distinguishing full-precision models from quantized variants.
• A significant "Stability Gap" was observed in commercial APIs, with distilled "Nano" models showing nearly double the decision instability (2.82% Flip Rate) compared to standard models (1.58% Flip Rate).
• Identified Sparse Mixture-of-Experts routing as a likely source of non-determinism in distilled models.

Conclusions:

• The "Flip Rate" is identified as a critical safety metric for evaluating LLMs in clinical contexts.
• Distilled and quantized LLMs require rigorous stability auditing before clinical deployment to ensure patient safety and reliable performance.