Which Similarity-Sensitive Entropy (Sentropy)?

Area of Science:

• Information Theory
• Data Science
• Machine Learning

Background:

• Traditional entropy measures quantify information based on element frequencies.
• Leinster, Cobbold, and Reeve (LCR) introduced similarity-sensitive entropy (S-entropy) to capture richer information.
• The Vendi Score (VS) emerged as an alternative, prompting a comparison with LCR.

Purpose of the Study:

• To compare the Leinster-Cobbold-Reeve (LCR) method and the Vendi Score (VS) for quantifying system entropy.
• To determine the practical implications and optimal use cases for LCR and VS.
• To analyze the conceptual, analytical, and experimental differences between LCR and VS.

Main Methods:

• Conceptual, analytical, and experimental comparisons using 53 machine-learning datasets.
• Investigation of the dependence of LCR and VS on similarity scaling and the introduction of "half distance".
• Mathematical proof of VS as an upper bound for LCR under specific conditions.

Main Results:

• LCR and VS values can differ significantly and capture complementary information.
• Both measures are sensitive to similarity scaling, parameterized by "half distance".
• VS is proven to be an upper bound for LCR for several Rényi-Hill order parameters.

Conclusions:

• VS is preferable for specific interpretations (ur-elements, quantum-like systems).
• LCR is generally favored for capturing similarity-encoded information in entropy calculations.
• LCR and VS can be complementary at certain "half distances".