Comparing frameworks for similarity-aware generalized entropy (sentropy).

Area of Science:

• Information theory
• Machine learning
• Data analysis

Background:

• Traditional Shannon entropy focuses on element frequencies in datasets.
• It overlooks crucial information embedded in element similarities and differences.
• Similarity-sensitive entropy measures like sentropy address this limitation.

Purpose of the Study:

• To compare the Leinster-Cobbold-Reeve framework (LCR) and Vendi Score (VS) for measuring sentropy.
• To determine which method is more suitable for machine learning datasets.
• To analyze the theoretical and numerical differences between LCR and VS.

Main Methods:

• Theoretical analysis and numerical computation.
• Utilized 53 diverse and large-scale imaging and tabular datasets.
• Introduced and utilized the concept of "half distance" to parameterize similarity scaling.

Main Results:

• LCR and VS values can vary significantly and are often complementary.
• Both metrics' results are dependent on the scaling of similarities.
• VS provides an upper bound for LCR under specific conditions of the Rényi-Hill order parameter.

Conclusions:

• VS is recommended for datasets interpretable as linear combinations of "ur-elements" or with quantum mechanical characteristics.
• LCR is generally favored for capturing rich information from element similarities, differences, and frequencies.
• LCR and VS can be complementary at specific "half distances".