sentropy: A Python Package for Revealing Hidden Differences in Complex Datasets.

Area of Science:

• Computational biology
• Machine learning
• Data science

Background:

• Machine learning datasets are typically evaluated by size and class balance.
• Existing diversity measures, incorporating element frequencies and similarities, are not readily accessible in Python for large datasets.
• There is a need for specialized tools to apply these richer diversity metrics to machine learning contexts.

Purpose of the Study:

• Introduce greylock, a Python package designed to calculate advanced diversity measures for large machine learning datasets.
• Provide tools for analyzing dataset composition beyond traditional metrics.
• Facilitate the application of frequency- and similarity-sensitive diversity measures in Python.

Main Methods:

• Developed greylock, a Python package for calculating diversity measures.
• Implemented frequency-sensitive measures from Hill's D-number framework.
• Extended functionality to include similarity-sensitive measures and beta diversities for dataset comparison.

Main Results:

• greylock efficiently calculates a comprehensive suite of diversity measures tailored for large datasets.
• The package supports both frequency-dependent and similarity-dependent diversity metrics.
• Demonstrated applicability across diverse fields including immunomics, metagenomics, computational pathology, and medical imaging.

Conclusions:

• greylock provides a powerful and accessible solution for advanced diversity analysis in machine learning.
• The package enhances the understanding of dataset characteristics by incorporating element frequencies and similarities.
• Offers broad utility for researchers across various scientific domains dealing with complex datasets.