Grade Inflation in Generative Models.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Generative models require reliable evaluation metrics for assessing synthetic data quality.
• Existing scores for comparing distributions often provide overly optimistic performance assessments.

Purpose of the Study:

• To identify and explain the "grade inflation problem" in commonly used generative model evaluation scores.
• To introduce a new class of "equidensity" scores designed to overcome this limitation.
• To present the Eden score as a novel equidensity score and evaluate its performance.

Main Methods:

• Analysis of widely used scores: correlation, Jaccard, earth-mover's, and Kullback-Leibler (relative-entropy).
• Introduction of the "equipoint" score concept, where all data points are valued equally.
• Development and testing of the Eden score, an example of an "equidensity" score.

Main Results:

• Commonly used "equipoint" scores (correlation, Jaccard, earth-mover's, KL divergence) exhibit "grade inflation."
• The proposed Eden score, an "equidensity" score, successfully avoids grade inflation.
• Eden score shows improved agreement with human perception of data distribution fit compared to equipoint scores.

Conclusions:

• Equipoint scores are inherently susceptible to grade inflation when evaluating generative models.
• Equidensity scores, exemplified by Eden, offer a more robust and perceptually aligned method for evaluating generative models.
• Equidensity scores are recommended for comparing low-dimensional distributions, particularly in the context of generative AI.