Understanding generative AI output with embedding models

Area of Science:

• Data Science
• Artificial Intelligence
• Machine Learning

Background:

• Feature engineering is crucial for quantitative analysis.
• Traditionally, feature engineering relied on domain expertise.
• Deep neural networks (DNNs) offer an automated approach to feature engineering.

Purpose of the Study:

• To explore the utility of dimensionality reduction on DNN embeddings.
• To investigate if embeddings from foundation models contain interpretable information.
• To assess the separability of real and AI-generated data using this framework.

Main Methods:

• Utilized deep neural networks (DNNs) to generate embeddings from input data.
• Applied principal components analysis (PCA) and other dimensionality reduction techniques to these embeddings.
• Analyzed the resulting low-dimensional representations for inherent data structures.

Main Results:

• Dimensionality reduction techniques successfully uncovered heterogeneity in the data embeddings.
• The revealed data structures corresponded to human-understandable explanations.
• Empirical evidence showed clear separability between real and AI-generated samples within the embedding space.

Conclusions:

• DNN embeddings, when analyzed with dimensionality reduction, offer insights into data heterogeneity.
• This approach provides a method for distinguishing between real and AI-generated data.
• Foundation model embeddings contain rich, interpretable information amenable to analysis.