Related Experiment Videos
RADS-PDD: Reproducibility-aware dynamic similarity-based pre-training data detection against LLMs
Xin Fan1, Miyamoto Ryoto1, Fan Mo2
1Department of CSCE, Waseda University, Tokyo, Japan.
Summary
Detecting large language model (LLM) pre-training data is challenging. A new method, RADS-PDD, uses dynamic similarity to improve detection accuracy using only final model outputs.
Area of Science:
- Artificial Intelligence
- Natural Language Processing
- Machine Learning
Background:
- Large language models (LLMs) show performance gains with larger pre-training corpora, raising ethical and legal concerns due to sensitive or copyrighted data.
- Detecting pre-training data is difficult, especially for commercial LLMs where only final outputs are accessible, unlike methods requiring intermediate model states.
- Existing output-based detection methods often fail to account for the dynamic nature of text reproducibility, limiting their effectiveness.
Purpose of the Study:
- To develop a novel method for detecting pre-training data in large language models (LLMs) using only their final outputs.
- To address the limitations of static similarity comparisons by incorporating dynamic reproducibility factors.
- To enhance the accuracy and reliability of pre-training data detection in black-box LLM scenarios.
Main Methods:
- Proposed RADS-PDD (reproducibility-aware dynamic similarity-based pre-training data detection) for black-box LLM data detection.
- Incorporated three reproducibility-aware mechanisms: positional gain weight, continual gain weight, and triplet occurrence probability.
- Modeled dynamic similarity to quantify varying reproducibility and amplify the similarity gap between seen and unseen texts.
Main Results:
- RADS-PDD consistently outperformed existing detection methods that rely solely on final LLM outputs.
- The proposed method achieved detection performance comparable to methods requiring access to intermediate LLM outputs.
- Experiments validated RADS-PDD's effectiveness across diverse datasets, data domains, and LLMs.
Conclusions:
- RADS-PDD offers a robust solution for detecting pre-training data in LLMs, even with black-box access.
- Dynamic similarity modeling is crucial for improving the accuracy of output-based detection methods.
- The approach effectively addresses ethical and legal concerns surrounding LLM data provenance.