Progressive discretization for generative retrieval: A self-supervised approach to high-quality DocID generation

Area of Science:

• Information Retrieval
• Machine Learning
• Natural Language Processing

Background:

• Generative retrieval utilizes large language models as differentiable indices for document memorization and retrieval.
• Traditional methods encode documents and queries separately, limiting performance.
• Current methods for document identifiers (DocIDs) in large-scale retrieval often suffer from information distortion due to unsupervised discretization.

Purpose of the Study:

• To propose a novel framework, Self-supervised Progressive Discretization (SPD), for generating high-quality, search-oriented document identifiers (DocIDs).
• To address the information distortion issue in unsupervised DocID generation.
• To enhance the performance of generative retrieval systems.

Main Methods:

• SPD distills document information into multi-perspective continuous representations using self-supervised learning.
• A progressive discretization algorithm transforms continuous representations into approximate vectors and discrete DocIDs.
• The self-supervised model, approximate vectors, and DocIDs are integrated into a query-side training pipeline.

Main Results:

• SPD successfully generates high-quality, search-oriented DocIDs.
• The proposed framework achieves state-of-the-art performance in generative retrieval tasks.
• Experiments on popular benchmarks validate the effectiveness of SPD.

Conclusions:

• Self-supervised Progressive Discretization (SPD) offers a robust solution for creating effective document identifiers for generative retrieval.
• SPD mitigates information distortion inherent in unsupervised DocID generation.
• The framework significantly advances the capabilities of generative retrieval systems.