Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Large language models (LLMs) require discrete data representations for efficient processing.
• Vector quantization (VQ) is a key technique for transforming continuous multimodal data into discrete tokens.
• Existing literature lacks a systematic survey of VQ methods tailored for LLM integration.

Purpose of the Study:

• To present the first structured taxonomy and analysis of discrete tokenization methods, specifically VQ, for LLM applications.
• To systematically categorize and analyze representative VQ variants within the context of LLM pipelines.
• To bridge the gap between VQ techniques and modern LLM development for multimodal systems.

Main Methods:

• Categorization of 8 representative VQ variants, spanning classical and modern approaches.
• Analysis of algorithmic principles, training dynamics, and integration challenges of VQ methods with LLMs.
• Review of existing research across classical, single-modality, and multimodal LLM systems.

Main Results:

• Identification of how quantization strategies influence alignment, reasoning, and generation in multimodal LLMs.
• Highlighting key challenges such as codebook collapse and unstable gradient estimation.
• Discussion of emerging research directions including dynamic quantization and unified tokenization frameworks.

Conclusions:

• This survey provides a foundational reference for developing efficient and generalizable multimodal LLM systems.
• The structured analysis of VQ techniques addresses a critical need in the rapidly advancing field of LLMs.
• Understanding VQ is crucial for optimizing the performance of LLM-based multimodal applications.