Efficient zero-shot post-training quantization of large language models

Recently, Large Language Models (LLMs) have revolutionized artificial intelligence; however, their massive parameter counts create a significant barrier to deployment on memory-constrained hardware. A critical challenge for ubiquitous AI is the memory bandwidth bottleneck, as billions of weights must be moved from memory to compute units for every generated token. While Post-Training Quantization (PTQ) offers a practical path to compression, existing state-of-the-art methods still struggle with major limitations: they either rely on computationally expensive calibration data, which risks overfitting or data unavailability, or use hardware-unfriendly mixed-precision representations to handle outliers. To address these limitations, this work develops a fast, calibration-free, and uniform-precision quantization framework. Specifically, we introduce two distinct methodologies: Hybrid Group Quantization (HGQ), which employs a Dual-Ruler strategy to statistically separate outliers from common weights, and Adaptive Grouped Vector Quantization (AGVQ), which learns local codebooks to capture sub-vector geometry. We test our methods across three diverse model families (Llama, OPT, and BLOOM), with model sizes ranging from 125M to 13B parameters. We outperform state-of-the-art calibration-based baselines in quantization runtime. HGQ reduces the quantization runtime for Llama-2-13B from over 30 minutes to under 20 seconds ( ∼ 100 ×  speedup) while maintaining competitive perplexity. Furthermore, AGVQ achieves highly competitive zero-shot accuracy on common sense reasoning tasks on the Llama-2-13B model, closely approximating the original FP16 performance while saving approximately 74% of the memory footprint. With fused reconstruction kernels, these savings further translate into measured speedup, HGQ reaching 1.28 ×  the decode throughput of the FP16 baseline on a 24 GB workstation GPU. These results demonstrate the potential of statistical, calibration-free approaches for democratizing accurate and efficient LLM deployment.