The Developer's Guide to LLM Quantization: AWQ, GPTQ, and Memory Trade-offs

Quantization reduces the mathematical precision of a neural network's weights and activations. Most LLMs train in 16-bit floating-point (FP16 or BF16). Compressing to 8-bit or 4-bit integers cuts memory consumption and increases inference speed. This is what makes running a 70B model on consumer hardware possible at all.

The numbers are straightforward. A model with N billion parameters needs roughly 2 × N GB of VRAM in FP16. A 70B model needs ~140 GB. The RTX 4090 has 24 GB. Without quantization, local inference on anything above 13B is off the table for most developers.

There is also a hardware energy argument. In 2014, Mark Horowitz's Stanford paper showed that INT8 addition consumes 30x less energy than FP32 addition, and INT8 multiplication consumes 18x less energy than FP32 multiplication. Lower-precision hardware is also faster and consumes less silicon area.

Two Paradigms: PTQ and QAT

All quantization methods fall into two categories.

Post-Training Quantization (PTQ)

PTQ compresses a model after training is complete. The weights are mapped from high precision to low precision using scaling factors derived from a small calibration dataset (typically a few hundred samples).

Key characteristics:

• Compute cost: Low. A large model can be quantized in minutes to hours on a single GPU.
• Data requirement: A small calibration set. No access to original training data needed.
• Accuracy at 8-bit: Near-lossless.
• Accuracy at 4-bit: Mild degradation, especially on models under 7B parameters.
• Use case: Rapid deployment of open-weight community models.

PTQ is the dominant method for open-source LLMs. If you have downloaded a quantized model from HuggingFace, it almost certainly used some form of PTQ.

Quantization-Aware Training (QAT)

QAT embeds quantization directly into the training loop. The model simulates quantization noise during the forward pass using "fake quantization" nodes. During the backward pass, gradients are approximated using the Straight-Through Estimator (STE) and applied to high-precision shadow weights. The model learns to compensate for the precision loss.

Key characteristics:

• Compute cost: High. Requires a full training/fine-tuning loop.
• Data requirement: Large, representative dataset.
• Accuracy at 4-bit: Near-lossless.
• Accuracy at 2-bit: Still usable (unlike PTQ).
• Use case: Mission-critical edge deployment, ultra-low bit-width targets.

[!NOTE] QAT is the method behind Google's Gemma 2 2B quantized variants and Meta's recent work on sub-4-bit Llama models. If you need extreme compression without losing coherence, QAT is the path. For everything else, PTQ is faster and cheaper.

Feature	PTQ	QAT
Compute Cost	Low (minutes to hours)	High (full training loop)
Data Requirement	Small calibration set	Large representative dataset
Accuracy (8-bit)	Near-lossless	Lossless
Accuracy (4-bit)	Mild degradation	Near-lossless
Accuracy (2-bit)	Severe degradation	Usable
Best For	Community model deployment	Edge computing, ultra-low bit

Algorithms: GPTQ vs AWQ

Simple round-to-nearest (RTN) quantization destroys model performance at 4-bit. The core problem is that naive rounding accumulates errors across millions of weights. Two algorithms address this differently.

GPTQ (Gradient Post-Training Quantization)

GPTQ is a one-shot weight quantization method that uses approximate second-order information. It processes weights layer by layer.

The core idea builds on Optimal Brain Quantization (OBQ). For a weight matrix W and input matrix X, the goal is to find a quantized weight matrix Wq that minimizes the squared error of the layer output:

$$\min_{W_q} \| WX - W_q X \|^2$$

The mechanism:

• Compute the Inverse Hessian of the layer's input activations (H = 2X·Xᵀ). This captures how sensitive the layer output is to each individual weight.
• Quantize weights one at a time (or in blocks of 128). After each weight is rounded, distribute the resulting error to the remaining unquantized weights in that row using the Hessian information.
• This error compensation is what separates GPTQ from naive RTN. The total output error of the layer stays minimal even though individual weights have been aggressively rounded.

GPTQ is optimized for GPU inference via AutoGPTQ and ExLlamaV2. Calibration is fast. A 175B model can be processed in a few hours on a single A100.

AWQ (Activation-Aware Weight Quantization)

AWQ starts from a different premise. Not all weights are equally important. A small fraction (0.1% to 1%) have an outsized impact on model accuracy.

The mechanism:

• Run a calibration dataset through the model and observe activation magnitudes per channel.
• Identify the weight channels that produce the largest activations. These are the "salient" channels.
• Scale up the salient weights (and inversely scale the corresponding activations to maintain mathematical equivalence) before applying the quantization grid.
• The salient weights now occupy a larger portion of the quantization range, giving them finer granularity and less rounding error.

AWQ tends to preserve accuracy better than GPTQ on smaller models (7B, 13B) at 4-bit. It does not require Hessian computation. It is the default quantization method in vLLM and TensorRT-LLM.

Feature	GPTQ	AWQ	RTN (Baseline)
Error Mitigation	Inverse Hessian compensation	Activation-aware weight scaling	None
Calibration Speed	Fast	Very Fast	Instantaneous
Accuracy (4-bit, 7B)	Good	Better	Poor
Accuracy (4-bit, 70B)	Very Good	Very Good	Acceptable
Primary Frameworks	AutoGPTQ, ExLlamaV2	AutoAWQ, vLLM	bitsandbytes

[!TIP] For small LLMs (under 13B), prefer AWQ over GPTQ. The activation-aware scaling makes a bigger difference when the model has fewer redundant weights to absorb quantization noise.

The GGUF Format

Algorithms define how weights are compressed. Formats define how those compressed weights are structured on disk and loaded into memory. The dominant format for local inference is GGUF.

From GGML to GGUF

GGML was the original format built for llama.cpp. It was groundbreaking. It let users run LLMs on CPUs without datacenter GPUs. But GGML had a fundamental flaw: it stored a hardcoded list of hyperparameters. Every new model architecture (different attention mechanism, new activation function, different vocabulary size) required a format update.

GGUF replaced GGML with an extensible key-value metadata system. All model details (vocabulary, RoPE scaling factors, architecture type, tensor shapes) are embedded directly in the file header. A single GGUF parser can load nearly any model architecture without code changes.

The k-quants System

GGUF introduced mixed-precision quantization within a single file. Instead of uniformly quantizing every layer to 4-bit, the k-quants system assigns different precisions to different layers based on sensitivity.

Common k-quant variants:

• Q4_K_M (medium): Most layers at 4-bit, attention and output layers at 6-bit. Best balance of size and quality.
• Q4_K_S (small): More aggressive 4-bit. Slightly worse quality, smaller file.
• Q5_K_M: Most layers at 5-bit. Better quality than Q4, about 25% larger.
• Q3_K_L: 3-bit with some 4-bit layers. Emergency compression for very tight VRAM budgets.
• Q8_0: Uniform 8-bit. Near-lossless quality, largest file size.

Here is how the file sizes compare for a 70B model:

Variant	Avg Bits/Weight	File Size	Perplexity Delta (vs FP16)
Q8_0	8.0	74 GB	+0.01 (negligible)
Q5_K_M	5.3	48 GB	+0.05
Q4_K_M	4.6	42 GB	+0.10
Q4_K_S	4.3	39 GB	+0.15
Q3_K_L	3.4	32 GB	+0.50

[!IMPORTANT] GGUF's hybrid execution model is its killer feature. You can offload N layers to your GPU (using --n-gpu-layers) and keep the rest on CPU RAM. This means a 70B Q4_K_M model can run on a machine with a 12 GB GPU and 32 GB of system RAM.

Memory and Performance Trade-offs

Quantization is a trade-off between three axes: memory footprint, inference speed, and accuracy (measured as perplexity).

The Deployment Matrix

Here is how different bit-widths impact a 70B parameter model:

Precision	VRAM	Accuracy Retention	Speed vs FP16	Practical Use Case
FP16	140 GB	100% (baseline)	1x	Cloud clusters, enterprise API
INT8	75 GB	99.9%	1.2x	Multi-GPU local workstations
INT4 (GPTQ/AWQ)	38 GB	96%	2-3x	Dual RTX 4090, A6000 setups
Q4_K_M (GGUF)	38 GB	97%	2x	CPU/GPU hybrid via llama.cpp
INT3	28 GB	85%	3-4x	Single high-end GPU (tight fit)
INT2	19 GB	50%	Fastest	Research only. Not practical.

Memory bandwidth is the primary bottleneck during autoregressive token generation. Each token requires reading the full weight matrix from memory. Lower precision means fewer bytes per weight, which means more tokens per second on the same hardware.

Why Quantization Speeds Up Decoding

LLM token generation is memory-bandwidth bound, not compute-bound. The GPU spends most of its time waiting for weights to arrive from HBM, not doing arithmetic. A 4-bit model moves 4x fewer bytes per weight read than FP16. On an RTX 4090 with 1 TB/s memory bandwidth, this translates directly to higher tokens/second.

The prefill phase (processing the input prompt) is compute-bound. Quantization helps less here because the bottleneck is matrix multiplication throughput, not memory reads.

Inference Engine Compatibility

The right format depends on your hardware target.

Quick decision guide:

• Serving at scale (H100/A100) >> AWQ or FP8 via vLLM or SGLang.
• Local GPU inference (RTX 4090) >> EXL2 format via ExLlamaV2 for maximum tokens/sec.
• Apple Silicon (M-series) >> GGUF via llama.cpp. Unified memory lets you load large models without a discrete GPU.
• CPU-only or hybrid >> GGUF with partial GPU offload (--n-gpu-layers).

[!WARNING] Do not mix up formats and algorithms. AWQ is an algorithm. GGUF is a format. You can have an AWQ-quantized model saved in GGUF format. The algorithm determines how the weights were compressed. The format determines how the compressed weights are stored and loaded.

Picking the Right Method

The best quantization strategy depends on your deployment target:

• Fine-tuning: For QLoRA fine-tuning, bitsandbytes NF4 quantization is the standard. It is tightly integrated with HuggingFace PEFT and loads the base model in 4-bit while training LoRA adapters in FP16.
• Production serving: AWQ quantized models loaded via vLLM or TensorRT-LLM provide the best throughput-per-dollar on datacenter GPUs. FP8 is gaining traction on Hopper (H100) and Blackwell architectures.
• Local GPU: For consumer GPUs with 24 GB VRAM, EXL2 (via ExLlamaV2) gives the highest tokens/sec. AWQ is a close second with broader framework support.
• Local CPU/Hybrid: GGUF Q4_K_M via llama.cpp is the most flexible option. It supports CPU offloading, Apple Silicon unified memory, and partial GPU acceleration via --n-gpu-layers.
• Browser: For browser-based inference, WebGPU-compatible quantized models use INT4 via WebLLM or Transformers.js.

As a general rule: start with Q4_K_M (GGUF) for local testing, AWQ for GPU serving, and QAT only when you control the training pipeline and need sub-4-bit precision.

References

• Integer Quantization Deep Dive
• Hugging Face Quantization Documentation
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• llama.cpp and GGUF Specification
• Computing's Energy Problem (Horowitz, 2014)
• Self-Hosting LLMs with vLLM, SGLang, and llama.cpp
• QLoRA Fine-Tuning Guide