Compression Without Regret: Quantization, QAT, and Shipping Smaller Models

Part 5 of “vLLM in 2026: An Infrastructure Engineer’s Field Guide”

---

Quantization is the closest thing inference has to a free lunch, and it isn’t free. Cutting precision shrinks weights, shrinks the KV cache, and speeds up memory-bound decode. It also risks quality regressions that don’t show up until a user finds them.

This post covers the precision ladder, the difference between post-training quantization (PTQ) and quantization-aware training (QAT), what each buys you in a vLLM deployment, and how to evaluate before shipping.

[Add from talk: “Highly Compressed Gemma Models with Custom Quantization-Aware Training” (Sara Smoot, Google DeepMind): the specific bit-widths, QAT recipe, quality-versus-size results, and vLLM integration details.]

---

1. Why quantization pays off

LLM decode is memory-bandwidth-bound: each step reads all active weights and the KV cache from HBM. Fewer bytes per parameter means:

• Smaller footprint: bigger models per GPU, or more KV cache headroom.
• Faster decode: less data to move per token.
• Cheaper hardware requirements: fewer GPUs per replica, less communication if TP degree drops.

Approximate weight memory for a 70B-parameter model:

Precision	Bytes per weight	Weights only
BF16 / FP16	2	~140 GB
FP8 / INT8	1	~70 GB
INT4 / FP4	0.5	~35 GB

(Add a few percent for scales and metadata. The KV cache and activations are extra.)

2. What you can quantize

Three separate targets, often confused:

Target	What changes	Main benefit	Main risk
Weights only (e.g., W4A16)	Weights stored in low precision, activations stay BF16	Memory and decode bandwidth	Small quality loss; kernel needs to dequantize on the fly
Weights + activations (e.g., W8A8, FP8)	Both use low precision; matmuls run natively in low precision	Compute speedup too, especially in prefill	Activation outliers make it harder to keep quality
KV cache	Cached keys/values stored in FP8 or similar	Roughly halves KV memory, so more concurrency and longer contexts	Attention accuracy at long contexts

Enabling KV cache quantization in vLLM is a single flag:

vllm serve <model> --kv-cache-dtype fp8

Halving KV bytes per token doubles the tokens that fit in the same memory. For long-context and agent workloads (Post 4), this can matter more than weight quantization.

3. The precision ladder

• FP8 is the common first step on hardware with native support. Quality loss is often small with proper scaling.
• INT4 weight-only (methods such as GPTQ and AWQ) is popular for memory-constrained serving.
• FP4-class formats are supported by newer hardware generations and offer more compression, but recipes matter more.

Hardware support drives everything (see Post 2): a format the GPU accelerates natively is fast; one it must emulate can be slower than BF16.

4. PTQ versus QAT

Post-training quantization (PTQ) takes a finished model, uses a small calibration dataset to pick scales, and converts weights. It is cheap and fast (minutes to hours), needs no training infrastructure, and works well down to about 8 bits and often 4 bits for weights.

Quantization-aware training (QAT) simulates quantization during (continued) training, so the model learns to tolerate the rounding error. It costs real training compute, requires access to training data and pipeline, and typically recovers much more quality at aggressive bit-widths.

Why QAT matters for model publishers: when a lab releases official QAT checkpoints, users get low-bit models that were trained to be low-bit, rather than squeezed afterward. That tends to keep quality closer to the full-precision model at a fraction of the memory.

For platform teams, the choice is usually: use vendor-provided QAT checkpoints when they exist; otherwise do PTQ with an open tool such as llm-compressor and evaluate carefully.

[Add from talk: the custom QAT approach: what is custom about it, which layers/tensors stay in higher precision, and how vLLM loads the result.]

5. A practical evaluation protocol

“It still sounds right” is not a test. Before shipping any quantized model:

1. Freeze a golden eval set covering your real tasks: not only public benchmarks, but also your prompts, tool calls, and long-context cases.
2. Compare against the BF16 baseline on the same set, same sampling settings.
3. Check the failure modes quantization tends to hit: long-context retrieval, multi-step reasoning, numerical/code tasks, structured output validity, and multilingual quality.
4. Measure the systems win: tokens/sec at fixed p99, max concurrency, memory headroom. If the win is small, the quality risk isn’t worth it.
5. Canary in production with shadow traffic or a small percentage, and compare downstream metrics (tool-call error rate, task success).
6. Re-evaluate per hardware backend: numerics differ by kernel implementation.

6. Interaction with the rest of the stack

• Parallelism. Smaller weights can reduce TP degree, which cuts all-reduce traffic (Post 3).
• Speculative decoding. The draft model and target must agree closely; quantizing the target can change acceptance rates (Post 6).
• Disaggregated serving. A quantized KV cache reduces the bytes to ship between prefill and decode pools.
• Cold start. Smaller checkpoints load faster (Post 1).

Design-review checklist

• Do we know which quantization target we’re applying (weights, activations, KV)?
• Is the format natively accelerated on our hardware?
• Do we have vendor QAT checkpoints available, or are we doing PTQ?
• Is there a frozen eval set with baseline scores, including long-context and structured-output tests?
• Did we measure the throughput and memory gain, not just assume it?
• Is there a rollback path and a canary?