Where Training Meets Serving: RL Rollouts and Speculative Decoding

Part 6 of “vLLM in 2026: An Infrastructure Engineer’s Field Guide”

---

For years, training and inference were separate worlds with separate clusters and separate teams. That boundary is dissolving. Reinforcement learning turns your inference engine into part of the training loop. Speculative decoding needs draft models that must be trained against the target you serve.

Both topics change what an inference platform team is responsible for. This post explains why vLLM ends up inside training systems, and what it takes to train the drafters that make serving faster.

[Add from talks: “GLM-5 RL Training with vLLM” (Sami Jaghouar, Prime Intellect) and “TorchSpec: Speculative Decoding Training at Scale” (Fred Wang, Together AI / LightSeek): actual architectures, cluster sizes, throughput numbers, and the problems they hit.]

---

Part A: RL training with vLLM as the rollout engine

1. Why RL needs an inference engine

In RL-based post-training (for reasoning, coding, agentic behavior), the loop looks like this:

The rollout step is pure text generation, often long chain-of-thought or multi-turn agent trajectories, and it typically dominates wall-clock time. A training framework’s own generation path is slow, so RL stacks delegate generation to a purpose-built engine: vLLM. Your serving expertise (batching, KV cache, parallelism) becomes the bottleneck-relief for training.

2. The systems problems

Weight synchronization. After each policy update, the inference workers need the new weights. For a large model that means moving hundreds of GB from trainer to rollout workers, repeatedly. Options include broadcasting over the network with collectives (NCCL-style), going through shared storage, or sharing memory when trainer and inference are colocated. The transfer path is a fabric design problem with the same considerations as Post 3.

Placement: colocated vs disaggregated.

	Colocated	Disaggregated
Layout	Trainer and vLLM share the same GPUs, alternating	Separate GPU pools for training and rollout
Pros	No idle pool; simpler weight sharing	Each side tuned independently; can overlap work
Cons	Must swap memory state between phases	Needs fast weight transfer; risk of one pool idling

For colocation, vLLM offers a sleep mode that releases GPU memory (weights and KV cache) so the trainer can use it, then restores it afterward. Check your version’s docs for the exact flags.

Long-tail rollouts. Response lengths vary widely: a few very long generations hold up a synchronous batch while most GPUs idle. Asynchronous designs let training proceed on completed rollouts while stragglers finish, at the cost of some off-policy staleness.

Numerical mismatch. The trainer and vLLM compute log-probabilities with different kernels and possibly different precision. Small differences between the sampling policy and the training policy can destabilize RL. Teams handle it with importance-sampling corrections, matching kernels/precision where possible, or explicitly monitoring the mismatch.

Scale and failure. A run spans many nodes for days; a failure mid-run must not lose the run. Checkpointing, worker health checks, and elastic rollout capacity all matter.

[Add from talk: how Prime Intellect structured GLM-5 RL: sync vs async, decentralized/heterogeneous workers if applicable, weight-sync path, measured utilization.]

3. What the inference team should know

• RL workloads are generation-heavy with long outputs, the opposite of agent serving’s prefill-heavy pattern (Post 4). Tune accordingly (decode throughput, large batch, KV headroom).
• Prefix caching helps when many rollouts share a prompt (grouped sampling).
• Version pinning matters: reproducibility of a training run depends on the vLLM version and kernels.

---

Part B: Training the speculative decoder

4. Speculative decoding in one page

Decode generates one token per forward pass of the big model, and it is bandwidth-bound, so the GPU is underused. Speculative decoding exploits that spare compute:

1. A small, fast draft proposes several tokens ahead.
2. The target model verifies all proposed tokens in one forward pass.
3. Accepted tokens are kept; at the first mismatch, the target’s own token is used.

The output distribution matches the target model (with correct verification), so quality is unchanged; only speed changes.

The key metric is acceptance length: the average number of tokens gained per target pass. Higher is better. The end-to-end speedup depends on acceptance length, draft cost, and batch size (speculation helps most at low-to-moderate batch sizes where the GPU has spare compute).

Approaches include separate small draft models and lightweight heads attached to the target (the EAGLE family), which reuse the target’s hidden states and generally achieve better acceptance. In vLLM you configure it with a speculative config:

# Illustrative: EAGLE-style draft
vllm serve <target-model> \
  --speculative-config '{"method":"eagle3","model":"<draft-model>","num_speculative_tokens":3}'

Method names and options change between versions; check the docs.

5. Why the drafter must be trained

A generic small model is a poor drafter for a specific target: its distribution differs, so acceptance is low. A drafter trained on the target’s own behavior (its hidden states and its outputs) tracks the target closely, so acceptance length rises and speedup follows. And every time you change the target (a fine-tune, a new quantization, an RL update), the drafter can drift out of alignment and needs retraining.

That last point ties back to this post’s first half: if you are RL-updating the target regularly, your speculative decoder is also on a training treadmill.

6. What “training at scale” involves

The drafter training pipeline is a data and systems problem:

• Data generation is expensive. Running the big target over lots of data and capturing hidden states produces enormous datasets; storage and I/O bandwidth become the constraint.
• Data distribution matters. Train on data resembling production traffic (including tool calls and code if you serve agents), or acceptance will disappoint in the real world.
• Evaluate in the real engine. Acceptance length measured in the actual serving stack, at realistic batch sizes, is the number that counts.
• A training framework native to your ecosystem reduces glue code; that is the space that tooling like TorchSpec addresses.

[Add from talk: Fred Wang’s TorchSpec design: what the framework automates, scaling results, acceptance-length gains on which models, integration with vLLM.]

---

The bigger picture

Serving, RL, and speculation form a loop: RL improves the model, which changes the target; the drafter is retrained; quantization is re-validated (Post 5); the fleet is redeployed (Post 3). A platform team that treats these as separate projects will be surprised by the dependencies. One that treats them as a pipeline can automate the loop.

Design-review checklist

• For RL: what is the weight-sync path, and is its bandwidth sized like a fabric design (not an afterthought)?
• Colocated or disaggregated placement chosen deliberately, with a plan for the idle-time tradeoff?
• Trainer/rollout numerical mismatch monitored?
• Rollout throughput tuned for long generations (decode-heavy), not chat defaults?
• For speculation: acceptance length measured on production-like traffic?
• Is there a retrain trigger for the drafter whenever the target changes?
• Is the whole loop (train, draft, quantize, evaluate, deploy) automated end to end?