Distributed Inference Is a Network Problem

Part 3 of “vLLM in 2026: An Infrastructure Engineer’s Field Guide”

---

If you’ve designed a lossless RoCEv2 fabric or debugged NCCL rings, you already know most of what matters here. Serving a frontier-scale model is no longer “one model on one GPU.” It is a distributed system whose performance is set by collectives, KV cache movement, and load balancing across ranks.

This post maps every parallelism strategy in vLLM to the traffic it creates, then covers disaggregated prefill/decode and the Kubernetes-native routing layer built around it (llm-d).

[Add from talks: “Production-Grade Distributed Inference with llm-d” (Red Hat) and “Serving MiniMax M3 at Scale” (MiniMax): architecture diagrams, cluster sizes, measured throughput/latency, failure-handling lessons.]

---

1. Why a single GPU stops being enough

Two independent reasons force you to distribute:

1. The weights don’t fit. A 400B+ parameter model at FP8 is 400+ GB before any KV cache.
2. The KV cache doesn’t fit at the concurrency you need. Even models that fit may need more memory for long-context batches.

Distributing solves capacity but introduces communication. The design question is which communication, and over which links.

2. The parallelism map

Strategy	What is split	Collective pattern	Frequency	Best link
Tensor parallel (TP)	Each weight matrix, across GPUs	All-reduce (roughly twice per layer)	Every layer, every step	NVLink / scale-up domain
Pipeline parallel (PP)	Layers, across stages	Point-to-point activations between stages	Per micro-batch per stage boundary	Can cross nodes
Data parallel (DP)	Whole model replicas (or attention replicas)	None for independent replicas; coordination for MoE	Low	Any
Expert parallel (EP)	MoE experts, across GPUs	All-to-all dispatch and combine	Every MoE layer, every step	Scale-up first, scale-out for wide EP

Rules of thumb:

• Keep TP inside the NVLink domain. All-reduce on every layer is latency-critical; pushing it over InfiniBand is expensive.
• Use PP or DP across nodes when the model outgrows a node but you don’t want cross-node all-reduce.
• MoE changes everything. Dense-model intuition breaks because the dominant collective becomes all-to-all.

3. MoE serving and the all-to-all problem

In a mixture-of-experts model such as DeepSeek-V3 (roughly 671B total parameters, about 37B active per token, 256 routed experts, top-8 routing), each token is routed to a few experts spread across GPUs. Each MoE layer therefore does:

1. Dispatch: send each token’s hidden state to the GPUs holding its selected experts.
2. Expert compute.
3. Combine: bring results back and merge.

That is two all-to-alls per MoE layer, on every decode step. Illustrative math (hidden size 7168, top-8): each token fans out roughly 8 copies of a 7168-wide vector, which is about 57 KB at FP8 on dispatch and about twice that in BF16 on combine, per token, per layer, for the portion that leaves the local node. Multiply by tokens per step and by dozens of layers, and the fabric is doing real work at every step.

Where this bites:

• Latency sensitivity. Decode steps are short (tens of milliseconds); all-to-all must complete inside that budget. Specialized dispatch/combine kernels and RDMA-friendly implementations exist for this reason.
• Load imbalance. If popular experts land on the same GPU, that rank becomes the straggler and everyone waits. Expert placement, replication of hot experts, and balancing strategies matter.
• Incast-like behavior. All-to-all is many-to-many. Congestion control, ECN/PFC tuning, and rail alignment decide whether it is smooth or bursty. If you’ve read the earlier posts on DCQCN and rail-optimized topologies, this is the same discipline applied to inference.

In vLLM, expert parallelism is enabled with a flag alongside TP/DP settings:

# Illustrative: single node, 8 GPUs, expert parallel enabled
vllm serve deepseek-ai/DeepSeek-V3 \
  --data-parallel-size 8 \
  --enable-expert-parallel

Multi-node deployments add the data-parallel addressing options (--data-parallel-address, --data-parallel-rpc-port, --data-parallel-size-local, and a start rank per node). Check the current docs for the exact layout for your version.

[Add from talk: MiniMax’s parallelism configuration for M3: TP/EP/DP split, node count, and how they handle expert imbalance.]

4. Prefill/decode disaggregation

Recall from Post 1 that prefill is compute-bound and decode is memory-bandwidth-bound. Running both on the same GPUs forces a compromise: big prefills interfere with decode latency, and decode-optimized settings underuse compute.

Disaggregation separates them into distinct pools:

Benefits:

• Tune each pool independently (batch sizes, parallelism, even different GPU types).
• Protect inter-token latency from prefill bursts.
• Scale prefill and decode separately as traffic mix shifts.

Cost: you must move the KV cache across the network. Size it:

Llama-3.1-8B:  ~128 KiB/token  →  8K-token prompt ≈ 1 GiB
70B-class:     ~320 KiB/token  →  8K-token prompt ≈ 2.5 GiB

At 400 Gb/s (about 50 GB/s) line rate, 1 GiB takes roughly 20 ms and 2.5 GiB roughly 50 ms, before protocol overhead and contention. That is fine if it overlaps well with compute and hurts if it lands on a congested fabric. Disaggregation is only a win when your transfer path is fast and predictable, so it is a network design decision, not just a software toggle.

vLLM supports KV transfer through pluggable connectors; one option uses NIXL for RDMA-based transfers:

vllm serve <model> \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'

Verify the connector names and roles against the version you deploy.

5. llm-d: Kubernetes-native serving

vLLM is an engine. Running it as a fleet needs routing, scheduling, and autoscaling. llm-d is a Kubernetes-native distributed inference framework built around vLLM. Its main ideas:

• Inference-aware gateway. Instead of round-robin, the router picks a replica using signals such as queue depth, KV cache utilization, and which replica already holds the prompt’s prefix in cache.
• Disaggregated serving as a first-class pattern, with prefill and decode deployments and KV transfer between them.
• Kubernetes primitives. Gateway API integration, standard deployments and autoscaling, so it fits existing platform tooling.

Why prefix-aware routing matters: a standard load balancer scatters requests that share a long prefix across replicas, so each replica recomputes the same prefill. Routing them to the replica that already has the blocks converts repeated prefill into cache hits. It is effectively free capacity, and it matters even more for agent workloads (Post 4).

[Add from talk: Greg Pereira and Michael Goin’s llm-d walkthrough: component names, well-lit-path deployment recipes, benchmark results.]

6. Failure and operations

Distributed inference has larger blast radii:

• One bad GPU or link can stall a whole EP group. Health checks need to be group-aware, and restart granularity is the whole group.
• Fabric hygiene is part of serving SLOs. Monitor RDMA errors, PFC pause counters, and ECN marking alongside vLLM metrics.
• Rollouts are heavier. Weight loading (Post 1) times group size means slow restarts; use surge capacity.
• Test with real failure injection, including a killed rank mid-request.

Design-review checklist

• Is TP confined to the scale-up (NVLink) domain?
• For MoE: have we sized all-to-all traffic per step and validated the fabric under it?
• Do we have an expert-load-imbalance metric?
• If disaggregating: KV transfer time budgeted and tested under contention?
• Is routing cache-aware, or are we round-robining away our prefix hits?
• Are fabric counters (PFC, ECN, RDMA errors) on the same dashboard as vLLM metrics?
• Is restart granularity and rollout capacity planned for group-level failures?