NVIDIA GPU Workloads on Kubernetes — Part 11: Key Concepts Deep Dive

Part 11 — the closing post of the Falcon AI workbook series. Four concepts have quietly appeared throughout every prior post without a full explanation: RuntimeClass, taints/tolerations, node affinity/topology, and gang scheduling. This post gives each one the depth it deserves, so the series is a complete reference, not just a build log.

Prerequisite

You’ve completed Part 10: MIG and time-slicing are both configured on gpu-wk-01, validated against the Grafana dashboards from Part 9. This post assumes the full stack from Parts 1–10 is in place.

Recap: where Falcon AI stands

Role	Count	Hostnames	Status after Part 10
Management (control plane)	3	mgmt-01, mgmt-02, mgmt-03	Ready, tainted NoSchedule
GPU worker	2	gpu-wk-01, gpu-wk-02	Ready, MIG + time-slicing configured and validated

Where each concept already showed up

Before diving in, worth being explicit about where you already used each of these — this post explains the “why” behind decisions made earlier without you necessarily seeing the full picture at the time.

Concept	First appeared	What it was doing
RuntimeClass	Part 7, every GPU pod spec	runtimeClassName: nvidia — told containerd which runtime to use
Taints/Tolerations	Part 1, Step 7	NoSchedule taint kept workloads off mgmt-01/02/03
Node Affinity/Topology	Part 5, GFD labels	nvidia.com/gpu.product labels exist specifically to be selected on
Gang Scheduling	Part 8, Volcano install	Prevented partial scheduling of multi-pod jobs

Concept 1 — RuntimeClass

RuntimeClass tells Kubernetes which container runtime configuration to use for a pod — normally invisible, but essential for GPU workloads because the NVIDIA Container Toolkit (Part 5, Layer 2) registered an alternate nvidia runtime in containerd alongside the default runc.

Without runtimeClassName: nvidia in a pod spec, the container starts under plain runc — it can still request nvidia.com/gpu as a resource and get scheduled onto a GPU node, but won’t actually have the driver libraries injected, and CUDA calls will fail inside the container. This is a common silent misconfiguration: the pod schedules fine, looks healthy, and only fails once your code touches the GPU.

bash

kubectl get runtimeclass
kubectl get runtimeclass nvidia -o yaml

Validation habit worth adopting: grep any new workload manifest for runtimeClassName before applying it — it’s easy to copy a pod spec that requests nvidia.com/gpu in resources but forgets the runtime class.

Concept 2 — Taints and Tolerations

You used this in Part 1 to keep GPU workloads off mgmt-01/02/03. The same mechanism runs in reverse to protect GPU nodes from non-GPU workloads accidentally landing on expensive hardware:

kubectl taint nodes gpu-wk-01 gpu-wk-02 nvidia.com/gpu=true:NoSchedule

Now any pod that wants to run on a GPU node must explicitly tolerate it:

tolerations:
- key: "nvidia.com/gpu"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"

Without this second taint, a regular CPU-only workload with no GPU request could still land on gpu-wk-01 and consume CPU/memory that GPU-bound pods need — a subtler form of the noisy-neighbor problem from Part 10, at the node level instead of the GPU level.

Concept 3 — Node Affinity and Topology

GFD (Part 5) labels nodes with GPU model, memory, and MIG capability. Node affinity is how you actually use those labels to place workloads correctly — critical once Falcon AI’s fleet stops being uniform (e.g., if newer nodes get added with different GPU generations later).

yaml

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: nvidia.com/gpu.product
          operator: In
          values:
          - NVIDIA-H100-80GB-HBM3

For multi-node training specifically, topology also matters at the network level — you want training pods landing on nodes connected via the same RoCE fabric segment (validated back in Part 6, Category C), not split across segments with worse cross-node bandwidth. This is typically expressed via pod affinity/anti-affinity combined with topology-aware labels from the Network Operator, rather than plain node affinity alone.

Concept 4 — Gang Scheduling

Introduced in Part 8 with Volcano. Worth revisiting the mechanism itself: Kubernetes’ default scheduler places pods independently — it has no concept of “these 16 pods belong together and should only start if all 16 can be placed.” For a distributed MPIJob, that’s a real problem: partial scheduling means some workers start, wait indefinitely for the rest, and hold GPUs idle in the meantime.

This is why Volcano existed as a prerequisite before Part 8’s MPIJob example would behave reliably under real multi-tenant load — without it, the NCCL test from Part 7 could technically still run (nothing else was competing for GPUs at the time), but it wouldn’t survive contact with a second team submitting jobs.

Bringing it together: one pod spec, all four concepts

yaml

apiVersion: v1
kind: Pod
metadata:
  name: falcon-training-worker
spec:
  runtimeClassName: nvidia                          # Concept 1
  tolerations:                                       # Concept 2
  - key: "nvidia.com/gpu"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"
  affinity:                                           # Concept 3
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nvidia.com/gpu.product
            operator: In
            values: ["NVIDIA-H100-80GB-HBM3"]
  schedulerName: volcano                              # Concept 4
  containers:
  - name: worker
    image: nvcr.io/nvidia/pytorch:24.01-py3
    resources:
      limits:
        nvidia.com/gpu: 8

Every field in this spec maps to something built across the previous ten posts — this pod could not run correctly on Falcon AI’s cluster without all four concepts working together underneath it.

Where we are — series complete

✅ RuntimeClass, taints/tolerations, node affinity/topology, and gang scheduling — each understood at the mechanism level, not just as YAML fields to copy ✅ Traced each concept back to where it silently appeared earlier in the series ✅ Falcon AI’s cluster: 3-node HA management plane, 2 validated GPU workers, full operator stack, AI platform layer, observability, GPU sharing modes, and now the scheduling primitives tying it all together

Series recap

1. Prerequisites & Cluster Readiness
2. Installing GPU & Network Operators
3. What Happens Inside the Cluster After Operators Deploy
4. Adding GPU Worker Nodes
5. What’s Actually Running on a GPU Node
6. Validating Cluster Readiness
7. Running a Test Job
8. The AI Platform & Application Layer
9. Observability Stack for GPU Clusters
10. MIG vs Time-Slicing
11. Key Concepts Deep Dive

From a bare Kubernetes cluster to a validated, observable, multi-tenant, production-ready AI platform — that’s the full journey the original infographic mapped out, now with a from-scratch, hands-on workbook behind every box in it.