NVIDIA GPU Workloads on Kubernetes — Part 11: Key Concepts Deep Dive

Prerequisite

You have completed Part 10, where MIG and time-slicing were configured on gpu-wk-01 and validated using the Grafana dashboards from Part 9. This post assumes the full stack from Parts 1–10 is in place.

Recap: where Falcon AI stands

Where each concept already showed up

Before diving in, worth being explicit about where you already used each of these — this post explains the “why” behind decisions made earlier without you necessarily seeing the full picture at the time.

Concept 1 — RuntimeClass

First, RuntimeClass tells Kubernetes which container runtime configuration a pod should use. In this case, that configuration is important because NVIDIA Container Toolkit registers the nvidia runtime alongside the default runc runtime. Without this configuration, a GPU pod can schedule successfully but still fail when an application attempts to access CUDA. Therefore, checking runtimeClassName should be part of your GPU workload validation process.

Without runtimeClassName: nvidia in a pod spec, the container starts under plain runc — it can still request nvidia.com/gpu as a resource and get scheduled onto a GPU node, but won’t actually have the driver libraries injected, and CUDA calls will fail inside the container. This is a common silent misconfiguration: the pod schedules fine, looks healthy, and only fails once your code touches the GPU.

bash

kubectl get runtimeclass
kubectl get runtimeclass nvidia -o yaml

Validation habit worth adopting: grep any new workload manifest for runtimeClassName before applying it — it’s easy to copy a pod spec that requests nvidia.com/gpu in resources but forgets the runtime class.

Concept 2 — Taints and Tolerations

tolerations:
- key: "nvidia.com/gpu"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"

Without this second taint, a regular CPU-only workload with no GPU request could still land on gpu-wk-01 and consume CPU/memory that GPU-bound pods need — a subtler form of the noisy-neighbor problem from Part 10, at the node level instead of the GPU level.

Concept 3 — Node Affinity and Topology

Similarly, node affinity controls where Kubernetes places workloads. For example, you can select nodes with a specific NVIDIA GPU model by using the nvidia.com/gpu.product label. In addition, topology-aware scheduling becomes important for distributed training. Therefore, multi-node AI workloads should consider both GPU characteristics and network topology when selecting nodes.

yaml

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: nvidia.com/gpu.product
          operator: In
          values:
          - NVIDIA-H100-80GB-HBM3

For multi-node training specifically, topology also matters at the network level — you want training pods landing on nodes connected via the same RoCE fabric segment (validated back in Part 6, Category C), not split across segments with worse cross-node bandwidth. This is typically expressed via pod affinity/anti-affinity combined with topology-aware labels from the Network Operator, rather than plain node affinity alone.

Concept 4 — Gang Scheduling

Finally, gang scheduling addresses a different scheduling problem. By default, Kubernetes schedules pods independently. Consequently, some pods belonging to a distributed training job may start while others remain pending. As a result, allocated GPUs can sit idle while the job waits for additional resources. For this reason, Volcano gang scheduling can improve resource utilization for distributed AI workloads.

This is why Volcano existed as a prerequisite before Part 8’s MPIJob example would behave reliably under real multi-tenant load — without it, the NCCL test from Part 7 could technically still run (nothing else was competing for GPUs at the time), but it wouldn’t survive contact with a second team submitting jobs.

Bringing it together: one pod spec, all four concepts

yaml

apiVersion: v1
kind: Pod
metadata:
  name: falcon-training-worker
spec:
  runtimeClassName: nvidia                          # Concept 1
  tolerations:                                       # Concept 2
  - key: "nvidia.com/gpu"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"
  affinity:                                           # Concept 3
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nvidia.com/gpu.product
            operator: In
            values: ["NVIDIA-H100-80GB-HBM3"]
  schedulerName: volcano                              # Concept 4
  containers:
  - name: worker
    image: nvcr.io/nvidia/pytorch:24.01-py3
    resources:
      limits:
        nvidia.com/gpu: 8

Every field in this spec maps to something built across the previous ten posts — this pod could not run correctly on Falcon AI’s cluster without all four concepts working together underneath it.

Where we are — series complete

✅ RuntimeClass, taints/tolerations, node affinity/topology, and gang scheduling — each understood at the mechanism level, not just as YAML fields to copy ✅ Traced each concept back to where it silently appeared earlier in the series ✅ Falcon AI’s cluster: 3-node HA management plane, 2 validated GPU workers, full operator stack, AI platform layer, observability, GPU sharing modes, and now the scheduling primitives tying it all together

Series recap

1. Prerequisites & Cluster Readiness
2. Installing GPU & Network Operators
3. What Happens Inside the Cluster After Operators Deploy
4. Adding GPU Worker Nodes
5. What’s Actually Running on a GPU Node
6. Validating Cluster Readiness
7. Running a Test Job
8. The AI Platform & Application Layer
9. Observability Stack for GPU Clusters
10. MIG vs Time-Slicing
11. Key Concepts Deep Dive

From a bare Kubernetes cluster to a validated, observable, multi-tenant, production-ready AI platform — that’s the full journey the original infographic mapped out, now with a from-scratch, hands-on workbook behind every box in it.

Conclusion: Kubernetes GPU Scheduling in Practice

Need Help Building Your AI Infrastructure?