NVIDIA GPU Workloads on Kubernetes — Part 5: What’s Actually Running on a GPU Node

Part 5 of the Falcon AI workbook series. gpu-wk-01 and gpu-wk-02 are up with nvidia.com/gpu: 8 Allocatable. Before we validate the cluster (Part 6) or run a workload (Part 7), let’s understand exactly what’s running on these nodes and why each piece exists — this is the knowledge that turns “it’s broken” into “I know which of these ten components to check first.”</em>

Prerequisite

You’ve completed Part 4: gpu-wk-01 and gpu-wk-02 are Ready, joined to the cluster, with GPU Operator and Network Operator DaemonSets deployed to both. kubectl describe node gpu-wk-01 shows nvidia.com/gpu: 8 under Allocatable.

Recap: where Falcon AI stands

Role	Count	Hostnames	Status after Part 4
Management (control plane)	3	mgmt-01, mgmt-02, mgmt-03	Ready, tainted NoSchedule
GPU worker	2	gpu-wk-01, gpu-wk-02	Ready, nvidia.com/gpu: 8 Allocatable each
Load balancer	1	k8s-lb	Fronting API server on :6443

The full node stack, layered

Everything installed on a GPU node falls into four layers, each depending on the one below it. This layering is the single most useful mental model for troubleshooting — a symptom in layer 3 is almost always caused by something in layer 1 or 2.

Why this order matters: if the driver (Layer 1) isn’t loaded, the container toolkit (Layer 2) has nothing to expose, the device plugin (Layer 3) has nothing to advertise, and DCGM (Layer 4) has nothing to measure. A CrashLoopBackOff on DCGM exporter is very often actually a driver problem one layer down — check bottom-up, not top-down.

Layer 1 — Hardware Enablement

Component	What it does	Falcon AI relevance
NVIDIA Driver	Kernel module that lets the OS talk to the physical GPU	Deployed as a DaemonSet pod per Part 2’s driver.enabled=true — no manual install
OFED / RDMA Driver	Kernel-level driver for InfiniBand/RoCE NICs	Required for the 400Gb RoCE NICs on gpu-wk-01/02 — without it, GPUs work but multi-node NCCL training falls back to slow TCP
SR-IOV Device Plugin	Carves a physical NIC into virtual functions (VFs)	Only relevant if Falcon AI later needs per-pod dedicated NIC bandwidth rather than shared RDMA

Inspect on mgmt-01:

kubectl get pods -n gpu-operator -l app=nvidia-driver-daemonset -o wide
kubectl exec -it -n gpu-operator <driver-pod-name> -- nvidia-smi

Expected: nvidia-smi output showing all 8 H100s, driver version, and CUDA version — run from inside the driver pod, confirming the kernel module actually loaded.

Layer 2 — Container Integration

Component	What it does
NVIDIA Container Toolkit	Injects GPU device nodes and driver libraries into containers at runtime — this is what makes docker run --gpus all (or the K8s equivalent) work
RDMA Device Plugin	Exposes RDMA devices to pods the same way the GPU device plugin exposes GPUs

This is the layer that modified containerd on your GPU nodes — recall from Part 1 we noted the runtime would get “an NVIDIA runtime hook injected” later. This is that moment:

# On gpu-wk-01
cat /etc/containerd/config.toml | grep -A3 nvidia

Expected: an nvidia runtime section added by the toolkit, alongside the default runc runtime you configured manually in Part 1.

Layer 3 — Kubernetes Resource Exposure

This is the layer that turns “GPU exists on this box” into “Kubernetes scheduler knows about it.”

Component	What it does
Device Plugin	Advertises GPUs to kubelet as Extended Resources (nvidia.com/gpu) — this is literally where the Allocatable: nvidia.com/gpu: 8 from Part 4 comes from
GPU Feature Discovery (GFD)	Adds detailed labels — GPU model, memory size, compute capability, MIG capability — used for node affinity later
MIG Manager	If MIG mode is enabled, partitions physical GPUs into isolated slices and updates advertised resources accordingly (we cover MIG vs time-slicing in depth in a later post)
CSI Node Plugin	Mounts persistent volumes (checkpoints, datasets, model weights) onto the node

kubectl get node gpu-wk-01 --show-labels | tr ',' '\n' | grep nvidia

Expected labels including nvidia.com/gpu.product=NVIDIA-H100-..., nvidia.com/gpu.memory=..., nvidia.com/mig.capable=true.

Layer 4 — Observability & Ops

Component	What it does
DCGM Exporter	Exposes GPU-level telemetry (utilization, temperature, ECC errors, power) as Prometheus metrics
Node Exporter	Standard CPU/memory/disk/network metrics — not GPU-specific, but essential context
Logging Agent (Fluent Bit/Promtail)	Ships container and system logs off-node
OpenTelemetry Agent	Collects traces/metrics/logs in a vendor-neutral format, if Falcon AI adopts distributed tracing for its inference services later

bash

kubectl get pods -n gpu-operator -l app=nvidia-dcgm-exporter -o wide
kubectl exec -it -n gpu-operator <dcgm-pod-name> -- dcgm-exporter --version

We’ll wire these into an actual Grafana dashboard in the Observability post later in the series — for now, just confirm the exporter pod is Running.

Full picture: one node, ten components

Troubleshooting checkpoint

Symptom	Check this layer first
nvidia-smi fails inside driver pod	Layer 1 — check Secure Boot, kernel headers, dmesg for module load errors
Pods can request GPU but toolkit errors at container start	Layer 2 — check containerd config was actually patched
Allocatable: nvidia.com/gpu missing or wrong count	Layer 3 — device plugin pod status and logs
DCGM metrics missing or zero	Layer 4, but verify Layer 1 first — DCGM reads from the driver

Where we are

✅ Understand the four-layer stack on every GPU node and why the order matters for troubleshooting ✅ Verified driver, toolkit, device plugin, GFD, and DCGM exporter are all functioning on gpu-wk-01 ✅ Know exactly which component to check first for each class of symptom

---

Next in the series: Part 6 — Validating Cluster Readiness, where we run the full six-category checklist (infrastructure, GPU health, network, storage, scheduling, observability) end to end, using the commands from this post plus a few new ones.