NVIDIA GPU Workloads on Kubernetes — Part 2: Installing the GPU & Network Operators

Part 2 of the Falcon AI workbook series. Follow along with real commands — by the end of this post the cluster from Part 1 will be running the operators that will recognize and configure GPU hardware, before any GPU node even joins.

Prerequisite

You already have the cluster built in Part 1: 3-node HA management plane (mgmt-01, mgmt-02, mgmt-03), CNI installed, control-plane nodes tainted, kubectl get nodes shows all three Ready. No worker nodes yet — that’s expected and correct at this stage.

Recap: where Falcon AI stands

Role	Count	Hostnames	Status after Part 1
Management (control plane)	3	mgmt-01, mgmt-02, mgmt-03	Ready, tainted NoSchedule
GPU worker	2 (growing)	gpu-wk-01, gpu-wk-02	Not yet built — arrives in Part 4
Load balancer	1	k8s-lb	Fronting API server on :6443

Why install operators before any GPU node joins?

This is the piece that confuses people coming from manual driver installs: you install the GPU Operator against a cluster that has no GPUs yet. That’s intentional. The Operator pattern means you’re installing a controller whose job is to watch for GPU hardware and react — not a one-time install script. It sits idle-but-ready until gpu-wk-01 joins in Part 4.

The two operators we’re installing

Operator	What it manages	Why Falcon AI needs it
NVIDIA GPU Operator	Driver, container toolkit, device plugin, DCGM exporter, MIG manager	Core requirement — every GPU node needs this stack
NVIDIA Network Operator	RDMA/RoCE drivers (OFED), SR-IOV device plugin	Falcon AI’s H100 nodes have 400Gb RoCE NICs for multi-node NCCL training — without this, GPUs work but distributed training is slow

Recommended-but-optional operators (Storage, Prometheus, Kubeflow) come later in the series when we actually need them — installing them now would just be noise you can’t validate yet.

How the Operator pattern actually works

Every Kubernetes Operator follows the same loop, and it’s worth internalizing this once because it explains everything that happens automatically in later parts:

This is why we install operators before hardware exists — the watch loop is what makes Part 4 (“add worker nodes”) almost fully automatic.

Step 1 — Install Helm (on mgmt-01)

We run all cluster-admin commands from mgmt-01, the same node holding your ~/.kube/config from Part 1.

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm version

Expected output:

version.BuildInfo{Version:"v3.15.x", GitCommit:"...", GoVersion:"go1.22.x"}

Step 2 — Add the NVIDIA Helm repos (on mgmt-01)

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

Step 3 — Install the GPU Operator (on mgmt-01)

kubectl create namespace gpu-operator

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --set driver.enabled=true \
  --set toolkit.enabled=true \
  --set devicePlugin.enabled=true \
  --set dcgmExporter.enabled=true \
  --set migManager.enabled=true

driver.enabled=true tells the Operator to manage the NVIDIA driver itself via a containerized driver pod, rather than expecting you to have pre-installed drivers on the host. This is the recommended approach for fleet consistency — Falcon AI won’t have driver-version drift between gpu-wk-01 and gpu-wk-02.

Step 4 — Install the Network Operator (on mgmt-01)

kubectl create namespace network-operator

helm install network-operator nvidia/network-operator \
  --namespace network-operator \
  --set deployCR=true \
  --set ofedDriver.deploy=true \
  --set rdmaSharedDevicePlugin.deploy=true

Step 5 — Verify the operators are up (on mgmt-01, with zero GPU nodes)

kubectl get pods -n gpu-operator
kubectl get pods -n network-operator
kubectl get crd | grep -i nvidia

Expected output — controller pods Running, nothing else yet:

NAME                                              READY   STATUS    RESTARTS
gpu-operator-7f9c9d4b8f-x2k9p                     1/1     Running   0
gpu-operator-node-feature-discovery-master-...    1/1     Running   0

You will not see driver/toolkit/device-plugin DaemonSet pods yet — there’s nothing to schedule them on. That’s correct. Compare against the diagram above: we’re sitting in the “idle — watching” state.

Troubleshooting checkpoint

If kubectl get pods -n gpu-operator shows Pending instead of Running, check the taint from Part 1 — Helm charts sometimes need an explicit toleration to schedule the controller itself onto a management node:

kubectl describe pod -n gpu-operator <pod-name> | grep -A5 Events

Where we are

✅ Helm installed on mgmt-01, NVIDIA repos added ✅ GPU Operator running, watching for GPU hardware ✅ Network Operator running, watching for RDMA-capable NICs ✅ CRDs, RBAC, and webhooks registered — nothing deployed to nodes yet (nothing to deploy to)

---

Next in the series: Part 3 — What Happens Inside the Cluster After Operators Deploy, a closer look at the reconcile loop before we actually add hardware — then Part 4 joins gpu-wk-01 and gpu-wk-02 and watches the automation described above fire for real.