NVIDIA GPU Workloads on Kubernetes — Part 1: From Bare Metal to a Ready Cluster

Part 1 of a hands-on workbook series. Follow along with real commands — by the end of this post you’ll have a working, validated Kubernetes control plane ready to receive GPU nodes.

The scenario

Throughout this series we’ll build out infrastructure for Falcon AI, a (fictional) company standing up an on-prem GPU cluster to fine-tune and serve LLMs. Their target architecture:

Role	Count	Hardware	Hostnames
Management (control plane)	3	16 vCPU / 64GB RAM / 500GB SSD	mgmt-01, mgmt-02, mgmt-03
GPU worker	2 (growing)	2× AMD EPYC, 8× NVIDIA H100, 2TB NVMe, 400Gb RoCE NIC	gpu-wk-01, gpu-wk-02
Load balancer (for HA API server)	1	Any small VM/appliance	k8s-lb

We’ll use this exact naming throughout the series — when Part 5 talks about “the GPU node,” it means gpu-wk-01.

Why management nodes and worker nodes are architecturally separate

This trips people up coming from single-node test clusters, so it’s worth being explicit:

• Management (control plane) nodes run kube-apiserver, etcd, kube-scheduler, kube-controller-manager. They hold cluster state and decisions. They do not run your GPU workloads — you taint them to prevent that.
• Worker nodes are where pods actually execute. GPU workers additionally carry the NVIDIA driver stack, container toolkit, and device plugins (that’s the whole subject of Parts 4–5).

Running etcd on the same box as your GPU workloads is a reliability anti-pattern: a training job that pins all CPU/memory can starve etcd, and etcd falling over takes your whole cluster’s control plane with it — GPUs and all. Keep them physically or at least logically separate.

Step 1 — Base Linux install (all nodes)

We’re using Ubuntu 22.04 LTS Server — the most common base for NVIDIA GPU Operator deployments, with predictable kernel/driver support. Same base OS on management and worker nodes; the divergence happens later.

During install:

• Partition with a separate /var (control plane nodes generate heavy etcd/container logs; GPU nodes need room for container images and driver packages)
• No desktop environment, minimal package set
• Enable OpenSSH server

After first boot, on every node (mgmt-01/02/03, gpu-wk-01/02):

# Update and set hostname
sudo apt update && sudo apt upgrade -y
sudo hostnamectl set-hostname mgmt-01   # adjust per node

# Disable swap — required by kubelet
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

# Load required kernel modules
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

# Required sysctl params
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sudo sysctl --system

Step 2 — Container runtime (all nodes)

sudo apt install -y containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

# Set SystemdCgroup = true — required for kubelet compatibility
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd

On GPU worker nodes, this same containerd will later get an NVIDIA runtime hook injected by the GPU Operator (Part 5) — no manual change needed now.

Step 3 — Install kubeadm, kubelet, kubectl (all nodes)

sudo apt install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | \
  sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | \
  sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Step 4 — Bootstrap the first management node

On mgmt-01 only:

sudo kubeadm init \
  --control-plane-endpoint "k8s-lb:6443" \
  --upload-certs \
  --pod-network-cidr=10.244.0.0/16

Save the two kubeadm join commands it prints — one for control-plane nodes (mgmt-02, mgmt-03), one for workers (gpu-wk-01, gpu-wk-02).

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Step 5 — Join the remaining management nodes

On mgmt-02 and mgmt-03, run the control-plane join command from Step 4’s output, e.g.:

sudo kubeadm join k8s-lb:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane --certificate-key <cert-key>

Step 6 — Install a pod network (CNI)

# On mgmt-01
kubectl apply -f <your chosen CNI manifest — Cilium, Calico, or Flannel>

(Which CNI you pick matters a lot once RDMA/Multus enters the picture in Part 4 — we’ll revisit this choice specifically when we add GPU worker networking.)

Step 7 — Taint management nodes (keep GPU workloads off them)

kubectl taint nodes mgmt-01 mgmt-02 mgmt-03 node-role.kubernetes.io/control-plane:NoSchedule

This is the practical enforcement of the architecture diagram above — control plane nodes host cluster brains, not tenant workloads.

Step 8 — Validate cluster readiness

This closes the loop back to the checklist from the original readiness guide:

kubectl get nodes -o wide
kubectl get pods -A
kubectl -n kube-system get pods
kubectl get sc          # will be empty until Part 5's storage step — expected at this stage

You should see mgmt-01/02/03 as Ready, with Roles showing control-plane. No worker nodes yet — that’s exactly right; gpu-wk-01 and gpu-wk-02 haven’t joined. That’s Part 4.

Where we are

✅ 3-node HA management plane, bootstrapped from bare Ubuntu installs ✅ CNI installed, control-plane nodes tainted ✅ Cluster validated and ready to receive workers

---

Next in the series: Part 2 — Installing the GPU and Network Operators via Helm, where we prepare the cluster (still with zero GPU nodes attached) to recognize NVIDIA hardware the moment it joins.