Part 4 of the Falcon AI workbook series. This is the payoff post — gpu-wk-01 and gpu-wk-02 join the cluster for real, and you’ll watch the entire reconcile chain from Part 3 fire in front of you. Prerequisite You’ve completed Part 3: GPU Operator and Network Operator are confirmed healthy and idle-watching, CRDs and NFD
Part 3 of the Falcon AI workbook series. Follow along with real commands — this post has no new infrastructure to build. Instead we go hands-on inside the operators installed in Part 2, so you know exactly what “idle and watching” actually means before we hand it real GPU hardware in Part 4. Prerequisite You’ve
Part 2 of the Falcon AI workbook series. Follow along with real commands — by the end of this post the cluster from Part 1 will be running the operators that will recognize and configure GPU hardware, before any GPU node even joins. Prerequisite You already have the cluster built in Part 1: 3-node HA
Part 1 of a hands-on workbook series. Follow along with real commands — by the end of this post you’ll have a working, validated Kubernetes control plane ready to receive GPU nodes. The scenario Throughout this series we’ll build out infrastructure for Falcon AI, a (fictional) company standing up an on-prem GPU cluster to fine-tune
In Part 1 of this series we built the vocabulary: the OSI layers as a debugging framework, physical-to-virtual appliance mapping, overlay networking as the answer to “how do pods get routable IPs without touching the physical fabric,” and the Linux primitives — network namespaces, veth pairs, and bridges — that make container networking possible at
Kubernetes is, at its core, a networking problem wearing an orchestration costume. Scheduling pods, managing deployments, scaling replicas — none of that matters if the pieces can’t talk to each other reliably. Before we get into cluster networks, pod networks, CNI plugins, Services, and Network Policies later in this series, we need a shared vocabulary.
Kubernetes was designed around a simple assumption: workloads are interchangeable, resources are fungible, and any pod can move to any node without much consequence. AI/ML training and inference break every part of that assumption. GPUs aren’t interchangeable — a workload that needs eight GPUs on the same NVLink domain fails just as hard on the
The muscle-memory layer: commands, config files, and scripts — for the networking person moving into HPC/AI Ops (Part two of a series — part one: From Jupyter Notebook to InfiniBand Fabric, covering what the ML/LLM team actually hands you before this stage begins.) Knowing what Slurm is and knowing what a Slurm admin does all
A network engineer’s field guide to the other side of the AI infrastructure stack If you’ve spent your career on fabric design and CCIE-grade network builds, the ML/LLM training world looks like a black box wrapped in a black box. A “job” shows up in the Slurm queue, it asks for 8 GPUs across 2
Every production LLM that answers a customer question or automates a workflow passed through the hands of five or six distinct roles before it ever served a single request. Each role hands off a well-defined artifact to the next — and when that handoff is fuzzy, that’s usually where projects stall for months. This post
