Part 2 of the Falcon AI workbook series. Follow along with real commands — by the end of this post the cluster from Part 1 will be running the operators that will recognize and configure GPU hardware, before any GPU node even joins. Prerequisite You already have the cluster built in Part 1: 3-node HA
Part 1 of a hands-on workbook series. Follow along with real commands — by the end of this post you’ll have a working, validated Kubernetes control plane ready to receive GPU nodes. The scenario Throughout this series we’ll build out infrastructure for Falcon AI, a (fictional) company standing up an on-prem GPU cluster to fine-tune
In Part 1 of this series we built the vocabulary: the OSI layers as a debugging framework, physical-to-virtual appliance mapping, overlay networking as the answer to “how do pods get routable IPs without touching the physical fabric,” and the Linux primitives — network namespaces, veth pairs, and bridges — that make container networking possible at
Kubernetes is, at its core, a networking problem wearing an orchestration costume. Scheduling pods, managing deployments, scaling replicas — none of that matters if the pieces can’t talk to each other reliably. Before we get into cluster networks, pod networks, CNI plugins, Services, and Network Policies later in this series, we need a shared vocabulary.
Kubernetes was designed around a simple assumption: workloads are interchangeable, resources are fungible, and any pod can move to any node without much consequence. AI/ML training and inference break every part of that assumption. GPUs aren’t interchangeable — a workload that needs eight GPUs on the same NVLink domain fails just as hard on the
The muscle-memory layer: commands, config files, and scripts — for the networking person moving into HPC/AI Ops (Part two of a series — part one: From Jupyter Notebook to InfiniBand Fabric, covering what the ML/LLM team actually hands you before this stage begins.) Knowing what Slurm is and knowing what a Slurm admin does all
A network engineer’s field guide to the other side of the AI infrastructure stack If you’ve spent your career on fabric design and CCIE-grade network builds, the ML/LLM training world looks like a black box wrapped in a black box. A “job” shows up in the Slurm queue, it asks for 8 GPUs across 2
Every production LLM that answers a customer question or automates a workflow passed through the hands of five or six distinct roles before it ever served a single request. Each role hands off a well-defined artifact to the next — and when that handoff is fuzzy, that’s usually where projects stall for months. This post
Every large-scale AI system — the model that answers your question, the recommendation engine on your favorite app, the vision model in a self-driving car — was trained and is served on infrastructure built from three foundational pieces: GPUs that do the actual computation, a scheduler that decides who gets to use them and when,
Enterprises are moving large language models from pilot projects into production faster than most security programs can adapt. The result is a new attack surface that doesn’t map cleanly onto traditional application security, network security, or data security playbooks — it borrows from all three and adds failure modes none of them anticipated. This post
