From Jupyter Notebook to InfiniBand Fabric: What an ML Team Actually Hands the Slurm Admin

A network engineer’s field guide to the other side of the AI infrastructure stack

If you’ve spent your career on fabric design and CCIE-grade network builds, the ML/LLM training world looks like a black box wrapped in a black box. A “job” shows up in the Slurm queue, it asks for 8 GPUs across 2 nodes, and somehow at the end of it a model checkpoint pops out. What actually happened in between, and who built what?

This post walks through the full chain — the same chain of questions a networking person naturally asks the first time they sit in front of a real training cluster.

---

The Two Worlds

There are, broadly, two teams involved in getting a large model trained on a cluster:

• The ML/LLM team — researchers, ML engineers, MLOps engineers. They own the model, the data, and the training code.
• The infrastructure/HPC team — Slurm admins, network engineers, storage engineers. They own the fabric, the scheduler, the drivers, and the runtime.

Neither team can do the other’s job well, and the handoff between them is where most real-world production incidents happen — a driver mismatch, a misconfigured RDMA device, a dataset that doesn’t fit the filesystem’s I/O profile.

“OK fine, Slurm — but what jobs does it actually run?”

This is the first question anyone from networking asks, because Slurm’s job (pun intended) is easy to misread as “an AI scheduler.” It isn’t. Slurm is a generic resource manager — it allocates nodes, CPUs, GPUs, memory, and time, then runs whatever executable you point it at. It has no idea what’s inside the job.

In a real HPC/AI environment, that translates into a handful of recurring categories:

• AI/ML training & inference — distributed PyTorch/DeepSpeed/Megatron jobs using NCCL over InfiniBand; this is the Core42/DGX SuperPOD territory.
• Scientific/numerical simulation — CFD, molecular dynamics (GROMACS, LAMMPS), weather models — usually MPI-based, predating the AI boom but running on the same fabric.
• Data processing/ETL — tokenization pipelines, feature engineering, usually CPU-heavy and less latency-sensitive.
• Benchmarks/diagnostics — all_reduce_perf (NCCL health), ib_write_bw (raw IB bandwidth), HPL/Linpack — exactly what you’d run on an idle shortq partition before releasing nodes to real training.
• Interactive/dev jobs — srun --pty bash for debugging on an actual compute node.

The partition (defq, shortq, whatever you’ve named yours) is just a policy bucket — different time limits and priorities — not a job-type restriction.

Who’s Actually on “The ML Team”

“The ML team” is really four different roles, and each one hands off a different artifact down the chain:

• Data Engineer — sourcing, cleaning, deduplicating data at terabyte-to-petabyte scale.
• ML Engineer / Research Scientist — architecture, training recipe, experiment design.
• LLM Pretraining Engineer — owns the distributed training system: sharding strategy, checkpoint cadence, throughput at scale.
• MLOps / Platform Engineer — builds the Docker image, owns CI/CD, and is usually the person who actually talks to you, the Slurm admin.

The Data Dimension — What They Actually Train On

Splitting by training stage:

• Pretraining data — massive, broad-coverage text (and increasingly code/math/multilingual/multimodal): Common Crawl derivatives, The Pile, RedPajama, FineWeb, or proprietary scrapes. Scale is in trillions of tokens.
• Fine-tuning/instruction data — much smaller, higher-quality instruction-response pairs.
• Preference data (RLHF/DPO) — paired responses with human or model-generated preference labels for alignment.

Why this matters to you as infra: raw text isn’t fed into training directly — it’s pre-tokenized and packed into formats optimized for fast, random-access reads: WebDataset (tar-shard based), Megatron-style .bin/.idx memory-mapped binary shards, or Parquet/Arrow. These formats are exactly why LLM pretraining jobs are often I/O-bound rather than compute-bound if the underlying filesystem (Lustre/Ceph/GPFS) isn’t tuned for the read pattern — thousands of small random reads, hitting every node simultaneously at the start of every epoch.

The Role of PyTorch — and Everything Sitting on Top of It

PyTorch is the tensor/autograd engine — forward pass, backward pass, gradients. Almost nobody trains at scale with vanilla PyTorch alone:

• PyTorch core — tensors, autograd, and torch.distributed, which gives you the low-level primitives (all_reduce, all_gather, process groups) everything else builds on.
• DeepSpeed / Megatron-LM / FSDP — implement the actual parallelism strategies that make training 70B+ parameter models across hundreds of GPUs possible: splitting the model itself (tensor/pipeline parallelism) and sharding optimizer states (ZeRO).
• HuggingFace transformers/accelerate/datasets — the friendlier glue layer most fine-tuning teams use daily.
• NCCL — not a training framework, but the library that actually moves gradients between GPUs across the fabric during all_reduce. NCCL builds rings or trees across GPUs based on detected topology (NVLink within a node, InfiniBand between nodes), and its performance depends entirely on correct topology awareness — a wrong NCCL_IB_HCA or missing RDMA device visibility silently degrades communication to sockets, with no error thrown.
• JAX — the PyTorch alternative, more common at Google-lineage labs.

“So the ML team just gives you a Docker image and it runs under Slurm?”

Close, but not quite — and this is the nuance that trips up most people coming from a non-container HPC background.

Docker itself typically does not run directly on compute nodes. Docker requires a root-privileged daemon. On a multi-tenant cluster where hundreds of users share nodes, a persistent root daemon is a security non-starter — one user’s container could escape and touch another user’s data or the whole node.

Instead, the image gets run through one of two paths:

1. Apptainer (formerly Singularity) — converts the Docker image to a .sif file, runs as the user’s own UID, no daemon:

   srun apptainer exec --nv myimage.sif python train.py

2. Enroot + Pyxis — the NVIDIA-native path on DGX SuperPOD-style clusters. Pyxis is a Slurm SPANK plugin that lets you invoke containers directly as sbatch/srun flags:

   srun --container-image=nvcr.io/nvidia/pytorch:24.01-py3 --container-mounts=/data:/data python train.py

This is almost certainly the pattern behind any NVIDIA-reference-architecture cluster (Base Command Manager, DGX SuperPOD).

Either way, the container has to be launched with GPU and InfiniBand/RDMA device passthrough explicitly configured — this is where “the job runs but is inexplicably slow” bugs live: a container that can see the GPU but not /dev/infiniband/* will still complete a training run, just over TCP sockets instead of RDMA, at a fraction of the speed, with nothing in the logs pointing at why.

What Gets Fabricated Inside the Docker Image

Six layers, bottom to top:

1. Base image — almost always an NVIDIA NGC container (nvcr.io/nvidia/pytorch:xx.xx-py3), because it ships a pre-validated, matched set of CUDA, cuDNN, NCCL, and InfiniBand-compatible libraries (OFED userspace). This is the single most important compatibility layer — an image built against CUDA 12.3 on a host whose driver only supports up to 12.1 either fails outright or silently underperforms.
2. Framework + dependencies — pip/conda installs for the training framework, tokenizer libraries, dataloading libraries.
3. Application code — model definitions, training loop, config files (Hydra/YAML), sometimes custom CUDA kernels.
4. Networking configuration — environment variables (and occasionally patched NCCL builds) to match the cluster’s NIC naming and topology.
5. Entrypoint/launch scripts — reads Slurm’s own env vars (SLURM_PROCID, SLURM_NTASKS, SLURM_NODELIST) and translates them into what PyTorch expects (RANK, WORLD_SIZE, MASTER_ADDR, MASTER_PORT). This is the literal seam where the ML world and the scheduler world are stitched together.
6. Checkpoint/signal-handling hooks — catches SIGTERM (Slurm sends this before killing a preempted or timed-out job) and triggers an emergency checkpoint save.

What the Admin Actually Checks Before Releasing the Job

This is the part that doesn’t show up in ML team documentation, because it’s not their layer — it’s yours:

• Driver/CUDA/NCCL alignment — does the image’s CUDA/NCCL version actually match the host driver?
• RDMA device visibility — is /dev/infiniband/* correctly mounted into the container runtime (Apptainer --nv / Enroot device passthrough)? A job that “runs” but silently falls back from IB to TCP is the single most common invisible failure mode in multi-node training.
• Topology-aware placement — does the job land on nodes that are actually close on the fabric (same leaf switch/rack), or does Slurm’s topology.conf need tuning so multi-node jobs aren’t spread across a suboptimal path?
• Filesystem I/O readiness — can the shared filesystem sustain the read pattern from tokenized shards across every node hitting it simultaneously at epoch start?
• Multi-tenancy isolation — cgroups for CPU/memory, GPU MIG/exclusive-mode settings, and (your world) NSX-T-style network segmentation so one tenant’s job traffic can’t interfere with another’s.
• Monitoring — DCGM exporters feeding Prometheus/Grafana so GPU utilization, ECC errors, and thermal data are visible once the job is running.
  
  

---

End to End: From Dataset to Gradient Sync on the Wire

Putting the whole chain together — this is the diagram that ties every question above into one flow, from raw data all the way to the actual all_reduce traveling across your fabric:

Why This Matters

Everything above the CUDA/NCCL line belongs to the ML team. Everything below it — driver versions, fabric topology, container runtime, RDMA visibility, topology-aware scheduling — belongs to you. Most of the failures that actually eat time on a real cluster live exactly at that seam: a job that “runs” but silently degrades to TCP, a checkpoint that doesn’t resume cleanly after preemption, a dataset that hammers the filesystem in a pattern nobody load-tested.

That seam — where deep fabric/networking expertise meets distributed ML training — is precisely where a CCIE-grade infrastructure background stops being “adjacent” to AI infrastructure and becomes directly load-bearing.