NVIDIA GPU Workloads on Kubernetes — Part 9: Observability Stack for GPU Clusters

Part 9 of the Falcon AI workbook series. Parts 5 and 6 confirmed DCGM Exporter and Node Exporter pods were Running — but “Running” isn’t the same as “wired into a dashboard someone actually looks at.” This post closes that gap: full-stack observability for cluster, nodes, GPUs, network, and the application layer from Part 8.

Prerequisite

You’ve completed Part 8: Kubeflow, KServe, Ray, MPI Operator, and Volcano are installed and running real jobs. DCGM Exporter and Node Exporter pods have been confirmed Running since Part 5 but aren’t yet feeding anywhere.

Recap: where Falcon AI stands

Role	Count	Hostnames	Status after Part 8
Management (control plane)	3	mgmt-01, mgmt-02, mgmt-03	Ready, tainted NoSchedule
GPU worker	2	gpu-wk-01, gpu-wk-02	Ready, running real Kubeflow/KServe/MPIJob workloads

Why this needed its own post instead of just “install Prometheus”

A generic Kubernetes observability setup misses the two things that actually matter for a GPU cluster: GPU-level telemetry (utilization, ECC errors, thermal throttling — none of which kube-state-metrics knows about) and fabric-level telemetry (RDMA link errors, NCCL collective performance — invisible to standard node monitoring). Everything in this post exists to surface those two blind spots alongside the metrics you’d expect from any cluster.

The three pillars, and how they connect

Metrics answer “what is the current state” (GPU util is 94%). Logs answer “what happened” (this pod crashed with OOM at 14:32). Traces answer “where did the time go” (this inference request spent 340ms in the KServe predictor). All three converge in Grafana as one pane of glass.

Step 1 — Install the Prometheus Operator via kube-prometheus-stack (on mgmt-01)

bash

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

kubectl create namespace monitoring

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

kubectl get pods -n monitoring

Pass criteria: prometheus-, alertmanager-, and grafana- pods Running. This single chart brings Prometheus, Alertmanager, Grafana, and kube-state-metrics together — the four boxes at the top and bottom-right of the diagram above.

Step 2 — Point Prometheus at DCGM Exporter

DCGM Exporter has been running since Part 5, but Prometheus won’t scrape it without a ServiceMonitor telling it where to look:

yaml

# dcgm-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dcgm-exporter
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: nvidia-dcgm-exporter
  namespaceSelector:
    matchNames:
    - gpu-operator
  endpoints:
  - port: gpu-metrics
    interval: 15s

kubectl apply -f dcgm-servicemonitor.yaml

Pass criteria:

kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090

Open localhost:9090/targets — dcgm-exporter should show state: UP for both gpu-wk-01 and gpu-wk-02.

Step 3 — Install Loki and Promtail (logs)

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install loki grafana/loki-stack \
  --namespace monitoring \
  --set promtail.enabled=true \
  --set loki.persistence.enabled=true \
  --set loki.persistence.size=50Gi

Pass criteria:

kubectl get pods -n monitoring -l app=promtail -o wide

One Promtail pod per node (including gpu-wk-01/gpu-wk-02), Running.

Step 4 — Install OpenTelemetry Collector and Tempo (traces)

Traces matter most for the KServe inference path from Part 8 — this is what lets Falcon AI see where latency actually goes inside a serving request.

helm install tempo grafana/tempo --namespace monitoring

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace monitoring \
  --set mode=daemonset \
  --set config.exporters.otlp.endpoint=tempo.monitoring.svc.cluster.local:4317

Pass criteria:

kubectl get pods -n monitoring -l app.kubernetes.io/name=opentelemetry-collector -o wide

One collector pod per node, Running.

Step 5 — Wire it all together in Grafana

kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80
# default credentials: admin / prom-operator (check Helm chart notes to confirm)

Import these dashboards (Grafana.com dashboard IDs, or build custom):

Dashboard	Grafana ID	Shows
NVIDIA DCGM Exporter	12239	Per-GPU utilization, memory, temperature, power, ECC errors
Node Exporter Full	1860	CPU, memory, disk, network per node
Kubernetes / Compute Resources	315	Namespace-level resource usage (Kubeflow, KServe, MPIJob pods)

Add Loki and Tempo as data sources in Grafana (Configuration → Data Sources) so logs and traces are queryable alongside metrics, not in separate tools.

Step 6 — Configure a first real alert

DCGM ECC errors are the single most useful GPU alert to have — they’re often the earliest signal of failing hardware, well before a job crashes outright:

yaml

# dcgm-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: dcgm-ecc-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
  - name: gpu-health
    rules:
    - alert: GPUUncorrectableECCErrors
      expr: DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Uncorrectable ECC error on {{ $labels.instance }}"

kubectl apply -f dcgm-alerts.yaml

Where we are

✅ Full metrics/logs/traces stack installed and wired to Grafana as one pane of glass ✅ DCGM Exporter, Node Exporter, and application-layer pods all confirmed feeding Prometheus ✅ Loki capturing logs and Tempo capturing traces from Kubeflow/KServe/MPIJob workloads ✅ First real alert configured for GPU ECC errors — Falcon AI now finds out about failing hardware before a job does

---

Next in the series: Part 10 — MIG vs Time-Slicing: Choosing a GPU Sharing Mode, where we put the observability stack built here to immediate use — comparing real utilization and isolation characteristics between the two sharing modes on Falcon AI’s H100s.