LLM Kubernetes Projects: 5 Hands-On AI Infrastructure Labs

Can you build, deploy, monitor, and scale an AI application using Kubernetes?

Understanding AI infrastructure concepts is important. However, building a working platform requires practical experience with real tools, deployment workflows, and performance testing.

That’s where these LLM Kubernetes projects come in.

This hands-on AI infrastructure workbook takes you through five connected projects. Each project builds on the previous one, helping you develop practical skills in LLM deployment, automation, observability, and GPU resource management.

#	Project	Layer it adds	Core tools	GPU needed
1	Dockerfile generation with an LLM	Tooling	Python, Ollama, Docker	No
2	Production LLM serving on Kubernetes	Serving	vLLM, Helm, Prometheus, Grafana	Yes
3	End-to-end DevOps + AIOps	Delivery and operations	GitHub Actions, Argo CD, Prometheus, Loki	No
4	GPU auto-scaling inference	Elasticity	KServe, k6	Yes
5	Scale-to-zero inference	Cost	KEDA, Redis, vLLM	Yes

Lab setup

You need one Kubernetes cluster with a GPU node for projects 2, 4 and 5; projects 1 and 3 run fine on a laptop. Pin every version you use and write it in your post, because vLLM, KServe and KEDA release quickly and older tutorials break.

Tool	Used in	Purpose
Docker, kind or minikube	1, 3	Local builds and a throwaway cluster
Ollama	1	Local LLM for Dockerfile generation
kubectl, Helm	2 to 5	Cluster access and chart installs
NVIDIA GPU Operator	2, 4, 5	Drivers, device plugin and metrics on GPU nodes
Prometheus, Grafana, Loki	2, 3	Metrics, dashboards and logs
Argo CD	3	GitOps sync from Git to the cluster
KServe, k6	4	Model serving and load generation
KEDA, Redis	5	Event-driven scaling and the job queue

Recommended repo layout, one folder per project plus shared config:

ai-infra-workbook/
  p1-dockerfile-gen/
  p2-vllm-stack/
  p3-devops-aiops/
  p4-gpu-autoscale/
  p5-scale-to-zero/
  shared/        # namespaces, monitoring values, model-cache PVC

Before project 2, confirm the base is healthy:

• kubectl get nodes shows the GPU node Ready
• kubectl describe node <gpu-node> lists nvidia.com/gpu under Capacity
• A test pod running nvidia-smi completes successfully
• Prometheus and Grafana are installed in a monitoring namespace
• A model-cache PVC exists so weights download once, not per pod

Project 1: Dockerfile generation using an LLM

First, you will build a Python script that generates Dockerfiles using a local LLM.

To begin, the script scans an application folder and collects the relevant configuration files. Next, it sends the information to a model running through Ollama.

However, generating a Dockerfile does not guarantee that it will work correctly.

Therefore, the workflow uses hadolint and Docker to validate the generated output.

If validation fails, the script sends the error information back to the model for another attempt.

As a result, you create a repeatable workflow that combines AI-assisted code generation with automated verification.

Approach

Decision	Choice	Why
Model location	Local model through Ollama	Source code never leaves the machine
Determinism	Temperature 0 and a fixed rule list in the system prompt	Same app should give the same Dockerfile
Context sent	File tree plus manifests only, capped in size	Keeps the prompt small and avoids leaking secrets
Validation	hadolint, then docker build	The toolchain is the source of truth, not the model
Retries	Feed the error back, maximum 3 attempts	Bounded cost and no infinite loops

Build steps

1. Install Ollama and pull a code model: ollama pull qwen2.5-coder:7b. Any instruction-tuned code model works; note which one you used.
2. Write a scanner that collects requirements.txt, package.json, go.mod or pyproject.toml, plus a file tree, each truncated to about 2,000 characters.
3. Write the system prompt with hard rules: pinned slim base image, multi-stage build where it applies, non-root user, no secrets in layers, .dockerignore respected, and output only a single fenced Dockerfile.
4. Call the Ollama chat API and extract the fenced block.
5. Run hadolint Dockerfile, then docker build -t test-app ..
6. On failure, send the Dockerfile and the error text back to the model and repeat up to 3 times.

Core of the script:

import pathlib, requests

OLLAMA = "http://localhost:11434/api/chat"
MODEL = "qwen2.5-coder:7b"

def scan(app: pathlib.Path) -> str:
    keep = ["requirements.txt", "pyproject.toml", "package.json", "go.mod"]
    parts = [f"## {n}\n{(app/n).read_text()[:2000]}" for n in keep if (app/n).exists()]
    tree = [str(p.relative_to(app)) for p in app.rglob("*") if p.is_file() and ".git" not in p.parts]
    parts.append("## tree\n" + "\n".join(tree)[:2000])
    return "\n\n".join(parts)

def ask(system: str, user: str) -> str:
    r = requests.post(OLLAMA, json={
        "model": MODEL, "stream": False, "options": {"temperature": 0},
        "messages": [{"role": "system", "content": system},
                     {"role": "user", "content": user}]}, timeout=300)
    return r.json()["message"]["content"]

Checkpoints

• The script produces a Dockerfile for a Flask or FastAPI sample app that builds on the first or second attempt
• The container runs as a non-root user (docker run --rm test-app id)
• hadolint reports no errors on the final Dockerfile
• A deliberately broken prompt (missing rules) produces a visibly worse Dockerfile, proving the rules matter
• Failed builds trigger the repair loop and stop after 3 attempts

Break it

• Add a fake .env with a token to the app folder and confirm it never appears in the prompt or the image.
• Try a Node app, then a Go app, and note where the 7B model gets multi-stage builds wrong.
• Swap the model for a smaller one and measure how many attempts the repair loop needs.

Blog prompts

• Success rate by model size across 5 sample apps, as a small table.
• Why “LLM proposes, toolchain disposes” is the pattern to reuse in every later project.
• The three most common mistakes the model made, and the prompt rule that fixed each.

Project 2: Production LLM serving stack on Kubernetes

After completing the first project, you will deploy a production-oriented LLM serving stack on Kubernetes.

In this project, vLLM handles model inference, while a router distributes requests across multiple replicas.

Meanwhile, Prometheus and Grafana collect metrics that help you understand inference performance.

For example, you will monitor request queues, KV cache usage, time to first token, and throughput.

Additionally, you will test how the deployment responds when a replica becomes unavailable.

Ultimately, this project establishes a measurable baseline for the GPU scaling experiments that follow.

Approach

Decision	Choice	Why
Packaging	Helm chart (vLLM production-stack)	Router, engines and monitoring in one versioned release
Model	Small open model such as Qwen/Qwen2.5-1.5B-Instruct	No gated-model token, fits one modest GPU, quick to reload
Replicas	3, one GPU each	Enough to see routing and failure behavior
Routing	Round-robin first, session affinity second	Compare both under the same load
Weights	Shared PVC cache	Restarts reuse downloaded weights
Metrics	Scrape router and engines	You need baselines before project 4

Build steps

1. Add the chart repo: helm repo add vllm https://vllm-project.github.io/production-stack.
2. Write values.yaml with one model spec, three replicas and a router. Check the chart’s default values for your version, since key names change.
3. Install: helm install vllm vllm/vllm-stack -f values.yaml -n llm --create-namespace.
4. Wait for pods, then port-forward the router service and list models with curl localhost:30080/v1/models.
5. Send a chat completion, then watch which replica logged it.
6. Import a Grafana dashboard for vLLM and confirm the panels fill.

Starting point for values.yaml:

servingEngineSpec:
  modelSpec:
    - name: "qwen"
      repository: "vllm/vllm-openai"
      tag: "<pinned-version>"
      modelURL: "Qwen/Qwen2.5-1.5B-Instruct"
      replicaCount: 3
      requestCPU: 4
      requestMemory: "16Gi"
      requestGPU: 1
      pvcStorage: "30Gi"
      vllmConfig:
        maxModelLen: 4096
routerSpec:
  routingLogic: "roundrobin"

Metrics worth a Grafana panel each: requests running, requests waiting, KV-cache usage, time to first token, and tokens per second. Exact metric names differ between vLLM versions, so confirm them at the /metrics endpoint.

Checkpoints

• All three engine pods are Ready and each reports a GPU in kubectl describe pod
• Ten identical requests land on more than one replica
• Grafana shows time to first token and queue depth moving under a simple loop of requests
• Deleting one replica pod causes no failed client requests after the router notices
• A pod restart reloads weights from the PVC instead of downloading again

Break it

• Set maxModelLen too high and read the out-of-memory error until you can explain it.
• Send one very long prompt while short ones run, and watch queue depth and latency for the others.
• Switch to session affinity and repeat a multi-turn chat to see the effect on cache hits.

Blog prompts

• A baseline table: requests per second, p50 and p95 time to first token, with 1 versus 3 replicas.
• What the router adds over a plain Kubernetes Service, and when you would not need it.
• Why GPU memory, not compute, usually decides how many requests fit.

Project 3: DevOps and AIOps Automation

Next, you will connect software delivery with infrastructure monitoring and AI-assisted incident analysis.

First, GitHub Actions builds and publishes application images. Then, Argo CD synchronizes the deployment configuration with Kubernetes.

At the same time, Prometheus and Loki collect operational evidence.

When an incident occurs, an LLM analyzes the available metrics, logs, and deployment changes to generate a possible root-cause summary.

However, the system does not automatically apply changes. Instead, it presents its findings for human review.

Consequently, the workflow supports faster investigation while maintaining operational control.

Approach

Decision	Choice	Why
Deploy model	GitOps: CI never touches the cluster	Git is the audit trail; Argo CD fixes drift
Image tag	Commit SHA, never latest	Every incident maps to one commit
AIOps trigger	Alertmanager webhook	Reuses alerts you already trust
Evidence	Metrics, logs, recent deploy diff, pod events	An LLM without evidence guesses
Output	Structured JSON, then a Slack message	Predictable format, easy to evaluate
Authority	Read-only; no auto-remediation	Earn trust before granting write access

Build steps

1. Create two repos: the app repo and a GitOps repo holding Kubernetes manifests or a Helm chart.
2. Write a GitHub Actions workflow that tests, builds, pushes ghcr.io/<you>/<app>:<sha>, then commits the new tag to the GitOps repo.
3. Install Argo CD and create an Application pointing at the GitOps repo with automated sync.
4. Install Prometheus and Loki, and add alert rules for error rate, latency and restarts.
5. Build a small RCA service that receives the Alertmanager webhook and gathers: the alert labels, PromQL results for the last 30 minutes, the last 100 error log lines from Loki, the previous and current Argo CD revisions with their Git diff, and recent pod events.
6. Send that bundle to your LLM (the project 2 endpoint works) with a prompt asking for summary, likely_cause, evidence, confidence and next_checks as JSON.
7. Post the result to Slack with the alert.

Minimal Argo CD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/<you>/gitops
    targetRevision: main
    path: apps/my-app
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app
  syncPolicy:
    automated: {prune: true, selfHeal: true}

Checkpoints

• A commit to main reaches the cluster with no manual kubectl step
• Manually editing a Deployment in the cluster is reverted by Argo CD (self-heal)
• A bad commit (crash on startup) fires an alert within your target time
• The Slack message names the failing commit and quotes at least one log line as evidence
• The RCA output parses as valid JSON on 10 out of 10 test alerts

Break it

• Ship a commit that raises latency without crashing, and see whether the RCA blames the right change.
• Remove logs from the bundle and compare the quality of the diagnosis with and without them.
• Inject an instruction into a log line (“ignore previous instructions”) and confirm the service treats logs as data, not commands.

Blog prompts

• Time from commit to running pod, and from fault to Slack message, as two measured numbers.
• A scorecard of 10 injected faults: how many times the RCA was right, partly right, or wrong.
• Why the AI stays read-only, and what evidence would justify giving it more authority.

Project 4: Kubernetes GPU Autoscaling

Once the serving platform is operational, the next challenge is adjusting GPU capacity to changing demand.

To address this, you will deploy an inference service with KServe and generate traffic using k6.

As demand increases, Kubernetes can add replicas when the configured scaling conditions are met.

However, GPU provisioning introduces additional delays. For example, new nodes may need to start, container images may need to download, and model weights must load.

Therefore, the project measures both scaling behavior and application latency.

Finally, you will compare the results against your service-level objectives to determine whether the configuration meets your requirements.

Approach

Decision	Choice	Why
Serving layer	KServe InferenceService with the Hugging Face runtime	Declarative model serving with built-in autoscaling hooks
Scaling signal	Concurrent requests per replica	Tracks real load; GPU utilization stays high even when overloaded
Bounds	minReplicas: 1, maxReplicas: 3	One warm replica avoids cold starts; the max caps cost
Node scaling	Cluster autoscaler or Karpenter on a GPU node group	Pods stay Pending until a node exists
Goal	An SLO: p95 latency under a stated limit	Scaling is judged against a target, not by feel

Signal	Good for LLMs?	Note
Concurrency per replica	Yes	Simple and reacts early
Requests waiting in the queue	Best	Needs a custom metric through KEDA or HPA
Requests per second	Partly	Ignores prompt and output length
GPU utilization	Poor	Saturates quickly and says little about queueing

Build steps

1. Install KServe and its dependencies, and confirm the GPU node group can scale from 1 to 3.
2. Deploy the InferenceService below and call it once to confirm it answers.
3. Write the k6 script and run a baseline at low load, recording p95 latency.
4. Run the ramp: up, hold, then down. Watch kubectl get pods -w and the node list in a second terminal.
5. Record when each replica became Ready, and how long the slowest one took from Pending.
6. Tune the target and scale-down delay, and rerun.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: qwen
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 3
    scaleMetric: concurrency
    scaleTarget: 8
    model:
      modelFormat:
        name: huggingface
      storageUri: hf://Qwen/Qwen2.5-1.5B-Instruct
      resources:
        limits:
          nvidia.com/gpu: "1"

k6 ramp, rate-based so slow responses do not hide overload:

import http from 'k6/http';

export const options = {
  scenarios: {
    ramp: {
      executor: 'ramping-arrival-rate',
      startRate: 1, timeUnit: '1s',
      preAllocatedVUs: 50, maxVUs: 200,
      stages: [
        { target: 5, duration: '2m' },
        { target: 20, duration: '5m' },
        { target: 20, duration: '5m' },
        { target: 2, duration: '5m' },
      ],
    },
  },
  thresholds: { http_req_duration: ['p(95)<5000'] },
};

export default function () {
  const body = JSON.stringify({
    model: 'qwen',
    messages: [{ role: 'user', content: 'Explain Kubernetes in 100 words' }],
    max_tokens: 128,
  });
  http.post(`${__ENV.URL}/openai/v1/chat/completions`, body,
    { headers: { 'Content-Type': 'application/json' } });
}

Checkpoints

• Baseline p95 latency is recorded at one replica before any ramp
• Replicas rise from 1 to 3 during the ramp and fall back after the hold
• You have a measured time from scale decision to Ready for a replica on an existing node and on a new node
• The k6 threshold fails when scaling is disabled, and passes when enabled
• Scale-down does not flap: no more than one scale-down event per quiet period

Break it

• Set scaleTarget far too high and watch latency breach the SLO before any scale-up.
• Set maxReplicas above the GPU quota and read the Pending pod events.
• Send long prompts instead of short ones and see how the same concurrency target behaves.

Blog prompts

• A timeline chart from your run: load, replica count and p95 latency on one time axis.
• Why GPU utilization is a poor autoscaling signal, with your own numbers.
• The cold-start budget: node provisioning, image pull and weight load, each timed.

Project 5: Scale-to-Zero LLM Inference

Finally, you will explore how to reduce idle GPU compute consumption using KEDA.

In this project, clients submit inference jobs to a Redis queue. KEDA then monitors the queue and adjusts the number of vLLM worker replicas.

When new jobs arrive, the system can start additional workers. Conversely, when the queue becomes empty and the cooldown period expires, the deployment can return to zero replicas.

However, this approach introduces cold-start delays.

For this reason, the project measures startup time, job processing behavior, and scaling efficiency.

Ultimately, the goal is to determine when scale-to-zero inference is appropriate and when always-on GPU capacity is the better choice.

Approach

Decision	Choice	Why
Request pattern	Asynchronous: enqueue a job, poll for the result	With zero pods there is nothing to answer an HTTP call
Trigger	KEDA Redis list scaler	Queue length is a direct measure of pending work
Pod layout	vLLM container plus a small worker sidecar	The worker pulls jobs and calls vLLM on localhost
Scaling range	0 to 3 replicas	KEDA handles 0 to 1; the HPA it creates handles 1 to N
Cold start	Weights on a PVC and images pre-pulled	Cuts the delay to the model load itself
Safety	Reliable queue pattern (move job to a processing list)	A pod killed mid-job must not lose it

For interactive traffic that must scale to zero, use the KEDA HTTP add-on instead, which holds requests while a pod starts.

Build steps

1. Install KEDA with Helm and Redis in a queue namespace.
2. Deploy the vLLM worker pod: a vLLM container and a sidecar that waits for vLLM readiness, then loops over jobs.
3. Create the ScaledObject below with minReplicaCount: 0.
4. Confirm the Deployment sits at 0 replicas when the queue is empty.
5. Push 10 jobs, watch a pod start, and time how long the first result takes.
6. Let the queue drain and time how long until replicas return to 0.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-worker
spec:
  scaleTargetRef:
    name: vllm-worker
  minReplicaCount: 0
  maxReplicaCount: 3
  pollingInterval: 15
  cooldownPeriod: 300
  triggers:
    - type: redis
      metadata:
        address: redis.queue.svc:6379
        listName: jobs
        listLength: "5"
        activationListLength: "1"

Sidecar worker, simplest version (it loses a job if the pod dies mid-request; Break it covers that):

import json, redis, requests

r = redis.Redis(host="redis.queue.svc", port=6379)
while True:
    _, raw = r.blpop("jobs")
    job = json.loads(raw)
    resp = requests.post("http://localhost:8000/v1/chat/completions",
                         json=job["request"], timeout=600)
    r.set(f"result:{job['id']}", resp.text, ex=3600)

Checkpoints

• With an empty queue, kubectl get deploy vllm-worker shows 0 replicas and the GPU is free
• One job wakes a pod, and its result appears in Redis
• 30 queued jobs scale the Deployment above 1 replica
• Replicas return to 0 after the cooldown period passes with an empty queue
• You have a measured cold-start time from first job to first result

Break it

• Kill the pod mid-job and see which job vanished, then switch the worker to move jobs to a processing list before working on them.
• Set cooldownPeriod to 30 seconds and push jobs in bursts to see flapping and repeated cold starts.
• Delete the model PVC and compare cold start with and without cached weights.

Blog prompts

• Cost comparison: an always-on GPU replica versus scale-to-zero for a workload with 2 hours of real work per day.
• The cold-start breakdown: scheduling, image pull, weight load and first token, each timed.
• When scale-to-zero is the wrong choice, and what latency target rules it out.

Putting it together

The five projects form one platform: project 1 packages the app, project 3 ships it, project 2 serves the model it calls, and projects 4 and 5 decide how much GPU that model gets and when. Each project’s numbers become the next one’s starting point.

Project	Feeds into	What you carry forward
1 Dockerfile generation	3	A repeatable way to containerize apps
2 LLM serving stack	3, 4, 5	A working endpoint and baseline latency numbers
3 DevOps + AIOps	Whole platform	GitOps delivery, observability and a read-only RCA assistant
4 GPU auto-scaling	5	The right scaling signal and a cold-start budget
5 Scale-to-zero	Whole platform	A cost floor of zero for async workloads

Failure modes seen across all five

• Unpinned versions. latest tags and chart defaults change under you; pin and record them.
• Trusting model output. Every LLM step here has a verifier: a build, a JSON parse, a human reader.
• Slow GPUs to start. Node, image and weights each add minutes; time all three.
• Wrong metric. Scaling on GPU utilization or CPU hides queueing; use concurrency or queue depth.
• Cache loss. Without a weights PVC, every restart is a fresh download.

Final self-assessment

• I can explain why the LLM proposes and a tool verifies in projects 1 and 3
• I have a measured baseline for latency, throughput and cold start
• I can state which signal each autoscaler uses and why
• I have one chart that shows load, replicas and latency together
• Each project has its repo folder, pinned versions and a short write-up
• I know which workload suits always-on, autoscaled or scale-to-zero, and can defend the choice with numbers

Conclusion: Build Practical AI Infrastructure Skills with Kubernetes

Building reliable AI infrastructure requires more than deploying a large language model. It also involves automation, observability, scalable computing resources, and continuous performance evaluation.

Throughout these five LLM Kubernetes projects, you have explored how different infrastructure components work together to support AI applications.

First, AI-powered Dockerfile generation demonstrates how LLMs can assist development workflows while automated tools validate the results.

Next, deploying vLLM on Kubernetes introduces the foundations of scalable model serving. In addition, Prometheus and Grafana provide visibility into inference performance and system behavior.

Furthermore, the DevOps and AIOps project connects automated software delivery with monitoring and AI-assisted incident investigation.

As workloads grow, GPU autoscaling with KServe helps you understand how infrastructure responds to changing demand. Meanwhile, KEDA-based scale-to-zero inference demonstrates how event-driven scaling can reduce idle GPU resource consumption.

However, successful AI infrastructure design requires more than implementing these technologies individually.

Organizations must also consider workload characteristics, GPU availability, cold-start latency, security, reliability, and operational costs.

Ultimately, these projects demonstrate how to build and evaluate an AI infrastructure platform using practical tools and measurable results.

Whether you are an infrastructure engineer, DevOps professional, AI developer, or enterprise architect, the most valuable outcome is the ability to make informed technical decisions based on real deployment experience.

The next step is simple: apply what you have learned, measure the results, and continue improving your AI infrastructure architecture.

Ready to Take Your AI Infrastructure from Lab to Enterprise?

Building an AI infrastructure lab is an important first step. However, deploying AI workloads in enterprise environments requires careful planning across networking, computing, security, scalability, and operations.

Need guidance on your infrastructure architecture?

TREZBON TECHNOLOGIES supports organizations with IT infrastructure consulting, cybersecurity, system integration, and technology architecture planning.

Whether you are evaluating GPU infrastructure, modernizing your data center, or planning an enterprise AI deployment, our team can help you assess your requirements and identify a practical approach.

Discuss your infrastructure requirements with TREZBON TECHNOLOGIES.

🌐 Contact Our Team

Want more hands-on technical learning?

Follow NETWORK BACHELOR for practical guides covering AI infrastructure, Kubernetes, enterprise networking, cybersecurity, and modern data center technologies.

📖 Explore More Technical Guides