Kubernetes for AI, minus the hype
Kubernetes solves the boring half of an AI programme — scheduling, packaging, rollback — and nothing of the half that decides whether the model is any good. Knowing which half you have is the whole decision.
Two different problems with the same symptom
Almost every AI programme that stalls between a working prototype and a production system is stuck on one of two things, and the two need entirely different answers.
The first is infrastructure. The notebook trains on one machine and there is no path to eight. GPUs are booked in a chat channel. The model that works locally will not start in staging because the CUDA version moved. Nobody can say what the cluster costs per team. These are scheduling, packaging, and quota problems, and they have well-understood solutions.
The second is epistemic. Nobody can say whether the new version is better than the old one. A prompt changed three weeks ago and no one knows who changed it. Quality is assessed by a senior engineer reading a handful of outputs and forming an impression. There is no way to replay the conversation that went wrong. These are not infrastructure problems, and no amount of infrastructure fixes them.
Kubernetes is an excellent answer to the first and no answer at all to the second. The trouble is that both present as “we cannot get our model into production”, so the second gets treated as the first — and a team spends two quarters building a platform for a model they still cannot evaluate.
- Packing GPUs onto nodes
- Restarting a crashed worker
- Rolling a model out, and back
- Same image, dev to prod
- Quotas across teams
- Deciding the model earns the GPU
- Noticing the answers got worse
- Knowing which rollout was better
- Same answer, dev to prod
- Whose eval blocks the release
What Kubernetes genuinely does for AI workloads
Taken for what it is, the value is real and worth having. Four things it does better than whatever you were doing instead:
- GPUs stop being pets
- Device plugins expose accelerators as a schedulable resource, so a job asks for a GPU rather than for a named machine. With quotas per namespace you get an actual answer to “who is using the cluster”, which is usually the first question finance asks and the one nobody can answer.
- Training jobs get a scheduler
- Distributed training across nodes is a coordination problem, and the Kubernetes Job and gang-scheduling primitives are a reasonable place to solve it. PyTorch, TensorFlow, and Ray all have operators that are honestly better than the shell scripts they replaced.
- Serving inherits the boring guarantees
- Load balancing, health checks, horizontal autoscaling, rolling updates, and a rollback that works. Serving stacks like KServe, TorchServe, or vLLM behind an ingress get all of it for free, and inference is a stateless HTTP workload — exactly what this machinery was built for.
- Reproducibility, up to a point
- A container pins the runtime, the CUDA version, and the library graph, which removes the “works on the training box” failure entirely. It pins nothing about the data or the weights. Those need their own versioning, and the image digest is the only part Kubernetes is holding.
None of this makes the model better. It makes the model cheaper to run, faster to replace, and possible to reason about — which is the precondition for improving it, not a substitute.
Where the work actually lands
The AI lifecycle maps onto Kubernetes unevenly. Some stages fit the primitives almost exactly; others fit badly enough that a managed service is the better answer. Where each one sits, and what to watch:
- Model training
- Batch jobs with a defined start and end, competing for expensive hardware — the workload Kubernetes handles most cleanly. Worth adding a queueing layer such as Kueue before you have contention rather than after, because the first three months of ad-hoc GPU booking is how a shared cluster acquires a shadow reservation system in a spreadsheet.
- Model serving
- The clearest win. Scale on the metric that actually predicts saturation — queue depth or tokens in flight, not CPU — and treat cold start as a first-class design problem. A ten-gigabyte model pulled on demand turns an autoscaling event into a multi-minute outage.
- Data and feature pipelines
- Airflow, Argo Workflows, or Kubeflow Pipelines all run well here, and pipeline steps are just containers with dependencies. The caution is scope: an orchestrator running on Kubernetes is not the same thing as data lineage, and teams routinely buy the first believing they got the second.
- Evaluation and experiment tracking
- MLflow or Weights & Biases give you the record; the cluster gives you somewhere to run the graded sets on every pull request. This is the integration we would build first, before serving is even in place, because it is the one that makes every later change reversible.
- Edge inference
- K3s or KubeEdge genuinely work for fleets of constrained devices, and the deployment model is the same one your platform team already knows. Budget for intermittent connectivity as the normal case rather than the exception, and for the fact that a rollback is much harder when the node is on a boat.
Before you stand up a cluster
Five things we would want settled first. The last one is the one most likely to save you a quarter.
- Take the managed control plane
- EKS, AKS, or GKE. Running your own control plane to serve models is an expensive way to acquire an operational speciality with no relationship to your product. Add the provider GPU node pools with autoscaling, and make sure scale-to-zero actually works — idle accelerators are the single largest line item we find on AI infrastructure bills.
- Instrument the model, not just the pod
- Prometheus and Grafana will tell you the pod is healthy while the answers get steadily worse. Model-level signals — latency at the tail, token throughput, refusal and error rates, drift against a reference set — need to be in the same dashboard as the infrastructure ones, or the two will be read by different people at different times.
- Treat weights and prompts as production artifacts
- They are deployable, versioned, and capable of causing an incident, so they belong under the same review, provenance, and rollback discipline as code. A prompt edited in a console is a production change with no author and no diff.
- Scope access before the first GPU lands
- RBAC per team, network policy around anything holding training data, and secrets that are not baked into images. Retrofitting isolation onto a shared research cluster is materially harder than starting with it, and research clusters have a way of quietly becoming production.
- Be honest about whether you need a cluster
- If you are serving one model to one product with predictable load, a managed inference endpoint will be cheaper and faster to stand up, and you should take it. Kubernetes starts paying when you have multiple teams, multiple models, contention for hardware, or a hard requirement to run somewhere specific.
The order we would do it in
Build the evaluation harness first — before the cluster, before the serving stack, before anything is described as a platform. Graded sets, a regression suite, and a threshold wired into CI, so that changing a model or a prompt becomes a pull request with a result attached. This is unglamorous, and it is the single highest-return thing on the list, because everything after it becomes reversible.
Then containerise and get one model served on managed Kubernetes with real observability — traces on every call, retry, and tool invocation, and model-quality signals next to the infrastructure ones. One model, end to end, with a rollback you have actually rehearsed.
Then add training, queueing, and multi-team quota at the point where contention is real rather than anticipated. Then, and only then, is there a case for calling any of it a platform.
The failure mode this ordering avoids is the common one: a beautifully engineered cluster serving a model nobody can prove is working. Kubernetes will happily scale that to a thousand replicas.
A non-deterministic system you cannot replay is one you cannot debug. Orchestration does not change that — it just gives you more of it.