Serving TensorFlow Models on Kubernetes with KServe: Autoscaling, Scale-to-Zero, and Canary Rollouts

Most TensorFlow models that reach production end up behind a single tensorflow/serving container on a fixed-size GPU node, sized for the worst hour of the week and idle for the other 167. That is fine for one model. It stops being fine the moment you have a dozen models, a Gemma endpoint that needs an 80 GB card, a finance team asking why the GPU bill tripled, and a release process that cannot roll back in under an hour.

This tutorial covers the layer above our TensorFlow Serving tutorial: running inference on Kubernetes with KServe, with request-driven autoscaling, scale-to-zero for cold models, canary rollouts, and the cost arithmetic that decides whether any of this is worth it.

Why KServe rather than a plain Deployment

A plain Deployment plus a Service works, and for a single always-on model it is the least machinery. KServe earns its keep when you need:

• Autoscaling on concurrency or RPS, not just CPU utilisation — the metric that actually tracks inference load on a GPU.
• Scale-to-zero for models that serve a few hundred requests a day.
• Traffic splitting between a current and a canary revision, declaratively, with one-field rollback.
• A consistent inference protocol (Open Inference Protocol / v2) across TensorFlow, ONNX, and custom runtimes, so clients do not rewrite when a model moves frameworks.
• Model-agnostic plumbing: transformers (pre/post-processing), explainers, and payload logging as separate containers.

If you need none of those, stay with a Deployment. We have told clients that more than once.

Export the model

Nothing changes about the export step. Keras 3 to SavedModel, with an explicit signature:

import keras, tensorflow as tf

model = keras.saving.load_model("churn.keras")

@tf.function(input_signature=[tf.TensorSpec([None, 32], tf.float32, name="features")])
def serve_fn(features):
    return {"probability": model(features, training=False)}

tf.saved_model.save(model, "gs://acme-models/churn/1", signatures={"serving_default": serve_fn})

KServe's TensorFlow runtime expects the versioned directory layout TF Serving uses: <storage-uri>/<version>/saved_model.pb. Point storageUri at the parent directory (gs://acme-models/churn), not the version folder.

Verify locally before you ever apply YAML:

saved_model_cli show --dir ./churn/1 --tag_set serve --signature_def serving_default
docker run -p 8501:8501 -v $PWD/churn:/models/churn \
  -e MODEL_NAME=churn tensorflow/serving
curl -d '{"instances": [[0.1, ... ]]}' localhost:8501/v1/models/churn:predict

Ninety per cent of "KServe is broken" tickets are a bad signature or a wrong path, and both are visible in thirty seconds locally.

A first InferenceService

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: churn
  namespace: ml-serving
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 10
    scaleTarget: 8            # target in-flight requests per replica
    scaleMetric: concurrency
    tensorflow:
      storageUri: gs://acme-models/churn
      resources:
        requests: {cpu: "2", memory: 4Gi}
        limits:   {cpu: "4", memory: 8Gi}

kubectl apply -f churn.yaml
kubectl get inferenceservice churn -n ml-serving

The status gives you a URL. Call it with the v1 (TF Serving-compatible) or v2 (Open Inference) protocol depending on your runtime configuration.

Picking scaleTarget

Measure, do not guess. Run a load test at fixed concurrency and record p50/p99 latency per replica:

hey -z 60s -c 8 -m POST -D payload.json \
  -T application/json http://churn.ml-serving.example.com/v1/models/churn:predict

Raise concurrency until p99 crosses your SLO; the concurrency one step below that is your scaleTarget. For CPU-bound tabular models the number is usually 4-16. For a GPU model with server-side batching it may be far higher, because the batcher is absorbing the parallelism.

GPU models and scale-to-zero

For a Gemma-class or large vision model, the predictor needs a GPU and an honest cold-start budget:

spec:
  predictor:
    minReplicas: 0            # scale to zero when idle
    maxReplicas: 4
    scaleMetric: concurrency
    scaleTarget: 4
    containerConcurrency: 4
    nodeSelector:
      cloud.google.com/gke-accelerator: nvidia-l4
    tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
    model:
      modelFormat: {name: tensorflow}
      storageUri: gs://acme-models/gemma-domain
      resources:
        limits:
          nvidia.com/gpu: "1"
          memory: 32Gi

minReplicas: 0 is the single biggest cost lever and the single biggest latency risk. A cold start on a GPU node is node provisioning (1-4 minutes if the cluster autoscaler must add a node) plus image pull (large CUDA images: minutes unless cached) plus model download and load (seconds to minutes depending on size). Mitigations we actually use:

• Keep a warm node pool, not a warm replica — a small number of pre-provisioned GPU nodes with the serving image pre-pulled, so scale-up is load time only.
• Slim the image. A 12 GB runtime image is a two-minute pull on every new node.
• Stream weights from a local SSD cache or a persistent volume rather than object storage.
• Set minReplicas: 1 for anything user-facing and reserve scale-to-zero for internal, batch, or evaluation endpoints.

Keep the serving batcher on for GPU endpoints — it is the difference between 15% and 70% utilisation:

    batcher:
      maxBatchSize: 32
      maxLatency: 20          # milliseconds

Tune maxLatency against your SLO: it is a direct latency-for-throughput trade.

Canary rollouts

This is the part that makes release night boring. KServe keeps the last-good revision and splits traffic:

spec:
  predictor:
    canaryTrafficPercent: 10
    model:
      modelFormat: {name: tensorflow}
      storageUri: gs://acme-models/churn-v7

Apply, and 10% of traffic hits v7 while 90% stays on the previous revision. Promote by removing canaryTrafficPercent; roll back by reverting storageUri — seconds, not a rebuild.

What to check before promoting, in order:

1. Error rate and p99 latency on the canary revision alone.
2. Prediction distribution. A model that is silently returning 0.5 for everything looks perfectly healthy on latency dashboards. Compare the canary's score histogram against the incumbent's over the same traffic.
3. Business metric, if you have fast feedback. If you do not, hold the canary longer.

Log payloads from the canary so you can replay them:

    logger:
      mode: all
      url: http://payload-collector.ml-serving.svc.cluster.local

Sample, do not log everything, and scrub PII in the collector — the same discipline we describe in the privacy-preserving training tutorial.

Observability that predicts failure

Scrape the predictor's Prometheus metrics and alert on:

• Queue / wait time, not just total latency. Rising queue time with flat compute time means you are under-replicated; the autoscaler reacts after users notice.
• Batch fullness on GPU endpoints. Consistently tiny batches mean maxLatency is too low or traffic is too thin to justify the GPU.
• GPU utilisation and memory via DCGM. Sustained utilisation under 30% on an always-on GPU is a signal to consolidate models or move to CPU with a quantised model.
• Cold-start count per day. If scale-to-zero is firing during business hours, it is costing you more in p99 than it saves in dollars.
• Model version served, as a label on every metric, so a dashboard anomaly can be attributed to a rollout.

Pair these with the drift monitoring from our TensorFlow Data Validation tutorial: infrastructure metrics tell you the server is healthy, drift metrics tell you the predictions still mean something.

The cost arithmetic

A quick framework we use in cost reviews, with order-of-magnitude cloud prices (check current rates for your region):

Setup	Rough monthly cost	When it wins
CPU replicas, quantised model	tens of dollars	Tabular, small text, low QPS
1 always-on mid-range GPU (L4-class)	hundreds of dollars	Steady traffic, latency SLO under 100 ms
GPU with scale-to-zero	a fraction of always-on	Spiky, internal, or batch traffic
Autoscaled pool of 2-8 GPUs	low thousands	Real product traffic with daily peaks

Three decisions move the number far more than cluster tuning:

1. Does this need a GPU at all? A distilled or quantised model on CPU is often within a point of accuracy — see our tutorials on quantization and pruning and knowledge distillation.
2. Can requests be batched offline? Nightly batch scoring into a key-value store costs a rounding error compared with an online endpoint.
3. How many models share a node? Several small models on one GPU (or one multi-model server) beats one node each.

A pragmatic rollout order

1. SavedModel verified locally with saved_model_cli and a local TF Serving container.
2. One InferenceService, minReplicas: 1, no autoscaling cleverness, behind an internal URL.
3. Load test; set scaleTarget from measured p99; enable autoscaling.
4. Add the batcher for GPU endpoints; re-measure.
5. Turn on payload logging and dashboards; define the promote/rollback checklist.
6. Only then consider scale-to-zero, multi-model serving, or spot/preemptible nodes.

Teams that do step 6 first are the ones who call us at 2 a.m.

Where this goes wrong

• Spot/preemptible GPUs for latency-critical serving. Great for training, risky for an endpoint with an SLO unless you keep an on-demand floor.
• Requests without limits, or limits without requests: either produces an autoscaler that oscillates.
• No PodDisruptionBudget, so a node upgrade takes the only replica down.
• One giant InferenceService serving five models through branching code. Split them; independent scaling is the whole point.
• Readiness probes that pass before the model is loaded, so traffic hits a pod that 500s for the first 90 seconds.

---

Serving is where ML projects quietly lose their margin. If you are standing up TensorFlow or Gemma endpoints on Kubernetes, or you already have and the bill or the p99 is the wrong shape, our TensorFlow consultants do short engagements that end with a working, measured, documented deployment. Get in touch with your traffic pattern and latency target and we will tell you what it should cost.