Training a model is the part that gets the demo. Serving it is the part that gets the pager. This tutorial covers the path we use on client engagements to take a trained Keras 3 / TensorFlow model and put it behind a stable, versioned, monitorable HTTP and gRPC endpoint with TensorFlow Serving - including the export details that quietly break production, dynamic batching, canarying a new version, and the metrics you should be watching before anyone else notices a problem.
None of our other tutorials cover the serving side: tf.data pipelines is about training throughput, LiteRT is about on-device, and Gemma fine-tuning stops at save_to_preset. This is what happens next.
1. Export a SavedModel with a signature you control
Under Keras 3, model.save("x.keras") writes the Keras format - great for retraining, useless to TF Serving. Serving needs a SavedModel directory, and it needs an explicit serving signature:
import tensorflow as tf, keras
model = keras.saving.load_model("model.keras")
export_archive = keras.export.ExportArchive()
export_archive.track(model)
export_archive.add_endpoint(
name="serving_default",
fn=lambda x: model(x, training=False),
input_signature=[tf.TensorSpec(shape=(None, 224, 224, 3), dtype=tf.float32)],
)
export_archive.write_out("serving/vision/1")
For the common case, model.export("serving/vision/1") does the same thing with an inferred signature. Use ExportArchive when you want to control dtypes, add preprocessing, or expose more than one endpoint.
Two rules we enforce on every project:
Bake preprocessing into the graph. If the client sends raw bytes and your Python service resizes and normalizes before calling the model, you now have two places where preprocessing can drift apart from training. Put it in the export instead:
@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])
def serve_jpeg(image_bytes):
def decode(b):
img = tf.io.decode_jpeg(b, channels=3)
img = tf.image.resize(img, [224, 224])
return tf.cast(img, tf.float32) / 255.0
x = tf.map_fn(decode, image_bytes, fn_output_signature=tf.float32)
return {"probabilities": model(x, training=False)}
Name your outputs. A dict return gives your callers probabilities instead of output_0, which means you can change the model without changing every client.
Always verify what you actually wrote:
saved_model_cli show --dir serving/vision/1 --tag_set serve --signature_def serving_default
The numbered subdirectory (/1) is not cosmetic - it is the version, and TF Serving requires it.
2. Run TensorFlow Serving
docker run --rm -p 8500:8500 -p 8501:8501 \
-v "$PWD/serving/vision:/models/vision" \
-e MODEL_NAME=vision \
tensorflow/serving:latest
Port 8501 is REST, 8500 is gRPC. Smoke test:
curl -s -X POST http://localhost:8501/v1/models/vision:predict \
-H 'Content-Type: application/json' \
-d '{"signature_name":"serving_default","instances":[{"b64":"<base64-jpeg>"}]}'
curl -s http://localhost:8501/v1/models/vision # status + version state
Use REST for internal tools and low volume. Use gRPC for anything latency-sensitive: it avoids JSON encoding of float arrays, which for image or embedding payloads is frequently a larger cost than the model itself.
3. Turn on dynamic batching
This is the single biggest throughput win on GPU, and it is off by default. Serving accumulates concurrent requests for a short window and runs them as one batch:
# batching.config
max_batch_size { value: 32 }
batch_timeout_micros { value: 5000 }
num_batch_threads { value: 8 }
max_enqueued_batches { value: 1000 }
docker run --rm -p 8500:8500 -p 8501:8501 \
-v "$PWD/serving/vision:/models/vision" \
-v "$PWD/batching.config:/etc/batching.config" \
-e MODEL_NAME=vision \
tensorflow/serving:latest \
--enable_batching=true --batching_parameters_file=/etc/batching.config
batch_timeout_micros is the dial that matters: it trades tail latency for throughput. Start at 1000-5000 microseconds, then measure. Under real traffic we routinely see 3-5x throughput on GPU with a few milliseconds added to p50. On CPU-only deployments the win is much smaller - and if your export has a fixed batch dimension, batching silently does nothing, which is one more reason to export with None as the leading shape.
4. Version, canary, and roll back
Point Serving at a model config instead of a single directory and you get controlled rollouts:
model_config_list {
config {
name: "vision"
base_path: "/models/vision"
model_platform: "tensorflow"
model_version_policy { specific { versions: 7 versions: 8 } }
version_labels { key: "stable" value: 7 }
version_labels { key: "canary" value: 8 }
}
}
Run with --model_config_file=/etc/models.config --model_config_file_poll_wait_seconds=30. Now:
- Both versions are loaded, so a rollback is a config edit, not a redeploy.
- Clients can request
/v1/models/vision/labels/stable:predictor.../canary:predict, and you shift traffic at the router. - Dropping a version from the list unloads it and frees the memory.
Before promoting a canary, run a parity check: replay a fixed set of production inputs against both labels and diff the outputs. Differences are expected when the model changed; unexplained differences in the unchanged preprocessing path are the ones that will hurt you. This is the same golden-set discipline as in our TensorFlow 2.21 upgrade guide.
5. Monitor the things that actually fail
Enable Prometheus metrics:
--monitoring_config_file=/etc/monitoring.config
prometheus_config { enable: true, path: "/monitoring/prometheus/metrics" }
What to alert on, in priority order:
:predictp99 latency, per model version. Averages hide the requests that time out.- Batch queue depth and
batching_sessionwait time. Rising queue depth is your early warning that you need another replica. - Model load status. A bad export leaves the previous version serving and the new one failing to load - healthy-looking traffic, stale model.
- Request/error rate split by status code, so a client sending malformed payloads doesn't read as a model outage.
- Input distribution drift. Serving will not tell you this. Log a sample of inputs and prediction distributions to your warehouse and compare weekly against the training distribution. Silent drift is the most common cause of "the model got worse" tickets, and it never shows up in infrastructure metrics.
Also log the model version with every prediction. When someone asks in six weeks why a specific decision was made, that field is the difference between an answer and a shrug.
6. Sizing and cost
A few heuristics we use for capacity planning:
- Measure single-request latency first, unbatched, on the target hardware. That is your latency floor.
- Then measure saturation throughput with batching on. The gap between those two numbers is your headroom.
- Pin one model per container where you can. Co-tenanting models makes a slow neighbour indistinguishable from a slow model.
- Reserve roughly 2x the model's memory footprint: during a version transition, two versions are resident simultaneously.
- If GPU utilization sits below ~30% after batching is tuned, you are probably paying for a GPU you don't need. Many production CNNs and small transformers serve fine on CPU with oneDNN.
When TF Serving isn't the answer
It is the right default for SavedModels with stable signatures. It is the wrong tool for autoregressive LLM generation (you want a serving stack with KV-cache and continuous batching), for models that need arbitrary Python at inference time, and for very low-traffic internal endpoints where a small FastAPI process is less operational surface. For on-device targets, see our Edge AI & LiteRT work instead.
Checklist
- Export a SavedModel with an explicit, named,
None-batch signature; verify withsaved_model_cli. - Bake preprocessing into the graph.
- Enable dynamic batching and tune
batch_timeout_microsagainst real traffic. - Serve from a model config with two versions and version labels.
- Parity-check the canary against a golden input set before shifting traffic.
- Export Prometheus metrics; alert on p99, queue depth, and load status; log model version per prediction.
- Track input drift outside of Serving.
If you have a model that works in a notebook and needs to survive contact with production traffic, that is exactly the kind of engagement our TensorFlow consultants take on. Get in touch with the model, the traffic profile, and the latency target, and we will tell you what it takes.