Every quarter a client asks the same question in a slightly different way: can we run this model in the user's browser instead of paying for GPU inference? For a surprising number of models the answer is now yes. WebGPU is shipping in Chrome and Edge, is on by default in recent Safari and Firefox builds, and the browser runtimes around it have matured enough that a Keras 3 model can go from model.export() to a tab on a laptop with no inference server in the loop.
This tutorial covers the two paths we actually use on client work - TensorFlow.js for the general case and LiteRT.js when you already have a .tflite artifact from the edge pipeline - plus the parts nobody writes down: model size budgets, warm-up latency, fallback when WebGPU is missing, and how to decide whether in-browser inference is the right call at all.
It is the missing piece between our serving tutorial, which is about servers, and our LiteRT migration guide, which is about phones.
When in-browser inference is the right answer
Run the model client-side when at least two of these are true:
- The data should not leave the device. Medical images, document scans, keystroke or webcam data. This is the strongest argument, and it is a compliance argument, not a performance one.
- You need interactive latency. Anything driven by a camera, a cursor, or every keystroke. A 20 ms local inference beats a 60 ms round trip you cannot control.
- Inference volume is high and per-inference value is low. Background classification, autocomplete ranking, image pre-filtering. Pushing that onto client hardware removes a line item from your cloud bill.
- The app must work offline. PWAs, field tools, kiosks.
Do not do it when the model is large (say, above ~100 MB of weights after quantization), when the weights themselves are proprietary and valuable - anything you ship to the browser is readable - or when the model changes daily and you need every user on the same version within minutes.
Path A: TensorFlow.js
1. Export and convert
Start from a SavedModel, not a .keras file:
import keras
model = keras.saving.load_model("model.keras")
model.export("export/vision") # SavedModel directory
Convert with the tensorflowjs converter, in its own virtualenv - its dependency pins fight with a normal training environment:
pip install tensorflowjs
tensorflowjs_converter \
--input_format=tf_saved_model \
--output_format=tfjs_graph_model \
--signature_name=serving_default \
--saved_model_tags=serve \
export/vision \
web/model
You get a model.json plus sharded group1-shard*of*.bin weight files. The sharding matters: shards are cached individually by the browser, so a 40 MB model downloads in parallel chunks and re-visits are free.
Quantize on the way out if you can accept the accuracy cost:
tensorflowjs_converter \
--input_format=tf_saved_model \
--quantize_float16='*' \
export/vision web/model-fp16
--quantize_float16 roughly halves download size and is usually loss-free enough for vision backbones. --quantize_uint8 halves it again but needs a real accuracy check on your holdout set - measure, do not assume. Our quantization tutorial covers how to evaluate that trade-off properly.
2. Load and run with the WebGPU backend
npm i @tensorflow/tfjs-core @tensorflow/tfjs-converter \
@tensorflow/tfjs-backend-webgpu @tensorflow/tfjs-backend-webgl
import * as tf from '@tensorflow/tfjs-core';
import { loadGraphModel } from '@tensorflow/tfjs-converter';
import '@tensorflow/tfjs-backend-webgpu';
import '@tensorflow/tfjs-backend-webgl';
async function initBackend() {
if (navigator.gpu) {
try {
await tf.setBackend('webgpu');
await tf.ready();
return 'webgpu';
} catch (e) {
console.warn('WebGPU init failed, falling back', e);
}
}
await tf.setBackend('webgl');
await tf.ready();
return tf.getBackend();
}
const backend = await initBackend();
const model = await loadGraphModel('/model/model.json');
Always keep the WebGL fallback. navigator.gpu existing does not guarantee an adapter; blocklisted drivers, headless environments, and older iOS all fail at adapter request time, which is why the try/catch above matters more than the feature check.
3. Warm up before the user notices
The first inference compiles shaders and uploads weights to the GPU. On a mid-range laptop that is 200-800 ms; subsequent calls are a fraction of it. Do it during page load, not on first click:
const warm = tf.zeros([1, 224, 224, 3]);
await model.executeAsync(warm).then(t => Array.isArray(t) ? t.forEach(x => x.dispose()) : t.dispose());
warm.dispose();
4. Inference without leaking memory
This is the number one bug in shipped TF.js code. Tensors live on the GPU and are not garbage collected. Wrap synchronous tensor work in tf.tidy and dispose anything async yourself:
async function classify(videoEl) {
const input = tf.tidy(() =>
tf.browser.fromPixels(videoEl)
.resizeBilinear([224, 224])
.toFloat()
.div(255)
.expandDims(0)
);
const out = await model.executeAsync(input);
const probs = await out.data(); // async read-back, do not use .dataSync() on the UI thread
input.dispose();
out.dispose();
return probs;
}
Put console.log(tf.memory().numTensors) behind a debug flag and watch it across a few hundred frames. If the count climbs, you have a leak, and the tab will die on a long session.
5. Get it off the main thread
Anything running per animation frame belongs in a Web Worker. Move model loading and classify() into the worker, post ImageBitmap objects in (they are transferable, so there is no copy), and post small typed arrays back. The UI stays responsive even when a frame takes longer than the budget, and you avoid the jank that makes a fast model feel slow.
Path B: LiteRT.js
If your team already produces .tflite artifacts for mobile - the pipeline from our LiteRT migration guide - LiteRT.js lets the web app run that same artifact with a WebGPU-accelerated runtime, instead of maintaining a second conversion path.
npm i @litertjs/core @litertjs/tfjs-interop
import { loadLiteRt, loadAndCompile } from '@litertjs/core';
await loadLiteRt('/wasm/'); // serve the wasm assets yourself
const model = await loadAndCompile('/model/vision.tflite', {
accelerator: 'webgpu', // 'wasm' for CPU fallback
});
const outputs = model.run(inputTensors);
The interop package lets you keep using tf.browser.fromPixels and the rest of the TF.js tensor API for pre- and post-processing while LiteRT executes the graph. Two practical notes: you must serve the WASM binaries from your own origin with the right MIME type, and coverage of exotic ops is narrower than TF.js graph mode - validate your specific model early rather than at the end of the sprint.
Choosing between them: one artifact for mobile and web, or a model already tuned for the edge, points to LiteRT.js. A model that only ever runs on the web, or one using ops outside the LiteRT set, points to TensorFlow.js.
Verify parity with the server
A converted model is a new model until proven otherwise. Before shipping, run a golden set - 200-500 representative inputs - through the Python model and the browser model, and diff:
const maxAbsDiff = Math.max(...browserProbs.map((v, i) => Math.abs(v - pythonProbs[i])));
Expect ~1e-5 for float32, ~1e-2 for float16, and larger for uint8. What you are hunting is not the numeric noise but the systematic gap: a resize with different interpolation, a channel order flip, a normalization constant that lives in your Python data pipeline and never made it into the JS preprocessing. That last one is the single most common cause of "the model works locally but is wrong in the browser", and it is why we recommend baking preprocessing into the exported graph wherever the runtime allows.
Budgets to hold yourself to
From projects that shipped and stayed shipped:
- Weights over the wire: under 25 MB is comfortable, 25-100 MB needs a loading state and a cache story, over 100 MB should be a server model.
- Cold start to first inference: under 1.5 s on a mid-range laptop, including download from cache.
- Steady-state inference: under 33 ms for anything driven by video at 30 fps; under 100 ms for click-driven interactions.
- Peak GPU memory: keep well under 1 GB. Mobile browsers will kill the tab, silently, with no error you can catch.
Serve the model files with long-lived immutable cache headers and a version in the path (/model/v7/model.json), so a re-visit costs nothing and a rollout is a URL change.
Where it fits in the stack
In-browser inference is not a replacement for a serving tier - it is a tier of its own. Most of the systems we build end up with both: a small quantized model in the browser for instant feedback, and the full model behind TensorFlow Serving for the cases that need accuracy or auditability. The interesting design work is deciding which requests are which.
If you are weighing a client-side deployment and want a second opinion on model size, browser coverage, or the privacy argument you are about to make to a regulator, get in touch - our Edge AI & LiteRT practice does this scoping work as a short fixed-scope engagement.