Every ML team we work with eventually gets an audio request. It arrives disguised as something else: "can the app respond to a wake word without sending audio to the cloud?", "can we transcribe these support calls and tag them?", "can we detect a failing pump from the microphone on the maintenance tablet?" Audio is the one modality most TensorFlow teams have never shipped, so it gets treated as exotic. It is not. An audio model is a vision model with a different front end — turn the waveform into a spectrogram, and the rest of the stack you already know applies.
This tutorial builds two things on the Keras 3 / TensorFlow stack: a small keyword spotting classifier that runs on-device through LiteRT, and a speech-to-text path using a pretrained encoder-decoder. Along the way it covers the parts that actually break audio projects — resampling, framing, augmentation, and the fact that your preprocessing must live inside the exported graph.
Setup
pip install -U tensorflow keras keras-hub tensorflow-io-gcs-filesystem
We use TensorFlow as the backend here because the audio ops (tf.signal) and the export path to LiteRT are TensorFlow-native:
import os
os.environ["KERAS_BACKEND"] = "tensorflow"
import tensorflow as tf
import keras
import numpy as np
Step 1: decode audio deterministically
The single most common bug in audio pipelines is a sample-rate mismatch between training and serving. Your training WAVs are 16 kHz; the phone hands you 44.1 kHz; nobody notices until accuracy is mysteriously 20 points lower in production. Fix the rate once, at the edge of the pipeline, and assert it:
TARGET_SR = 16000
CLIP_SECONDS = 1.0
CLIP_SAMPLES = int(TARGET_SR * CLIP_SECONDS)
def load_wav_16k(path):
audio_binary = tf.io.read_file(path)
waveform, sr = tf.audio.decode_wav(audio_binary, desired_channels=1)
tf.debugging.assert_equal(sr, TARGET_SR, message="resample upstream to 16 kHz")
waveform = tf.squeeze(waveform, axis=-1) # mono, float32 in [-1, 1]
return waveform
tf.audio.decode_wav does not resample. If your corpus is mixed, resample offline (ffmpeg, librosa) as a dataset-prep step rather than inside tf.data — it is cheaper and it keeps the graph simple. Then pad or crop every clip to a fixed length:
def fix_length(waveform, samples=CLIP_SAMPLES):
waveform = waveform[:samples]
return tf.pad(waveform, [[0, samples - tf.shape(waveform)[0]]])
For keyword spotting, fixed-length one-second clips are the standard framing — the Speech Commands dataset is built that way, and it is the right place to start a prototype.
Step 2: waveform to log-mel spectrogram
A short-time Fourier transform turns the waveform into a time-frequency image; a mel filterbank compresses the frequency axis the way human hearing does; a log makes the dynamic range trainable.
FRAME_LENGTH = 400 # 25 ms at 16 kHz
FRAME_STEP = 160 # 10 ms hop
FFT_LENGTH = 512
N_MELS = 64
mel_matrix = tf.signal.linear_to_mel_weight_matrix(
num_mel_bins=N_MELS,
num_spectrogram_bins=FFT_LENGTH // 2 + 1,
sample_rate=TARGET_SR,
lower_edge_hertz=20.0,
upper_edge_hertz=7600.0,
)
def log_mel(waveform):
stft = tf.signal.stft(
waveform, frame_length=FRAME_LENGTH,
frame_step=FRAME_STEP, fft_length=FFT_LENGTH,
)
magnitude = tf.abs(stft) # (frames, bins)
mel = tf.tensordot(magnitude, mel_matrix, axes=1)
return tf.math.log(mel + 1e-6)[..., tf.newaxis] # (frames, mels, 1)
Two numbers worth understanding rather than copying: 25 ms frames with a 10 ms hop is the near-universal default for speech, because speech is roughly stationary over that window. 1e-6 inside the log is a floor that stops silent frames from producing -inf; if your loss goes NaN on the first step, this is the first place to look.
Note that log_mel is pure TensorFlow ops. That matters in step 6.
Step 3: build the tf.data pipeline with augmentation
Audio augmentation is where small datasets become adequate datasets. The three that pay for themselves:
- Random time shift — the keyword rarely starts exactly at t=0 in the wild.
- Background noise mixing — add café, road, or fan noise at a random SNR.
- SpecAugment — mask random frequency bands and time steps in the spectrogram itself.
def random_shift(waveform, max_shift=1600): # up to 100 ms
shift = tf.random.uniform([], -max_shift, max_shift, dtype=tf.int32)
return tf.roll(waveform, shift, axis=0)
def mix_noise(waveform, noise, snr_db_range=(5.0, 20.0)):
snr_db = tf.random.uniform([], *snr_db_range)
sig_p = tf.reduce_mean(waveform ** 2) + 1e-9
noise_p = tf.reduce_mean(noise ** 2) + 1e-9
scale = tf.sqrt(sig_p / (noise_p * tf.pow(10.0, snr_db / 10.0)))
return tf.clip_by_value(waveform + scale * noise, -1.0, 1.0)
def spec_augment(spec, freq_mask=8, time_mask=12):
f = tf.random.uniform([], 0, freq_mask, dtype=tf.int32)
f0 = tf.random.uniform([], 0, N_MELS - f, dtype=tf.int32)
spec = tf.tensor_scatter_nd_update(
tf.transpose(spec, [1, 0, 2]),
tf.range(f0, f0 + f)[:, None],
tf.zeros((f, tf.shape(spec)[0], 1)),
)
return tf.transpose(spec, [1, 0, 2])
Wire it together with the same tf.data discipline we cover in tf.data Pipelines That Keep Your GPU Busy — augment on CPU, prefetch, and cache the decode step if the corpus fits in memory:
AUTOTUNE = tf.data.AUTOTUNE
def make_ds(paths, labels, training):
ds = tf.data.Dataset.from_tensor_slices((paths, labels))
ds = ds.map(lambda p, y: (fix_length(load_wav_16k(p)), y), num_parallel_calls=AUTOTUNE)
ds = ds.cache()
if training:
ds = ds.shuffle(4096)
ds = ds.map(lambda w, y: (random_shift(w), y), num_parallel_calls=AUTOTUNE)
ds = ds.map(lambda w, y: (log_mel(w), y), num_parallel_calls=AUTOTUNE)
if training:
ds = ds.map(lambda s, y: (spec_augment(s), y), num_parallel_calls=AUTOTUNE)
return ds.batch(64).prefetch(AUTOTUNE)
Split by speaker, not by clip. If the same voice appears in train and validation, your validation score is measuring memorisation and will not survive contact with new users. This is the audio equivalent of the time-leak problem in forecasting.
Step 4: a keyword spotting model that fits on a microcontroller-class budget
Once you have (frames, mels, 1) tensors, this is image classification. A small depthwise-separable CNN is the workhorse — it is what most production wake-word models still are, because it is tiny and it quantizes cleanly:
def build_kws(num_classes, input_shape):
inputs = keras.Input(shape=input_shape)
x = keras.layers.Normalization()(inputs) # adapt() on training spectrograms
x = keras.layers.Conv2D(32, 3, strides=2, padding="same", use_bias=False)(x)
x = keras.layers.BatchNormalization()(x)
x = keras.layers.ReLU()(x)
for filters in (64, 64, 128):
x = keras.layers.SeparableConv2D(filters, 3, padding="same", use_bias=False)(x)
x = keras.layers.BatchNormalization()(x)
x = keras.layers.ReLU()(x)
x = keras.layers.MaxPooling2D()(x)
x = keras.layers.GlobalAveragePooling2D()(x)
x = keras.layers.Dropout(0.3)(x)
outputs = keras.layers.Dense(num_classes, activation="softmax")(x)
return keras.Model(inputs, outputs)
model = build_kws(num_classes=12, input_shape=(98, N_MELS, 1))
model.compile(
optimizer=keras.optimizers.AdamW(1e-3),
loss="sparse_categorical_crossentropy",
metrics=["accuracy"],
)
model.fit(train_ds, validation_data=val_ds, epochs=30,
callbacks=[keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True)])
Do not skip the _silence_ and _unknown_ classes. A wake-word model trained only on your ten target words will confidently fire on anything. Roughly 10% silence and 10% out-of-vocabulary speech in the training mix is a reasonable starting ratio, tuned against your false-accept budget.
The metric that matters is not accuracy
Top-1 accuracy on a balanced test set tells you almost nothing about a wake word. The operating metric is false accepts per hour at a fixed false reject rate. Run the model over hours of continuous background audio containing no keywords, count the firings, and tune the detection threshold and the smoothing window against that curve:
probs = model.predict(streaming_windows) # sliding 1 s windows, 100 ms hop
smoothed = np.convolve(probs[:, KEYWORD_ID], np.ones(5) / 5, mode="same")
fires = (smoothed > THRESHOLD).sum()
print("false accepts/hour:", fires / hours_of_audio)
Posterior smoothing across consecutive windows removes most single-frame spikes and is the cheapest accuracy win available.
Step 5: speech-to-text without training from scratch
Nobody should train an ASR model from scratch in 2026 unless the language or domain genuinely has no pretrained coverage. Use a pretrained encoder-decoder and fine-tune it on your audio if the domain vocabulary (drug names, part numbers, ticker symbols) is unusual. KerasHub ships Whisper-family speech-to-text models with matching audio preprocessing:
import keras_hub
asr = keras_hub.models.WhisperBackbone.from_preset("whisper_small_en")
# or the task-level wrapper, which bundles the audio feature extractor and tokenizer
Check the current KerasHub model list for the exact preset names and task classes available in your installed version — they move between releases, and pinning the version in your lockfile is the difference between a reproducible pipeline and a Monday morning surprise.
Practical guidance we give clients on ASR:
- Measure WER on your own audio before choosing anything. Vendor benchmarks are read speech in quiet rooms; your audio is a phone call in a warehouse. The ranking often inverts.
- Domain vocabulary is usually a decoding problem, not a training problem. Biasing the decoder toward a term list fixes more errors, faster and cheaper, than fine-tuning does.
- Diarisation, punctuation, and timestamps are separate models. Budget for them; they are frequently what the customer actually meant by "transcription".
- If audio is sensitive, on-prem inference may be a requirement rather than a preference — see Training on Sensitive Data for the surrounding controls.
Step 6: export with preprocessing inside the graph
Here is where audio projects go wrong at the finish line. If your Python client has to compute the log-mel spectrogram before calling the model, then every client — Android, iOS, the Python batch job — has to reimplement tf.signal.stft with identical parameters. They will not. Bake the front end into the exported model so its input is a raw waveform:
waveform_in = keras.Input(shape=(CLIP_SAMPLES,), dtype="float32", name="waveform")
spec = keras.layers.Lambda(lambda w: tf.map_fn(log_mel, w))(waveform_in)
logits = model(spec)
serving_model = keras.Model(waveform_in, logits)
converter = tf.lite.TFLiteConverter.from_keras_model(serving_model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
def rep_data():
for waveforms, _ in raw_val_ds.take(200):
yield [tf.cast(waveforms, tf.float32)]
converter.representative_dataset = rep_data
converter.target_spec.supported_ops = [
tf.lite.OpsSet.TFLITE_BUILTINS_INT8,
tf.lite.OpsSet.SELECT_TF_OPS, # FFT ops are not all builtin
]
tflite_model = converter.convert()
open("kws_int8.tflite", "wb").write(tflite_model)
Two caveats worth knowing before you promise a delivery date:
- FFT ops may pull in Select TF ops, which grows the runtime binary. On a phone that is fine. On a microcontroller it is not, and you will need to move the spectrogram into the device firmware and export a spectrogram-input model instead.
- Quantize the classifier, keep the front end in float. Int8 quantization of an STFT rarely goes well. Use a representative dataset of real waveforms so the activation ranges are calibrated on real signal levels — the method is the same one in Shrinking TensorFlow Models.
Verify on-device numerics before shipping. Run the same 500 clips through the Keras model and the LiteRT interpreter, and compare predicted classes, not just logits:
interp = tf.lite.Interpreter(model_content=tflite_model)
interp.allocate_tensors()
inp, out = interp.get_input_details()[0], interp.get_output_details()[0]
agree = 0
for w, y in zip(check_waveforms, check_labels):
interp.set_tensor(inp["index"], w[None].astype(np.float32))
interp.invoke()
agree += int(np.argmax(interp.get_tensor(out["index"])) == np.argmax(serving_model(w[None])))
print("agreement:", agree / len(check_labels))
Anything below ~99% agreement means a quantization or op-compatibility problem, not noise. The deployment mechanics — delegates, benchmarking, model updates in the field — are covered on our Edge AI & LiteRT Deployment page and in Migrating from TensorFlow Lite to LiteRT.
Monitoring audio models in production
Audio drifts faster than most teams expect: a new hardware revision changes the microphone frequency response, a firmware update adds noise suppression, a product launch adds a new accent mix. Log spectrogram summary statistics (mean energy per mel band, clipping rate, estimated SNR) alongside predictions, and alert on distribution shift the same way you would for tabular features — see Catching Data Drift Early. Log raw audio only where your privacy policy and consent flow genuinely permit it, and keep a sampled, labelled regression set that every new model must clear before rollout.
When to call someone
Audio projects fail on data logistics far more often than on modelling. If you have no labelled audio, no consented recording path, or no way to measure false accepts per hour on real background audio, the model is not your bottleneck. DeepAdvisors' TensorFlow consultants have shipped keyword spotting, acoustic anomaly detection, and transcription pipelines on the TensorFlow and LiteRT stack, and we are usually blunt about which of the three you actually need. If you have an audio problem and a deadline, get in touch and we will scope it honestly.