Every team that fine-tunes an open-weight model eventually hits the same wall. Supervised fine-tuning (SFT) teaches the model what a good answer looks like — but only from examples you wrote. It has no way to learn what a bad answer looks like, and no way to learn the difference between two answers that are both plausible, where one is noticeably better. That difference is where most of the perceived quality of a production assistant lives: tone, refusal behaviour, citation discipline, verbosity, willingness to say "I don't know".
Preference tuning closes that gap. You collect pairs — a prompt, a chosen response, a rejected response — and train the model to raise the likelihood of the chosen one relative to the rejected one. Direct Preference Optimization (DPO) does this with a plain classification-style loss and no separate reward model and no RL loop, which is why it has become the default alignment step for teams working outside a frontier lab.
This tutorial builds that step on the Keras 3 / TensorFlow stack: dataset shape, a DPO loss you can read, a training loop over a LoRA-adapted Gemma, the evaluation you actually need, and the failure modes we see most often on client projects. It assumes you have already done an SFT pass — see our Gemma fine-tuning tutorial for that half.
Where preference tuning belongs in the pipeline
pretrained base -> SFT (instruction format, domain voice) -> preference tuning (ranking quality) -> eval + guardrails -> export
Do not skip SFT. DPO optimizes a relative objective; if the policy cannot produce a competent answer at all, making it relatively more likely than a bad answer buys nothing. The rule of thumb we use: SFT until the model reliably produces answers in the right shape, then preference-tune until it reliably picks the better of two right-shaped answers.
Step 1: the preference dataset
A DPO record is three fields:
{
"prompt": "Customer asks: my invoice shows tax twice. What do I reply?",
"chosen": "Apologise briefly, confirm the duplicate line, explain the credit note will issue within 2 business days, and give the ticket reference. Do not speculate about the cause.",
"rejected": "That's a known bug in our billing system, sorry! Our engineers are working on it."
}
Three practical sources, in descending order of quality:
- Production edits. If humans review or rewrite model output today, the pre-edit output is
rejectedand the shipped version ischosen. This is free, in-domain, and reflects your real quality bar. - Pairwise human ranking. Sample two responses per prompt (temperature ~0.9, same SFT model), have a reviewer pick. Cheaper per item than writing gold answers.
- AI feedback (RLAIF). A stronger model judges pairs against a written rubric. Fast and scalable, but it imports the judge's biases — length bias especially. Use it to bootstrap, then re-validate a sample by hand.
Volume: 2k–10k clean pairs moves a 2B–9B model noticeably. 500 careful in-domain pairs beat 50k scraped generic ones. Hold out 10% by prompt (never by row — the same prompt in train and eval leaks).
Tokenize each pair into a sequence that shares the prompt prefix:
import os
os.environ["KERAS_BACKEND"] = "jax" # or "tensorflow"
import keras, keras_hub, tensorflow as tf
MAX_LEN = 1024
policy = keras_hub.models.GemmaCausalLM.from_preset("gemma2_2b_en")
tokenizer = policy.preprocessor.tokenizer
def encode(prompt, response):
p = tokenizer(f"<start_of_turn>user\n{prompt}<end_of_turn>\n<start_of_turn>model\n")
r = tokenizer(f"{response}<end_of_turn>")
ids = (p + r)[:MAX_LEN]
# loss mask: score only the response tokens
mask = ([0] * len(p) + [1] * len(r))[:MAX_LEN]
pad = MAX_LEN - len(ids)
return ids + [0] * pad, mask + [0] * pad
The loss mask matters. DPO compares the log-probability of the responses; if you let the shared prompt tokens into the sum, you add an identical constant to both sides in the best case and noise in the worst.
Step 2: the reference model
DPO regularizes against a frozen reference policy — normally your SFT checkpoint. The loss only rewards moving away from the reference where preference data justifies it, which is what stops the model collapsing into degenerate high-reward text.
With LoRA you get the reference for free: the frozen base weights are the reference, and disabling the adapters recovers it. That halves memory versus loading two copies.
policy.backbone.enable_lora(rank=8) # only adapters train
If your framework version cannot toggle adapters mid-step, precompute reference log-probabilities once over the whole dataset and cache them as two extra columns. Preference sets are small; this is a few GPU-minutes and it removes a whole model from the training step.
Step 3: the DPO loss
Given sequence log-probabilities for chosen and rejected under the policy (pi) and the reference (ref):
import keras.ops as ops
def seq_logps(model, ids, mask):
logits = model(ids)[:, :-1, :]
targets = ids[:, 1:]
logp = ops.take_along_axis(
ops.log_softmax(ops.cast(logits, "float32"), axis=-1),
targets[..., None], axis=-1)[..., 0]
return ops.sum(logp * ops.cast(mask[:, 1:], "float32"), axis=-1)
def dpo_loss(pi_c, pi_r, ref_c, ref_r, beta=0.1, label_smoothing=0.0):
logits = beta * ((pi_c - ref_c) - (pi_r - ref_r))
loss = -(1 - label_smoothing) * ops.log_sigmoid(logits) \
- label_smoothing * ops.log_sigmoid(-logits)
return ops.mean(loss), logits
That is the whole algorithm. logits is the implicit reward margin: positive means the policy prefers the chosen response more strongly than the reference did.
beta is the one hyperparameter that matters. It controls how far the policy may drift from the reference. 0.1 is the standard starting point; 0.3–0.5 when your preference data is noisy or you are protecting a carefully tuned SFT voice; 0.01–0.05 only with clean, plentiful, in-domain pairs. Too low and the model degenerates (repetition, runaway length, forgotten domain facts); too high and nothing changes.
Set label_smoothing=0.05–0.1 (this is cDPO) when your labels come from AI feedback or from annotators you have not agreement-checked — it caps how much any single mislabelled pair can push the model.
Step 4: the training loop
optimizer = keras.optimizers.AdamW(learning_rate=5e-6, weight_decay=0.01,
global_clipnorm=1.0)
@tf.function
def train_step(batch):
with tf.GradientTape() as tape:
pi_c = seq_logps(policy, batch["chosen_ids"], batch["chosen_mask"])
pi_r = seq_logps(policy, batch["rejected_ids"], batch["rejected_mask"])
loss, margin = dpo_loss(pi_c, pi_r,
batch["ref_chosen_logp"], batch["ref_rejected_logp"])
vars_ = policy.trainable_variables # LoRA adapters only
optimizer.apply_gradients(zip(tape.gradient(loss, vars_), vars_))
return loss, ops.mean(ops.cast(margin > 0, "float32"))
Settings that hold up in practice:
| Knob | Value | Why |
|---|---|---|
| Learning rate | 1e-6 – 5e-6 | 10–50x lower than SFT; DPO destabilises easily |
| Epochs | 1–2 | Beyond 2, reward hacking dominates |
| Batch (pairs) | 16–64, via gradient accumulation | Small batches make the margin very noisy |
| LoRA rank | 8–16 | Higher rank mostly adds drift, not quality |
| Schedule | warmup 10%, then cosine | Early steps are the destabilising ones |
Build the tf.data input the usual way — shuffle → padded_batch → prefetch(AUTOTUNE); our tf.data tutorial applies unchanged here.
Step 5: read the right metrics
Loss going down tells you almost nothing. Log these instead:
- Accuracy (
margin > 0): fraction of held-out pairs the policy ranks correctly. Expect 0.55 at the start, 0.65–0.80 after a good run. Above ~0.95 on train with flat eval means you memorised the annotator. - Chosen and rejected rewards separately (
pi - reffor each). The healthy pattern is chosen roughly flat or slightly up, rejected clearly down. If both fall hard, the model is learning "say less" rather than "say better" — raisebeta. - KL proxy (mean
|pi - ref|): drift budget. A sharp rise precedes degeneration. - Mean response length: DPO's signature failure is length inflation, because longer answers are usually preferred. If length grows more than ~20% and win rate does not, you have length bias, not quality.
Then run a real win-rate eval: fixed prompt set, generate from SFT and DPO checkpoints, judge blind and position-swapped. Wire it into the harness from our generative AI evaluation and guardrailing tutorial, and keep a regression slice of domain-fact questions to catch capability loss — alignment gains routinely cost a point or two of task accuracy, and you want to see the trade, not discover it in production.
Variants worth knowing
- IPO replaces the log-sigmoid with a squared loss. More robust to near-deterministic preferences; a good fallback when DPO overfits at any
beta. - KTO trains on single responses labelled good/bad, no pairs. Use it when your feedback is thumbs-up/thumbs-down telemetry — far cheaper to collect.
- ORPO folds preference into SFT with an odds-ratio penalty, needs no reference model. Attractive when you have preference data before you start training and want one pass instead of two.
- Online / iterative DPO: sample from the current policy, label, retrain, repeat. Closes the distribution gap between training pairs and live output, and gives most of the remaining headroom — at the cost of a labelling loop that has to keep running.
Export and serve
Nothing special: merge the LoRA adapters, export a SavedModel, ship it the way you ship any other Gemma checkpoint — see serving TensorFlow models in production. Keep the SFT checkpoint deployable behind a flag for at least one release; preference regressions are qualitative and often surface days after rollout.
policy.backbone.merge_lora() # fold adapters into the base weights
policy.save_to_preset("gemma2-2b-dpo-v1")
Version the preference dataset alongside the checkpoint. When someone asks in six months why the assistant started hedging, the answer is almost always in the pairs, not in the code.
The honest summary
DPO is cheap to run and easy to get subtly wrong. The code is fifty lines; the work is the data — pair quality, prompt coverage, annotator agreement — and the evaluation discipline to tell real improvement from length inflation. Budget your effort accordingly: one day on the loop, two weeks on the pairs.
If you are weighing SFT against preference tuning, building the labelling pipeline, or trying to work out why an aligned model got worse at its actual job, get in touch. Our TensorFlow consultants build and evaluate these pipelines on client infrastructure, and we are happy to review a plan before you spend GPU budget on it.