Catching Data Drift Early with TensorFlow Data Validation

Every model we are asked to rescue has the same story. It launched, it worked, and then over a few months it quietly got worse. Nobody changed the model. The infrastructure dashboards stayed green the whole time. What changed was the data: an upstream team renamed a category, a mobile client started sending nulls, a marketing campaign shifted the user mix, a currency field switched from cents to dollars.

Our TensorFlow Serving tutorial ends with the advice to watch input distributions. This tutorial is how you actually do it, with TensorFlow Data Validation (TFDV) — the piece of the TFX ecosystem that computes dataset statistics, infers a schema, and tells you when today's data no longer looks like the data you trained on.

What TFDV gives you

Three things, and they map onto three distinct failure modes:

TFDV concept	Catches	Typical cause
Schema validation	Missing feature, wrong type, unexpected category value	Upstream contract change
Skew comparison	Training data ≠ serving data	Different preprocessing paths
Drift comparison	Yesterday's serving data ≠ today's	The world changed

The point is to turn all three into a build-style pass/fail, not a chart somebody looks at when they remember to.

Install

pip install tensorflow-data-validation

TFDV pins closely to TensorFlow and Apache Beam versions. Install it into the same locked environment as your training job — mismatched pins here are one of the most common installation headaches, and the same discipline we recommend in the 2.21 upgrade guide applies.

1. Compute statistics over the training set

import tensorflow_data_validation as tfdv

train_stats = tfdv.generate_statistics_from_csv("data/train.csv")
# or, for the format you are probably already training from:
# tfdv.generate_statistics_from_tfrecord("data/train-*.tfrecord")
tfdv.visualize_statistics(train_stats)

visualize_statistics renders in a notebook and is worth five minutes of your attention the first time: it shows missing-value rates, min/max, and the category distribution for every feature. Half the data bugs we find on new engagements are visible in that first render — a "percentage" feature with a max of 10000, a categorical with 40 % empty strings.

For anything bigger than a laptop's memory, the same call runs on Beam:

tfdv.generate_statistics_from_tfrecord(
    "gs://bucket/train-*.tfrecord",
    pipeline_options=beam_options,   # DataflowRunner, Spark, etc.
)

2. Infer, then edit, the schema

schema = tfdv.infer_schema(train_stats)
tfdv.display_schema(schema)

Inference is a starting draft, never the final artifact. It only knows what it saw in one sample. Tighten it by hand:

from tensorflow_metadata.proto.v0 import schema_pb2

# This feature must always be present
tfdv.get_feature(schema, "user_id").presence.min_fraction = 1.0

# Optional feature we tolerate missing 10 % of the time
tfdv.get_feature(schema, "referrer").presence.min_fraction = 0.9

# Numeric range you know from the domain
tfdv.set_domain(schema, "age", schema_pb2.IntDomain(name="age", min=13, max=120))

# Allow new payment methods to appear without failing the build
tfdv.get_feature(schema, "payment_method").distribution_constraints.min_domain_mass = 0.95

tfdv.write_schema_text(schema, "schema.pbtxt")

Then commit schema.pbtxt to version control. This is the whole trick. The schema is a data contract, changes to it show up in code review, and "who allowed nulls in this column?" becomes a question with an answer in the git log.

3. Validate every new batch against the schema

new_stats = tfdv.generate_statistics_from_csv("data/2026-02-14.csv")
anomalies = tfdv.validate_statistics(statistics=new_stats, schema=schema)
tfdv.display_anomalies(anomalies)

In a pipeline you want a boolean, not a table:

if anomalies.anomaly_info:
    for feature, info in anomalies.anomaly_info.items():
        print(f"{feature}: {info.short_description} — {info.description}")
    raise SystemExit("Data validation failed; refusing to train.")

Typical output on a real feed:

payment_method: Unexpected string values — Examples contain values missing from the schema: apple_pay.
session_length: Out-of-range values — Unexpectedly large value 98213.
country: High number of missing values — Column dropped below min_fraction 0.99.

The first is benign and you fix it by updating the schema in a pull request. The second and third are bugs upstream, and training on them would bake the bug into the model.

4. Detect training/serving skew

Skew is the difference between the data you trained on and the data arriving at the endpoint. Log a sample of real serving requests (post-decode, pre-model) and compare:

serving_stats = tfdv.generate_statistics_from_tfrecord("logs/serving-sample-*.tfrecord")

tfdv.get_feature(schema, "device_type").skew_comparator.infinity_norm.threshold = 0.01
tfdv.get_feature(schema, "price").skew_comparator.jensen_shannon_divergence.threshold = 0.05

skew = tfdv.validate_statistics(
    statistics=serving_stats, schema=schema, previous_statistics=train_stats
)
tfdv.display_anomalies(skew)

Use the infinity norm for categorical features (it flags the single most-shifted category) and Jensen–Shannon divergence for numeric ones. Thresholds are empirical: run the comparison over a few weeks of known-good history, look at the values you get on ordinary days, and set the threshold above that band. A threshold copied from a tutorial will either never fire or fire every morning, and an alert that fires every morning is not an alert.

The best fix for skew is structural, not statistical: bake preprocessing into the exported graph so there is only one code path. That is exactly why the serving tutorial insists on it.

5. Detect drift over time

Drift is the same comparison run against yesterday's data instead of the training data:

tfdv.get_feature(schema, "product_category").drift_comparator.infinity_norm.threshold = 0.01

drift = tfdv.validate_statistics(
    statistics=today_stats, schema=schema, previous_statistics=yesterday_stats
)

Two practical notes. First, compare like windows — a Monday against a Monday, a full day against a full day — or seasonality will masquerade as drift. Second, drift is a signal to investigate, not to retrain automatically. Retraining on drifted data that turned out to be an upstream bug propagates the bug into the model, and now you have two problems.

6. Wire it into the pipeline

The minimum viable version, running as a scheduled job before training:

raw data ──> generate_statistics ──> validate_statistics(schema)
                                        │
                              anomalies?─┴─ no ──> train ──> export ──> serve
                                        │
                                       yes ──> fail the run, page the data owner

If you are already on TFX, this is three standard components:

from tfx import v1 as tfx

example_gen = tfx.components.CsvExampleGen(input_base=DATA_ROOT)
statistics_gen = tfx.components.StatisticsGen(examples=example_gen.outputs["examples"])
schema_importer = tfx.dsl.Importer(
    source_uri=SCHEMA_DIR, artifact_type=tfx.types.standard_artifacts.Schema
).with_id("schema_importer")
example_validator = tfx.components.ExampleValidator(
    statistics=statistics_gen.outputs["statistics"],
    schema=schema_importer.outputs["result"],
)

Note the importer: use your reviewed, committed schema rather than SchemaGen re-inferring one on every run. A schema that regenerates itself can never detect a violation — it just adapts to whatever arrived.

If you are not on TFX, do not adopt it just for this. TFDV is a plain Python library; a 60-line Airflow or Cloud Run job that computes stats, validates, writes the anomaly proto to object storage, and exits non-zero gets you 90 % of the value with none of the orchestration cost.

7. Keep the artifacts

For every run, store the statistics proto, the schema version used, and the anomalies proto, keyed by date and by the model version that consumed the data. Combined with logging the serving model version on every prediction, that gives you the ability to answer the question that arrives six months later: what did the data look like when this model was trained, and when did it stop looking like that? Without those artifacts, the honest answer is a shrug.

Cheap monitoring rule of thumb: statistics generation over a day of tabular data is minutes of CPU, costing far less than one bad retrain. There is no reason to sample aggressively until you are into billions of rows a day.

What TFDV does not do

It is a data monitor, not a model monitor. It will not tell you accuracy dropped — for that you need labels, which usually arrive days or weeks later, and a separate continuous-evaluation job (TFMA, or your own metrics job) that scores each model version against labelled outcomes as they land. TFDV also will not catch semantic drift where the distribution is stable but the meaning changed: same range, same histogram, different definition. Only the data owner catches that, which is why the alert should page a human who knows the domain.

Where to start Monday morning

1. Generate statistics on your current training set and look at them.
2. Infer a schema, tighten it by hand, commit it.
3. Add a validation step in front of training that fails the run on anomalies.
4. Sample serving inputs, and add a weekly skew comparison.
5. Only then worry about thresholds, dashboards, and automated retraining.

Steps 1–3 usually take an afternoon and catch the majority of the incidents we get called about. If you would like help putting this in front of a model that is already in production — or a second opinion on why one quietly got worse — get in touch; this is a standard part of our TensorFlow ML consulting work.