How to Set Up WhyLabs Model Monitoring with whylogs

October 27, 2026

WhyLabs is a monitoring platform for machine learning models in production. It tracks data drift, data quality, and model performance by ingesting statistical profiles generated locally by the open-source whylogs library. Model setup happens in the WhyLabs dashboard under Models → New Model.

The architecture matters: your data does not leave your infrastructure. whylogs computes a compact statistical sketch of each batch and only that sketch is uploaded. For teams who cannot ship raw PII to a third party, this is usually the deciding feature.

What does whylogs actually compute?

For each column, whylogs records a profile: counts, nulls, min/max, quantiles, distinct estimates, and a histogram for numeric data or a frequent-items list for categoricals. Typical profile size is a few kilobytes per batch regardless of row count — it uses a mergeable sketch structure, so it does not scale with data volume.

That is why uploading is cheap. A million-row batch produces roughly the same profile size as ten thousand rows.

How do you install and create profiles?

Start in Python:

  1. pip install whylogs — current 1.x line. The 0.x API is different; do not follow old blog posts.
  2. Build a session and log a reference dataset (typically a training or validation sample).
  3. Write the profile to disk as a file that WhyLabs ingests.

A minimal pattern: create whylogs.logger, call logger.log(df) on your reference dataframe, then profile.writer(path).write(). For inference batches, repeat with the same schema, one profile per batch.

The whylogs Python package ships a writer for WhyLabs upload — you pass your org ID, dataset ID, and an API key from Settings → Access Tokens.

How do you structure datasets in WhyLabs?

One WhyLabs dataset per thing you monitor. Common patterns:

Dataset Contains
model_input The features your model receives at inference
model_output Predictions and scores
model_performance Joined actuals, once labels arrive

The reason to split: input drift and output drift mean different things. If inputs drift but outputs do not, your model is robust to that shift. If outputs drift with stable inputs, something changed in the model or the deployment.

Set the first batch as the reference profile. WhyLabs compares every subsequent batch against it. Choosing a bad reference — like a week when your traffic was 80% bots — poisons every alert afterwards.

What drift thresholds should you set?

WhyLabs default alerting uses a drift distance metric per column, and the defaults are too sensitive for high-cardinality categoricals. Two adjustments worth making early:

  • Exclude ID-like columns. User IDs and timestamps drift constantly by definition and generate permanent alerts. Mark them as non-monitored.
  • Raise the threshold on low-importance features. A 5% drift in a feature with 0.01 importance is not worth a page.

Use the monitor config under Models → [Model] → Monitors. Start with 3–5 monitors: input drift on your top features, null rate on any column that must not be null, and output distribution.

How long before you know it is working?

Backfill if you have historical batches, or let it run two weeks. The first useful output is not an alert — it is the column drift ranking. You will see which of your features actually move. Often two or three features account for all the real drift, and the rest are stable noise.

Performance monitoring is the slow one. If labels arrive monthly, you have no performance signal for a month. Plan for that: use input drift as the leading indicator and treat performance as the lagging confirmation.

What are the real limitations?

  • Not real-time by default. Profile upload is batch-oriented. Sub-minute streaming monitoring is not the design centre.
  • Performance monitoring needs labels. If your labels come from a manual review process, expect a pipeline job, not an integration.
  • Structured data focus. Text, image, and embedding drift are supported to varying degrees but are not where the product is strongest. For LLM-specific monitoring, other tools in this space are more purpose-built.
  • Dashboard, not an incident tool. WhyLabs alerts; routing to PagerDuty is yours to wire.

Who should use it

Teams running tabular models in production with compliance constraints on data leaving their environment — credit, healthcare, insurance. The privacy-preserving architecture is genuinely unusual and it is the reason to pick WhyLabs over a platform that wants your raw rows.

Teams with a single model and a part-time ML engineer should probably start with a Grafana dashboard over batch statistics. WhyLabs pays off when you have several models to watch and no one whose full-time job is watching them.

Key Takeaways

  • WhyLabs ingests statistical profiles from the open-source whylogs library, not raw data.
  • Set a meaningful reference profile from a clean period; every alert compares against it.
  • Split datasets into input, output, and performance — drift means different things in each.
  • Exclude ID-like columns from drift monitoring or you will get permanent alerts.
  • Performance monitoring lags by however long your labels take to arrive.
Tags: