Data Failures That Break Models

When a UEBA deployment underperforms, the instinct is to blame the model. In practice the cause is almost always upstream, and the failure mode is silence rather than error.

The failures, in order of frequency

A source stops sending. An agent stops, a forwarder fills its disk, a firewall rule changes, an API credential expires. Events simply stop arriving. Every feature depending on that source becomes constant, the entities it covered stop generating anomalies, and the system reports normal.

This is the most common and most damaging failure in the field. It is detected only by monitoring for absence.

Partial coverage. The source is sending, from eighty percent of hosts. The other twenty percent are unmonitored and nobody knows which.

Duplicate events. The same event arriving twice through two paths inflates every volume feature. Baselines shift, and an entity whose events are duplicated appears twice as active as it is.

Clock skew. A source whose clock is wrong places events in the wrong window. Sequence-based analytics break entirely.

Retention shorter than the baseline window. Configured for ninety days, retained for thirty. The model trains on what exists, which is a third of what was intended, and reports nothing unusual.

Sampling. Some sources sample under load. A feature counting events is then counting a fraction of them, varying with volume.

Backfill. A source recovering from an outage delivers a day of events at once. Every volume feature spikes for every entity simultaneously.

Field-level nulls. The event arrives, one field is empty. Features using it silently exclude those events.

Why it is silent

Analytics pipelines are built to be resilient. Bad events are dropped, missing fields are defaulted, failed parses are counted and discarded. Each decision is individually reasonable and collectively produces a system that degrades without complaining.

The model has no way to distinguish "this entity did nothing unusual" from "this entity's data stopped arriving". Both look like an absence of anomalies, and absence of anomalies is what success looks like on a dashboard.

The monitoring that catches it

Four measures, checked daily, catch the large majority:

Event volume per source, against its own baseline. Alert on absence and on drops beyond normal variance. This is behavioural analytics applied to your own pipeline, and it is the highest-value monitoring in the deployment.

Distinct entity count per source per day. Catches entity resolution and coverage failures that volume alone misses.

Field completeness per source. Percentage of events with each critical field populated.

Event timestamp lag. The distribution of arrival time minus event time. A widening tail means a delivery problem; a bimodal distribution means backfill.

None of these requires the UEBA product. All can be built on the log platform in a day.

Coverage as a first-class metric

Report what fraction of your estate each source actually covers:

Hosts with the endpoint agent, against the asset inventory. Cloud subscriptions with audit logging enabled, against the subscription list. Applications with logging configured, against the application inventory.

The gap between "deployed" and "covering everything" is routinely twenty percent or more, and it is almost never on the dashboard because the dashboard reports what arrived, not what should have.

Before blaming the model

When output looks wrong, check in this order:

Is every source still delivering? Has volume per source changed? Has the distinct entity count changed? Are timestamps sane? Did a source change format recently? Did coverage change — a migration, a decommission, a new subnet?

Most investigations into model quality end at one of these six questions, and the remaining ones are much easier to reason about once the data is known to be intact.

A daily pipeline check

Four numbers, on one screen, reviewed each morning. This is the cheapest meaningful control in a UEBA deployment.

Events per source, against that source's own weekly baseline. Alert on absence and on any drop beyond normal variance.

Distinct entities per source. Catches coverage and resolution failures that volume alone hides — a source can maintain volume while covering half the hosts it did last week.

Field completeness for the three or four fields your features depend on.

Timestamp lag distribution. A widening tail means delivery problems; a second cluster means backfill.

Any of these moving is a reason to check before interpreting the day's output. Most mornings they are flat and the check takes thirty seconds.

Common false positives

Data defects are the largest single source of apparent anomalies, and they are recognisable:

Backfill producing a synchronised volume spike across many entities at once. The tell is that it is population-wide rather than entity-specific.

Duplicate ingestion doubling one entity's volume with no change in distinct-resource counts.

A source resuming after an outage, where the first day back contains several days of activity.

Agent upgrades changing the granularity of what is reported, so counts shift without behaviour changing.

Clock correction on a host, moving a block of events into a different window.

Checking the four numbers above distinguishes all of these from behaviour in under a minute.

Blind spots and assumptions

That the platform monitors itself. Most monitor their own health, not the completeness of what they receive.

That someone would notice. Nobody notices the absence of alerts.

That backfill is harmless. It corrupts baselines for the period it covers, and rolling baselines carry that corruption forward.

That duplicate detection is on. Frequently it is not, and it is not free to enable.