Statistical Baselines vs Machine Learning

Products are sold on their machine learning. Practitioners frequently respond that a percentile threshold would work as well. Both positions contain something true, and the useful version of the argument is more specific.

What simple statistics actually achieve

A per-entity, per-feature model holding a median and a high percentile catches:

Volume anomalies. Access to systems never accessed before. Authentication from countries never seen. Activity outside a range of hours never previously used. Counts of distinct resources beyond anything historical.

That list covers a large share of real detections in production deployments. Simple methods are also transparent, cheap, debuggable, and explainable to an analyst, an employee and a regulator.

A deployment that implements only this, on well-normalised data with correct entity resolution, will outperform a sophisticated model on poor data. That is the single most important fact in this subject and it is not disputed by anyone who has run both.

What simple statistics miss

Feature combinations. Each feature is individually within range, and the combination is unprecedented. Moderate volume, plus a system rarely touched, plus an unusual hour, plus a new source address. Nothing crosses a threshold; the joint pattern is novel.

Univariate models cannot see this. This is the genuine argument for multivariate methods and it is a real gap.

Sequence. The order of operations carries information — reconnaissance, then collection, then transfer. A model over independent counts discards ordering entirely.

Relationships. Which entities interact with which. Lateral movement is a property of the graph, not of any node.

Rarity across a population. "How unusual is this value across everyone" is different from "how far is it from this entity's mean", and it is frequently the more useful question.

What machine learning genuinely adds

Density estimation over feature space. Isolation forests, autoencoders and similar methods score how unusual a combination is, not how unusual each part is. This is the real capability that thresholds cannot replicate.

Learned peer grouping. Clustering entities by behaviour rather than by org chart, which is frequently more accurate.

Sequence models. Where ordering matters and the alphabet of events is manageable.

Graph methods. Relationship anomalies — a host talking to a host it never talks to, an account used from a device it never uses.

What it does not add

Intent. No model infers purpose. It infers unusualness.

Precision on its own. Unsupervised methods produce a ranking. Whether the top of that ranking is worth an analyst's time is determined by data quality and enrichment, not by the algorithm.

Freedom from tuning. Learned models have hyperparameters, contamination assumptions and retraining schedules. They trade explicit configuration for implicit configuration, which is harder to audit.

Explainability, without deliberate work. A z-score explains itself. An autoencoder reconstruction error does not, and if a finding may lead to a consequence for a person, that difference is decisive.

The honest sequencing

Start statistical. Deploy per-entity percentile models on a small feature set. Measure what they catch. This establishes the floor and, crucially, exposes data problems that a complex model would absorb silently.

Add multivariate methods for the combination gap, once the data is trusted and the simple detections are tuned.

Add sequence and graph methods last, and only where you can articulate what they are for.

Deployments that begin with the sophisticated model and never establish the floor cannot tell whether the model is contributing anything, because they have nothing to compare against.

The evaluation nobody runs

Run both in parallel for a month. Compare what each surfaces.

If the learned model finds nothing the percentile model missed, you have learned something valuable and cheap. If it does, you can name what kind of thing and defend the additional complexity.

Very few deployments do this, which is why the argument between the two positions is conducted with assertion rather than evidence.

Running the comparison

The evaluation that settles the argument in your environment, and it costs one month.

Implement percentile baselines on a dozen features. A few days of work on a platform that already holds the data.

Run them alongside whatever learned model you have. Same period, same entities, same enrichment.

Compare the top fifty from each. How much overlap? What does each surface that the other does not?

Have analysts adjudicate both sets blind, without knowing which model produced which alert.

Then ask the specific question: what did the learned model find that the simple one missed, and can you characterise it? If the answer is a category — feature combinations, sequences, relationship anomalies — the complexity is justified and you know what it buys. If there is no answer, you have saved yourself an operating cost.

Common false positives

Each approach has characteristic noise, which is useful for diagnosis:

Statistical models produce noise from skewed distributions treated as normal, from single extreme values dominating a mean, and from cycles absorbed into a baseline.

Learned models produce noise from contamination assumptions set wrongly, from high dimensionality flattening the distinction between points, and from correlated features inflating a joint anomaly score.

Both produce noise from the same underlying causes: immature baselines, poor entity resolution, and organisational change nobody signalled.

If the noise profile of a sophisticated model resembles that of a simple one, the model is probably reporting data problems rather than behaviour.

Blind spots and assumptions

That complexity implies capability. It implies more parameters and more ways to be wrong quietly.

That unsupervised means unconfigured. Contamination rates, window sizes and feature selection are all choices, made by someone, usually the vendor, on someone else's data.

That the model handles data problems. It does not. It absorbs them and produces confident output.

That explainability can be added later. For anything touching a person, it is a requirement at design time, and retrofitting it to a model chosen without it in mind is frequently not possible.