Labels: Using Analyst Judgement

Supervised learning is largely unavailable in UEBA because labelled attack data does not exist. But a partial version is available and routinely wasted: every analyst triage decision is a label, and most deployments discard them.

Why supervised detection of attacks does not work

Confirmed incidents are rare. An organisation may have a handful of substantiated insider cases in a decade. That is not a training set.

Labels are unbalanced beyond usefulness. True positives are a vanishing fraction of events. Standard classifiers on such data learn to predict the majority class and achieve excellent accuracy while detecting nothing.

Historical labels are unreliable. What was investigated is a function of what was detected, which is a function of what was monitored. Training on that reproduces the existing blind spots.

Novel attacks are unrepresented by definition.

What supervision is available

Analyst adjudication. Every alert closed carries a judgement: legitimate business activity, error, policy violation, compromise, unresolved. That is a label, generated as a by-product of work already being done.

Approved workflow identification. Every exclusion written is a statement that a pattern is normal. That is a label too, and a particularly clean one.

Confirmed incidents, however few.

Red team activity, with known ground truth. Small volume, high quality.

Capturing adjudication properly

The value depends entirely on capture discipline, and this is where deployments fail.

Fixed categories, small in number. Free-text closure notes are unusable at scale. Five or six categories: legitimate business activity, user error, policy violation, suspected compromise, insufficient information, technical false positive.

Mandatory at closure. If it is optional it will be empty within a month.

A reason field, structured where possible. "Legitimate — approved workflow, finance month-end" is worth far more than "legitimate".

Store with the full feature vector. The label is useless without what the model saw at the time. This is the step most often missed: products store the outcome and discard the input.

Record the analyst and timestamp. Different analysts adjudicate differently, and knowing that is itself information.

What you can then build

Precision measurement. The proportion of alerts that were something. The only quality metric with a defensible meaning, and it requires exactly this data.

A false positive classifier. Train on adjudicated alerts to predict which new alerts will be closed as legitimate. Use it to reorder the queue rather than to suppress — suppression on a model trained on past dismissals entrenches past blind spots.

Per-detector precision. Which detectors produce findings and which produce noise. This is the input to retirement decisions and it is unavailable without labels.

Feature contribution analysis. Which features appear in true positives versus dismissals. Directly informs feature selection.

Drift detection. A rising dismissal rate for a detector that was previously precise means something changed.

The feedback loop trap

Training on analyst decisions creates a loop worth naming.

Analysts dismiss what they consider uninteresting. The model learns to deprioritise it. Those alerts stop surfacing. Analysts never see that category again, and cannot revise the judgement.

If the original dismissals were wrong, the error is now permanent and invisible.

Mitigations:

Reserve a random sample that bypasses any learned prioritisation and is reviewed regardless. This is the only reliable way to detect the loop, and it costs a small fraction of review capacity.

Periodically re-examine a sample of dismissed alerts with fresh eyes.

Use learned models to reorder, never to suppress entirely.

Track the dismissal rate per category over time. A category dropping to zero is either solved or invisible, and the two are indistinguishable without the random sample.

Realistic expectations

You will not build an attack classifier. You will build a noise classifier, which is genuinely valuable: it makes the queue reviewable, and reviewability is the binding constraint in most deployments.

Reframing the goal from "detect attacks with machine learning" to "reduce the review burden using analyst judgement" produces something achievable within a year, on data you are already generating.

A minimal adjudication schema

What to capture at closure, kept small enough that analysts will actually complete it.

Outcome, from a fixed list: legitimate business activity, user error, technical or data issue, policy violation, suspected compromise, insufficient information.

Reason, structured where possible: approved workflow, role change, project work, covering absence, tool rollout, data defect, unexplained.

Confidence, three levels. An analyst who was unsure should be able to say so, because a low-confidence dismissal is a different label from a confident one.

Time spent, approximate. This is how you measure whether enrichment improvements are working.

Analyst identity and timestamp.

The feature vector, stored automatically alongside.

Six fields, two of them automatic. Anything larger degrades within a month, and the degradation is silent.

Common false positives

Labels themselves carry error, and knowing the pattern helps:

Rushed dismissals during high-volume periods, which enter the label set as confident negatives.

Analyst variation, where the same alert would be labelled differently by different people.

Category drift, as the meaning of a category shifts informally over time.

Confirmation of prior dismissals — an analyst who sees the entity was cleared before will clear it again without independent examination.

Post-hoc labels applied in bulk during a cleanup, which are guesses rather than judgements.

Quarterly calibration sessions catch most of this, and without them a label set becomes noise within a year.

Blind spots and assumptions

That analyst judgement is correct. It is the best available and it is not ground truth. Dismissals include missed incidents.

That categories are applied consistently. They are not, without periodic calibration between analysts.

That labels transfer. A model trained on one organisation's adjudications does not apply to another. Vendor-supplied classifiers were trained elsewhere.

That more labels are always better. Labels from a rushed queue are noise, and training on them degrades the model.