Testing Detection With Red Teams

You cannot measure recall without knowing what should have been detected. A red team exercise manufactures exactly that: real adversarial behaviour, in your environment, with a known record of what happened.

It is the most informative evaluation available for behavioural analytics, and it is routinely wasted because nobody arranges to measure the analytics against it.

Setting it up for this purpose

Tell the red team what you want to test. A generic exercise optimises for reaching an objective, frequently by a path your behavioural analytics could never see. If the question is whether UEBA detects lateral movement, the exercise should include lateral movement.

Do not tell the blue team the timing. The value is in whether normal operation surfaces it.

Require a detailed timeline. Every action, timestamped, with the account and host used. Without this you cannot align activity against alerts afterwards, and reconstructing it later is unreliable.

Agree the scope in writing. Which systems, which accounts, what is out of bounds. This is standard practice and it matters more than usual here, because the exercise will generate behavioural profiles for real accounts.

Decide what happens to the data. Exercise activity is now in the baselines. Whether to exclude it from training is a decision to make in advance.

What to measure

Was it detected at all? Binary and decisive.

How long from first action to detection? The number that matters most operationally.

Which detector fired? Frequently a rule rather than the behavioural model, which is itself worth knowing.

What scored, and how highly? An action that produced an elevated score below the alert threshold is a threshold finding, not a detection failure.

What produced nothing? The most valuable output. Every action that generated no signal is a documented blind spot.

Would an analyst have understood it? Take the alerts the exercise generated and have someone triage them cold. Surfacing an alert nobody can interpret is not detection.

The three outcomes and what each means

Detected quickly. Good, and check which detector. If a rule caught it, the behavioural layer contributed nothing and that is worth knowing.

Scored but below threshold. The model saw it. This is a tuning finding and it is the most actionable result: you know the signal exists and where the threshold should sit.

Nothing at all. Either the data source was missing, the feature does not exist, or the behaviour fell within baseline. Each has a different remedy and the distinction is worth establishing.

Feeding it back

Build detection for what was missed. Then verify with a repeat of that specific action.

Record what remains uncovered. Every exercise leaves gaps that will not be closed. Writing them down converts an unknown into an accepted risk.

Check whether exercise activity poisoned baselines. If an account's baseline now includes adversarial behaviour, reset it.

Re-run annually. A single exercise is a snapshot. The value compounds with repetition, because you can measure whether detection improved.

The purple team variant

For behavioural analytics specifically, a collaborative exercise is frequently more productive than a blind one.

Run an action, watch what the model produces, adjust, run it again. This iterates far faster than an annual blind test and it builds the analysts' intuition for what the system can and cannot see.

The trade: it measures capability, not operational readiness. A blind exercise tests whether the queue gets reviewed. Both are worth running, and they answer different questions.

What this does not tell you

Recall in general. It measures detection of what the exercise did, scoped by rules of engagement.

Anything about insider scenarios, unless the exercise deliberately simulated one — which is unusual, and worth requesting, because the behavioural profile of an insider differs substantially from an external intruder.

Whether the model would catch a patient adversary. Exercises are time-boxed. Low-and-slow behaviour is precisely what they cannot simulate.

An exercise scorecard

What to record, per action, so that the exercise produces measurement rather than an anecdote.

Action and timestamp, from the red team's log.

Was any signal generated? Yes, no, or unknown.

Which detector, and at what score?

Was an alert raised at the operating threshold?

Was it reviewed, and how long after the action?

What was the analyst's disposition when triaged cold?

If nothing was generated: was the source missing, the feature absent, or the behaviour within baseline?

That last distinction is the most valuable output of the whole exercise, because each cause has a different and known remedy, and confusing them leads to buying tools that would not have helped.

Common false positives

Exercise artefacts that distort the measurement:

Detection by coincidence, where an unrelated alert happened to surface the same entity.

Blue team awareness, which changes review attention for the period.

Scope constraints preventing the red team from taking the path an actual adversary would.

Accounts created for the exercise having no baseline, so everything they do is anomalous and detection looks better than it is.

Compressed timelines, where a week of simulated activity in an afternoon produces volume signals a patient adversary would never generate.

Exercise activity entering baselines, which both distorts the measurement and degrades subsequent detection unless the period is excluded from training.

Blind spots and assumptions

That the exercise is representative. It is one path chosen by one team.

That the timeline is accurate. Verify it against logs rather than accepting the report.

That failure is the red team's success. Framing it that way discourages the honest reporting that makes the exercise useful.

That detection means the programme works. It means one path was visible on one day.