Measuring Quality Without Ground Truth

"Is it working?" is the question every deployment faces and few answer honestly. The difficulty is structural: you can measure what you found, and you cannot measure what you missed.

Precision: measurable

Of the alerts raised, what fraction were something?

This requires adjudication capture and nothing else. It is the one quality metric with a defensible meaning, and it should be reported per detector rather than in aggregate, because an aggregate hides the single detector generating most of the noise.

Report it as a fraction with the denominator visible. "Twelve findings from 340 alerts" is honest. "Precision 3.5%" invites a demand to improve it, and the improvement will come from suppressing detection.

Expect it to be low. In behavioural analytics, single-digit precision is normal and not necessarily a failure. The question is whether the findings were worth the review cost.

Recall: not measurable

Of everything that should have been detected, what fraction was?

The denominator is unknowable. Undetected incidents are undetected. Any vendor quoting a detection rate is quoting it against simulated data of their own construction.

What you can obtain instead:

Recall against red team exercises. Known ground truth, generated adversarially, in your environment. The best available proxy and the only one worth taking seriously. If the exercise was not surfaced, that is decisive.

Recall against historical incidents. Replay the model over a period containing a confirmed case. Rare and valuable every time it is possible.

Recall against injected activity. Establishes sensitivity to what you simulated, and nothing about anything else. Useful for verifying that a detector works at all.

The sampled tier. If random review of unalerted activity regularly finds things, your ordering is wrong. If it finds nothing over months, that is weak evidence of coverage — weak, because random sampling of a rare event finds nothing most of the time regardless.

Process metrics that mean something

Given the limits above, most honest reporting is about the machinery rather than the outcome.

Proportion of the queue reviewed. The binding constraint.

Time from event to review.

Coverage per data source, against the estate.

Entity resolution rate. The percentage of log identifiers resolving to a known entity.

Baseline maturity distribution. What fraction of entities have a usable baseline.

Exclusion count and age.

Detector inventory with last true positive date.

None of these is a detection rate. All of them are true, and together they describe whether the system is in a condition to detect anything.

What to tell an executive

The temptation is to produce a number. The number will be either alert counts, which are meaningless, or a detection rate, which is invented.

A better shape: what we can see and what we cannot, by data source and entity population. What we found this quarter and what it would have cost. What the red team exercise showed. What we changed as a result. What remains uncovered and what closing it would take.

The last item builds more credibility than any metric. A programme that states its blind spots is believed about the rest.

The uncomfortable truth to state early

A quiet quarter may mean strong detection or complete blindness, and no metric distinguishes them.

Saying this at the start of a programme, in writing, prevents a much worse conversation later when an incident is found by an external party. It also makes the case for the red team exercise, which is the only thing that resolves the ambiguity.

A reporting template

What to put in front of a security committee, quarterly, that is both honest and useful.

Coverage. Which populations and sources are monitored, which are not, and what the gaps would cost to close.

Health. Entity resolution rate, source completeness, baseline maturity distribution. Three numbers that describe whether the system can detect anything.

Volume and review. Alerts raised, proportion reviewed, median time to review.

Findings. What was confirmed, in categories, with what outcome.

Testing. What the last red team exercise showed, including what it missed.

Changes. What was tuned, retired, added, and why.

Known blind spots. Explicitly listed, with an owner where one exists.

The last section is what makes the rest credible. A programme willing to publish what it cannot see is believed about what it can.

Common false positives

Metrics themselves mislead in recurring ways:

Alert count as a proxy for value, which falls during good tuning and rises with coverage.

Precision improved by suppression, which reduces unmeasurable recall.

A quiet quarter read as success, when it may be a broken pipeline.

Vendor detection rates, measured on simulated data.

Benchmark comparisons against organisations with different data, populations and adjudication standards.

Time-to-detect measured only on detected cases, which excludes everything that was missed and therefore always looks good.

Blind spots and assumptions

That precision improvement is always good. It can be achieved by suppressing everything marginal, which reduces recall you cannot measure.

That a benchmark transfers. Precision figures from another organisation reflect their data, their tuning and their adjudication standards.

That the red team is representative. It tests what the exercise attempted, scoped by rules of engagement.

That measurement is free. Adjudication capture, calibration and sampling all consume analyst time. They are the cost of knowing whether the deployment works.