Log Normalisation and Schemas

A behavioural feature such as "number of distinct hosts accessed" requires that "host" means the same thing across every source contributing to it. Achieving that is the bulk of a UEBA deployment and it is invariably underestimated.

What normalisation involves

Field mapping. One source calls it user, another account_name, a third principalId, a fourth puts it inside a nested message string. All must map to one field.

Value normalisation. DOMAIN\jsmith, [email protected], jsmith and S-1-5-21-... may be the same account. Case differs. Domain prefixes come and go.

Type normalisation. An authentication success is expressed as event ID 4624 in one source, "result": "SUCCESS" in another, and HTTP 200 in a third.

Timestamp normalisation. Different formats, different precision, different timezones, sometimes no timezone at all.

Enum harmonisation. Logon types, protocol names, action verbs. Each source has its own vocabulary.

Use an existing schema

Do not invent one. Several mature schemas exist and adopting one saves months and makes your work portable.

OCSF — the Open Cybersecurity Schema Framework, vendor-neutral and increasingly broadly supported.

ECS — Elastic Common Schema, widely implemented and well documented.

CIM — Splunk's model, if that is your platform.

Adopting a published schema means source mappings frequently already exist, analysts moving between organisations already know it, and replacing the platform later does not mean redoing the mapping.

The main argument against — that none fits your environment exactly — is true and cheaper to work around than a bespoke schema nobody else understands.

Where normalisation goes wrong

Silent field loss. A parser that fails on a field drops it rather than erroring. The feature depending on it quietly becomes constant, and no alert fires because nothing broke visibly.

Partial mapping. Ninety percent of events map correctly, and the other ten percent are a specific event type that happens to be the interesting one.

Schema drift. A source updates and changes its format. The parser continues to run, producing fewer fields. Detection degrades over weeks.

Over-normalisation. Forcing everything into a common shape discards source-specific detail that carried the signal. Keep the raw event alongside the normalised one.

Character encoding. Non-ASCII usernames, mangled by a parser assuming otherwise, become separate entities.

Testing the pipeline

Normalisation is code and deserves the treatment code gets.

Fixture tests. A stored set of real events per source with expected normalised output. Run them on every parser change.

Field completeness monitoring. For each source, the percentage of events with each critical field populated. Alert when it moves. This single measure catches most silent failures.

Volume monitoring per source. A source that stops sending is invisible unless you watch for absence. Absence is the most common and most damaging pipeline failure, and the dashboard looks perfectly healthy while it happens.

Cardinality monitoring. The number of distinct users seen per day. A sudden change means a mapping problem, not a behavioural one.

The ordering that saves rework

Normalise before enriching. Enrichment joins on normalised fields; doing it earlier means redoing it.

Normalise before baselining. Obvious and routinely violated when a source is added mid-deployment with a different mapping, silently changing what the baseline contains.

Keep the raw event. Investigations need it, and it is the only way to reprocess when a mapping is found to be wrong.

Testing a mapping by hand

Automated checks confirm that fields are populated. Only manual sampling confirms they are correct, and the difference matters.

Take thirty events per source. Put the raw event and the normalised output side by side.

Check the entity field resolves to the right account, in the right format, with domain and case handled consistently.

Check the timestamp against the raw value, including the timezone.

Check the event type maps to the right category. Authentication failures miscategorised as successes is a real and common error that silently disables a whole class of detection.

Check the fields that were dropped. Anything the parser discarded that a feature might have wanted.

This takes about an hour per source. It is the highest-value hour in the whole normalisation effort, and it is skipped almost universally because the pipeline appears to be working.

Common false positives

Normalisation defects present as behavioural anomalies:

Duplicate delivery through two collection paths, doubling every volume feature for affected entities.

Partial parsing where one event subtype fails and its absence makes an entity look inactive.

Enum mismatches where the same action from two sources maps to different event types, so a feature counting it undercounts.

Encoding failures splitting one account into several entities.

Nested field extraction errors where a value is taken from the wrong position in a message string, producing entity identifiers that are actually hostnames or process names.

Blind spots and assumptions

That the vendor's connector is correct. Out-of-the-box parsers are written against a reference version of a source and diverge from yours. Verify by sampling.

That parsing errors are visible. Most pipelines count them and few alert on them.

That normalisation is a phase. It is continuous. Every new source, every upgrade, every configuration change touches it.

That completeness equals correctness. A field can be populated with the wrong value. Sampling by hand is the only reliable check, and thirty events per source takes an hour.