Sequence and Session Analytics

Most behavioural features are counts over a window, which throws away ordering. Yet the patterns that matter most in intrusions are defined by order: reconnaissance, then access, then collection, then transfer.

Sequence analytics recovers that information, at meaningful cost.

Sessions first

Before sequence, you need sessions: coherent periods of activity attributable to one continuous episode.

Gap-based sessionisation is standard — events separated by more than some interval belong to different sessions. Thirty minutes is a common default and it is a parameter worth examining rather than accepting.

Too short fragments one working period into many sessions, and patterns spanning a morning become invisible.

Too long merges a day into one session, and a burst of activity is diluted by hours of ordinary work.

Boundaries matter more than the interval. A session split in the middle of an attack sequence destroys the pattern. Overlapping sessionisation, or hierarchical sessions at several intervals, reduces sensitivity to the choice.

Session-level features are frequently more useful than window-level ones: session duration, event count, distinct systems within a session, whether a session crossed a privilege boundary.

Methods

N-gram models. Count the frequency of short subsequences and score rare ones. Simple, interpretable, and effective where the event alphabet is modest. The main limitation is fixed length: an n-gram model cannot see a dependency spanning more steps than n.

Markov models. Model transition probabilities between event types. A sequence traversing improbable transitions scores high. Interpretable, and it handles variable-length sequences.

Sequence neural models. Recurrent or transformer architectures over event sequences. Expressive, capable of long-range dependencies, expensive, and opaque.

Process mining. Derive the typical process from observed sequences and score deviation from it. Underused in security and well suited where the underlying activity genuinely is a process — a finance approval chain, a deployment pipeline.

The alphabet problem

Sequence methods need a manageable vocabulary of event types. Raw security telemetry has an enormous one: every process name, every command line, every resource path.

Abstraction is required, and it is the main design decision.

Too coarse — "authentication", "file access", "network" — and every sequence looks alike. Too fine and every sequence is unique, so everything is rare.

A workable middle: abstract to event type plus a small number of qualifiers. "Authentication, interactive, privileged account, unfamiliar host" rather than either "logon" or the full event.

Getting this right takes iteration on real data and it determines whether the method works at all.

Where sequence genuinely earns its cost

Lateral movement. A chain of authentications across hosts, each individually authorised, forming a path that has never occurred.

Privilege escalation chains. Account creation, group membership change, then use of the new privilege. Each step is a normal administrative action; the sequence is not.

Collection then exfiltration. Enumeration of a repository, followed by bulk read, followed by transfer. Volume features see the transfer; sequence sees the preparation.

Process anomalies. A deployment that skipped approval. A record modified outside the normal workflow.

Where it does not

Highly variable activity. Human working patterns have little sequential structure. Sequence models over general user activity mostly learn that anything can follow anything.

Sparse data. Sequence methods need many examples of normal sequences to model them. Entities with limited history have none.

Interleaved activity. A person doing three things at once produces an interleaved event stream that no sessionisation cleanly separates.

Where the alphabet is unbounded. Command lines, URLs and file paths without heavy abstraction.

Machine entities are a much better target than humans here: service accounts and automated processes have genuinely repetitive sequences, and deviation is correspondingly meaningful.

Practical guidance

Start with sessionisation and session-level features. These are cheap and frequently capture most of the available value before any sequence model is built.

Apply sequence methods to machine entities first.

Use n-grams or Markov before neural models. They are interpretable, they establish whether sequential structure exists in your data at all, and if they find nothing, a more complex model probably will not either.

Verify event ordering before trusting anything. Clock skew across sources makes ordering unreliable, and a sequence model on misordered events learns noise.

Choosing the event abstraction

The decision that determines whether sequence analytics works, and it is made by whoever configures the pipeline rather than by design.

Start from the detections you want. If the target is lateral movement, the alphabet needs authentication events distinguished by type, privilege and target familiarity — not by hostname.

Include a small number of qualifiers, not the full event. "Authentication, interactive, privileged, unfamiliar host" is four bits of information. "Logon" is too coarse; the raw event is too fine.

Bound the alphabet. Aim for tens of event types, not thousands. If your abstraction produces more, it is too specific and every sequence will be unique.

Test on real data. Compute the frequency distribution of your abstracted event types. If a handful dominate overwhelmingly, or if the distribution is uniform, the abstraction is not carrying structure.

Iterate. The first abstraction is always wrong, and the second is usually usable.

Common false positives

Sequence methods generate noise from specific, identifiable causes:

Interleaved activity, where a person doing several things at once produces a sequence that no sessionisation separates cleanly.

Event ordering errors from clock skew across sources, producing impossible transitions.

Session boundary artefacts, where a natural break splits a routine sequence in two.

Rare-but-routine orderings — the same tasks done in an unusual order because of an interruption.

Sparse entities whose limited history makes every sequence novel.

Batch and automation timing changes, where a schedule shift alters the observed order of machine activity.

Blind spots and assumptions

That order is recorded accurately. Across sources, frequently not.

That sessions correspond to intent. A session is a gap heuristic, not a unit of purpose.

That rare sequences are meaningful. Most rare sequences are legitimate work that happened in an unusual order.

That the abstraction is neutral. It determines what the model can see, and it is a security design decision, not a preprocessing detail.