Log Processing at Scale: Parse, Normalise, Enrich, Store

Security teams rarely lack data. They lack data they can use. Firewalls, identity providers, cloud audit trails, endpoints, and SaaS tools each write events in their own format, with their own field names, timestamps, and ideas about what a "user" is. Security log processing is the pipeline that turns that flood into one consistent, enriched, queryable stream, and it's the part of a security platform that decides whether detection, investigation, and AI features work at all.

The reason it matters is compounding. A detection rule that has to understand five log dialects is five times harder to write and five times more likely to miss something. An analyst who has to look up IP ownership and user identity by hand is slower during exactly the incidents where speed counts. And a schema decision made in the first month either lets you add new sources cheaply for years or forces a painful migration once billions of events depend on it.

This article walks through the six-stage pipeline we use: receive, parse, normalise, enrich, store, and analyse. It covers why parsing is harder than it looks, how to design a normalisation schema that can grow, which enrichment steps earn their cost, and how columnar storage with hot, warm, and cold tiers keeps queries fast and bills sustainable. It closes with what a good pipeline looks like, what it realistically costs to build, and answers to the questions teams ask most.

The Line Nobody Can Read

A Linux server writes this to its auth log every time someone fails to log in:

Failed password for root from 10.0.0.15

A human security analyst reads that in half a second. A firewall, a Windows domain controller, an AWS CloudTrail event, and a Microsoft 365 sign-in log all describe the same kind of thing — an authentication attempt — in five completely different shapes. None of them agree on field names, timestamp formats, or even what counts as a "user."

Now multiply that by millions of events a day, arriving continuously, from dozens of source types that each speak their own dialect. That is the problem the log processing engine solves. It sits directly after data collection and before everything intelligent: detection, threat intelligence, the AI copilot. If this layer is sloppy, every layer above it inherits the mess.

This is the least glamorous module in an AI-native security platform and quietly the most important. Get it right and everything downstream becomes queryable, correlatable, and cheap to reason about. Get it wrong and you spend the next two years writing special-case code to paper over decisions you made in week three.

The Log Processing Pipeline in Six Stages

Every event that enters the platform travels the same road:

Receive → Parse → Normalise → Enrich → Store → Analyse

Each stage has one job.

Receive accepts events from agents, syslog, cloud APIs, and webhooks without dropping anything under load. At real volume this is a streaming buffer — Kafka or Redpanda — that absorbs bursts so a spike in log traffic doesn't take the pipeline down.

Parse turns an unstructured or semi-structured line into fields. Failed password for root from 10.0.0.15 becomes {action: failed_login, user: root, source_ip: 10.0.0.15}.

Normalise maps those fields onto one shared schema, so a login failure from Linux and a login failure from Windows land in the same shape.

Enrich adds context the raw event never carried: where the IP is, who owns the machine, whether the user is a real employee, whether the address appears on a threat feed.

Store puts the record somewhere it can be queried fast at scale — and keeps the untouched original somewhere cheap.

Analyse is everything downstream: the detection engine, correlation, the copilot. It only works if the five stages before it did their job.

The interesting engineering lives in the middle three.

The Parsing Problem: Dozens of Dialects

There is no universal log format. There never has been. A realistic platform ingests, at minimum: Linux syslog and journald, Windows Event Logs, AWS CloudTrail, Azure Activity logs, firewall logs from three vendors who each invented their own layout, VPN logs, DNS query logs, and JSON from a dozen SaaS APIs. Some are neat structured JSON. Many are free-text lines written by a developer in 2009 who never imagined a machine would parse them.

So parsing is not one parser. It's a library of source-specific parsers, each of which knows how to pull fields out of one format, plus a routing layer that recognises which parser a given event needs. New sources arrive constantly — a customer adds a new SaaS tool, a new appliance shows up on the network — and each may need its own parser.

The practical discipline here is that parsing must fail safely. When a parser hits a line it doesn't recognise, it must not crash the pipeline and must not silently discard the event. It stores the raw line as-is, tags it "unparsed," and moves on. Unparsed volume becomes a metric you watch: a sudden spike usually means a source changed its format, and you'd far rather see that on a dashboard than discover it three weeks later when a detection quietly stopped firing.

Normalisation: One Schema to Rule Them All

Normalisation is where the platform earns its keep, and where the single most consequential decision in the whole build gets made.

The goal is simple to state: every event, regardless of source, ends up in one common schema. A failed login is a failed login whether it came from a Linux box, a Windows server, or an Okta sign-in log. Here is the before and after for our example line:

Raw (Linux auth log):

Failed password for root from 10.0.0.15

Normalised (common schema):

{
  "user": "root",
  "action": "login",
  "status": "failed",
  "ip": "10.0.0.15",
  "country": "India",
  "severity": "medium",
  "device": "Ubuntu Server"
}

That transformation is the entire point of the module. Once every source produces records in this shape, a question like "show me every failed login for root across the whole estate in the last hour" is one query — not a per-source integration nightmare. The detection engine writes one rule instead of five. The copilot answers in plain language because the underlying data is already uniform.

The hard part is not writing the schema. The hard part is that the schema is nearly impossible to change later. Every parser writes into it. Every detection rule reads from it. Every stored record — potentially billions of them — is shaped by it. Rename a field or change what severity means six months in, and you're rewriting parsers, migrating historical data, and re-testing every rule that depended on the old shape. A bad schema decision doesn't cost you once; it taxes every later module, forever.

Two principles keep you out of that hole. First, design the schema to be extensible from day one — a common core of fields that every event shares (timestamp, source, action, actor, status, severity) plus room for source-specific extras that don't force a migration when a new log type arrives. Second, be ruthless about shared vocabulary: login and authentication and signin must not be three different values for the same concept, or every query has to know all three. This is exactly the "how does the data model handle a source we haven't thought of yet" question from the platform architecture — and it's answered here, in the schema, or not at all.

Enrichment: Turning Data Into Context

A normalised event is structured, but it's still thin. 10.0.0.15 is just an address. root is just a string. Enrichment adds the context that turns a data point into something an analyst — or a detection rule — can act on. There are four kinds worth building in:

Geolocation. Map the source IP to a country, region, and network owner. This is what lets a rule notice that an account which logged in from London an hour ago is now logging in from another continent — the "impossible travel" pattern.

Asset ownership. Tie the event to the actual machine and team. web-prod-03 owned by the platform team, in production, running Ubuntu. A failed login against a forgotten test box and a failed login against a production database matter very differently, and only the asset context tells you which is which.

User identity. Resolve root or a raw username to a real person (or confirm it's a service account). An action by a departed employee's account, or by a service account doing something a service account never does, is only visible once identity is attached — the same identity-resolution work that identity security depends on downstream.

Threat-intelligence tagging. Cross-reference the IP, domain, or file hash against known-bad indicators. If 10.0.0.15 appears on a threat feed as a known scanner or command-and-control node, the event gets flagged the instant it lands. This is the handoff to threat intelligence enrichment, which maintains those feeds and does the deeper correlation.

Enrichment is what makes the volume of security data manageable. An analyst can't read a million raw lines. But a million lines where the twelve genuinely interesting ones are already tagged with location, owner, identity, and threat context — that's a workload a small team, backed by AI agents, can actually stay on top of.

Storage: Built for Real Volume

Millions of events a day, retained for months or years, is not a job for a normal database. Try to run security analytics on a single PostgreSQL instance and it falls over — not because Postgres is bad, but because it's the wrong tool for append-heavy, query-across-billions-of-rows workloads. Storage for a security platform is really three stores doing three jobs:

A columnar store for analytics. ClickHouse is the common choice. Security queries are overwhelmingly "aggregate across a huge number of rows on a few columns" — count failed logins per IP over 24 hours, group events by country. Columnar databases are built for exactly that shape and stay fast where a row store would grind. This is where the normalised, enriched records live for querying.

A search index for investigation. OpenSearch (or Elasticsearch) handles the "find me the needle" queries — full-text search across log bodies, "show me everything mentioning this hostname," the free-form pivoting an analyst does mid-investigation.

Object storage for raw retention. The untouched original of every event goes to cheap object storage (S3 or equivalent). This is your compliance archive and your safety net: if you ever improve a parser or need to prove what actually arrived, the raw truth is still there, cheaply, long after the processed copy has aged out of the hot tier.

Underpinning all of it is hot / warm / cold tiering, because retaining everything at full query speed is ruinously expensive. Recent data (say the last 30 days) stays hot — instantly queryable. Older data moves to warm — slower, cheaper. Data past your active window goes cold in object storage — rarely touched but retained for compliance. A sensible tiering policy is often the difference between a storage bill that's sustainable and one that isn't.

Normalise at Ingest vs Query-Time Parsing

The main alternative to the pipeline described here is to keep raw logs and parse them only when someone asks a question, sometimes called schema-on-read. Both approaches exist in real products, and the trade-off is worth seeing side by side.

Query-time parsing is attractive early on because it costs almost nothing to start. You point sources at storage and search them. For a small estate with a handful of sources and occasional investigations, that can be enough. The problem is that the cost never goes away: every query, every detection, and every analyst pivot pays the parsing and correlation price again, and the person writing the query has to know how each source formats its fields.

Normalising at ingest moves that cost to one place and pays it once. The price is upfront design work, especially on the schema, and an ongoing parser library to maintain. In return, detection rules become simpler and more reliable, enrichment is available to every consumer, and an AI copilot can answer questions against uniform data rather than guessing at field meanings.

In practice the strongest designs combine both. The normalised, enriched record serves analytics and detection, while the raw original sits in cheap object storage. That raw copy gives you a schema-on-read fallback for formats you haven't parsed yet and lets you reprocess history when a parser improves.

Benefits of Security Log Processing

Detection Rules That Work Across Every Source

When every authentication event lands in the same shape with the same vocabulary, a rule for brute-force attempts or impossible travel is written once and applies to Linux, Windows, cloud, and SaaS sources alike. Without normalisation, the same rule has to be rewritten per source, and every new source is a chance for coverage to silently lapse. A shared schema turns detection engineering from a translation exercise into actual threat logic, which is where a small security team's time should go.

Faster Investigations

Analysts lose most of their time in an incident gathering context: who owns this machine, is this a real employee, where is this IP, has it been seen before. Enrichment at ingest answers those questions before anyone asks them. An analyst opening an alert sees location, asset owner, identity, and threat-feed matches together, and can pivot across sources with one query. During an active incident, that difference compounds across every step of the investigation.

Predictable Storage Costs

Tiered storage, with recent data hot, older data warm, and raw originals cold, keeps the bill tied to how data is actually used. Columnar storage compresses repetitive security fields well and answers aggregation queries efficiently. The result is a platform that can retain months or years of data for compliance without paying hot-tier prices for events nobody has queried since the week they arrived.

Resilience to Change

Source formats change without warning. A pipeline that stores unparsed events raw, tags them, and tracks unparsed volume turns a silent detection gap into a visible metric. Combined with an extensible schema, new sources become a new parser rather than a migration, so the platform keeps pace as the organisation adopts new tools.

A Usable Foundation for AI Features

Copilots and AI-assisted triage depend on consistent, well-labelled data. A model asked about failed logins across the estate gives far better answers when the data already uses one vocabulary and carries enrichment, rather than having to infer meaning from five formats. Clean processing is what makes AI features in a security platform trustworthy instead of a demo.

Security Log Processing Use Cases

Each of the following depends on the pipeline doing its job well. They are also the scenarios we use to test whether a schema and enrichment design is good enough before committing to it.

Detecting Credential Attacks

Brute-force and password-spraying attempts show up across many systems at once: VPN, identity provider, servers, and SaaS logins. The problem is that each source describes a failed login differently. Normalising them into one schema lets a single rule count failures per user and per IP across the whole estate, while geolocation and threat-feed enrichment add the context to judge severity. The outcome is earlier detection of attacks that would look like noise in any one source.

Impossible Travel and Account Takeover

A login from one country followed shortly by another from a different continent is a classic takeover signal. It only works if every login event carries geolocation and a resolved user identity, regardless of source. Enrichment in the pipeline makes the pattern a straightforward query, and identity resolution separates real employees from shared or service accounts that legitimately behave differently.

Incident Investigation and Forensics

When an incident is confirmed, analysts need to reconstruct what an account or machine did over days or weeks. A search index supports free-form pivots on hostnames and strings, columnar storage answers timeline aggregations quickly, and the raw archive provides the untouched original if evidence needs to be shown later. Together they turn a multi-day reconstruction into hours of focused work.

Compliance Retention and Audit

Many organisations must retain security logs for set periods and produce them on request. Tiered storage with raw originals in object storage meets retention requirements cheaply, while the normalised copy makes audit questions, such as who accessed a system and when, answerable without hand-assembling data from each source.

Feeding an AI Security Copilot

Teams adding a natural-language interface to their security data need the data to be consistent first. A normalised, enriched store lets the copilot translate a question into one query and explain results with context attached, which is what makes the answers useful to a small team rather than a source of fresh confusion.

Common Security Log Processing Mistakes

Most pipeline problems we're asked to fix trace back to a handful of early decisions. They are cheap to avoid at the start and expensive to unwind once billions of events depend on them.

Writing Parsers Before Designing the Schema

Teams eager to see data flowing often start with parsers and let the schema emerge from whatever the first few sources produce. That schema then reflects those sources' quirks, and the fifth source doesn't fit. Because every parser, rule, and stored record depends on the schema, fixing it later means migrations and re-testing everything. Design the core fields and vocabulary first, then write parsers into it.

Dropping Events That Don't Parse

Discarding unrecognised lines keeps the pipeline tidy and hides problems. When a vendor changes its log format, events start failing to parse and simply disappear, along with the detections that depended on them. Store unparsed events raw, tag them, and alert on unparsed volume so format changes surface as a metric rather than a missed incident.

Allowing Vocabulary Drift

If one parser writes login, another signin, and a third authentication, every query and rule must know all three, and someone eventually forgets. Shared vocabulary needs an owner, a reference list, and review whenever a new parser is added.

Enriching at Query Time

Looking up geolocation, identity, and threat context only when someone queries makes every investigation slower and leaves detection rules without that context. It also means enrichment results can change between queries. Attach it in the pipeline so every consumer sees the same context.

Using One Store for Everything

Running analytics, full-text search, and long-term retention on a single general-purpose database works at small scale and then fails as volume grows. Match each job to a store built for it, and tier data by age so costs stay sustainable as retention periods lengthen.

Security Log Processing Best Practices

A well-built log processing layer follows a short list of practices consistently. None of them is exotic; the difficulty is holding to them as sources multiply and deadlines press.

• Never silently drop an event. Unrecognised events are stored raw and tagged, never discarded. Unparsed volume is a monitored metric with an alert, because a spike usually means a source changed format.
• Keep one schema with a shared vocabulary. Every source normalises to the same shape, with consistent field names and values. login is login everywhere, and new values go through a review rather than appearing ad hoc in a parser.
• Design the schema to grow. Define a common core that every event shares, such as timestamp, source, action, actor, status, and severity, plus room for source-specific extras. New log sources should slot in as a new parser, not a migration.
• Enrich in the pipeline, not at query time. Attach geolocation, asset ownership, identity, and threat tags as events flow through, so downstream queries are fast and analysts see context immediately.
• Buffer at the front. Put a streaming buffer before parsing so bursts of traffic are absorbed rather than dropped, and so parsing workers can scale out independently.
• Match storage to the query pattern. Use a columnar store for analytics, a search index for investigation, and object storage for raw retention, with hot, warm, and cold tiering to keep costs sane.
• Always keep the raw original. However aggressively you process, retain the untouched event cheaply and independently. It is your compliance archive and lets you reprocess history when a parser improves.
• Treat parsers as production code. Version them, test them against sample logs from each source, and monitor their error rates, because vendors change formats without announcing it and a broken parser looks exactly like a quiet day.

What It Costs and How Long It Takes

Honest framing: a processing pipeline that handles a couple of real source types end-to-end — receive, parse, normalise into a well-designed schema, basic enrichment, and a columnar store you can query — is typically a four-to-six-week piece of work as part of a Phase 1 build. It's rarely a standalone project; it's the plumbing that makes collection and detection worth having.

The cost that matters isn't the initial build — it's the schema. Time spent designing the normalisation schema before you write a single parser is the highest-impact time in the whole project. A schema you can extend without migrations saves months later; a schema you have to rip out and replace in month nine can effectively force a partial rebuild of every module that reads from it. This is the single decision we'd most encourage a team to slow down on. Everything else in this module is competent engineering; the schema is architecture.

Related guides

• The data collection layer that feeds this pipeline
• Threat intelligence enrichment: matching events to known-bad indicators
• Building the detection engine on top of normalised events
• The complete AI-native security platform architecture
• AI agent security: what business owners need to know
• Our AI development services

We Build the Unglamorous Layers Properly

The log processing engine is exactly the kind of work that separates a platform that scales from one that stalls at month nine. We'd start by designing a normalisation schema for your actual sources — the one decision that's cheap to get right now and painful to fix later — and build the pipeline around it: fail-safe parsing, in-flight enrichment, and a storage layout that survives real volume.

If you're planning a security platform, or any system that has to turn millions of messy events into something queryable and useful, we're happy to walk through the data engineering for your specific case.

Talk to us about your platform — no commitment, just a conversation.

Frequently Asked Questions

Why can't I just store raw logs and query them when I need to?

You can, but you'll pay for it on every query. Raw logs are in dozens of incompatible formats, so any question that spans more than one source ("all failed logins everywhere in the last hour") requires per-source parsing logic every single time you ask it. Normalising once, up front, means every downstream query, detection rule, and AI feature works against one consistent shape. The processing pipeline moves that cost from query time — where you pay it repeatedly — to ingest time, where you pay it once.

What is log normalisation, in plain terms?

It's mapping events from different sources onto one common schema. A failed login looks completely different in a Linux auth log, a Windows event, and an Okta sign-in record, but they all describe the same thing. Normalisation turns each of them into an identical structured record — same field names, same values for the same concepts — so the rest of the platform can treat them uniformly. The line Failed password for root from 10.0.0.15 becomes a JSON object with user, action, status, ip, and severity fields that mean exactly the same thing regardless of where the event came from.

Why is ClickHouse recommended for security logs?

Security analytics is overwhelmingly aggregation across enormous numbers of rows on a small number of columns — counting, grouping, and time-bucketing billions of events. Columnar databases like ClickHouse are built for precisely that access pattern and stay fast at a volume where a traditional row-based database would grind to a halt. It's usually paired with a search index like OpenSearch for full-text investigation and object storage for cheap raw retention, because no single store is ideal for all three jobs.

Why is the schema so hard to change later?

Because everything depends on it. Every parser writes into the schema, every detection rule reads from it, and every stored record — potentially billions — is shaped by it. Change a field name or the meaning of a value after the fact, and you have to update every parser, migrate all the historical data, and re-test every rule that relied on the old shape. That's why a well-designed, extensible schema up front is the highest-impact decision in the whole module, and why a bad one quietly taxes every later stage of the platform.

What does enrichment actually add to a log event?

Context the raw event never carried. Geolocation turns an IP into a country and network owner. Asset ownership ties the event to a real machine and the team that owns it. Identity resolution connects a username to an actual person or confirms it's a service account. Threat-intelligence tagging flags the event if its IP, domain, or file hash appears on a known-bad feed. Together these turn a thin structured record into something an analyst or a detection rule can act on immediately — and they're attached in the pipeline, so downstream queries stay fast.

How much data can a pipeline like this realistically handle?

A properly built pipeline scales horizontally, so the ceiling is mostly a function of how much infrastructure you're willing to run rather than a hard architectural limit. The streaming buffer at the front (Kafka or Redpanda) absorbs bursts, parsing and enrichment scale out across workers, and a columnar store with hot/warm/cold tiering keeps both query speed and cost under control as volume grows. Typical mid-sized deployments comfortably handle millions of events a day; the design pattern is the same whether you're processing a few million or a few billion, which is why getting the foundations right early matters so much.

Conclusion

Security data arrives in dozens of incompatible shapes, and every layer above the pipeline, from detection rules to threat intelligence to an AI copilot, inherits whatever mess the processing layer leaves behind. Turning raw events into one normalised, enriched record at ingest time is what makes the rest of a security platform affordable to query and practical to build on.

The decisions that matter most are few. Never silently drop an event you can't parse; store it raw and track the volume. Design the normalisation schema before writing parsers, and design it to grow. Attach enrichment such as geolocation, asset ownership, identity, and threat tags in the pipeline rather than at query time. Match storage to the query pattern, with columnar analytics, a search index for investigation, and tiered retention for cost.

The caveat is that none of this is finished at launch. New sources arrive, vendors change log formats without warning, and parsers need monitoring like any other production code. If you're planning a security platform or any system that has to make millions of messy events useful, talk to our backend infrastructure team about designing the pipeline and schema for your sources.