The Line Nobody Can Read
A Linux server writes this to its auth log every time someone fails to log in:
Failed password for root from 10.0.0.15
A human security analyst reads that in half a second. A firewall, a Windows domain controller, an AWS CloudTrail event, and a Microsoft 365 sign-in log all describe the same kind of thing — an authentication attempt — in five completely different shapes. None of them agree on field names, timestamp formats, or even what counts as a "user."
Now multiply that by millions of events a day, arriving continuously, from dozens of source types that each speak their own dialect. That is the problem the log processing engine solves. It sits directly after data collection and before everything intelligent: detection, threat intelligence, the AI copilot. If this layer is sloppy, every layer above it inherits the mess.
This is the least glamorous module in an AI-native security platform and quietly the most important. Get it right and everything downstream becomes queryable, correlatable, and cheap to reason about. Get it wrong and you spend the next two years writing special-case code to paper over decisions you made in week three.
The Pipeline in Six Stages
Every event that enters the platform travels the same road:
Receive → Parse → Normalise → Enrich → Store → Analyse
Each stage has one job.
Receive accepts events from agents, syslog, cloud APIs, and webhooks without dropping anything under load. At real volume this is a streaming buffer — Kafka or Redpanda — that absorbs bursts so a spike in log traffic doesn't take the pipeline down.
Parse turns an unstructured or semi-structured line into fields. Failed password for root from 10.0.0.15 becomes {action: failed_login, user: root, source_ip: 10.0.0.15}.
Normalise maps those fields onto one shared schema, so a login failure from Linux and a login failure from Windows land in the same shape.
Enrich adds context the raw event never carried: where the IP is, who owns the machine, whether the user is a real employee, whether the address appears on a threat feed.
Store puts the record somewhere it can be queried fast at scale — and keeps the untouched original somewhere cheap.
Analyse is everything downstream: the detection engine, correlation, the copilot. It only works if the five stages before it did their job.
The interesting engineering lives in the middle three.
The Parsing Problem: Dozens of Dialects
There is no universal log format. There never has been. A realistic platform ingests, at minimum: Linux syslog and journald, Windows Event Logs, AWS CloudTrail, Azure Activity logs, firewall logs from three vendors who each invented their own layout, VPN logs, DNS query logs, and JSON from a dozen SaaS APIs. Some are neat structured JSON. Many are free-text lines written by a developer in 2009 who never imagined a machine would parse them.
So parsing is not one parser. It's a library of source-specific parsers, each of which knows how to pull fields out of one format, plus a routing layer that recognises which parser a given event needs. New sources arrive constantly — a customer adds a new SaaS tool, a new appliance shows up on the network — and each may need its own parser.
The practical discipline here is that parsing must fail safely. When a parser hits a line it doesn't recognise, it must not crash the pipeline and must not silently discard the event. It stores the raw line as-is, tags it "unparsed," and moves on. Unparsed volume becomes a metric you watch: a sudden spike usually means a source changed its format, and you'd far rather see that on a dashboard than discover it three weeks later when a detection quietly stopped firing.
Normalisation: One Schema to Rule Them All
Normalisation is where the platform earns its keep, and where the single most consequential decision in the whole build gets made.
The goal is simple to state: every event, regardless of source, ends up in one common schema. A failed login is a failed login whether it came from a Linux box, a Windows server, or an Okta sign-in log. Here is the before and after for our example line:
Raw (Linux auth log):
Failed password for root from 10.0.0.15
Normalised (common schema):
{
"user": "root",
"action": "login",
"status": "failed",
"ip": "10.0.0.15",
"country": "India",
"severity": "medium",
"device": "Ubuntu Server"
}
That transformation is the entire point of the module. Once every source produces records in this shape, a question like "show me every failed login for root across the whole estate in the last hour" is one query — not a per-source integration nightmare. The detection engine writes one rule instead of five. The copilot answers in plain language because the underlying data is already uniform.
The hard part is not writing the schema. The hard part is that the schema is nearly impossible to change later. Every parser writes into it. Every detection rule reads from it. Every stored record — potentially billions of them — is shaped by it. Rename a field or change what severity means six months in, and you're rewriting parsers, migrating historical data, and re-testing every rule that depended on the old shape. A bad schema decision doesn't cost you once; it taxes every later module, forever.
Two principles keep you out of that hole. First, design the schema to be extensible from day one — a common core of fields that every event shares (timestamp, source, action, actor, status, severity) plus room for source-specific extras that don't force a migration when a new log type arrives. Second, be ruthless about shared vocabulary: login and authentication and signin must not be three different values for the same concept, or every query has to know all three. This is exactly the "how does the data model handle a source we haven't thought of yet" question from the platform architecture — and it's answered here, in the schema, or not at all.
Enrichment: Turning Data Into Context
A normalised event is structured, but it's still thin. 10.0.0.15 is just an address. root is just a string. Enrichment adds the context that turns a data point into something an analyst — or a detection rule — can act on. There are four kinds worth building in:
Geolocation. Map the source IP to a country, region, and network owner. This is what lets a rule notice that an account which logged in from London an hour ago is now logging in from another continent — the "impossible travel" pattern.
Asset ownership. Tie the event to the actual machine and team. web-prod-03 owned by the platform team, in production, running Ubuntu. A failed login against a forgotten test box and a failed login against a production database matter very differently, and only the asset context tells you which is which.
User identity. Resolve root or a raw username to a real person (or confirm it's a service account). An action by a departed employee's account, or by a service account doing something a service account never does, is only visible once identity is attached — the same identity-resolution work that identity security depends on downstream.
Threat-intelligence tagging. Cross-reference the IP, domain, or file hash against known-bad indicators. If 10.0.0.15 appears on a threat feed as a known scanner or command-and-control node, the event gets flagged the instant it lands. This is the handoff to threat intelligence enrichment, which maintains those feeds and does the deeper correlation.
Enrichment is what makes the volume of security data manageable. An analyst can't read a million raw lines. But a million lines where the twelve genuinely interesting ones are already tagged with location, owner, identity, and threat context — that's a workload a small team, backed by AI agents, can actually stay on top of.
Storage: Built for Real Volume
Millions of events a day, retained for months or years, is not a job for a normal database. Try to run security analytics on a single PostgreSQL instance and it falls over — not because Postgres is bad, but because it's the wrong tool for append-heavy, query-across-billions-of-rows workloads. Storage for a security platform is really three stores doing three jobs:
A columnar store for analytics. ClickHouse is the common choice. Security queries are overwhelmingly "aggregate across a huge number of rows on a few columns" — count failed logins per IP over 24 hours, group events by country. Columnar databases are built for exactly that shape and stay fast where a row store would grind. This is where the normalised, enriched records live for querying.
A search index for investigation. OpenSearch (or Elasticsearch) handles the "find me the needle" queries — full-text search across log bodies, "show me everything mentioning this hostname," the free-form pivoting an analyst does mid-investigation.
Object storage for raw retention. The untouched original of every event goes to cheap object storage (S3 or equivalent). This is your compliance archive and your safety net: if you ever improve a parser or need to prove what actually arrived, the raw truth is still there, cheaply, long after the processed copy has aged out of the hot tier.
Underpinning all of it is hot / warm / cold tiering, because retaining everything at full query speed is ruinously expensive. Recent data (say the last 30 days) stays hot — instantly queryable. Older data moves to warm — slower, cheaper. Data past your active window goes cold in object storage — rarely touched but retained for compliance. A sensible tiering policy is often the difference between a storage bill that's sustainable and one that isn't.
What Good Looks Like
A well-built log processing layer has these properties:
Nothing is silently dropped. Unrecognised events are stored raw and tagged, never discarded. Unparsed volume is a monitored metric.
One schema, shared vocabulary. Every source normalises to the same shape, with consistent field names and values. login is login everywhere.
The schema was designed to grow. New log sources slot in without a migration. Adding a SaaS tool is a new parser, not a rewrite.
Enrichment happens in the pipeline, not at query time. Geolocation, ownership, identity, and threat tags are attached as events flow through — so downstream queries are fast and analysts see context immediately.
Storage matches the query pattern. Columnar for analytics, search index for investigation, object storage for raw retention, with hot/warm/cold tiering to keep costs sane.
The raw original always survives. However aggressively you process, the untouched event is retained cheaply and independently.
What It Costs and How Long It Takes
Honest framing: a processing pipeline that handles a couple of real source types end-to-end — receive, parse, normalise into a well-designed schema, basic enrichment, and a columnar store you can query — is typically a four-to-six-week piece of work as part of a Phase 1 build. It's rarely a standalone project; it's the plumbing that makes collection and detection worth having.
The cost that matters isn't the initial build — it's the schema. Time spent designing the normalisation schema before you write a single parser is the highest-leverage time in the whole project. A schema you can extend without migrations saves months later; a schema you have to rip out and replace in month nine can effectively force a partial rebuild of every module that reads from it. This is the single decision we'd most encourage a team to slow down on. Everything else in this module is competent engineering; the schema is architecture.
Related guides
- The data collection layer that feeds this pipeline
- Threat intelligence enrichment: matching events to known-bad indicators
- Building the detection engine on top of normalised events
- The complete AI-native security platform architecture
- AI agent security: what business owners need to know
- Our AI development services
We Build the Unglamorous Layers Properly
The log processing engine is exactly the kind of work that separates a platform that scales from one that stalls at month nine. We'd start by designing a normalisation schema for your actual sources — the one decision that's cheap to get right now and painful to fix later — and build the pipeline around it: fail-safe parsing, in-flight enrichment, and a storage layout that survives real volume.
If you're planning a security platform, or any system that has to turn millions of messy events into something queryable and useful, we're happy to walk through the data engineering for your specific case.
Talk to us about your platform — no commitment, just a conversation.
Frequently Asked Questions
Why can't I just store raw logs and query them when I need to?
You can, but you'll pay for it on every query. Raw logs are in dozens of incompatible formats, so any question that spans more than one source ("all failed logins everywhere in the last hour") requires per-source parsing logic every single time you ask it. Normalising once, up front, means every downstream query, detection rule, and AI feature works against one consistent shape. The processing pipeline moves that cost from query time — where you pay it repeatedly — to ingest time, where you pay it once.
What is log normalisation, in plain terms?
It's mapping events from different sources onto one common schema. A failed login looks completely different in a Linux auth log, a Windows event, and an Okta sign-in record, but they all describe the same thing. Normalisation turns each of them into an identical structured record — same field names, same values for the same concepts — so the rest of the platform can treat them uniformly. The line Failed password for root from 10.0.0.15 becomes a JSON object with user, action, status, ip, and severity fields that mean exactly the same thing regardless of where the event came from.
Why is ClickHouse recommended for security logs?
Security analytics is overwhelmingly aggregation across enormous numbers of rows on a small number of columns — counting, grouping, and time-bucketing billions of events. Columnar databases like ClickHouse are built for precisely that access pattern and stay fast at a volume where a traditional row-based database would grind to a halt. It's usually paired with a search index like OpenSearch for full-text investigation and object storage for cheap raw retention, because no single store is ideal for all three jobs.
Why is the schema so hard to change later?
Because everything depends on it. Every parser writes into the schema, every detection rule reads from it, and every stored record — potentially billions — is shaped by it. Change a field name or the meaning of a value after the fact, and you have to update every parser, migrate all the historical data, and re-test every rule that relied on the old shape. That's why a well-designed, extensible schema up front is the highest-leverage decision in the whole module, and why a bad one quietly taxes every later stage of the platform.
What does enrichment actually add to a log event?
Context the raw event never carried. Geolocation turns an IP into a country and network owner. Asset ownership ties the event to a real machine and the team that owns it. Identity resolution connects a username to an actual person or confirms it's a service account. Threat-intelligence tagging flags the event if its IP, domain, or file hash appears on a known-bad feed. Together these turn a thin structured record into something an analyst or a detection rule can act on immediately — and they're attached in the pipeline, so downstream queries stay fast.
How much data can a pipeline like this realistically handle?
A properly built pipeline scales horizontally, so the ceiling is mostly a function of how much infrastructure you're willing to run rather than a hard architectural limit. The streaming buffer at the front (Kafka or Redpanda) absorbs bursts, parsing and enrichment scale out across workers, and a columnar store with hot/warm/cold tiering keeps both query speed and cost under control as volume grows. Typical mid-sized deployments comfortably handle millions of events a day; the design pattern is the same whether you're processing a few million or a few billion, which is why getting the foundations right early matters so much.