The Data Collection Layer: How an AI Security Platform Sees Everything

You Cannot Detect What You Cannot See

Every clever thing an AI security platform does — the enrichment, the correlation, the conversational copilot, the automated response — depends on one unglamorous fact: it has the data. If an event never gets collected, no amount of AI further up the stack can reason about it. It simply didn't happen, as far as the platform is concerned.

This is why the data collection layer is the foundation the whole AI-native security platform stands on. It is also the layer most likely to be quietly incomplete. A platform that ingests AWS CloudTrail but misses the SSH logins on the servers, or watches the servers but ignores the SaaS apps where half the company's data actually lives, has blind spots — and attackers are drawn to blind spots the way water finds the crack in a wall.

The goal of this module is deceptively simple to state and genuinely hard to deliver: collect security-relevant events from every possible source, reliably, at a cost you can live with. This piece walks through what "every source" really means, the engineering tensions you have to resolve to get there, and how to tell a thorough collection layer from a superficial one.

What "Every Source" Actually Covers

Coverage is not one checklist. It's several overlapping domains, each with its own log formats, access methods, and failure modes. A serious collection layer reaches into all of them.

Cloud providers

For most modern businesses, the cloud control plane is the crown jewels. The events that matter here are less about traffic and more about who did what: IAM changes, new access keys, role assumptions, and permission grants. On AWS that means CloudTrail as the backbone, plus VPC Flow Logs for network movement, S3 access logs, RDS and Lambda logs, and the findings that GuardDuty and Security Hub already produce. Azure and GCP have their equivalents — activity logs, audit logs, and their native threat-detection findings.

Pulling in the provider's own findings (GuardDuty, Security Hub) is worth calling out: you are not just collecting raw events, you are collecting the cloud's own opinions about them, which the platform can then correlate with everything else.

Linux and Windows servers

Servers are where a foothold turns into a breach. On Linux, the high-value signals are SSH logins, sudo usage, cron jobs, auditd records, kernel messages, and syslog — plus the application logs that sit on top: nginx, Apache, MySQL, PostgreSQL, Redis, and Docker. On Windows, it's the Windows Event Logs, PowerShell activity, RDP sessions, Active Directory events, service and registry changes, and Defender output. A failed-password line or an unexpected sudo at 3am is often the first honest sign that something is wrong.

Containers and Kubernetes

Containerised estates move fast and leave thin trails. At the Docker level you want container creation and deletion, image pulls, known image vulnerabilities, and container network traffic. At the Kubernetes level, the events that matter are pod creation, secrets access, RBAC changes, node status, ingress logs, and — critically — the API server logs, which are the audit trail for the entire cluster. A pod that suddenly reads a secret it never touched before is exactly the kind of signal that only exists if you collected it.

SaaS applications

This is the domain teams most often forget, and it's often where the sensitive data actually lives. Microsoft 365 and Google Workspace hold email and files. Slack and Zoom hold conversations. Dropbox and Salesforce hold documents and customer records. GitHub, GitLab, and Bitbucket hold your source code and deployment keys. Each exposes an audit API — sign-ins, sharing changes, admin actions, repository access — and collection here is almost always agentless, pulling from those APIs on a schedule.

Network devices

Finally, the network layer: firewalls, VPN concentrators, routers, switches, proxies, DNS resolvers, and IDS/IPS sensors. DNS logs in particular are underrated — a lot of malware phones home over DNS, and the request to a suspicious domain is often visible long before anything else fires.

The Engineering Tensions You Have to Resolve

Listing the sources is the easy part. The reason collection is hard is that every source forces a set of trade-offs, and getting them wrong is expensive to fix later.

Agent-based vs agentless

The first fork in the road. An agent is a small piece of software you install on the server or endpoint. It sees everything locally — processes, file changes, kernel events — and can collect deep signals that no remote API exposes. The cost is operational: you have to deploy it, keep it updated, and account for the resources it consumes on every host.

Agentless collection pulls from an API or a log endpoint the source already exposes — CloudTrail, the Microsoft 365 audit API, a firewall's syslog stream. Nothing to install, which makes it fast to onboard and impossible to break the host. The cost is depth: you only see what the API chooses to expose, and often at a delay.

In practice a good platform is not dogmatic about this. Cloud and SaaS are almost always agentless — there's no host to put an agent on, and the APIs are the right interface. Servers and endpoints usually justify an agent, because the deep signals (a suspicious process spawning, a config file changing) simply aren't available any other way. The right answer is a blend, chosen per source.

Push vs pull

Related but distinct. In a push model, the source sends events to the platform — a server's agent streams logs out, a firewall forwards syslog. In a pull model, the platform reaches out on a schedule and asks the API for what's new. Push gives you lower latency and works well for high-volume streaming sources. Pull is simpler to secure (the platform initiates the connection, so you're not opening inbound ports on every host) and is the natural fit for SaaS APIs. Most real deployments run both, and the collection layer has to handle each cleanly.

Log volume and cost

This one blindsides people. Security logs are enormous. VPC Flow Logs alone can dwarf everything else. A mid-sized estate can easily generate hundreds of gigabytes a day, and if you're paying a per-gigabyte ingest fee — the pricing model of most incumbent SIEMs — costs scale in a way that quietly punishes you for having good visibility. The engineering answer is to filter at the edge (drop the genuinely useless noise before it travels), tier your storage (hot store for recent data, cheap object storage for older retention), and choose a log store that doesn't fall over at real volume. This is where the choice of the log processing pipeline downstream — and a columnar store like ClickHouse — earns its keep.

Coverage and reliable delivery

The last two tensions are about trust. Coverage is the discipline of knowing what you're not collecting. A collection layer should track which sources are configured and, more importantly, surface the ones that aren't — an unmonitored server is worse than a known gap because nobody's thinking about it. Reliable delivery is the promise that once an event is generated, it actually arrives. That means buffering when the pipeline is busy, retrying on failure, and detecting silence — if a source that normally sends 10,000 events an hour goes quiet, that's either a broken collector or an attacker who disabled logging, and both need an alert.

Thorough vs Superficial Collection

The difference between a collection layer that protects you and one that looks like it does comes down to details that are invisible in a demo.

Neither column is a demo you can tell apart in five minutes — both light up dashboards. The difference shows the day an incident happens: whether the event you need to reconstruct what happened was actually captured, or whether it fell into a gap nobody was tracking.

What Good Data Collection Looks Like

If you're evaluating or specifying this layer, these are the properties that separate a foundation you can build on from one you'll have to rip out.

Breadth by design. The architecture assumes new source types will keep arriving and absorbs them without a rewrite. A new SaaS app or a new cloud service should be a configuration exercise, not an engineering project.

A blend of agent and agentless. Deep host signals where they matter, API pulls where that's the right interface — chosen per source, not applied dogmatically.

Explicit coverage tracking. The platform knows what it's collecting and what it isn't, and treats an unmonitored source as a visible gap rather than silence.

Reliable delivery with silence detection. Events are buffered and retried, and a source going unexpectedly quiet raises an alert — because disabling logging is a classic early move in an intrusion.

Cost-aware ingestion. Filtering at the edge and tiered storage, so good visibility doesn't become a bill that pressures you into collecting less.

Clean handoff downstream. Collected events land in a form the log processing pipeline and detection engine can immediately work with, rather than a mess the next layer has to untangle.

A Concrete Example

Picture a 40-person logistics firm running a modest but real estate: a handful of AWS accounts, a dozen Linux servers, Microsoft 365 for email and files, GitHub for code, and a firewall at the edge.

Before a proper collection layer: they had CloudTrail flowing into an S3 bucket nobody read, and their servers wrote logs locally. When a developer's laptop was compromised and the attacker used a leaked access key to spin up resources, the evidence existed — in CloudTrail — but it was never being watched, and the M365 sign-in from an unfamiliar country that preceded it was never collected at all. They found out from their cloud bill.

After: CloudTrail, VPC Flow Logs, and GuardDuty findings feed the platform in near real time; a lightweight agent on each server streams SSH and sudo events; the M365 and GitHub audit APIs are pulled on a schedule; and the firewall forwards DNS and connection logs. The same attack now surfaces as a sequence — an anomalous M365 sign-in, followed by GitHub key access, followed by unusual IAM activity in CloudTrail — that correlates into a single alert hours before any resources spin up. Nothing changed about the attack. What changed is that every step of it was being collected.

Cost and Timeline Reality

Honest framing: the collection layer is unglamorous work, and it's where a surprising amount of the effort in a security platform actually goes. Onboarding a source is rarely just "point it at the API." Each one has its own auth model, its own quirks, its own rate limits and edge cases, and the long tail of sources is where the time disappears.

A useful first cut — cloud collection for one provider plus Linux server agents, delivered reliably into a processing pipeline — is typically a matter of a few weeks for a focused team. Broad coverage across cloud, servers, containers, SaaS, and network, with proper coverage tracking and silence detection, is a longer effort measured in months and never quite "finished," because new sources keep appearing.

The expensive mistake here is the same one that haunts the rest of the platform: architecting the first version in a way that can't absorb new source types without a rewrite. Getting the ingestion abstraction right at the start is cheap. Retrofitting it once ten source-specific collectors have hard-coded their own assumptions is not. The same design discipline that matters when securing any AI agent applies here — decide the hard structural questions early, when they're still cheap to change.

Related guides

• Building an AI-native security operations platform: the full architecture
• The log processing pipeline: parse, normalise, enrich at scale
• Building the detection engine
• Cloud security posture management
• AI agent security: what business owners need to know
• Our AI development services

We Build the Foundation Right

Data collection is the layer that decides whether everything above it works. We treat it as core engineering, not plumbing to rush through — a blend of agent and agentless collection chosen per source, explicit coverage tracking so you know what you're not seeing, and reliable delivery with silence detection built in from the start.

If you're planning a security platform, or you suspect your current visibility has gaps you can't quite name, we're happy to map out what a complete collection layer would look like for your specific estate.

Talk to us about your platform — no commitment, just a conversation.

Frequently Asked Questions

What is the data collection layer in a security platform?

It's the module responsible for gathering security-relevant events from every source in your environment — cloud providers, servers, containers, SaaS applications, and network devices — and delivering them reliably to the rest of the platform for processing and analysis. It's the foundation everything else depends on: enrichment, detection, and the AI copilot can only reason about events that were actually collected. If a source isn't being collected, it's a blind spot no amount of downstream intelligence can compensate for.

Should I use agent-based or agentless collection?

Both, chosen per source. Agentless collection — pulling from APIs like CloudTrail or the Microsoft 365 audit log — is the right fit for cloud and SaaS, where there's no host to install anything on and nothing to break. Agent-based collection makes sense for servers and endpoints, where deep signals like process activity and file changes simply aren't exposed by any remote API. A platform that insists on only one approach is either missing depth on your servers or adding operational overhead where an API would have done the job.

Which log sources are most commonly missed?

SaaS applications and DNS logs, in our experience. Teams tend to focus on cloud and servers because those feel like "infrastructure," and forget that Microsoft 365, Google Workspace, GitHub, and Slack hold much of the sensitive data and are common initial attack vectors. DNS logs are the other blind spot — a lot of malware communicates over DNS, and a request to a malicious domain is often the earliest visible signal of a compromise, but only if you're collecting it.

How much do security logs cost to collect and store?

More than most people expect, and the pricing model matters as much as the volume. A mid-sized estate can generate hundreds of gigabytes of security logs a day, and incumbent SIEMs typically charge per gigabyte ingested — which means good visibility quietly becomes an expensive habit. A well-designed collection layer controls this by filtering genuinely useless noise at the edge before it travels, and tiering storage so recent data sits in a fast store while older retention moves to cheap object storage. The store you choose downstream has a large effect on the total bill.

How do I know if a log source has stopped sending data?

Through silence detection, which a thorough collection layer builds in deliberately. Each source has a normal baseline volume, and the platform watches for unexpected drops — if a source that usually sends thousands of events an hour suddenly goes quiet, that raises an alert. This matters for two reasons: a silent source might be a broken collector you need to fix, or it might be an attacker who disabled logging to cover their tracks, which is a classic early move in an intrusion. Silence should never pass unnoticed.

Can we start with a few sources and add more later?

Yes, and that's the sensible way to do it — provided the architecture is built to absorb new source types without a rewrite. A good first phase is often one cloud provider plus Linux server agents, delivered reliably into the processing pipeline, which already gives real visibility. The critical constraint is the ingestion abstraction: it has to treat "a new source" as a configuration exercise rather than an engineering project. Get that right at the start and expanding coverage is easy; get it wrong and every new source becomes a fight.