Security and operations teams are not short of alerts. They are short of time to investigate them properly. Every alert that matters starts the same way: someone has to gather context from half a dozen systems before any real thinking can happen. That gathering is slow, repetitive, and easy to get wrong when the same analyst is on their fifth alert of the night. It is also why so many alerts get closed after a glance rather than a proper look.
An AI investigation agent targets that specific bottleneck. It does not replace the analyst, and it should not be allowed to take action on its own. What it does is run the standard evidence-gathering queries automatically, connect the results, compare them against past incidents, and hand a human a cited summary with a proposed root cause and an honest confidence rating.
This article explains what the agent does step by step, how it gathers evidence through narrow read-only tools, how retrieval over past incidents improves its conclusions, and a worked example of a crypto-mining investigation. It also covers the guardrails that keep it safe, what a good implementation looks like, questions to ask before building one, and realistic cost and timeline expectations.
The First Hour Is the Expensive Part
An alert fires. Something is wrong with a server, or a login looks suspicious, or a process is behaving oddly. What happens next, in most organisations, is that a human being opens six or seven browser tabs — exactly the manual work an AI investigation agent is built to replace.
They check the cloud console for recent changes. They SSH into the box and run top and ps. They pull the last day of auth logs. They cross-reference an IP against a threat feed. They squint at a DNS query history. They read a firewall rule. Forty-five minutes later, they have assembled — in their head, or in a hurried notepad file — a picture of what happened. Most of that time was not analysis. It was fetching, formatting, and pivoting between tools that don't talk to each other.
That first hour is where an AI investigation agent earns its place. It is the module that does the fetching and the pivoting automatically, then hands a human a structured, evidence-backed summary with a proposed root cause. It doesn't replace the analyst's judgement. It replaces the tab-switching. This piece is part of the broader AI-native security platform architecture, and sits directly downstream of the detection engine that raises the alert in the first place.
What the Agent Actually Does
The investigation agent is best understood as a small, disciplined workflow rather than a magic box. When a detection fires — or when a human simply asks a question — the agent works through a sequence of evidence-gathering steps, each one a call into the data layer:
- Collect the alert context — what triggered this, on which asset, at what time.
- Collect the process list — what was running on the affected host around that time.
- Collect user and session history — who logged in, from where, and when.
- Collect cloud events — API calls, instance changes, IAM modifications from the provider's audit trail.
- Collect network and firewall logs — inbound and outbound connections, blocked and allowed.
- Collect DNS queries — what the host was trying to resolve, which is often where command-and-control betrays itself.
- Collect threat and malware intelligence — is this IP, hash, or domain known-bad?
- Summarise and propose a root cause — with a confidence rating and the evidence that supports it.
The important thing is that none of these are guesses. Each step is a real query against real data — the same queries an analyst would run by hand, executed in seconds and assembled into one coherent narrative rather than seven disconnected browser tabs.
How the Agent Gathers Evidence
Under the hood, the agent doesn't have free rein over your systems. It has a defined set of tools — strictly typed functions that fetch specific kinds of data. Something like get_process_list(host, time_window), get_cloud_events(account, time_window), get_dns_history(host, time_window), lookup_threat_intel(indicator). The model decides which tools to call and in what order, but it can only call the tools it has been given, and each tool returns only the data it is scoped to return.
This is the same architectural discipline that matters for any AI agent that touches business systems: the model's power comes from reasoning over data, not from open-ended access. The agent is smart about sequencing — if the process list turns up an unfamiliar binary, it will pivot to DNS and network logs for that process rather than blindly collecting everything. But it is fetching through a narrow, auditable interface, and every call it makes is logged.
The reasoning loop is straightforward: look at what's been gathered so far, decide what evidence would be most useful next, call the relevant tool, incorporate the result, and repeat until the picture is complete enough to summarise. A well-built agent knows when to stop — it isn't trying to boil the ocean, it's trying to answer one question.
Learning From History: RAG Over Past Incidents
Raw evidence gathering is only half the value. The other half is context: has anything like this happened before?
This is where retrieval-augmented generation (RAG) over your incident history comes in. Every past investigation — the evidence, the conclusion, the remediation — is embedded and stored. When a new investigation runs, the agent retrieves the most similar historical incidents and folds them into its reasoning. So instead of analysing a spike in isolation, the agent can note: "This pattern of DNS queries to a mining pool matches an incident from three weeks ago on a different host — the root cause then was an exposed Docker API." That's institutional memory that would otherwise live only in a departed analyst's head.
RAG also grounds the agent. Rather than free-associating from its training data, it reasons from your environment's actual history, which sharply reduces the confident-but-wrong failure mode that plagues naive LLM deployments.
A Worked Example: "Why Did the Server CPU Reach 100%?"
Here is the kind of investigation the agent handles end to end. A user — an ops engineer, not a security specialist — types a plain question into the security copilot: "Why did server web-prod-03 hit 100% CPU last night?"
The agent works through it:
| Step | Tool call | What it found |
|---|---|---|
| 1 | get_metrics(host, window) | CPU pinned at 100% from 02:14 onward; sustained, not a spike |
| 2 | get_process_list(host, 02:14) | A process named kdevtmpfsi consuming 98% CPU — not a known system binary |
| 3 | get_containers(host) | The process was spawned inside a Docker container with an exposed API port |
| 4 | get_cron(host) | A cron entry re-downloading and relaunching the binary every 10 minutes |
| 5 | get_network(host, window) | Outbound connections to a known mining pool on port 3333 |
| 6 | get_dns_history(host, window) | Repeated lookups of a pool domain flagged in threat intelligence |
| 7 | lookup_threat_intel(hash) | The binary's hash matches a well-documented crypto-mining family |
The agent's summary, in plain language:
Root cause (confidence: high): web-prod-03 is running an unauthorised crypto-miner. The Docker API on this host was exposed to the internet, which allowed an attacker to launch a container running the miner
kdevtmpfsi. A cron job re-installs it every 10 minutes to survive restarts, and it is connecting outbound to a mining pool. This is why CPU is pinned at 100%.Recommended next steps (for human review): isolate the host from the network, remove the malicious cron entry and container, and — critically — restrict the exposed Docker API, which is the actual entry point. Rotate any credentials that were reachable from this host.
What took the agent under a minute would have taken an analyst most of an hour, and the analyst might not have connected the cron persistence to the exposed Docker API on the first pass. Notice, too, what the agent did not do: it did not kill the process, remove the cron job, or touch the firewall. It investigated and recommended. It did not act.
Benefits of an AI Investigation Agent
Analysts spend their time on judgement, not fetching
Most of the first hour of an investigation is spent opening consoles, running the same queries, and copying results into notes. The agent runs those queries in parallel and assembles the output into one narrative. The analyst starts from a structured summary with cited evidence, so their attention goes to the part only a human can do well: deciding whether the conclusion holds and what to do about it. The work becomes reviewing a case file rather than building one from scratch.
More alerts get a proper look
When every investigation costs an hour, teams quietly ration them, and many alerts are closed after a glance. Cutting the evidence-gathering step to minutes changes that arithmetic. Alerts that would have been dismissed get a consistent first pass, which means fewer real incidents slip through simply because nobody had time to check. This is especially valuable overnight and at weekends, when coverage is thinnest.
Consistent investigations regardless of who is on shift
A tired analyst on their fifth alert of the night skips steps. A junior analyst may not know that DNS history is often where command-and-control shows itself. The agent runs the same evidence checklist every time and pivots based on what it finds, so the quality of the first pass no longer depends on experience or time of night. Senior staff can then spend their review time on the cases that are genuinely ambiguous.
Institutional memory that does not leave with people
Retrieval over past incidents means the agent can point out that a new pattern matches something resolved weeks ago, along with the root cause found then. That knowledge usually lives in one experienced analyst's head and disappears when they move on. Folding it into every investigation makes the team's history usable by everyone, including people who joined after the original incident.
Security answers for non-specialists
Because the agent accepts plain-language questions through a copilot interface, ops engineers can ask why a server is misbehaving without learning every security console. They receive an evidence-backed explanation and recommended next steps for a human to approve, which shortens the loop between operations noticing a symptom and security understanding the cause.
AI Investigation Agent Use Cases
Resource abuse on cloud hosts
Unexplained CPU spikes, unusual outbound traffic, or unexpected containers often point to crypto-mining or other abuse of exposed services. The agent checks metrics, process lists, containers, scheduled tasks, network connections, and threat intelligence in sequence, as in the worked example above. The outcome is a root cause that names the entry point, not just the symptom, so remediation closes the hole rather than killing a process that will be relaunched ten minutes later.
Suspicious login triage
A login from an unfamiliar location or at an odd hour may be an attacker or just a travelling employee. The agent pulls session history, previous logins for that user, related cloud API activity, and any threat intelligence on the source IP. The analyst receives a summary showing whether the session did anything unusual after login, which makes the call to dismiss or escalate faster and better documented.
Cloud configuration and IAM changes
Detections often fire on new access keys, permission changes, or instances launched in unusual regions. The agent reconstructs who made the change, from where, and what happened next using the provider's audit trail. That context distinguishes a planned change by the platform team from a compromised identity expanding its own privileges. When the change matches a ticket or a known deployment pattern, the analyst can close the alert with the evidence attached; when it does not, the case arrives already half-investigated.
Malware and command-and-control indicators
When an unfamiliar binary or a suspicious domain appears, the agent connects process data with DNS lookups, outbound connections, and hash reputation. Correlating these sources is tedious by hand but quick for the agent, and it produces a clear picture of whether the host is communicating with known malicious infrastructure.
Plain-language questions from operations teams
Not every investigation starts with a security alert. An engineer asking why a service slowed down or why a host rebooted can get an evidence-based answer that either rules out a security cause or flags one early. The agent becomes a shared investigative tool rather than something only the security team touches.
The Guardrails: Read-Only by Default
That last point is the whole game, and it's worth being emphatic about it.
The investigation agent is read-only. Every tool it holds fetches data. None of them change anything. It cannot stop a process, quarantine a host, revoke a credential, or modify a firewall rule — because it has not been given tools that do those things. This is a deliberate architectural boundary, not a policy you hope the model respects.
Its access is scoped. The agent can read process lists and logs; it cannot read the contents of your database or your customers' files. Each tool is bounded to exactly the telemetry an investigation needs, following the same principle of least privilege that governs securing any AI agent.
Any action requires a human. The output is a recommendation. A person reads the evidence, agrees or disagrees with the proposed root cause, and decides what to do. The confidence rating exists precisely to support that judgement — a "high confidence" root cause backed by seven corroborating pieces of evidence gets treated differently from a "low confidence" hypothesis based on two.
The reason you keep investigation and response separate is not squeamishness — it's blast radius. An investigation agent that got manipulated (through, say, a poisoned log entry crafted to mislead it) can, at worst, produce a wrong summary that a human then sanity-checks. An agent that could both investigate and execute destructive actions could be talked into quarantining a critical production host or revoking a valid admin credential. You never let the thing that reads also be the thing that destroys. Automated response — the playbooks that actually contain a threat — belongs in a separate module with its own, much stricter, approval gates. That's a different conversation, and a different post.
Common AI Investigation Agent Mistakes
Giving the agent tools that change state
It is tempting to add "just one" remediation tool, such as killing a process or blocking an IP, so the agent can finish the job. That single addition turns a worst case of a wrong summary into a worst case of a destructive action triggered by a poisoned log entry. Keep every investigation tool read-only and put containment in a separate module with its own approval gates.
Building the agent before the data layer
Teams often start with the model and prompts because that is the visible, exciting part. If process lists are patchy or the cloud audit trail has gaps, the agent will investigate incomplete evidence and reach shaky conclusions with a confident tone. Fix collection and normalisation for your most common sources first, then build tools on top of them.
Accepting conclusions without citations
A summary that reads well but does not link to the log lines and events behind it cannot be checked. Analysts under time pressure will start trusting it anyway. Require every claim in the root cause to cite its evidence, and treat an uncited conclusion as a defect rather than a style choice.
Hiding uncertainty
Agents that always present a single confident answer train their users to stop questioning them. When evidence is thin or contradictory, the summary should say so and hand the case to a human labelled as uncertain. A low-confidence flag on the right cases is more valuable than a polished guess.
Letting tools collect everything
Broad tools that return entire log streams or arbitrary data look convenient but waste context, raise costs, and expand what a compromised agent could read. Narrow, typed tools scoped to a host and time window keep investigations focused and keep access limited to the telemetry an investigation actually needs.
AI Investigation Agent Best Practices
A well-built investigation agent has these properties:
- Every conclusion is backed by cited evidence. The summary links to the actual log lines, process entries, and events it reasoned from. You can check its work, and reviewers can spot a weak link in the chain before acting on it.
- It rates its own confidence honestly. It distinguishes a well-corroborated root cause from an educated guess, and says which is which. Low-confidence cases are routed to a human with the open questions listed.
- It is read-only, and provably so. Not "instructed not to act" — incapable of acting, because it holds no tools that mutate state. Review the tool list whenever a new capability is added.
- Its access is scoped to telemetry. It reads security-relevant logs and metadata, not arbitrary business data. Each tool takes a specific host, account, or indicator and a bounded time window.
- Every tool call is logged. You have a full audit trail of exactly what the agent looked at and when, which also helps when tuning which evidence it gathers first.
- It learns from your history. RAG over past incidents means it gets more useful as your environment accumulates resolved cases. Make sure analysts record the confirmed root cause and remediation when they close a case, or there is nothing good to retrieve.
- Recommendations go to a named human owner. Every proposed next step, such as isolating a host or rotating credentials, is assigned to a person who approves or rejects it. The decision and the reasoning are recorded alongside the agent's summary so the case history stays complete.
- It is evaluated against known incidents. Replay resolved cases through the agent and compare its proposed root cause with what analysts concluded. Repeat after changes to prompts, tools, or models so regressions are caught before they reach a live investigation.
Questions to Ask Before You Build One
"What tools does the agent have, and can any of them change state?" The answer should be a specific list, and every tool should be read-only. If anything on the list can modify, delete, or execute, that's a design you need to challenge.
"How does it show its work?" A root cause with no cited evidence is just an assertion. You should be able to trace every conclusion back to the raw data that supports it.
"How is confidence calculated, and what happens at low confidence?" A good agent hands ambiguous cases to a human clearly labelled as uncertain, rather than presenting a shaky guess with false authority.
"Where does investigation end and response begin?" There should be a hard, architectural line between the agent that recommends and the system that acts. If that line is blurry, the blast radius of a mistake grows dramatically.
What It Costs and How Long It Takes
An investigation agent is not a standalone product — it sits on top of a working data layer. If your logs are already being collected and normalised, building a genuinely useful first version of the agent — evidence-gathering tools for the most common sources, a summarisation loop, and confidence rating — is typically a six-to-ten-week engagement for a small team. Adding RAG over historical incidents comes shortly after, once you have a corpus of resolved cases worth retrieving from.
The honest caveat: the agent is only as good as the data beneath it. If your process lists are patchy or your cloud audit trail has gaps, the agent will faithfully investigate incomplete evidence and reach shakier conclusions. Most of the real engineering effort in this module is not the AI — it's the plumbing that makes clean, queryable telemetry available for the tools to fetch. Budget accordingly, and be sceptical of any timeline that assumes the data layer is already perfect.
Related guides
- Building an AI-native security operations platform
- Building the detection engine that raises the alerts
- The AI security copilot: natural-language queries over your data
- Multi-agent architecture for security operations
- AI agent security: what business owners need to know
- Our AI agent development services
We Build Investigation Agents That Know Their Limits
The investigation agent is one of the most satisfying modules to build well, because the value is so immediate and the guardrails are so clear. We'd start by scoping the evidence-gathering tools around your actual sources, wiring in a read-only data layer, and proving out the worked-example flow — an alert in, a cited, confidence-rated root cause out — before layering in RAG over your incident history.
Crucially, we build it read-only by design and keep investigation cleanly separated from response, so that automating the analyst's first hour never means handing an unsupervised agent the keys to your production estate.
Talk to us about your platform — no commitment, just a conversation.
Frequently Asked Questions
What is an AI investigation agent in security operations?
It's an AI system that automates the evidence-gathering an analyst does at the start of an investigation. When an alert fires, the agent calls a set of read-only tools to collect the relevant process lists, logs, cloud events, DNS history, and threat intelligence, then reasons over that evidence to produce a summarised, confidence-rated root cause. It replaces the tedious pivoting across consoles that eats the first hour of most investigations — but it recommends rather than acts, leaving the decision and any remediation to a human.
How is an investigation agent different from an automated response system?
Investigation agents read and recommend; response systems act. The investigation agent gathers evidence and proposes a root cause, but it holds no tools that can change anything — it cannot kill processes, quarantine hosts, or revoke credentials. Automated response (a SOAR-style module) runs containment playbooks that do change state, and it lives behind much stricter approval gates. Keeping the two separate is a deliberate safety boundary: the thing that investigates should never also be the thing that can destroy, because that dramatically increases the blast radius of any mistake or manipulation.
Can I trust the root cause the agent proposes?
Trust it the way you'd trust a junior analyst's write-up — as a well-organised starting point, not a final verdict. A well-built agent cites the specific evidence behind every conclusion and rates its own confidence, so you can check its work rather than take it on faith. High-confidence conclusions backed by several corroborating pieces of evidence are usually reliable; low-confidence hypotheses are flagged as such and handed to a human. The agent is a force multiplier for analyst judgement, not a replacement for it.
Why does the investigation agent have to be read-only?
Because an agent that can both investigate and take destructive action is far more dangerous if it's ever wrong or manipulated. A read-only agent that's fed a misleading log entry can, at worst, produce a wrong summary that a human sanity-checks. An agent with the power to act could be talked into quarantining a critical production server or revoking a valid credential. By giving the investigation agent only tools that fetch data — and none that change state — the worst-case outcome is a bad recommendation, not a bad action. This is enforced architecturally, not by trusting the model to behave.
What data does the investigation agent need to work well?
It needs clean, queryable telemetry: host process lists, authentication and session logs, cloud provider audit trails, network and firewall logs, DNS query history, and a threat intelligence feed. The agent is only as good as this underlying data — gaps or patchy collection lead to shakier conclusions. In practice, most of the engineering effort in building an investigation agent goes into the data layer that makes this telemetry reliably available, not into the AI reasoning on top. Adding RAG over your own resolved-incident history further improves results by grounding the agent in your environment's actual patterns.
How long does it take to build an investigation agent?
Assuming your logs are already being collected and normalised, a useful first version — read-only evidence-gathering tools for your most common sources, a summarisation and reasoning loop, and confidence rating — is typically a six-to-ten-week engagement for a small, focused team. Adding retrieval over historical incidents follows once you've accumulated a corpus of resolved cases. The timeline stretches considerably if the data layer isn't ready, because the plumbing that supplies clean telemetry is where most of the real work lives.
Is an AI investigation agent worth it for a small security team?
Often it matters more for small teams than large ones. A two- or three-person team cannot spend an hour gathering evidence on every alert, so many alerts get closed with a quick look. An agent that does the first pass in minutes lets a small team investigate more alerts properly. The prerequisite is the same at any size: logs must already be collected and queryable. If they are not, start with the data layer, because an agent over patchy telemetry will produce confident-sounding but unreliable summaries.
Conclusion
The expensive part of most security investigations is not the analysis. It is the first hour of fetching process lists, logs, cloud events, and threat intelligence from tools that do not talk to each other. An AI investigation agent automates that gathering and turns it into a cited, confidence-rated root cause a human can check quickly.
The design choices matter more than the model. The agent should hold only read-only, strictly typed tools, so the worst outcome of a mistake or a poisoned log entry is a wrong summary rather than a destructive action. Every conclusion should link back to the evidence behind it. Retrieval over your own past incidents makes it more useful over time, and a hard line between investigation and automated response keeps the blast radius small. Frameworks like MITRE ATT&CK are useful for mapping which techniques your evidence tools should cover.
The main caveat is data quality. An agent investigating incomplete telemetry will faithfully reach shaky conclusions, so most of the real engineering effort sits in the data layer, not the AI.
If you are planning a security platform and want an investigation agent built read-only from the start, our AI agent development team can help you scope it.