Most security teams do not suffer from a shortage of data. They suffer from too many alerts and too little confidence in any of them. A security detection engine is the part of a platform that fixes or causes that problem: it takes millions of authentication logs, process events, and network records and decides which few deserve a human's attention today.
That decision has consequences in both directions. Miss a real intrusion and the rest of the stack, from collection to response, was wasted effort. Surface hundreds of low-value alerts and the team stops reading them, which is how real incidents get dismissed as noise. For small teams without a dedicated security operations centre, the quality of detection largely determines whether a security platform is usable at all.
This article walks through how a detection engine works in practice. It starts with what a detection actually is and why correlation across events matters more than any single event. It then covers the three detection strategies, static rules, behavioural analytics, and anomaly detection, with example rules mapped to MITRE ATT&CK techniques. From there it explains risk scoring, the alert fatigue problem, where AI triage fits, what a trustworthy engine looks like, the questions to ask before building one, and realistic effort and timelines.
The Layer That Decides What Matters
Every other part of a security platform exists to feed this one. Collection gathers the events, processing normalises them, threat intelligence enriches them with context — and then the detection engine has to answer the only question that counts: out of the millions of events that arrived today, which handful should a human actually look at?
Get this wrong and nothing else matters. A platform that surfaces a thousand alerts a day is functionally identical to a platform that surfaces none — because a team that can't triage a thousand alerts will triage zero. The detection engine is where a security product either earns trust or quietly loses it.
This is the heart of the system. It's also the part where "AI security" is most often oversold and least often built well. Below is what a detection engine actually does, how the different detection strategies fit together, and why the hardest problem isn't catching attacks — it's not crying wolf.
What a Detection Actually Is
At its simplest, a detection is a statement of the form: when this pattern appears in the event stream, raise an alert. The pattern might be a single event ("a user was added to the domain admins group") or — far more often — a correlation across many events over time ("more than 20 failed SSH logins from one source within five minutes").
That word correlation is the whole game. Individual events are almost always innocent. One failed login is someone fat-fingering a password. One large file upload is someone sending a video. One PowerShell command is an admin doing their job. Attacks reveal themselves in sequences and volumes — the failed logins that suddenly succeed, the upload that's a thousand times larger than usual, the PowerShell that downloads and then executes.
So the correlation engine is the core primitive. It holds a rolling window of normalised events in memory (or in a fast store like ClickHouse) and continuously evaluates conditions against that window: counts within a time span, sequences in a particular order, the same actor appearing across different event types. When a condition matches, it emits a detection — not yet an alert a human sees, but a candidate.
Three Ways to Detect, and Why You Need All Three
There is no single detection technique that catches everything. A serious engine layers three, because each covers the blind spots of the others.
Static rules are explicit, human-written conditions. "More than 20 failed logins in five minutes." "A process named powershell.exe spawned by a Word document." "A login from a country we've never seen for this user." Rules are fast, cheap, and — crucially — explainable: when one fires, you can say exactly why. Their weakness is that they only catch what someone thought to write down. A novel attack that doesn't match any rule sails straight through.
Behavioural analytics fix part of that gap by learning what "normal" looks like per user and per asset, then flagging deviations. The engine builds a baseline: this employee usually logs in from London on weekdays, touches three internal systems, and uploads around 100MB a week. When the same account suddenly uploads 120GB overnight from a new location, no static rule needs to exist — the deviation from the baseline is itself the signal. This is often called UEBA (user and entity behaviour analytics), and it's where a lot of the genuine value sits, because it catches misuse of legitimate credentials, which static rules struggle with.
Anomaly detection is the statistical and machine-learning layer that catches what neither rules nor per-user baselines can express. It looks across the whole population for outliers: a server whose outbound traffic pattern doesn't match any of its peers, a login time that's a statistical anomaly for the department, a rare parent-child process relationship that almost never occurs in the estate. This is the fuzziest layer — the most prone to noise — and it works best as a ranking signal rather than a standalone alarm.
The mistake is treating these as competitors. They aren't. Static rules give you precision and explainability, behavioural analytics catch credential misuse, and anomaly detection surfaces the genuinely novel. A mature engine runs all three and lets them reinforce each other — a static rule that fires and corresponds to a behavioural anomaly is a much stronger signal than either alone.
Example Detection Rules
Concrete rules make the idea tangible. Here are four that most engines ship early, spanning the categories above, with the MITRE ATT&CK technique each maps to (more on that mapping below).
| Detection | Condition | Type | MITRE ATT&CK | Typical severity |
|---|---|---|---|---|
| SSH brute force | More than 20 failed logins from one source IP within 5 minutes | Static rule | T1110 Brute Force | Medium |
| Impossible travel | Successful login from India, then a successful login from Germany 10 minutes later — physically impossible | Behavioural | T1078 Valid Accounts | High |
| Data exfiltration | User who normally uploads ~100MB uploads 120GB in a day | Behavioural (UEBA) | T1567 Exfiltration Over Web Service | High |
| Suspicious process chain | powershell.exe downloads a payload, then executes a Base64-encoded command | Static + anomaly | T1059 Command & Scripting Interpreter | Critical |
Notice that the impossible-travel rule isn't magic — it's geometry. Two successful logins, two geolocations from enrichment, and the distance divided by the time gap exceeds any plausible travel speed. That's it. Most powerful detections are this mundane underneath, which is exactly why explainability matters: a security lead can understand and trust "these two logins are 6,000km apart, ten minutes apart" far more readily than "the model scored this 0.87."
Risk Scoring: Turning Detections Into a Queue
If every detection produced an equal-priority alert, you'd be back to the thousand-alerts-a-day problem. So the next job is ranking. Every detection gets a risk score, and the engine surfaces the top of the queue.
A workable risk score combines several factors: the inherent severity of the detection type, the value of the asset involved (a detection on a payroll server outranks the same detection on a test box), the sensitivity of the identity (an admin account outranks a contractor), corroboration from threat intelligence (is that source IP a known-bad indicator?), and — importantly — whether multiple detections cluster around the same entity in a short window. Ten weak signals on one host in an hour is a stronger story than one medium signal in isolation.
Scoring is what lets a small team operate the platform. Instead of "here are 400 alerts, good luck," the interface says "here are the six things most likely to be real, ranked, with the evidence attached." That framing is the difference between a tool a generalist IT lead can run and one that needs a trained analyst.
The Real Enemy: False Positives and Alert Fatigue
Here is the honest centre of the whole subject. A detection engine that cries wolf is worse than none at all. A platform with no detections gives a team an accurate sense of its own blindness. A platform that fires forty false alarms a day trains the team to ignore alerts entirely — and the one real alert arrives in a stream the team has already learned to dismiss.
Alert fatigue is not a minor tuning issue. It is the failure mode of security operations, and it's why so many expensive deployments end up as dashboards nobody reads. Every over-eager rule, every anomaly detector left at a hair-trigger threshold, every alert without enough context to act on, contributes to the erosion of trust that makes the whole system worthless.
Fighting it is continuous work, not a one-off. Rules need tuning against the specific environment — the "20 failures in 5 minutes" threshold that's right for a small office is wrong for a service that legitimately retries. Behavioural baselines need a learning period and a way to absorb legitimate change (a genuine office relocation shouldn't fire impossible-travel forever). And there needs to be a feedback loop: when an analyst marks an alert as a false positive, that judgement should feed back into scoring, so the engine gets quieter and sharper over time rather than louder.
Mapping to MITRE ATT&CK, and Where AI Triage Sits
Two things separate a serious engine from a pile of ad-hoc rules.
The first is mapping every detection to MITRE ATT&CK — the industry-standard catalogue of adversary techniques (the T-numbers in the table above). This isn't box-ticking. It gives you a coverage map: which techniques can we actually detect, and where are the gaps? It gives every alert a shared vocabulary, so "T1078 Valid Accounts" means the same thing to your engine, your analyst, and the threat report they read yesterday. And it lets detections chain into a narrative — a brute force (T1110) followed by valid-account use (T1078) followed by exfiltration (T1567) reads as a coherent attack, not three unrelated blips.
The second is AI triage sitting on top of the raw detections, before a human ever sees them. This is where an AI-native platform genuinely differs from a bolted-on chatbot. When a detection fires, an investigation agent gathers the surrounding evidence — recent activity for that user, the process tree, related events on the same host, the reputation of the IP — and produces a plain-language assessment: this looks like a real brute-force attempt that succeeded, here's the evidence, here's the likely next step. Or, just as valuable: this is almost certainly benign — a monitoring script retrying — here's why, deprioritised.
That triage layer is the single biggest lever on alert fatigue. It doesn't replace the detection engine; it filters and enriches its output so that the alerts reaching a human are already contextualised and ranked. The detection engine decides what is suspicious. The AI decides what's worth your attention right now — the same discipline that matters when securing any AI agent that acts on real systems.
Benefits of a Well-Built Detection Engine
When the layers above work together, the payoff is less about catching more and more about what the team can do with what it catches.
A small team can run security operations
Ranked, explained, and triaged alerts mean a generalist IT lead can work the queue without a dedicated analyst. The difference between six contextualised alerts and four hundred raw ones is the difference between a team that investigates and a team that gives up. For companies without a security operations centre, this is often what makes a detection platform usable at all.
Credential misuse becomes visible
Attackers increasingly log in rather than break in. Static rules struggle with that because a valid login looks like any other. Behavioural baselines catch the account that suddenly works from a new country, touches systems it never used, or moves far more data than usual. That covers a class of intrusion that would otherwise only surface after damage is done.
Alerts get trusted again
Explainable detections, a feedback loop on false positives, and AI triage that labels likely-benign activity all push in one direction: fewer alerts, each one more credible. When people trust that an alert is worth reading, they respond faster and more carefully. That trust is the real asset a detection engine builds, and it is easy to lose with a few weeks of noise.
Coverage gaps are known rather than guessed
Mapping detections to MITRE ATT&CK turns "are we protected?" into a concrete list of techniques you can and cannot see. That list guides where to invest next, whether that is a new data source, a new rule, or a behavioural model. It also gives a clear answer when leadership or a customer asks what the platform actually covers.
Incidents read as stories, not fragments
Correlation and technique mapping let the engine connect a brute-force attempt, a subsequent valid login, and a large upload into one narrative. Responders start from a likely sequence of events rather than three unrelated alerts in different queues. That shortens investigations and makes the decision about containment far easier to justify.
Security Detection Engine Use Cases
The same engine handles very different threats. These are the scenarios most detection layers are built to cover first, using the rules and strategies described above.
Brute force against exposed services
SSH, remote desktop, VPN gateways, and web login pages attract constant password-guessing traffic. A correlation rule counting failed logins per source in a short window catches the attempt. The higher-value detection is the follow-up: failures that suddenly turn into a success. Risk scoring raises that combination well above the background noise of failed attempts that never get anywhere, so the team sees the one that worked.
Compromised accounts and impossible travel
When credentials are phished or reused from a breach, the attacker's logins come from unfamiliar places and at unfamiliar times. Impossible-travel checks and per-user baselines flag those sessions even though the password was correct. Enrichment with threat intelligence adds weight when the source address is already known to be malicious. The result is an alert on account takeover early in the intrusion, before the attacker has done much with the access.
Data exfiltration and insider misuse
A user who normally moves modest amounts of data and suddenly uploads a very large volume to an external service is a classic behavioural signal. It might be a departing employee, a compromised account, or a legitimate project. The detection's job is to surface it with the baseline comparison attached, so a person can decide quickly. Combined with asset value, an upload from a system holding customer data ranks above one from a test environment.
Malicious use of built-in tools
Attackers often avoid custom malware and use tools already on the machine, such as PowerShell or scripting interpreters. Static rules catch known patterns like an office document spawning a shell, and anomaly detection flags rare parent-child process relationships nobody wrote a rule for. Together they cover both the familiar techniques and the unusual variants, and the process tree gathered by triage shows the analyst exactly what ran.
Privilege changes and persistence
Adding a user to an administrator group, creating a new service account, or changing authentication settings can be routine administration or the first step of an attacker settling in. Single-event rules on these changes are cheap and explainable. Correlating them with recent suspicious logins on the same account is what separates the worrying ones from the everyday.
Detection Engine Best Practices
A detection engine you'd trust in production comes from a few habits applied consistently, not from any single clever model.
- Layer your detection strategies. Run static rules, behavioural baselines, and anomaly detection together, and give extra weight when two layers agree on the same entity. Each covers the others' blind spots, and agreement between them is one of the strongest signals you have.
- Make every alert explain itself. Each alert should state why it fired in terms a non-specialist can follow, with the triggering events attached. A bare model score invites the team to ignore it.
- Rank before you show. Lead the interface with the few detections most likely to be real, scored on severity, asset value, identity sensitivity, threat intelligence, and clustering. An undifferentiated list pushes all the prioritisation work back onto people.
- Map every detection to MITRE ATT&CK. Tag rules with technique IDs from the moment they are written. The resulting coverage map shows where you are blind and gives everyone a shared vocabulary for alerts.
- Close the false-positive loop. When an analyst marks an alert as benign, feed that judgement back into thresholds and scoring. Review the noisiest rules every month and either tune them, add context, or retire them.
- Put AI triage in front of people. Have an investigation agent gather surrounding evidence and give a plain-language assessment before a human opens the alert, so the queue a person sees is short and already contextualised.
- Tune thresholds to your environment. Defaults from a vendor or another company are a starting point. Measure what normal looks like for your services and users, and set thresholds against that rather than against a generic benchmark.
- Track precision alongside coverage. Record how many alerts each week turned out to be real. A rising false-positive rate is an early warning that the engine is drifting toward the noise that makes teams stop reading.
Questions to Ask Before You Build One
"How do you tune out false positives, and how does that improve over time?" If there's no feedback loop from analyst decisions back into scoring, the engine will get louder, not quieter.
"Which detections are static rules versus behavioural versus anomaly-based?" A serious answer names all three and explains what each covers. "It's all AI" is a red flag, not a feature.
"Can every alert explain why it fired?" If the only answer is a model score, your team will never trust it — and untrusted alerts get ignored.
"How are detections mapped to a framework like MITRE ATT&CK?" This is how you know your coverage and your gaps, rather than guessing.
Common Detection Engine Mistakes
Most failing detection layers fail in predictable ways. These are the ones worth checking for first.
Chasing coverage before precision
Writing hundreds of rules to cover every technique on the ATT&CK matrix looks thorough and produces a queue nobody can work. Early on, a small set of high-confidence detections that fire rarely and are usually right does far more good. Expand coverage only once the existing rules are tuned and the false-positive rate is under control.
Shipping vendor defaults untouched
Default thresholds are written for an average environment that doesn't exist. A retry-heavy service trips a brute-force rule constantly, and a team that travels a lot keeps triggering impossible travel. Leaving defaults in place guarantees noise. Budget time after deployment specifically for tuning each rule against how your systems and people actually behave.
Adding machine learning before the data is clean
Anomaly models trained on messy, partially normalised events find anomalies in the data pipeline rather than in user behaviour. The result is a stream of confident, unexplainable alerts that erode trust quickly. Get collection and processing stable, field names consistent, and rules tuned before introducing models, and introduce them as ranking signals rather than standalone alarms.
Alerts with no context
An alert that says only "suspicious login" forces the analyst to go and collect everything else: who the user is, what they usually do, what happened before and after. Multiply that by dozens of alerts a day and investigation stops. Attach enrichment, baseline comparisons, and related events at detection time, or let a triage agent gather them.
No feedback from analysts to the engine
When analysts dismiss the same false positive every day and nothing changes, they learn that their judgement doesn't matter and stop looking carefully. Without a feedback loop, the engine gets louder as the environment grows. Make marking an alert benign or real a single action, and route those decisions into scoring and the monthly rule review.
What It Costs and How Long It Takes
Honest framing: a basic detection layer — a set of well-chosen static rules over a working processing pipeline, with simple risk scoring — is typically a four-to-six-week piece of work once collection and processing are in place. That alone catches a meaningful share of the obvious attacks and is genuinely useful.
Behavioural analytics add more: you need a learning period to build baselines, and tuning time to get the thresholds right for the specific environment. Realistically that's a further six-to-ten weeks, and it keeps improving with feedback rather than being "done." Anomaly detection and AI triage sit on top of that and are best added once the fundamentals are stable — building them first, before you have clean data and tuned rules, just produces expensive noise.
The most expensive mistake is optimising for detection coverage over detection precision early on. A demo that fires on everything looks impressive and is useless in production. The engineering effort that matters most isn't writing more rules — it's the tuning, scoring, and triage that keep the alert queue short enough that a real team will actually work it.
Related guides
- The full AI-native security platform architecture
- Threat intelligence enrichment: turning indicators into context
- The AI investigation agent: auto-gathering evidence
- The log processing pipeline that feeds detection
- AI agent security: what business owners need to know
- Our AI development services
We Build Detection Engines That Stay Quiet Until They Shouldn't
The hard part of a detection engine isn't catching attacks in a demo — it's building one a small team will still trust in month nine, when the novelty has worn off and the alert queue has to be short, ranked, and explainable every single day. That's a data-engineering and tuning problem as much as an AI one, and it's exactly the kind of system we like to build: real correlation underneath, behavioural baselines in the middle, and AI triage on top that keeps the noise off a human's desk.
If you're scoping a security platform, or trying to fix a detection layer that's drowning your team in false positives, we're happy to walk through your specific case.
Talk to us about your platform — no commitment, just a conversation.
Frequently Asked Questions
What is a security detection engine?
It's the component of a security platform that turns a stream of normalised events into a small number of meaningful alerts. It runs conditions and correlations against the event stream — counts within a time window, sequences of actions, deviations from normal behaviour — and emits a detection whenever a suspicious pattern appears. It sits downstream of collection and processing and upstream of investigation and response, and it's the layer that decides which of the day's millions of events a human should actually look at.
What's the difference between rules, behavioural analytics, and anomaly detection?
Static rules are explicit human-written conditions ("more than 20 failed logins in five minutes") — fast, precise, and explainable, but they only catch what someone thought to write down. Behavioural analytics learn what's normal per user and asset, then flag deviations, which catches misuse of legitimate credentials that no static rule would spot. Anomaly detection is a statistical or machine-learning layer that finds outliers across the whole population — the novel things neither of the others can express. A good engine runs all three, because each covers the others' blind spots.
What is impossible-travel detection?
It's a detection that flags two successful logins for the same account from locations too far apart to have travelled between in the elapsed time — for example, a login from India followed ten minutes later by one from Germany. Underneath it's simple geometry: two geolocations from enrichment, the distance between them, divided by the time gap. If the implied speed exceeds any plausible travel, it's physically impossible, so one of those sessions is almost certainly using stolen credentials.
Why are false positives such a big problem?
Because a detection engine that cries wolf is worse than none at all. A platform with no alerts at least gives a team an accurate sense of its own blind spots. A platform that fires dozens of false alarms a day trains the team to ignore alerts entirely — and then the one real alert arrives in a stream everyone has already learned to dismiss. Alert fatigue is the single biggest failure mode of security operations, which is why tuning, risk scoring, and AI triage matter more than raw detection coverage.
What is MITRE ATT&CK and why map detections to it?
MITRE ATT&CK is an industry-standard catalogue of adversary techniques, each with an identifier (like T1110 for brute force or T1078 for valid-account abuse). Mapping every detection to it gives you three things: a coverage map showing which techniques you can and can't detect, a shared vocabulary so an alert means the same thing to your engine and your analyst, and the ability to chain detections into a coherent attack narrative rather than a list of unrelated blips.
How does AI triage reduce alert noise?
AI triage sits on top of the raw detections, before a human sees them. When a detection fires, an investigation agent automatically gathers the surrounding evidence — the user's recent activity, the process tree, related events on the same host, the reputation of the IP — and produces a plain-language assessment of whether it's likely real or benign, with the reasoning attached. The detection engine decides what is suspicious; the AI decides what's worth attention right now. Done well, it's the single biggest lever for keeping the alert queue short enough that a small team will actually work it.
Do small companies need a custom detection engine, or is an off-the-shelf SIEM enough?
For many small companies, a managed SIEM or XDR product with a well-tuned default rule set is the right starting point, because building and maintaining detection logic takes ongoing engineering time. A custom engine makes sense when you have unusual data sources, a product that needs embedded detection for your own customers, or a commercial tool that produces so much noise your team has stopped trusting it. Either way, the same principles apply: layered detection, explainable alerts, risk ranking, and a feedback loop for false positives.
How long does it take to build a detection engine?
Once collection and processing are working, a basic layer of well-chosen static rules with simple risk scoring typically takes four to six weeks and already catches many obvious attacks. Behavioural analytics usually add another six to ten weeks, because baselines need a learning period and thresholds need tuning for your environment. Anomaly detection and AI triage come after that. None of these layers is ever truly finished, since rules and baselines need ongoing tuning as your systems, users, and attackers change.
Conclusion
A detection engine has one job that matters: turning an overwhelming event stream into a short, ranked list of things a person should look at. Catching attacks is only half of that job. The other half is staying quiet when nothing is wrong, because an engine that fires constantly teaches its users to ignore it.
The practical takeaways are consistent. Layer static rules, behavioural baselines, and anomaly detection rather than betting on one technique. Make every alert explain why it fired. Rank detections with a risk score that accounts for asset value, identity sensitivity, threat intelligence, and clustering. Map coverage to MITRE ATT&CK so gaps are visible. And build a feedback loop so analyst decisions make the engine quieter and sharper over time, with AI triage enriching detections before they reach a human.
The main caveat is sequencing. Behavioural analytics and anomaly models depend on clean, normalised data and tuned rules, and adding them too early mostly produces expensive noise. Start with a small set of well-tuned rules over a reliable pipeline, measure your false-positive rate, and expand from there. If you are scoping that first layer or fixing a noisy one, our AI and machine learning engineering team can help you plan the build.