Detecting Anomalies with Elastic Machine Learning Jobs

Wylie Bayes

• 3 min read

Writing static KQL or EQL rules is highly effective for catching known threats (e.g., "Alert me if mimikatz.exe runs"). However, static rules struggle immensely against the "unknown unknowns" or attacks that mimic legitimate behavior.

How do you write a static rule for data exfiltration? If you set a threshold rule to alert when a user uploads more than 1GB of data, you will generate massive false positives for the video editing team, but you will completely miss an attacker slowly exfiltrating 500MB of sensitive text documents. You cannot write a static rule for "unusual."

To catch these subtle behavioral shifts, modern Security Operations Centers (SOCs) rely on Unsupervised Machine Learning (ML). In this post, we will explore how Elastic's ML engine operates and how to deploy it for advanced threat detection.

How Elastic Machine Learning Works

Unlike supervised machine learning (which requires training a model on millions of labeled "malicious" and "benign" datasets), Elastic utilizes unsupervised machine learning for anomaly detection.

When you activate an ML job, it looks at your historical data and builds a mathematical baseline of what is "normal" for your specific environment. It understands seasonality—it learns that traffic spikes at 9:00 AM on Monday, but a similar spike at 3:00 AM on a Sunday is abnormal.

As new data streams in, the ML model compares it to the baseline. If it detects a deviation, it assigns it an Anomaly Score from 0 to 100. A score of 90+ represents a massive, highly unusual deviation from normal behavior.

Pre-Built Security ML Jobs

Elastic Security makes leveraging this complex math incredibly easy. The platform ships with dozens of pre-configured ML jobs mapped to the MITRE ATT&CK framework.

To enable them, navigate to Security -> Machine Learning -> Anomalies and click Enable. Some of the most powerful pre-built jobs include:

  • Unusual Network Destination: Flags when a host connects to an IP address or geolocation that no one in the company has ever connected to before (excellent for catching new C2 servers).
  • Rare Process by User: Flags when an employee in HR runs a command-line utility (like PowerShell or whoami) that they have never executed in their entire employment history, even if that utility is standard for the IT team.
  • High DNS Query Volume: Detects subtle, sustained spikes in DNS queries to a single domain, which is the primary indicator of a DNS Tunneling attack.

Creating Custom ML Jobs

If the pre-built jobs don't cover your specific use case, you can build your own using the Machine Learning App in Kibana.

Let's say you want to detect unusual cloud infrastructure deployment (a common tactic of cryptojackers who steal AWS credentials to spin up massive mining rigs).

  1. Navigate to Machine Learning -> Anomaly Detection -> Create Job.
  2. Select your cloud audit logs index (e.g., logs-aws.cloudtrail-*).
  3. Choose a Multi-metric Job.
  4. Set the Field to model to event.action (e.g., RunInstances).
  5. Set the Split field to user.name.

This job will build a unique behavioral baseline for every single user in AWS. If developer Alice normally spins up 2 EC2 instances a week, and suddenly spins up 50 instances in an hour, her individual baseline is broken, and a high-scoring anomaly is generated.

Integrating ML with the Detection Engine

Machine Learning anomalies are fascinating to look at in a graph, but to be useful to a SOC, they must generate actionable tickets.

Elastic seamlessly integrates ML with the Detection Engine. You can create a new Detection Rule and select the Machine Learning rule type.

You simply link the rule to your specific ML Job and set the threshold: Generate a SIEM Alert if the Anomaly Score exceeds 75. This creates a highly refined, low-volume alerting pipeline. Because the alert is based on advanced mathematics rather than a static threshold, the false positive rate is drastically lower.

Conclusion

Attackers can easily evade static file hashes and IP blocklists, but they cannot easily evade the mathematical reality of their behavior. By leveraging Elastic's unsupervised Machine Learning, security teams can automatically model the baseline of their entire enterprise and detect the subtle ripples that indicate an active, stealthy compromise.