Key takeaway
- How threat analytics uses machine learning, Apache Metron and Apache Spot to detect cyber threats in real time, plus what to check before you build one.
Threat analytics is the use of data pipelines, machine learning and behavioural analysis to find, score and investigate cyber threats from network and security data as it arrives. Threat detection is the art of identifying attacks on a computer, and analytics automates that work at a scale people cannot match by hand.
This page is for security and data leaders. It covers the main attack types, the machine learning workflow, how Apache Metron and Apache Spot were used, where AI agents fit and what to check before you build. Related reading: agentic analytics.
What is advanced threat analytics?
Cyberattacks have become more pervasive, persistent and proficient at evading traditional security architecture. Many companies face stealthy attacks that target intellectual property, steal consumer information or encrypt essential data for ransom. To protect IT assets, secure digital interactions, detect and manage inevitable breaches and safeguard regulatory compliance, you must know what is coming.
Most attacks fit one of four categories: probe, denial of service, user to root and remote to user. This use case shows how a threat detection system is built with machine learning.
What makes real-time threat intelligence hard?
- Running threat intelligence in real time on very large volumes of messages.
- Storing and processing unstructured security data.
- Combining machine learning and predictive analytics so that analysis happens as events arrive.
How is a threat detection model built?
The process is automated in steps, from a network dataset to a trained model.
- Start with a network dataset.
- Pre-process the data.
- Extract features.
- Reduce the amount of data.
- Improve accuracy.
- Avoid overfitting.
- Train and test classification models such as decision tree, random forest, naive Bayes and KNN, then analyse the results.
A machine learning approach automates insight extraction from file samples, generalises better to unknown variations and reduces human analysis time. A multi-layered deep learning system is robust, scalable and adaptable, and attaches a risk score to each identified incident or pattern so analysts can prioritise their investigation.
How do Apache Metron and Apache Spot compare?
The Apache Metron site describes Metron as a scalable advanced security analytics framework for detecting cyber anomalies, built with the Hadoop community and evolved from the Cisco OpenSOC project. The Apache Incubator page for Spot describes Spot as a platform for network telemetry built on an open data model and Apache Hadoop.
| Question | Apache Metron | Apache Spot |
|---|---|---|
| Focus | Streaming security data, detection and response | Network flow, packet and DNS analysis |
| Core idea | Ingest, enrich and store security telemetry at high rates | Machine learning over network telemetry on an open data model |
| Typical use | Alerting, replay, evidence storage, threat intelligence | Suspicious DNS, threat hunting, known and unknown threats |
| Status | Retired in December 2020; move to the Attic completed April 2021 | Retired on 17 March 2023 |
| Source | Apache Attic | Apache Incubator |
Metron has four key capabilities: a security data lake for search and operational analytics, a pluggable framework for enrichment services and threat feeds, security applications such as alerting and evidence storage, and a threat intelligence platform that applies anomaly detection and machine learning to events in real time. Spot uses a parallel ingest framework, machine learning and operational analytics to detect suspicious DNS packets, support threat hunting and incident response, and find known and unknown threats.
Both projects have retired, so treat them as design references. The Apache Attic page for Metron and the Apache Incubator page for Spot confirm their status. Check what is maintained today before you choose components.
Where do AI agents fit?
Models and rules raise alerts, but an analyst still has to triage them. An AI agent can group related alerts, pull the supporting network and DNS records, summarise the likely attack type and propose a response for an analyst to approve. It should work within limits the security team sets and leave a record of every step. ElixirClaw is the AI agents computing platform from XenonStack, and is the page to read for agent and automation use cases.
What should you check before you build?
- Data. Confirm which telemetry you can collect and where it is stored.
- Components. Check that each open-source project you choose is still maintained.
- Labels. Plan how attack examples are labelled before training classifiers.
- Overfitting. Test models on traffic they have not seen.
- Response. Decide which actions an agent may take and which need a person.
Next Steps with Threat Analytics
Talk to our experts about implementing compound AI system, How Industries and different departments use Agentic Workflows and Decision Intelligence to Become Decision Centric. Utilizes AI to automate and optimize IT support and operations, improving efficiency and responsiveness.
To map one decision your security agents make, and the data and policy it depends on, book a Context Workshop. See what happens in the workshop.
Frequently asked questions
What is threat analytics?
It is the use of data pipelines, machine learning and behavioural analysis to detect and investigate cyber threats from network and security data in real time.
What are the main categories of cyber attack?
Most attacks fit into four categories: probe, denial of service, user to root and remote to user.
Which machine learning models are used for threat detection?
Common classification models include decision tree, random forest, naive Bayes and KNN, trained and tested on a network dataset after pre-processing and feature extraction.
Are Apache Metron and Apache Spot still active?
No. Apache Metron retired in December 2020 and Apache Spot retired on 17 March 2023, so use them as references and check what is maintained today.
How can AI agents help security teams?
An agent can group alerts, gather supporting records and propose a response for an analyst to approve, within limits the security team sets.
Ready to put AI agents to work? See an ElixirClaw demo: the agentic OS to build, orchestrate and govern AI agents across your enterprise systems.
Explore the XenonStack platform family: ElixirData · ElixirClaw · ElixirHub · NexaStack · MetaSecure · Neural AI · AgentGRC · AgentSRE · AgentAnalyst · AgentInstruct · AgentRAI · AgentScrum · AgentSearch · AgentForce · AI Aviator · SAIF Aviator · CohortIQ · XenonStack AI · XenonStack UAE