Key takeaway
- How to run real-time analytics for streaming apps on Kubernetes with Spark, Kafka, Cassandra and Akka, and how to handle storage and scaling.
Big data analytics on Kubernetes for streaming applications is the practice of running stream ingestion, processing and storage components, such as Apache Kafka, Apache Spark and Apache Cassandra, as containers on one cluster so they can be scaled and recovered together. This pattern is often called the SKACK stack: Spark, Kubernetes, Akka, Cassandra and Kafka.
This page is for data engineers and platform teams building real-time analytics. It covers the main challenges, a three-node cluster layout, how the components fit, where AI agents help, and what to check first.
What is the SKACK stack for streaming analytics?
SKACK is an open source stack for real-time analysis of big data. It combines Apache Spark for processing, Kubernetes for running containers, Akka for building concurrent services, Apache Cassandra for storage and Apache Kafka for moving events.
Each part has a distinct job, which is why they are used together.
| Component | Role in the stack | What its documentation says |
|---|---|---|
| Apache Kafka | Collects and routes event streams | Kafka runs as a cluster of servers, and if one fails the others take over its work |
| Apache Spark | Processes live data streams | Spark describes scalable, high-throughput, fault-tolerant stream processing |
| Apache Cassandra | Stores results across nodes | Cassandra is a distributed NoSQL database that scales horizontally |
| Akka | Builds concurrent, event-driven services on the JVM | Used for the reactive application layer |
| Kubernetes | Schedules and restarts the containers | Runs the other four as workloads |
Note that Spark's own documentation now calls the older Spark Streaming API a legacy project and recommends Structured Streaming for new applications. Check this before you design a new pipeline.
What are the challenges of a multi-node SKACK cluster?
The hard parts are operational rather than conceptual.
- Setting up a multi-node cluster with documented, repeatable steps.
- Giving stateful components persistent storage, because a container's file system is not persistent by default.
- Scaling Spark workers as processing load grows.
- Scaling Cassandra nodes as data volumes grow.
- Scaling Kafka brokers as event rates grow.
How do you set up the cluster and storage?
A simple on-premises layout uses three nodes: one master and two workers. The master runs the API server, the scheduler and the controller manager. The workers run the containers. The cluster also reports resource use, availability and performance back to the API server, so you can see how it is doing.
For storage, the Kubernetes documentation says a PersistentVolume has "a lifecycle independent of any individual Pod that uses the PV". That means data can outlive a container restart. A distributed file system such as GlusterFS, which supports multi-mount, can sit behind those volumes so data is available on every node.
- Install the cluster and confirm nodes report their resource use.
- Create persistent storage and claim it for Kafka and Cassandra.
- Deploy Kafka first, then Cassandra, then Spark jobs.
- Test a node failure and confirm the data and jobs recover.
- Add monitoring before you add load.
What is reactive application development and why does Akka matter?
Reactive application development uses event-driven design patterns to build systems that stay responsive under load. Akka is a toolkit for concurrent, distributed, fault-tolerant, event-driven applications on the JVM.
Reactive systems aim for four qualities: responsiveness, resilience, elasticity and message-driven communication. Typical uses are online shopping, high-volume banking transactions and share trading, where many requests arrive at once.
Where do AI agents fit in streaming analytics?
Agents sit above the pipeline. They can watch consumer lag and job failures, propose a scaling change, and summarise an incident for an engineer to approve. Treat any change to a running cluster as a decision that needs a record. ElixirData is XenonStack's Context OS, a place to keep the data and policy those decisions depend on.
What should you check before adopting it?
- Do you need true streaming, or would scheduled batch jobs do?
- Which components need persistent storage, and how is it backed up?
- Who owns scaling for Kafka, Cassandra and Spark?
- Which Spark streaming API will you use for new code?
- How will you test a node failure before production?
To map one decision your agents make, and the data and policy it depends on, book a Context Workshop. See what happens in the workshop.
Frequently asked questions
What is the SKACK stack?
It is an open source stack for real-time analysis of big data made of Apache Spark, Kubernetes, Akka, Apache Cassandra and Apache Kafka.
Why do streaming apps on Kubernetes need persistent storage?
A container's file system is not persistent by default, so stateful components such as Kafka and Cassandra need volumes whose data outlives any one Pod.
How many nodes do you need to start?
A small on-premises cluster can use three nodes, one master and two workers. Add workers as processing and storage needs grow.
Where do AI agents help?
Agents can watch the pipeline, flag failures or lag, and propose scaling changes for an engineer to approve.
What should we set up first?
Start with a three-node cluster, persistent storage and one Kafka to Spark to Cassandra flow, then test a node failure before adding load.
Ready to put AI agents to work? See an ElixirClaw demo: the agentic OS to build, orchestrate and govern AI agents across your enterprise systems.
Explore the XenonStack platform family: ElixirData · ElixirClaw · ElixirHub · NexaStack · MetaSecure · Neural AI · AgentGRC · AgentSRE · AgentAnalyst · AgentInstruct · AgentRAI · AgentScrum · AgentSearch · AgentForce · AI Aviator · SAIF Aviator · CohortIQ · XenonStack AI · XenonStack UAE