Skip to main content
XenonStack

DataOps

XenonStack Use Case Big Data Analytics on Kubernetes for Streaming Apps

| Published | Updated

Big data analytics on Kubernetes: cluster, persistent storage, Spark Kafka Cassandra and Akka, real-time analytics

Key takeaway

  • How to run real-time analytics for streaming apps on Kubernetes with Spark, Kafka, Cassandra and Akka, and how to handle storage and scaling.
. #}

Big data analytics on Kubernetes for streaming applications is the practice of running stream ingestion, processing and storage components, such as Apache Kafka, Apache Spark and Apache Cassandra, as containers on one cluster so they can be scaled and recovered together. This pattern is often called the SKACK stack: Spark, Kubernetes, Akka, Cassandra and Kafka.

This page is for data engineers and platform teams building real-time analytics. It covers the main challenges, a three-node cluster layout, how the components fit, where AI agents help, and what to check first.

What is the SKACK stack for streaming analytics?

SKACK is an open source stack for real-time analysis of big data. It combines Apache Spark for processing, Kubernetes for running containers, Akka for building concurrent services, Apache Cassandra for storage and Apache Kafka for moving events.

Each part has a distinct job, which is why they are used together.

ComponentRole in the stackWhat its documentation says
Apache KafkaCollects and routes event streamsKafka runs as a cluster of servers, and if one fails the others take over its work
Apache SparkProcesses live data streamsSpark describes scalable, high-throughput, fault-tolerant stream processing
Apache CassandraStores results across nodesCassandra is a distributed NoSQL database that scales horizontally
AkkaBuilds concurrent, event-driven services on the JVMUsed for the reactive application layer
KubernetesSchedules and restarts the containersRuns the other four as workloads

Note that Spark's own documentation now calls the older Spark Streaming API a legacy project and recommends Structured Streaming for new applications. Check this before you design a new pipeline.

What are the challenges of a multi-node SKACK cluster?

The hard parts are operational rather than conceptual.

Two-column diagram pairing four SKACK cluster challenges with four matching practices.
  • Setting up a multi-node cluster with documented, repeatable steps.
  • Giving stateful components persistent storage, because a container's file system is not persistent by default.
  • Scaling Spark workers as processing load grows.
  • Scaling Cassandra nodes as data volumes grow.
  • Scaling Kafka brokers as event rates grow.

How do you set up the cluster and storage?

A simple on-premises layout uses three nodes: one master and two workers. The master runs the API server, the scheduler and the controller manager. The workers run the containers. The cluster also reports resource use, availability and performance back to the API server, so you can see how it is doing.

For storage, the Kubernetes documentation says a PersistentVolume has "a lifecycle independent of any individual Pod that uses the PV". That means data can outlive a container restart. A distributed file system such as GlusterFS, which supports multi-mount, can sit behind those volumes so data is available on every node.

  1. Install the cluster and confirm nodes report their resource use.
  2. Create persistent storage and claim it for Kafka and Cassandra.
  3. Deploy Kafka first, then Cassandra, then Spark jobs.
  4. Test a node failure and confirm the data and jobs recover.
  5. Add monitoring before you add load.

What is reactive application development and why does Akka matter?

Reactive application development uses event-driven design patterns to build systems that stay responsive under load. Akka is a toolkit for concurrent, distributed, fault-tolerant, event-driven applications on the JVM.

Reactive systems aim for four qualities: responsiveness, resilience, elasticity and message-driven communication. Typical uses are online shopping, high-volume banking transactions and share trading, where many requests arrive at once.

Where do AI agents fit in streaming analytics?

Agents sit above the pipeline. They can watch consumer lag and job failures, propose a scaling change, and summarise an incident for an engineer to approve. Treat any change to a running cluster as a decision that needs a record. ElixirData is XenonStack's Context OS, a place to keep the data and policy those decisions depend on.

What should you check before adopting it?

  • Do you need true streaming, or would scheduled batch jobs do?
  • Which components need persistent storage, and how is it backed up?
  • Who owns scaling for Kafka, Cassandra and Spark?
  • Which Spark streaming API will you use for new code?
  • How will you test a node failure before production?

To map one decision your agents make, and the data and policy it depends on, book a Context Workshop. See what happens in the workshop.

Frequently asked questions

What is the SKACK stack?

It is an open source stack for real-time analysis of big data made of Apache Spark, Kubernetes, Akka, Apache Cassandra and Apache Kafka.

Why do streaming apps on Kubernetes need persistent storage?

A container's file system is not persistent by default, so stateful components such as Kafka and Cassandra need volumes whose data outlives any one Pod.

How many nodes do you need to start?

A small on-premises cluster can use three nodes, one master and two workers. Add workers as processing and storage needs grow.

Where do AI agents help?

Agents can watch the pipeline, flag failures or lag, and propose scaling changes for an engineer to approve.

What should we set up first?

Start with a three-node cluster, persistent storage and one Kafka to Spark to Cassandra flow, then test a node failure before adding load.

Ready to put AI agents to work? See an ElixirClaw demo: the agentic OS to build, orchestrate and govern AI agents across your enterprise systems.

Table of Contents

Get the latest articles in your inbox

Subscribe Now

Interested in solving your challenges with the XenonStack team?
Get Started

Tell us about you and your priorities so we can prepare the right conversation. It takes about 2 minutes.

Please Select your Industry
Banking
Fintech
Payment Providers
Wealth Management
Discrete Manufacturing
Semiconductor
Machinery Manufacturing / Automation
Appliances / Electrical / Electronics
Elevator Manufacturing
Defense & Space Manufacturing
Computers & Electronics / Industrial Machinery
Motor Vehicle Manufacturing
Food and Beverages
Distillery & Wines
Beverages
Shipping
Logistics
Mobility (EV / Public Transport)
Energy & Utilities
Hospitality
Digital Gaming Platforms
SportsTech with AI
Public Safety - Explosives
Public Safety - Firefighting
Public Safety - Surveillance
Public Safety - Others
Media Platforms
City Operations
Airlines & Aviation
Defense Warfare & Drones
Robotics Engineering
Drones Manufacturing
AI Labs for Colleges
AI MSP / Quantum / AGI Institutes
Retail Apparel and Fashion

Continue