XenonStack

Enterprise AI

XenonStack Blog Real-Time Analytics Architecture: Layers and Design

and | Published | Updated

Real-Time Analytics Architecture: Layers and Design
19:48

Key takeaway

  • Real-time analytics architecture for CTOs and architects: six layers, Lambda vs Kappa, serving stores, governance, cost and a step-by-step adoption plan.

A real-time analytics architecture is the set of layers that moves an event from its source to a decision while the event is still useful: capture, a durable log, stream processing, a serving store that answers queries quickly, and a governance layer that says who may see and act on the result. The design choice that matters most is where you accept delay and where you do not, because each step towards lower latency adds cost and operating effort.

This guide is about the architecture, not the product shortlist. If you are choosing engines and managed services, read our streaming analytics tools buyer's guide; if you are deciding what to build around them, the sections below cover the layers, the patterns, the serving store, governance and a staged adoption plan for CTOs, CDOs and platform architects.

What layers make up a real-time analytics architecture?

Most production designs have the same six layers. The names of the tools change; the responsibilities do not.

Two-column diagram mapping four layers (event log, stream processing, serving store, governance) to the design decision each one needs: retention, state, latency and ownership.
LayerWhat it doesTypical optionsDecision to make
Sources and captureCollects events from applications, logs, sensors, databases (change data capture) and partner feedsApplication events, CDC, IoT gatewaysWhich sources justify streaming at all
Durable event logBuffers events, keeps order within a partition and lets consumers replayApache Kafka, managed cloud streamsRetention period and replay needs
Stream processingFilters, enriches, joins and aggregates events, often with state and event-time rulesApache Flink, Spark Structured Streaming, managed Flink servicesHow much state and how late-data tolerant
Serving storeAnswers user-facing and agent-facing queries on fresh dataReal-time OLAP stores such as Apache Pinot or Apache Druid; open table formats such as Apache Iceberg for historyQuery latency, concurrency and freshness targets
ConsumptionDashboards, alerts, APIs and AI agents that act on resultsBI tools, alerting, natural-language analyticsWho or what is allowed to act automatically
Governance and operationsLineage, access control, data quality checks, cost and incident monitoringCatalogues, policy engines, observabilityWho owns each stream and its quality

Start from the decision, not the data. List the three to five decisions that lose value if they are made late, such as blocking a fraudulent payment or rerouting a delivery, and size each layer to those decisions only. Our guide to real-time analytics use cases shows where that value appears by industry.

Which pattern fits: Lambda, Kappa or streaming with a lakehouse?

Three patterns cover most enterprise designs. They are general architecture styles, so judge them by your team's skills and your accuracy requirements rather than by fashion.

PatternHow it worksWhen it fitsTrade-off
LambdaA batch layer recomputes accurate results from full history while a speed layer gives quick approximate ones; a serving layer merges bothExisting batch estate you cannot replace yet, and a need for a fast view alongside itTwo code paths to build, test and keep consistent
KappaOne stream processing path handles live and reprocessed data, replaying the log when logic changesEvent-centric products and teams that can run a replayable logReplay at scale needs retention capacity and careful state handling
Streaming into a lakehouseStreams land continuously in open table formats, with a low-latency store for hot queriesNeed for both fresh operational answers and governed history for analysts and modelsMore moving parts to govern; table maintenance becomes an operational task

Apache Flink describes state as a first-class part of its model and says its checkpointing keeps application state consistent after failures, which is what makes the single-path Kappa style practical for stateful logic (source: Apache Flink applications). Apache Kafka is the common choice for the log, and Apache Iceberg lets several engines work on the same tables at once, which suits the lakehouse pattern.

How do you choose the serving layer?

The serving store is where architectures often fail. A stream processor can compute results quickly, but dashboards and agents need a store that handles many concurrent queries on fresh data. Compare stores against your own latency and concurrency targets, using vendor figures only as a starting point.

OptionWhat its own site saysWhen it fitsCurrent status
Apache PinotAn open-source distributed OLAP database for user-facing and agent-facing real-time analytics, with sub-second queries on fresh data (vendor-published, varies with hardware and schema)High-concurrency, customer-facing or agent-facing queriesActive Apache project
Apache DruidA real-time analytics database with sub-second queries on streaming and batch data and native streaming ingestionEvent and time-series exploration with streaming ingestionActive Apache project
Apache Iceberg tablesA high-performance format for huge analytic tables that several engines can use safely at onceGoverned history, backfills and model training dataActive Apache project
Warehouse or lakehouse engineDepends on the platform you already runMinute-level freshness where sub-second answers are not requiredCheck your vendor's streaming ingestion limits

If a business user or an AI agent needs a plain-language answer from this layer, AgentAnalyst lets teams ask questions of enterprise data in plain language and get real-time insights, working on top of the stores you already run.

Video: real-time analytics in practice.

What has changed in real-time architecture design?

ChangeWhat it means for the designSource
Amazon Kinesis Data Analytics for SQL applications is discontinued: no new applications from 15 October 2025, deletion from 27 January 2026; Amazon Managed Service for Apache Flink is the successorDo not design new pipelines on the SQL service; plan migration for existing onesAWS documentation
Spark Streaming with DStreams is legacy and receives no updates; Structured Streaming is the recommended APIChoose Structured Streaming for new Spark workApache Spark guide
Serving stores now position themselves for agent-facing queries as well as dashboardsPlan for machine query volume and for policy on what an agent may read or doApache Pinot
Event time and watermarks are standard in modern enginesDesign for late and out-of-order events instead of assuming clean arrivalApache Flink

How do you govern and secure a real-time pipeline?

Real-time pipelines move personal and commercial data faster than review cycles can follow, so controls must be built in rather than added later.

  • Ownership: name an owner for each stream, with a data contract that fixes schema, quality checks and retention.

  • Access: classify data at the source, apply least-privilege access at the log and the serving store, and mask sensitive fields early.

  • Privacy: map streams that carry personal data to the rules that apply to you, such as GDPR or CCPA, and set retention and deletion accordingly.

  • Automated actions: where results trigger actions, record what context and policy the decision used. ElixirData Context OS keeps governed context, versioned policy and decision traces in one decision layer above AI agents.

  • Cost: the main drivers are retention, partition count, state size, always-on compute and query concurrency. Set a budget per stream and review it with the business owner.

How should you adopt it, step by step?

  1. Pick one decision and its latency target. Write down the value of acting early and the cost of a wrong answer.

  2. Run a proof of concept on one source. Use real data, measure end-to-end delay and note what breaks.

  3. Fix data quality at the edge. Validate schema, duplicates and late events before they reach the serving store.

  4. Add the serving store and one consumer. Load test for peak concurrency, not average.

  5. Test resilience. Replay from the log, kill a node and confirm results stay correct.

  6. Scale by stream, not by platform. Add sources one at a time, each with an owner and a budget.

Related reading: batch and real-time data ingestion with Apache NiFi, real-time streaming data visualisation and continuous intelligence with real-time analytics.

When is a real-time architecture the wrong choice?

Skip it when the decision is made daily or weekly, when source systems cannot emit events reliably, or when nobody can act on an answer within seconds. A well-run batch or micro-batch pipeline is cheaper and easier to audit. Build real-time layers only for the decisions that justify them, and keep the rest on batch.

Our streaming data platform page describes how we approach the build, and the streaming analytics architecture guide goes deeper on tooling.

To map one real-time decision your agents should make, and the context and policy it depends on, book a Context Workshop. See what happens in the workshop.

Frequently asked questions

What is a real-time analytics architecture?

It is the set of layers that captures events, keeps them in a durable log, processes them as streams, serves fast queries and governs access, so decisions use data while it is still fresh.

What is the difference between Lambda and Kappa architecture?

Lambda runs a batch layer and a speed layer and merges their results. Kappa uses one stream path and replays the event log when logic changes, which removes the second code path but needs replayable retention.

Do I need Kafka and Flink for real-time analytics?

Not always. Kafka is a common event log and Flink a common engine for stateful processing, but managed cloud services or Spark Structured Streaming can meet many needs. Match the choice to your latency target and skills.

What serving store should I use for real-time dashboards?

Real-time OLAP stores such as Apache Pinot and Apache Druid are built for sub-second queries on fresh data at high concurrency. Test them with your own queries, because published figures vary with hardware and schema.

What has changed for AWS streaming analytics?

Amazon Kinesis Data Analytics for SQL applications is discontinued, with no new applications from 15 October 2025 and deletion from 27 January 2026. Amazon Managed Service for Apache Flink is the successor.

When should I use batch instead of real-time?

Use batch when decisions are made daily or weekly, when sources cannot emit reliable events or when no one can act within seconds. It is cheaper and simpler to audit.

Ready to put AI agents to work? Explore ElixirClaw, the agentic OS platform to build, orchestrate and govern AI agents.

Table of Contents

navdeep-singh-gill

Navdeep Singh Gill

Global CEO and Founder of XenonStack

LinkedIn profile

Navdeep Singh Gill is serving as Chief Executive Officer and Product Architect at XenonStack. He holds expertise in building SaaS Platform for Decentralised Big Data management and Governance, AI Marketplace for Operationalising and Scaling. His incredible experience in AI Technologies and Big Data Engineering thrills him to write about different use cases and its approach to solutions.

Get the latest articles in your inbox

Subscribe Now

Interested in Solving your Challenges with XenonStack Team
Get Started

Get Started with your requirements and primary focus, that will help us to make your solution

Please Select your Industry
Banking
Fintech
Payment Providers
Wealth Management
Discrete Manufacturing
Semiconductor
Machinery Manufacturing / Automation
Appliances / Electrical / Electronics
Elevator Manufacturing
Defense & Space Manufacturing
Computers & Electronics / Industrial Machinery
Motor Vehicle Manufacturing
Food and Beverages
Distillery & Wines
Beverages
Shipping
Logistics
Mobility (EV / Public Transport)
Energy & Utilities
Hospitality
Digital Gaming Platforms
SportsTech with AI
Public Safety - Explosives
Public Safety - Firefighting
Public Safety - Surveillance
Public Safety - Others
Media Platforms
City Operations
Airlines & Aviation
Defense Warfare & Drones
Robotics Engineering
Drones Manufacturing
AI Labs for Colleges
AI MSP / Quantum / AGI Institutes
Retail Apparel and Fashion

Proceed Next