XenonStack Recommends

Big Data Architecture: Stack 6 Layers, Patterns and their Features

Acknowledging Data Management
          Best Practices with DataOps Image


XenonStack White Arrow Image

Big Data Architecture helps design the Data Pipeline with the various requirements of either the Batch Processing System or Stream Processing System. This architecture consists of 6 layers, which ensure a secure flow of data.

Big Data Architecture Layers

  • Data Ingestion Layer
  • Data Collector Layer
  • Big Data Processing Layer
  • Data Storage Layer
  • Data Query Layer
  • Big Data Visualization Layer
  • Data Security Layer
  • Data Monitoring Layer

The data lake has proven as a viable approach for business insights. Along with this, Splunk and ThingWorx's architecture has also flourished in terms of handling Big Data. Source: Big Data Requires a Big, New Architecture

Big Data Ingestion Layer

This layer of Big Data Architecture is the first step for the data coming from variable sources to start its journey. Data ingestion means the data is prioritized and categorized, making data flow smoothly in further layers in the Data ingestion process flow.

Data Collector Layer

In this Layer, more focus is on the transportation of data from the ingestion layer to the rest of the data pipeline. It is the Layer of data architecture where components are decoupled so that analytic capabilities may begin.

Data Processing Layer

In this primary layer of Big Data Architecture, the focus is to specialize in the data pipeline processing system. We can say the data we have collected in the previous layer is processed in this layer. Here we do some magic with the data to route them to a different destination and classify the data flow, and it’s the first point where the analytic may occur.

Data Storage Layer

Storage becomes a challenge when the size of the data you are dealing with becomes large. Several possible solutions, like Data Ingestion Patterns, can rescue from such problems. Finding a storage solution is very much important when the size of your data becomes large. This layer of Big Data Architecture focuses on “where to store such large data efficiently.”

Data Query Layer

This is the architectural layer where active analytic processing of Big Data takes place. Here, the primary focus is to gather the data value to be more helpful for the next layer.

Data Visualization Layer

The visualization, or presentation tier, probably the most prestigious tier, where the data pipeline users may feel the VALUE of DATA. We need something that will grab people’s attention, pull them into, make your findings well-understood.

Big Data Ingestion Layer

Data ingestion mystery can be well understood using the Layered Architecture of Big Data. The Layered Architecture of the Big Data ingestion pipeline is divided into different layers, where each layer performs a particular function.

Learn more about Big Data Ingestion Architecture Layer

Big Data Processing Layer (Tools, Use Cases, Features)

In this layer of Big data Architecture, we gathered the data from different sources and made it available to go through the rest of the pipeline.

Our task is to do magic with data; as now data is ready, we only have to route the data to different destinations.

In this main layer, the focus is to specialize the Data Pipeline processing system, or we can say the data we have collected by the last layer in this next layer we have to do processing on that data.

Big Data Batch Processing System is a simple batch processing system for offline analytics. For doing this tool used is Apache Sqoop.

What is Apache Sqoop?

It efficiently transfers bulk data between Apache Hadoop and structured datastores such as relational databases. Apache Sqoop can also extract data from Hadoop and export it into external structured data stores.

Apache Sqoop works with relational databases such as Teradata, Netezza, Oracle, MySQL, Postgres, and HSQLDB.

Functions of Apache Sqoop

  • Import sequential data sets from the mainframe
  • Data imports
  • Parallel Data Transfer
  • Fast data copies
  • Efficient data analysis
  • Load balancing

Near Real-Time Processing System

A pure online processing system for online analytics. For this type of processing, use Apache Storm. The Apache Storm cluster makes decisions about the event's criticality and sends the alerts to the warning system (dashboard, e-mail, other monitoring systems).

What is Apache Storm?

It is a system for processing streaming data in real-time during Data ingestion. It adds reliable real-time data processing capabilities to Enterprise Hadoop. Storm on YARN is powerful for scenarios requiring real-time analytics, machine learning, and continuous monitoring of operations.

6 Key Features of Apache Storm

  1. Fast: It can process one million 100 byte messages per second per node.
  2. Scalable: It can do parallel calculations that run across a cluster of machines.
  3. Fault-tolerant: When workers die, Storm will automatically restart them. If a node dies, the worker will be restarted on another node.
  4. Reliable: Storm guarantees that each data unit (tuple) will be processed at least once or exactly once. Messages are only replaying when there are failures.
  5. Easy to operate: It consists of Standard configurations that are suitable for production on day one. Once deployed, Data ingestion, Storm is easy to work.
  6. Hybrid Processing system: This consists of Batch and Real-time processing System capabilities. This type of processing tool used is Apache Spark and Apache Flink.

What is Apache Spark?

Apache Spark Optimization is a fast, in-memory data processing engine with elegant and expressive development APIs to allow data workers to efficiently execute streaming, machine learning, or SQL workloads that require fast iterative access to data sets.

With Spark running on Apache Hadoop YARN, developers everywhere can now create applications to exploit Spark’s power, derive insights, and enrich their data science workloads within a single, shared data set in Hadoop.

What is Apache Flink?

Apache Flink is an open-source framework in the Data ingestion pipeline for distributed stream processing that provides accurate results, even in out-of-order or late-arriving data or Distributed Data Processing Apache Flink. Some of its features are –

Key Features of Apache Flink

  1. Performs Data ingestion at a large scale, running on thousands of nodes with excellent throughput, latency characteristics, and Data ingestion framework.
  2. It’s streaming data flow execution engine, APIs, and domain-specific libraries for Batch, Streaming, Machine Learning, and Graph Processing.

Apache Flink Use Cases

  • Optimization of e-commerce search results in real-time
  • Stream processing-as-a-service for data science teams
  • Network/Sensor monitoring and error detection
  • ETL for Business Intelligence Infrastructure

Big Data Storage Layer

Next, the data ingestion process flow's major issue is to keep data in the right place based on usage. We have relational Databases that were a successful place to store our data over the years.

But with the new Big data analytics strategic enterprise applications, you should no longer be assuming that your persistence should be relational in Data ingestion.

We need different databases to handle the different varieties of data, but using different databases creates overhead. That’s why there is an introduction to the new concept in the database world, i.e., the Polyglot Persistence.

Polyglot persistence is the idea of using multiple databases to power a single application. Polyglot persistence is the way to share or divide your data into multiple databases and leverage their power together. It takes advantage of the strength of different databases. Here various types of data are arranged in a variety of ways. In short, it means picking the right tool for the right use case.

It’s the same idea behind Polyglot Programming, which is the idea that applications should be written in a mix of languages in Data ingestion to take advantage of the fact that different languages are suitable for tackling different problems using the correct Data ingestion framework.

Advantages of Polyglot Persistence

  • Faster response times: In this, we leverage all the features of databases in one app, which makes your app's response times very quick.
  • Helps your app to scale well: Your app scales exceptionally well with the data. All the NoSQL databases scale well when you model databases correctly for the data you want to store.
  • A rich experience: You have a vibrant experience when you harness the power of multiple databases simultaneously. For example, if you want to search for Products in an e-commerce app, you use ElasticSearch, which returns the results based on relevance, which MongoDB cannot do.

Read more to get insight on IoT Architecture

Big Data Storage Tools

Different types of Data Storage tools used for handling Big Data are as follows:

HDFS: Hadoop Distributed File System

  • HDFS is a Java file system that provides scalable and reliable data storage, and it helped to span large clusters of commodity servers.
  • It holds a huge amount of data and provides easier access.
  • To store such massive data, the files are stored on multiple machines. These files are stored redundantly to rescue the system from possible data losses in case of failure.
  • HDFS also makes applications available for parallel processing in Data ingestion. HDFS is built to support applications with large data sets, including individual files that reach the terabytes.
  • It uses a master/slave architecture, with each cluster consisting of a single NameNode that manages file system operations and supporting DataNodes that manage data storage on individual compute nodes.
  • When HDFS takes in data, it breaks the information down into separate pieces and distributes them to different nodes in a cluster, allowing for parallel processing.
  • The file system in Data ingestion also copies each piece of data multiple times. It distributes the copies to individual nodes, placing at least one copy on a different server rack.
  • HDFS and YARN form the data management layer of Apache Hadoop in the Data ingestion framework.

Features of HDFS

  • It is suitable for distributed storage and processing.
  • Hadoop provides a command interface to interact with HDFS.
  • The built-in servers of the name node and data node help users quickly check the cluster's status.
  • Streaming access to file system data in Data ingestion process flow.
  • HDFS provides file permissions and authentication.

GlusterFS: Dependable Distributed File System

As we know, a good storage solution must provide elasticity in both storage and performance without affecting active operations. Scale-out storage systems based on GlusterFS are suitable for unstructured data such as documents, images, audio and video files, and log files. GlusterFS is a scalable network filesystem. Using this, we can create large, distributed storage solutions for media streaming, data analysis, data ingestion, and other data- and bandwidth-intensive tasks.

  • It’s Open Source.
  • You can deploy GlusterFS with the help of commodity hardware servers.
  • Linear scaling of performance and storage capacity.
  • Scale storage size up to several petabytes, which thousands of servers can access.

GlusterFS Use Cases

  • Cloud Computing
  • Streaming Media
  • Content Delivery

Amazon S3 Storage Service

  • Amazon Simple Storage Service (Amazon S3) is object storage with a simple web service interface to store and retrieve any data from anywhere on the internet.
  • It delivers 99.99% durability and scales past trillions of objects worldwide. Customers use S3 as primary storage for cloud-native applications, as a bulk repository, or “data lake,” for analytics, as a target for backup & recovery and disaster recovery. With the serverless architecture of big data computing.
  • It’s simple to move large volumes of data into or out of S3 with Amazon’s cloud data migration options.
  • Once data is stored on Amazon S3, it can be automatically tiered into lower cost, longer-term cloud storage classes like S3 Standard – Infrequent Access and Amazon Glacier for archiving.

Big Data Query Layer

It is the layer of data architecture where active analytic processing takes place. This is a field where interactive queries are necessary, and it’s a zone traditionally dominated by SQL expert developers. Before Hadoop, we had insufficient storage, due to which it takes a long analytics process.

At first, it goes through a Lengthy process, i.e., ETL, to get a new data source ready to be stored, and after that, it puts the data in a database or data warehouse. Data ingestion and data analytics became two essential steps that solved problems while computing such a large amount of data while making a Data ingestion framework.

Companies from all industries use Big data analytics to –

  • Increase revenue
  • Decrease costs
  • Increase productivity

Big Data Analytics Query Tools

Apache Hive Architecture

  • Apache Hive is a data warehouse infrastructure built on top of Apache Hadoop for providing data summarization, ad-hoc query, and analysis of large datasets.
  • Data analysts use Hive to query, summarize, explore, analyze that data, and then turn it into actionable business insight.
  • It provides a mechanism to Data ingestion project structure Hadoop'so the doop and to query that data using a SQL – like language called HiveQL (HQL).

Features of Apache Hive

  • Query data with a SQL – based language.
  • Interactive response times, even over massive datasets.
  • It’s scalable as data variety and volume grows, more commodity machines can be added without a corresponding reduction in performance. Works with traditional data integration and data analytics tools.

Apache Spark SQL

Spark SQL includes a cost-based optimizer, columnar storage, and code generation to make queries fast.

At the same time, it scales to thousands of nodes and multi-hour queries using the Spark engine, which provides full mid-query fault tolerance.

Spark SQL is a Spark module for structured data processing. Some of the Functions performed by Spark SQL are –

  • The interfaces provided by Spark SQL provide Spark with more information about the structure of both the data and the computation.
  • Internally, Spark SQL uses this extra information to perform additional optimizations.
  • One use of Spark SQL is to execute SQL queries.
  • Spark SQL helps to read data from an existing Hive installation.

Amazon Redshift

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. We use Amazon Redshift to load the data and run queries on the data. We can also create additional databases as needed by running an SQL command. Most important, we can scale it from a hundred gigabytes of data to a petabyte or more.

It enables you to use your Data ingestion to acquire new insights for your business and customers. The Amazon Redshift service manages all of setting up, operating, and scaling a data warehouse.

Creating a Data ingestion framework includes provisioning capacity, monitoring, and backing of the cluster, and applying patches and upgrades to the Amazon Redshift engine.

Presto – SQL Query Engine For Big Data

Presto is an open-source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes.

It was designed and written for interactive analytics and approaches and commercial data warehouses' speed while scaling to organizations like Facebook.

Presto Capabilities

  • Presto allows querying data where it lives, including Hive, Cassandra, relational databases, or even proprietary data stores.
  • A single Presto query can combine data from multiple sources, allowing for analytics across your entire organization.
  • Presto targets analysts who expect response times ranging from sub-second to minutes in Data ingestion process flow.
  • Presto breaks the false choice between having fast analytics using an expensive commercial solution or using a slow “free” solution that requires excessive hardware.

Who Uses Presto?

Facebook uses Presto for interactive queries against several internal data stores, including its 300PB Data Warehouse. Over 1,000 Facebook employees use Presto daily to run more than 30,000 queries in the complete scan over a petabyte each per day for Data ingestion. Leading internet companies, including Airbnb and Dropbox, are using Presto.

Big Data Visualization Layer (Tools, Features)

This layer of Big Data Architecture is the thermometer that measures the success of the project. This is the user perceives the data value user. While it helps to handle and store volumes of data, Hadoop and other tools have no built-in provisions for data visualization and information distribution, leaving no way to make that data easily consumable by end business users in the Data ingestion pipeline.

Tools For Building Data Visualization Dashboards

Various tools that help in building Data Visualization dashboards are below with their features:

Custom Dashboards for Data Visualization

Custom dashboards are useful for creating unique overviews that present data differently. For example, you can:

  • Show the web and mobile application information, server information, custom metric data, and plugin metric data all on a single custom dashboard.
  • Create dashboards that present charts and tables with uniform size and arrangement on a grid.
  • Select existing New Relic charts for your dashboard, or create your charts and tables.

Real-Time Visualization Dashboards

Real-Time Dashboards save, share, and communicate insights. It helps users generate questions by revealing the depth, range, and content of their data stores.

  • Data Visualization dashboards always change as new data arrives.
  • In Zoomdata, you have the flexibility to create a data analytics dashboard with just a single chart and then add to it as needed.
  • Dashboards can contain multiple visualizations from multiple connections side by side.
  • You can quickly build, edit, filter, and delete dashboards and move and resize them and then share them or integrate them into your web application.
  • Can export a dashboard as an image or as a file configuration like JSON.
  • You can also make multiple copies of your dashboard in the Data ingestion process flow or talk with Data Visualization Experts.

Data Visualization with Tableau

Tableau is the richest data visualization tool available in the market, with Drag and Drop functionality.

  • Tableau allows users to design Charts, Maps, Tabular, Matrix reports, Stories, and Dashboards without any technical knowledge.
  • It helps anyone quickly analyze, visualize, and share information. Whether it’s structured or unstructured, petabytes or terabytes, millions or billions of rows, you can turn Graph Databases in Big Data Analytics into big ideas.
  • It connects directly to local and cloud data sources or import data for fast in-memory performance during Data ingestion.
  • Make sense of big data with easy-to-understand visuals and interactive web dashboards.

You may also like to read: Apache Kafka Security with Kerberos on Kubernetes.

Exploring Data sets With Kibana

  • A Kibana dashboard displays a collection of saved visualizations. You can arrange and resize the visualizations according to requirements and save dashboards, to reload and share.
  • Kibana acts as analytics and visualization platform built on Elasticsearch to understand your Data ingestion framework better.
  • Application Performance Monitoring is one key area to implement in projects to ensure proper and smooth operations from day 1. APM solutions provide development and operations teams with near real-time insights on how the applications and services perform in the production, allowing for a proactive tune of services and early detection of possible production issues.
  • It gives you the freedom to select the way you give shape to your data. And you don’t always have to know what you’re looking for in Data ingestion using Parallel Processing Applications.
  • Kibana core ships with the classics: histograms, line graphs, pie charts, sunbursts, and more. They leverage the full aggregation capabilities of Elasticsearch in Data ingestion process flow.

The Kibana interface is of four main sections:

  1. Discover
  2. Visualize
  3. Dashboard
  4. Settings

Introduction to Intelligence Agents

An intelligent agent is a software that assists people and acts on their behalf. Intelligent agents work by allowing people to delegate work they could have done to the agent software. Agents can perform repetitive tasks, remember things you forgot, intelligently summarize complex data, learn from you, and even make recommendations.

An intelligent agent can help you find and filter information when looking at corporate data or surfing the Internet without knowing where the right information is. It could also customize information to your preferences, thus saving you from handling it as more and more new information arrived each day on the Internet. An agent could also sense changes in its environment and responds to these changes.

An agent continues to work even when the user is gone in the Data ingestion pipeline, which means that an agent could run on a server, but in some cases, an agent runs on the user systems.

Recommendation Systems

  • Recommender systems provide personalized information by learning the user’s interests from traces of interaction with that user. For a recommender system to make predictions about a user’s inter has to determine a user model.
  • A user model contains data about the user and should be represented so that the data can be matched to the items in the collection.
  • The question here is what kind of data can be used to construct a user profile during Data ingestion. Obviously, the items that users have seen in the past are important. Simultaneously, other information such as the items' content, users' perception of the items, or information about users themselves could also be used.
  • Most recommender systems focus on information filtering, which deals with delivering elements selected from an extensive collection that the user is likely to find interesting or useful.
  • Recommender systems are unique types of information filtering systems that suggest items to users. Some of the largest e-commerce sites use recommender systems applying a marketing start, referred to as mass customization.
  • A content-based filtering system often uses many of the same techniques as an information retrieval system (such as a search engine). Both systems require a content description of the items in their domain. A recommender system also requires modeling the user’s preferences for a longer period, which is unnecessary for an information retrieval system.
  • There are several techniques of Data ingestion that can be used to improve recommender systems in different ways.

Click to Discover about Stream Analytics

Angular.JS Framework

AngularJS is a very powerful JavaScript Framework. Use it in Single Page Application (SPA) projects in the Data ingestion framework. It extends HTML DOM with additional attributes and makes it more responsive to user actions. AngularJS is open source, completely free, and used by thousands of developers around the world. React is a library for building composable user interfaces.

It encourages the creation of reusable UI components that present data that changes over time.

Understanding React. JS React is a JavaScript library that helps for building User Interface, focuses on the UI, not a framework. One-way reactive data flow(no two-way Data Binding), Virtual DOM. React is a front-end library developed by Facebook. It’s used for handling the view layer for the web and mobile apps. ReactJS allows us to create reusable UI components. It is currently one of the most popular JavaScript libraries, and it has a strong foundation and a large community behind it.

Useful Features of React

  • JSX − JSX is JavaScript syntax extension. It isn’t necessary to use JSX to React for development, but it is recommended.
  • Components − React is all about components. You need to think of everything as a component. This will help you to maintain the code when working on larger-scale projects.
  • Unidirectional data flow and Flux − React implements one-way data flow, making it easy to reason about your app. Flux is a pattern that helps to keep your data unidirectional.

Big Data Security and Data Flow Layer

Security is the crucial part of any sort of data and also is an essential aspect of Big Data Architecture. It is the primary task of any work. Implement security at all layers of the lake, starting from Ingestion, through Storage, Analytics, Discovery, all the way to Consumption. For providing security in Data ingestion to data pipeline, few steps are there that are:-

Big Data Authentication

Authentication will verify the user’s identity and ensure they are who they say they are. Using the Kerberos protocol provides a reliable mechanism for authentication.

Access Control

Defining which datasets can be consulted by the users or services is the best step to secure the information. Access control will restrict users and services to access only that data they have permission for; they will access all the data in the Data ingestion framework.

Encryption and Data Masking

Encryption and data masking is required to ensure secure access to sensitive data. Sensitive data in the cluster should be secured at rest as well as in motion.

Auditing Data Access by users

Another aspect of data security requirement is Auditing data access by users in the Data ingestion pipeline. It can detect the log & access attempts as well as the administrative changes.

Read More about Data Warehouse Big Data Architecture

Data Monitoring Layer

Data in enterprise systems is like food – it has to be fresh. Also, it needs nourishment. Otherwise, it goes wrong and doesn’t help you in making strategic and operational decisions. Just as consuming spoiled food could make you sick, using “spoiled” data may be bad for your organization’s health.

There may be plenty of data in the Data ingestion process flow, but it has to be reliable and consumable to be valuable. While most of the focus in enterprises is often about storing and analyzing large amounts of data, keeping this data fresh and flavorful is also essential.

So we can do this?

The solution is for monitoring, auditing, testing, managing, and controlling the data. Continuous monitoring of data is an important part of the governance mechanisms.

Apache Flume is useful for processing log data. Apache Storm is desirable for operations monitoring Apache Spark for streaming data, graph processing, and machine learning. Monitoring can happen in the data storage layer. It includes the following steps for data monitoring:-

Data Profiling and lineage

These are the techniques to identify the quality of data and the data's lifecycle through various phases. In these systems, it is important to capture the metadata at every layer of the stack for verification and profiling. Talend, Hive, Pig.

Data Quality

Data in Data ingestion is high quality. If it meets business needs, it satisfies the intended use to make business decisions successfully. So, understanding the dimension of greatest interest and implementing methods to achieve it is important.

Data Cleansing

It means implementing various solutions to correct incorrect or corrupt data.

Data Loss and Prevention

Policies have to be in place to make sure the loopholes for data loss are taken care of. Identification of such data loss needs careful monitoring and quality assessment processes in Data ingestion process flow.


Big data architecture can handle the processing, ingestion, and analysis of data that is too complex or large for traditional database systems. It is the overarching system used to manage large amounts of data to be analyzed for business purposes, steer data analytics, and provide an environment in which big data analytics tools can extract vital business information, moreover big data architecture framework serves as a reference blueprint for big data infrastructures and solutions.

Related blogs and Articles