XenonStack Recommends

Big Data Engineering

IoT Analytics Platform for Real-Time Data Ingestion

Navdeep Singh Gill | 29 May 2023

What is IoT Analytics Platform?

IoT analytics is an application that helps to understand the huge volume of data generated connected IoT devices. In other words,  ‘Internet of Things’ is coined by Kevin Ashton co-founder of the Auto-ID Center at the Massachusetts Institute of Technology (MIT). In other words, the Internet of Things is just an ecosystem, the physical things which are connected to the internet and we see results through its platform. You may also explore more about Real-Time Anomaly Detection for Cognitive Intelligence in this blog. For instance, these physical things include
  • Smartphones
  • Tablets
  • Wearable Device
  • Sensors etc.
This IoT analytics has the ability to transfer data over the network. Learn more about Learning Analytics in this insight.
Genomics analytics is responsible not only for handling non-invasive generational diagnostic tests. Click to explore about, Guide to Analytics Platform

What is the architecture of IoT Analytics?

There are 4 layers in IoT architecture are highlighted below:

  • Hardware/Sensing Layer - This is the very first layer of the IoT architecture. Also, it consists of the hardware device which is connected to the Network layer. Wireless Sensor Networks (WSN) and radio-frequency identification (RFID) are considered as the two main building blocks of the IoT. The Arduino Microcontrollers are directly connected with the sensors. The Arduino microcontrollers are also connected to the Raspberry Pi which is also connected to the internet using Ethernet or WiFi. It transmits the data collected from the sensor in real-time to the server.
  • Network/Gateway Layer -  This layer acts as the bridge in between Hardware /Sensing Layer and Management Layer. This layer receives the digitized data and routes it over wired LANs, Wi-Fi, or the Internet for further processing. There is a different protocol which can be used to communicate between IoT gateway and servers like MQTT, AMQP, COAP, and HTTP. We have discussed different protocols used in the next section.
  • Management Layer - This is the layer which is responsible for data modeling and security control. This is the layer where all the data handling or data processing operation will be done. At this layer, the necessary data is extracted from the data transferred by the sensor.
  • Application Layer - This is the final layer of the IoT analytics architecture. This layer uses the data processed by the management layer.
Apache Nifi is composed of processing elements, ETL with a graphical interface. Click to explore about, Building Data Ingestion Platform Using Apache Nifi

What is Real-Time Stream Processing for IoT?

Real-Time Stream processing refers to the data processing with the data stream collected from the IoT device in Real-Time. Learn more about Open Source tools for real-time Analytics Platform. These are the tasks which can be included in this processing -

  • Transformation - It includes the conversion of the data which is collected from the IoT device. After this conversion, the resulting data is transferred for further analytics.
  • Data Enrichment - Data enrichment process is the operation in which the sensor collected raw data is combined with the other dataset to get the results.
  • Storing Data - This task includes storing the data at the required storage location.

Which protocol is most widely used in IoT Analytics?

The below mentioned protocol are most widely used:

  • MQTT - MQTT protocol uses a publish/subscribe architecture. The central communication point of this protocol is MQTT broker. Every client includes a topic name while publishing data to the broker. Topics are responsible for routing information for the broker. Each client that wants to receive messages subscribes to a particular topic and the broker delivers all messages with the matching topic to the client.
  • COAP - COAP Protocol (Constrained Application Protocol) is a web-based protocol that has been designed to connect with lightweight devices to the Internet of things(IoT). Like HTTP protocols, COAP is also used Request-Response model. It also allows to make API calls GET, PUT, POST, DELETE data via URL.
  • AMQP - AMQP (Advanced Messaging Queuing Protocol) is an open standard for passing messages in between applications and organizations. It connects system, provides business processes with the information they need.
  • HTTP - This is the standard protocol for web services and still is used in it solutions. The most popular architectural style called RESTFul is widely used on mobile and web application and must be considered on IoT Solutions.
  • DDS - DDS stands for Data Distribution Service which is a standard for real-time IoT analytics, scalable and high-performance machine-to-machine communication. DDS can be deployed in both low footprint devices and on the cloud as well.
With Human-centric, sustainability and Resilient Industry 5.0 strategy for empowering connected IoT Solutions AI powered Robots. Explore our IoT Application Development Services

Xenonify IoT Architecture

Xenonify IoT Architecture

Data Collection and Ingestion

This will be the first step in data handling from the IoT device. We have different option to collect data from the sensors. In the upcoming paragraph,  we are going to discuss data ingestion using a different platform.

Using Apache NiFi

  1. Using Apache MiNiFi on Raspberry - We can use Apache MiNiFi which is a sub-project of Apache NiFi on the Raspberry Pi. This is a very lightweight and low resource-consuming agent. From this, we can collect data from the sensor and route it to the Apache NiFi. From here we can route the data to multiple required destinations.  
  2. Direct ingestion from MQTT Broker - The second way is to ingest data directly from the MQTT broker and route it to the multiple required destinations. For this, we need to design the data flow pipeline using different Apache NiFi processors.

Using Stream sets

  1. By Using StreamSets Data Collector Edge -  StreamSets data collector Edge is an ultralight agent for the IoT. Which collects the data and sends it to the stream sets.
  2. By Direct Ingestion from MQTT broker - The second way is to ingest data directly from the MQTT broker and route it to the multiple required destinations. For this, we need to design the data flow pipeline using different Stream sets processors.

Data Processing

Xenonify provides you with a platform that you can use to build your own IoT analytics platform for the data collected from the sensor. Apache Spark is being used by Xenonify at the data processing stage. The speed of the data stream can also vary according to the time. So, Our Data Pipeline should be able to handle data of any size and at any velocity. The platform should be intelligent enough to scale up and down automatically according to the load on the data pipeline.

Data Storage

Xenonify offers you multiple data storage option which includes HBase, HDFS, ElasticSearch, and Druid as the Time-Series database.


After the process now the most important part is the visualization of the data. Xenonify offers the custom dashboard for the visualization of the data. We can build our dashboard by using Kibana and Grafana. We can also build our custom dashboard using React JS and D3.JS.

Google IoT Architecture

Google IoT Architecture In Google, IoT analytics architecture Cloud Pub/Sub is used to listen to the data from MQTT broker. Also, Cloud Pub/Sub is a messaging system that can handle a real-time data stream. Cloud Pub/Sub also natively connects to other Cloud Platform services, gluing together data import, data pipelines, and storage systems. After ingesting the data from the IoT device Cloud Pub/Sub further sends the data to the Cloud Dataflow for the data processing. Cloud Dataflow is used to create the data pipeline to perform some data transformations. Cloud Dataflow sends the processed data to the BigQuery. BigQuery provides a fully managed data warehouse with a familiar, SQL-like interface. And further Cloud DataLab is used for the data representation.

Microsoft Azure IoT Architecture

xenonstack-aws-iot-architecture-1Microsoft Azure Events Hub connects to MQTT via Cloud Gateway and consumes the data published by Raspberry Pi on the MQTT broker. Azure provides Stream Analytics for data processing. Above all, we can use the data stream from Events Hub and process the real-time data stream. After processing, the data is sent to Azure CosmoDB and can be further visualized on Power BI by using the CosmoDB connector.

AWS IoT Architecture

xenonstack-microsoft-azure-iot-architecture-1Events Hub connects to MQTT via Cloud Gateway and consumes the data published by Raspberry Pi on the MQTT broker. Azure provides Stream Analytics for data processing. Above all, we can use the data stream from Events Hub and process the real-time data stream. After processing, the data is sent to Azure CosmoDB and can be further visualized on Power BI by using the CosmoDB connector.

Configuring Sensor DHT22 with Raspberry Pi 3

Requirements for Configuring Sensor
  • Raspberry Pi 3
  • DHT 22 or AM2302 temperature and humidity sensor
  • 10K resistor
  • Breadboard
  • Jumper wires
DHT22 is a 4-pin sensor. It can be operated on 3.3V which makes it compatible with Raspberry Pi 3.
  • Pin 1 is the VCC pin
  • Pin 2 is the data pin
  • Pin 3 is the No connection pin
  • Pin 4 is the ground pin
Pin 1 requires being connected with a 3.3V power source. Pin 2 to the GPIO (General-purpose input-output) on the Raspberry Pi. Pin 4 to the Ground (GND) and Pin 3 remain disconnected. A 10kΩ resistor is connected between pin 1 and pin 2.

Configuring Raspberry Pi 3

Firstly, we need to make sure that SSH login is enabled on our Raspberry Pi. Then we need to connect to Raspberry Pi using ssh.

Updating Raspberry Pi Packages

It is always necessary to ensure that your installed packages are up to date. First of all, we need to run the sudo apt-get update to update the list of available packages and versions. Now we can update our installed packages using sudo apt-get dist-upgrade.

Install additional Raspberry Pi packages

Now, we are using sensor DHT22 which needs some additional packages. DHT22 is accessed by using the python library. So we need to install python libraries using this - sudo apt-get install python

Installing DHT22 Sensor Libraries

Now, we are going to download the libraries which we will use for reading data from the sensor. To add more, we will use the libraries provided by Adafruit. So, we will clone this repository into our Raspberry Pi. Furthermore, all the installations have been completed and we can test the readings from the sensor. Now, we need to know on which GPIO pin the data pin of the sensor is connected and which sensor we are using. This is the pin diagram for Raspberry Pi 3 - Pin Diagram for Raspberry Pi Now, we are using the sensor DHT22 and GPIO pin is GPIO22. So, we will execute this - ./AdafruitDHT.py 22 22 This will give results as below - So, we can modify the AdafruitDHT.py script and enable that to push data on the MQTT broker. This is all we need to set up on the Raspberry Pi 3 and push data on the MQTT broker.

What are the key use cases of IoT Analytics?

The best use cases of IoT Analytics are listed below:


The Healthcare sector is one of the fastest sectors which is adopting the Internet of Things. There are a lot of sensors coming into the scenario and the healthcare sector is adopting them very fast. These devices are allowing the doctor to track their patients whether they are stuck to the treatment or not.


IoT-enabled machinery will enable operation managers to manage the factory units remotely. Hence, the use of IoT sensors in manufacturing equipment enables condition-based alerts that are designed to function within specific temperature and vibration ranges.

Smart Transportations

Keep vehicles on the road by predicting maintenance needs and Streamlining logistics using real-time data and alerts to optimize delivery routes, monitor performance, and quickly respond to delays or issues as they happen. Therefore, we can ease traffic congestion with the help of using real-time analytics.

Enables Identity and Access Management solutions integrated with existing enterprises LDAP and Windows AD Access Management. Taken From Article, IoT Application Development Services

Smart Retail

Predictive equipment maintenance is used for managing energy, predicting equipment failure, or detecting other issues. Hence, IoT analytics in retail helps to enable precise inventory management and most importantly enhances the consumer’s shopping experience.

Smart Buildings

No doubt, IoT helps to Connect and reduce management and utility costs by learning from the data you collected. For example, IoT analytics for smart buildings helps to personalize and automate your building’s heating and cooling, and room utilization, which further helps create a more comfortable and productive work environment.

Smart Agriculture

Smart Agriculture is growing rapidly in the field of agriculture. Therefore, farmers can use meaningful insights from the IoT-driven data for sensing soil moisture and nutrients and controlling water usage for plant growth to get better outcomes.

Holistic Approach

Real-Time Streaming Analytics Solutions for Enterprises & Startups helps them to accelerate their business growth for enhanced productivity. To know more about the enterprise IoT Analytics platform, you are advised to go through the below steps: