Event-Driven Architecture for Microservices and Cloud-Native in Kubernetes

What is Event-Driven Architecture?

An Event-Driven Architecture is a software architecture that operates services by the production, detection, and consumption of, and reaction to, significant changes in a system’s state (known as events). This architecture consists of event creators and event consumers. Creators are the ones who create events and have knowledge about what events occurred while consumers are the ones who get affected by events occurrence and they are involved in processing events. You can also learn about Cloud Native Applications Architecture in this blog. How to Adopt Event-Driven Architecture?

  • Multiple subsystems must process the same events.
  • Processing at Real-Time with the minimum time lag.
  • High velocity and a high volume of data, such as IoT.
  • Complex event processing, such as pattern matching or aggregation over time windows.

Event-Driven Architecture for Cloud-Native in Kubernetes

Event-Driven Architecture (EDA) is a model or an architectural paradigm for software that supports the production, detection, consumption of, and reaction to the event or a significant change in system state.

This structure consists of event creators and event consumers. Creators are the ones who produce events and have knowledge about what events occurred on the other hand’s consumers are the ones who become affected by events occurring and they are also involved in processing events.

The event-driven architecture is also sometimes related to stream processing architecture or message-driven architecture based on its fluctuations. Message Driven Architecture centre is to combine the categorized systems by transferring messages from one module to another module of the system using a standard Message Broker. Event stream processing runs by controlling a data set by one data point at a time. Instead of view data as a whole set, event stream processing is about


Why Event-Driven Architecture Matters?

  • Loosely Coupled Structure – Event creators and consumers loosely coupled with the help of event bus which helps them to the couple and decouples into different networks in response to various events. It also helps in giving different developers to develop different event processors. And, these components can be used and reused by many different networks.
  • In Real-time Analytics – Event-driven architecture is highly optimized for real-time analytics. It has a much higher capacity to find, analyze, and then respond to patterns in time before critical events happen. Complex event processing is used to achieve real-time analytics.
  • Versatility – A system that follows event-driven architecture facilitates greater responsiveness because event-driven systems are, by design, normalized to unpredictable, nonlinear, and asynchronous environments. These systems are highly adaptable in different circumstances.

How Event-Driven Architecture Works?

Publish/Subscribe (Pub/Sub) Messaging

It is a form of asynchronous service-to-service communication used in microservices and serverless architectures. When an event gets published or generated, it sends the message to each subscriber. A message doesn’t repeat after its received, and new subscribers do not see the event. Subscribers do not need to know anything about the publishers, and vice-versa other than the publisher is a legitimate publisher and subscriber is entitled to receive information.

Event Source

The event source sends events to the event processing system. These sources can be one or more which can send an event to a single or multiples event processor.

Event Processor

This system can perform various actions on events –

  • Merely enrich a single event; for example, adding a timestamp to the event data called as Simple Event Processing.
  • Add information about the source of the event. Process multiple single events, from multiple event sources against event patterns to produce a new derived or compound event called Complex Event Processing.

Event Consumer

The event consumer reacts to the event. An event consumer can be as simple as updating a database or business dashboard, or as complex as required, carrying out new business processing as a result of the event.

Here are some examples of consuming an event –

  • Updating a business dashboard
  • Updating a database
  • Interacting with Web 2.0 components to dynamically update web pages
  • Sending an alert by using email or SMS based on event data
  • Putting event data on a message queue
  • Starting a new piece of application work

Event processing categorized by processing complexity and sophistication having two categories –

  • Simple Event Processing – Simple events are events that are independent of other events and these events process under this category. These events do not summarize. Such events can provide considerably and provide excellent value business information. Simple processing includes processing such as augmenting the event payload with additional data, changing the events schema from one form to another, generating multiple events based on the payload of a single event, and redirecting the event from one channel or stream to another.
  • Complex Event Processing – Complex events are events that are dependent on other events and these events process under this category. These events do summarize. Complex processing involves applying a collection of evaluation constraints or conditions over an event set. Results of processing should not be complicated for the user, distinguish between the complex and vibrant nature of the information available from this processing.

Implementing Event Source in Kubernetes

Event Sources are Kubernetes Custom Resources which implement a tool for expressing interest in a collection of events from a particular software system. For different events, sources may be defined by different Custom Resources.

Kubernetes Events

Kubernetes events are a source type in Kubernetes that are automatically generated when other sources have state changes, errors, or other messages that should be declared to the system.

While there is not a lot of documentation available for events in Kubernetes, they are an invaluable source when debugging issues in the Kubernetes cluster.

It is necessary to understand that events do not develop in a way that you might expect. For example, if the container is weak to start because the docker image cannot be pulled the events are only visible on the pod, not on any Replica-set, StatefulSet, Deployment, or any other resource that generated the pod.

Tools to Implement Event-Driven Architecture

A message broker is a middle-ware software, tool or SaaS used to send events between event producers and consumers in a publish-subscribe pattern. All implementations of event-driven architecture need to use some technology in the role of an event broker or message broker. there are various Tools for message broker but top three tools which are used:-

KubeMQ

KubeMQ is a Cloud Native, enterprise-grade, message broker for shared services architecture. KubeMQ is given as a small, lightweight Docker container, created for any workload and architecture working in Kubernetes or any new container orchestration system which supports Docker. KubeMQ is also Lightweight and Small Footprint Container. The capacity of each container is 30MB, which gives it natural to perform in a microservice architecture. The capacity of a cluster of three containers is 90 MB. KubeMQ was developing in GO to maximize computer resources and to work smoothly with Kubernetes. According to Kubernetes experts when they benchmarked KubeMQ using a cluster with 1k message size and sent one million messages with the same hardware specs. KubeMQ beat another message broker by 20% in speed.

Apache Kafka

Apache Kafka is a very successful publish/subscribe system, which can be used probably to process a stream of data. The central concept in Kafka is a topic, which can be replicated across a cluster giving a safe data warehouse. By performing processed message offsets following to Kafka, it is nearly straightforward to perform guaranteed “at-least-once” processing. Kafka is a stateful service, and this does make the Kubernetes configuration more difficult than stateless micro-services. Zookeeper is a centralized set of services for managing configuration data, labelling, providing shared synchronization, and providing group services. To manage a cluster, both Kafka and Zookeeper should be deployed. Each Kafka container weighs about 600 MB, and each Zookeeper container weighs about 100MB, totalling 2.1GB (three Kafka containers + three Zookeeper containers). Kafka is heavy and manages large microservice without any problem.

NATS

NATS is a simple, secure and high-performance, open-source messaging system for cloud-native applications, IoT messaging, and microservices architectures. The NATS server develops in the Go language. The centre design systems of NATS are performance, ease of use and scalability. A data streaming system powered by NATS that combines persistence, message replay, durable subscriptions to NATS.NATS supports four main patterns for communicating messages across entities. Those are Subject-Based messaging, Publish-Subscribe, Request-Reply, and Queue Group.

Shows a quick summary of Difference between KubeMQ, Kafka and NATS
Difference between KubeMQ, Kafka and NATS

Challenges in Event-Driven Architecture

The key difficulties in an EDA are Guaranteed delivery, Processing events in order or exactly once and managing consistency across service boundaries.

Almost once: The producer transfers a message, and the consumer demand may / may not receive it.

At least once: The producer transfers a message, and the consumer may prepare duplicate occurrences of the message.

Exactly once: The producer transfers a message exactly once, and the consumer processes it exactly once in EDA.


Top Terminologies of Event-Driven Architecture

Event Bus

An event bus is a system or program or application which receives events from event creators and sends that event to any event consumers used in decoupling event creators and event consumers. Consumers can also post events to event buses if they are dependent on other events and the event bus is not responsible if the receiver does not receive any message.

Event Creator/Producers

An event creator or producer or publisher is a program or application which creates events and sends events to the event bus. The event creator knew what events occurred.

Event Consumer/Processor

An event consumer or processor or subscriber is a program or application which receives events from the event bus without knowing which creator creates that event. The event processor processes the events and from that take particular actions written in events.

Event Log

The messages written to event logs stored on a disk and these messages are the ones written to an event queue. If the system crashes, the system can resume its state by merely replaying the events logs. Backup of event logs can be taken and used in performance tests on new releases before deploying them to production.


Topologies In Event-Driven Architecture

Mediator Topology

It has a single event queue in which all events are stored in order and pop out in order only. A mediator which is used to direct each event to relevant event processors. Each event is filtered or pre-processes in an event channel before fed into the event processors.

Broker Topology

There is no event queue in this topology. Broker in this topology is event processors that are responsible for everything like to chain events, obtaining events, processing and publishing another event indicating the end. Once a processor processes an event, another event published so that another processor can proceed.


Best Practices of Event-Driven Architecture

  • Supports the business demands for better services (no batch, less waiting).
  • No point-to-point integrations.
  • High performance and highly scalable systems.
  • Components can remain autonomous, being capable of coupling and decoupling into different networks in response to different events.
  • Advanced analytics can be done using complex event processing.

Event Driven Approach

The event-driven architecture enables real-time, complex transactions among IoT devices, data streams and workflow systems. Every enterprise needs to respond quickly and effectively to evolving markets or circumstances. Greater operating efficiencies – a loose combining of the EDA without disrupting the layering, which allows for smooth, cost-effective service gives enterprises the competitive advantage they need.