Xenonstack Recommends

Event-Driven Architecture for Cloud-Native in Kubernetes

Acknowledging Data Management
          Best Practices with DataOps

Subscription

Overview of Event-Driven Architecture

Event-Driven Architecture (EDA) is a model or an architectural paradigm for software that supports the production, detection, consumption of, and reaction to the event or a significant system state change. This structure consists of event creators and event consumers. Creators are the ones who produce events and have knowledge about what events occurred. On the other hand's consumers are the ones who become affected by events occurring, and they are also involved in processing events. The event-driven architecture is also sometimes related to stream processing architecture or message-driven architecture based on its fluctuations. Message Driven Architecture center combines the categorized systems by transferring messages from one module to another module of the system using a standard Message Broker. Event stream processing runs by controlling a data set by one data point at a time.
The cloud-native products are mostly open source with a commercially hosted version available in the cloud. Source - Beyond Kubernetes - 5 Promising Cloud-Native Technologies

Why Event-Driven Architecture Matters?

  • Loosely Coupled Structure – Event creators and consumers loosely coupled with event bus help, which helps them to the couple and decouples into different networks in response to various events. It also helps in giving different developers to develop different event processors. And use these components and also reused by many different networks.
  • In Real-time Analytics – Optimize event-driven architecture for real-time analytics. It has a much higher capacity to find, analyze, and then respond to patterns in time before critical events happen. Use complex event processing to achieve real-time analytics.
  • Versatility – A system that follows event-driven architecture facilitates greater responsiveness because event-driven systems are, by design, normalized to unpredictable, nonlinear, and asynchronous environments. These systems are highly adaptable in different circumstances.

What are the Characteristics & Considerations for choosing EDA?

The central Characteristics & Considerations of an Event-Driven architecture:-
  1. Multicast communications: The publishers can send events to various systems that have subscribed to them. This is opposite to unicast communication in which publishers can send the information only to one subscriber.
  2. Real-time transmission: Publishers generate the events and when the event occurs in real-time to the subscribers. The mode of processing is real-time rather than batch processing.
  3. Asynchronous communication: The publisher doesn't wait for the receiver to prepare an event before sending the new event. Event-Driven Architecture can perform asynchronous operations in correspondence. This results in a more reliable performance of the architecture, irrespective of the time stay involved in queuing and dequeuing messages.
  4. Fine-grained communication: Publishers keep publishing small and specific events instead of waiting for a single aggregated event.
  5. Event Ontology: Event-Driven Architecture systems always have a technique to classify events in terms of group/hierarchy based on their general characteristics which give flexibility to the subscribers to subscribe to a specific event and subscribers to subscribe to a specific category
  6. Deployment and Development: In Event-Driven Architecture, pattern elements are loosely coupled, and development is also easy because of the pattern's asynchronous nature.

Read more about the Top five Application Architecture That Enables Digital Transformation

Event-driven Architecture in Cloud-Native World

Event-driven architectures and reactive programming models aren't new ideas. But, during Cloud-Native architectures with microservices, container-based workloads, and serverless computing, you can revisit event-driven approaches in the cloud-native context. Two phases of a cloud-native architecture are necessary for developing an event-driven architecture:
  1. Micro-services implement the loosely coupled application architecture that allows deployment in highly distributed patterns for flexibility, agility, and scale.
  2. Cloud-native platforms with containers and serverless deployments present the application platform and tools that realize the flexibility, agility, and scalability of the microservices architecture.

How Event-Driven Architecture Works?

Listed below are the working stages of Event-Driven architecture:

Publish/Subscribe (Pub/Sub) Messaging

Pub/Sub is a form of asynchronous service-to-service communication used in microservices and serverless architectures. When publish or generate, an event it sends the message to each subscriber. A massage doesn’t repeat after its received, and new subscribers do not see the event. Subscribers do not need to know anything about the publishers, and vice-versa other than the publisher is a legitimate publisher and entitle subscriber is to receive information.

Event Source

The event source sends events to the event processing system. These sources can be one or more which can send an event to a single or multiples event processor.

Event Processor

This system can perform various actions on events –

  • Merely enrich a single event; for example, adding a timestamp to the event data called Simple Event Processing.
  • Add information about the source of the event. Process multiple single events, from multiple event sources against event patterns to produce a new derived or compound event called Complex Event Processing.

Event Consumer

The event consumer reacts to the event. An event consumer can be as simple as updating a database or business dashboard, or as complex as required, carrying out new business processing as a result of the event.

Here are some examples of consuming an event –

  • Updating a business dashboard
  • Updating a database
  • Interacting with Web 2.0 components to dynamically update web pages
  • Sending an alert by using email or SMS based on event data
  • Putting event data on a message queue
  • Starting a new piece of application work

Event processing categorized by processing complexity and sophistication having two categories –

  • Simple Event Processing – Simple events are events that are independent of other events and these events process under this category. These events do not summarize. Such events can provide considerably and provide excellent value business information. Simple processing includes processing such as augmenting the event payload with additional data, changing the events schema from one form to another, generating multiple events based on the payload of a single event, and redirecting the event from one channel or stream to another.
  • Complex Event Processing – Complex events are events that are dependent on other events and these events process under this category. These events do summarize. Complex processing involves applying a collection of evaluation constraints or conditions over an event set. Results of processing should not be complicated for the user, distinguish between the complex and vibrant nature of the information available from this processing.

Because of their flexibility, failure, and self-healing properties, Kubernetes implementation is the best for production purposes. Source - Implementing a Kubernetes Strategy in Your Organization

Implementing Event Source in Kubernetes

Event Sources are Kubernetes Custom Resources which implement a tool for expressing interest in a collection of events from a particular software system. For different events, sources may be defined by different CustomResources.

Kubernetes Events

Kubernetes events are a source type in Kubernetes that are automatically generated when other sources have state changes, errors, or other messages that should be declared to the system. While there is not a lot of documentation available for events in Kubernetes, they are an invaluable source when debugging issues in the Kubernetes cluster. It is necessary to understand that events do not develop in a way that you might expect. For example, if the container is weak to start because the docker image cannot be pulled the events are only visible on the pod, not on any Replica-set, StatefulSet, Deployment, or any other resource that generated the pod.

Best Tools to Implement Event-Driven Architecture

A message broker is a middle-ware software, tool or SaaS used to send events between event producers and consumers in a publish-subscribe pattern. All implementations of event-driven architecture need to use some technology in an event broker or message broker's role. there are various Tools for message broker but the top three tools which are used:-

KubeMQ

KubeMQ is a Cloud Native, enterprise-grade, message broker for shared services architecture. It is given as a small, lightweight Docker container, created for any workload and architecture working in Kubernetes or any new container orchestration system which supports Docker. Each container's capacity is 30MB, which gives it natural to perform in a microservice architecture. The capacity of a cluster of three containers is 90 MB. KubeMQ developed in GO to maximize computer resources and work smoothly with Kubernetes. According to Kubernetes experts when they benchmarked KubeMQ using a cluster with 1k message size and sent one million messages with the same hardware specs. KubeMQ beat another message broker by 20% in speed.

Kafka

Apache Kafka is a very successful publish/subscribe system that can probably process a stream of data. Kafka's central concept is a topic that can be replicated across a cluster giving a safe data warehouse. By performing processed message offsets following to Kafka, it is nearly straightforward to perform guaranteed "at-least-once" processing. Kafka is a stateful service, and this does make the Kubernetes configuration more difficult than stateless micro-services. Zookeeper is a centralized set of services for managing configuration data, labeling, providing shared synchronization, and providing group services. Deploy to manage a cluster, both Kafka and Zookeeper. Each Kafka container weighs about 600 MB, and each Zookeeper container weighs about 100MB, totaling 2.1GB (three Kafka containers + three Zookeeper containers). Kafka is heavy and manages large microservice without any problem.

NATS

NATS is a simple, secure, and high-performance, open-source messaging system for cloud-native applications, IoT messaging, and microservices architectures. The NATS server develops in the Go language. The center design systems of NATS are performance, ease of use, and scalability. A data streaming system powered by NATS combines persistence, message replay, and durable subscriptions to NATS. NATS supports four main patterns for communicating messages across entities. Those are Subject-Based messaging, Publish-Subscribe, Request-Reply, and Queue Group. Shows a quick summary of difference between KubeMQ, Kafka and NATS

What are the Challenges in Event-Driven Architecture?

The key difficulties in an EDA are Guaranteed delivery, Processing events in order or exactly once, and managing consistency across service boundaries.
  • Almost once: The producer transfers a message, and the consumer demand may / may not receive it.
  • At least once: The producer transfers a message, and the consumer may prepare duplicate occurrences of the message.
  • Exactly once: The producer transfers a message exactly once, and the consumer processes it exactly once in EDA.

Top Terminologies of Event-Driven Architecture

Event Bus

An event bus is a system or program or application which receives events from event creators and sends that event to any event consumers used in decoupling event creators and event consumers. Consumers can also post events to event buses if they are dependent on other events and the event bus is not responsible if the receiver does not receive any message.

Event Creator/Producers

An event creator or producer or publisher is a program or application which creates events and sends events to the event bus. The event creator knew what events occurred.

Event Consumer/Processor

An event consumer or processor or subscriber is a program or application which receives events from the event bus without knowing which creator creates that event. The event processor processes the events and from that take particular actions written in events.

Event Log

Store the messages written to event logs on a disk. If the system crashes, the system can resume its state by merely replaying the events logs. Take the backup of event logs and use them in performance tests on new releases before deploying them to production.


Topologies In Event-Driven Architecture

Mediator Topology

It has a single event queue in to store in order and pop out in order only. Use a mediator to direct each event to relevant event processors. Filtre each event is or pre-processes in an event channel before fed into the event processors.

Broker Topology

There is no event queue in this topology. Broker in this topology is event processors that are responsible for everything like chain events, obtaining events, processing, and publishing another event indicating the end. Once a processor processes an event, another event published so that another processor can proceed.


You may like to read this Cloud-Native Application Architecture Pattern and Design


Best Practices of Event-Driven Architecture

  • Supports the business demands for better services (no batch, less waiting).
  • No point-to-point integrations.
  • High performance and highly scalable systems.
  • Components can remain autonomous, being capable of coupling and decoupling into different networks in response to different events.
  • Complete advanced analytics using complex event processing.

Conclusion 

The event-driven architecture enables real-time, complex transactions among IoT devices, data streams and workflow systems. Every enterprise needs to respond quickly and effectively to evolving markets or circumstances. Greater operating efficiencies: a loose combining of the EDA without disrupting the layering, which allows for smooth, cost-effective service gives enterprises the competitive advantage they need. 

Related blogs and Articles

Adopt KEDA to Deploy Serverless, Event-Driven Containers

Kubernetes

Adopt KEDA to Deploy Serverless, Event-Driven Containers

Overview of Kubernetes Based Event Driven Autoscaling Implement event-driven processing on Kubernetes using Kubernetes-Based Event-Driven Autoscaling (KEDA). The IT industry is now moving towards Event-Driven Computing. Today it’s becoming so popular due to the ability of engaging users with the app. Popular games like PUBG and COD are using this approach to provide the user with a quick and...