Introduction to Data Mesh
Data mesh builds a layer of connectivity that takes away the complexities of connecting, managing, and supporting data access. It is a way to fasten the data together that is held across multiple data silos. It combines the data distributed data across different locations and organizations. It provides data that is highly available, easily discoverable, and secure. It is beneficial in an organization where a team generates data from many data-driven use cases and access patterns in it.Cloud technology used in conjunction with the “data mesh” concept provides a promising approach to solving regulatory reporting problems. Source: Cloud-based Data Mesh Technology
We can use it, like when we need to connect cloud applications to sensitive data that lives in a customer's cloud environment. Also, when we need to create virtual data catalogs obtained from various data sources that can't be centralized. There is also a situation in which it is used, for instance, when we create virtual data warehouses or data lakes for analytics and ML training that can be done without consolidating data into a single repository.
What is Data Mesh Architecture?
Data Mesh Architecture is an architecture paradigm that offers self configure, self-service architecture design embracing the distributed, diverse nature of the data while leveraging Domain-Driven Design. A paradigm where database, pipelines, structure, and code language correspond with the business domain. Unlike centralized monolithic architecture, Data Mesh is a distributed architecture. Big data to provision innovation must have accountability and ownership with domain-driven owners who can provide data as a product. InterOperability, Customer satisfaction, scalability are the main focus points of Data Mesh.
Data mesh architecture also focuses on building Data Infrastructure as a platform for self-serve use. With this architecture, Data consumers like Data scientists, ML Engineers, Product Teams can have easy and direct access to Data Products with agreed SLA’s. It also allows micromanaging the access rights to data.
Example of Data Mesh Architecture
Think of a data platform that has to be built for eCommerce. In Mesh Architecture, you define domains as products, orders, carts, sessions, leads, etc. These domains are cohesive to each other and complete on their own at the same time. Each domain will have its pipeline, own storage while allowing the flow of data to cross domains. To control access rights on data, each domain will have a governance layer to take care of Policies, granular Access control, Data Monitoring.
Click to explore Adopt or not to Adopt Data Mesh? - A Crucial Question
What are the Challenges of Data Mesh Architecture?
With the ever-increasing diversity of data, the modern data tech stack has evolved. Data teams are getting more diverse than ever. With this evolution, the world is now moving from Old monolithic architectures to Distributed Data Mesh.
Integration and ingestion of data in a Central Monolithic Data Lake/Data Warehouses Architecture is a good idea for analyzing and reporting data. Still, they welcome the organizations with a huge cost, unmanaged technical debt, the technical debt of hundreds of ETL jobs, Batch processing jobs, and Databases on which only a set of few specialized people can work, thus delivering over-promised, no ownership underperformed results and huge bills.
In the following sections, we will be talking about Data Mesh Architecture, which necessarily is not a perfect solution for the above issues but allows us to address and avoid most of the problems.
What are the benefits of Data Mesh?
- It provides agility. In this, each node works independently. The node is containerized and can be deployed as soon as any changes are ready.
- Construct and deploy new nodes to the mesh, whenever new data arises. Many portals and teams can access the same node, allowing the organization to scale the data mesh. This way, it provides scalability.
- Use it under various circumstances, like connecting cloud applications to sensitive data that lives in a customer's on-premise or cloud environment. Use it while creating virtual data catalogs from various data sources. We need to create virtual data warehouses or data lakes for analytics and machine learning training without consolidating data into a single repository.
What is the Logical Architecture Design for Data Mesh?
The four principles act as a base for Data Mesh Architecture.
Domain Oriented
With a shift to microservices architecture, product thinking has been built with domain-driven development. Domain-driven teams provide Access to their product data by building APIs, and pub/sub-messaging systems. The teams focus on delivering the best developer experience by providing understandable and discoverable documentation. For self-serve data platforms, Domain-driven data teams must apply the same product thinking to their data set, considering their customers as Data Scientists, ML Engineers, and various cross teams.
Key points to be taken care of before serving data as the product are
- Data must be Reliable
- Data must be easy to understand semantics and well-defined schema.
- It should be easily discoverable.
- Security of data should be taken care of in both motion and rest.
- It should be InterOperable.
Decentralized Data Ownership
Domain data teams must be fed with new skill sets. A single domain data team must be accompanied with
Data Product Owner: The person who is Accountable and sets the path, makes decisions, and creates a vision around creating data as a product, the one who takes decision-related to architecture and decides KPIs for their data products
Data Engineers: In order to build data pipelines, and organize and convert data to useful information, the team must have Data Engineers.
Self Serve Data Platform
For example, we are building a data infrastructure to automate data ingestion through configurations, self-registering a data product with the catalog, etc.
A Success Criterion of Data Infrastructure as a platform can be measured by how much time it takes to register new data as a product on the platform infrastructure.
The checklist that must be considered but not limited to while building data infrastructure as a platform :
- Data governance according to defined standards
- Scalable data storage
- Data versioning
- Confederated identity management
- Data lineage
- Data Monitoring and proper logging
- Data Caching
- Data Quality Benchmarks
- Encryption of data and secrets
Monitoring and Governance
Movement, Accessing, and storing of data has a primary security concern, whether it is a centralized or a decentralized architecture. In Mesh architecture, Access is controlled at a finer granular level. Defining a role-based access control policy is a convenient way to monitor Access across the platform.
What is Anthos Service Mesh?
It is a fully managed service mesh that is used for complex microservices architectures. It is a suite of tools that monitor and manage a reliable service mesh on-premises or Google Cloud. It's powered by Istio, which is a highly configurable and one of the powerful open-source service mesh platforms that have tools and features that enable industry best practices. It defines and manages configuration centrally at a higher level. It is deployed as a uniform layer across the full infrastructure. Service developers and operators can use a rich feature set without making a single change to the application code.
Anthos Service Mesh relies on Google Kubernetes Engine (GKE ) GKE On-Premise Observability features. Microservices architectures provide many benefits, but on the other hand, there are challenges like added complexity and fragmentation for different workloads. It solves the problem like it unburdens operations and development teams by simplifying service delivery across the board, from traffic management and mesh telemetry to securing communications between services.
What are the features of Anthos Service Mesh?
Here are some of the features of Anthos Service MeshDeep visibility built-in [beta]
Anthos Service Mesh is integrated with Cloud Logging, Cloud Monitoring, and Cloud Trace that provides many benefits, such as monitoring SLOs at a per-service level and setting targets for latency and availability.Easy authentication, encryption
Anthos Service Mesh ensures easy authentication and encryption. It transport authentication through MTLS (Mutual Transport Layer Security) has never been more effortless. It secures service-to-service as well as end-user-to-service communications with just a one-click mTLS installation or incremental implementation.Flexible authorization
It provides flexible authorization like we only need to specify the permissions after that grant access to them at the level that we choose, from namespace down to users.Fine-grained traffic controls
Anthos Service Mesh opens up many traffic management features as it decouples traffic flow from infrastructure scaling and includes dynamic requests. Routing for A/B testing, canary deployments, and gradual rollout, and that also all outside of your application code.Failure recovery out of the box
It provides many critical failure-recovery features out of the box, to configure dynamically at runtime.
There's no better way to pick which data storage platform suits your enterprise than to assess it primarily based on use case and organization demand. Click to explore about our, Data Warehouse vs Data Lake vs Data Mesh
What is Azure Service Fabric Mesh?
Azure Service Fabric Mesh helps the developers deploy microservices applications, and there is no need to manage virtual machines, storage, or networking. The applications hosted on Service Fabric Mesh run and scale without worrying about the infrastructure powering it. Service Fabric Mesh has clusters of many machines, and every one of these cluster operations is hidden from the developer.
You only need to upload the code and mention the resources we need, availability requirements, and resource limits. It automatically allocates the infrastructure and handles infrastructure failures as well, and we need to make sure the applications are highly available. We need to take care of the health and responsiveness of the application and not the infrastructure. Azure Service Fabric has three public offerings: Service Fabric Azure Cluster service, Service Fabric Standalone, and Azure Service Fabric Mesh service.
A framework that allows storing Big Data in distributed mode and allows for the distributed processing on that large dataset. Click to explore about our, Big Data Open Source Tools
What is AWS App Mesh?
AWS App Mesh helps to run services by providing consistent visibility and network traffic controls. For services built across multiple computing infrastructure types. App Mesh abolishes the necessity to update the application code. To vary how monitoring data is collected or traffic is routed between services. It configures each service to export monitoring data and implements consistent communications control logic across your application. When any failure occurs or when code changes must be deployed, therein situation makes it easy. To pinpoint the precise location of errors quickly and automatically reroute network traffic.What are the advantages of AWS App Mesh?
Following are the advantages of AWS App Mesh: Provides End-to-end visibility because it captures metrics, logs, and traces from all of your applications. We can combine and export this data to Amazon CloudWatch, AWS X-Ray, and community tools for monitoring, helping to quickly identify and isolate issues with any service to optimize your entire application.Ensure High Availability
App Mesh gives controls to configure how traffic flows between your services. Implement easily custom traffic routing rules to ensure that every service is highly available during deployments, after failures, and as your application scales.Streamline Operations
App Mesh configures and deploys a proxy that manages all communications traffic to and from your services. This removes the requirement to configure communication protocols for every service, write custom code, or implement libraries to control the application.Enhance Any Application
Users can use App Mesh with services running on any compute services like AWS Fargate, Amazon EKS, Amazon ECS, and Amazon EC2. App Mesh can also monitor and control communications for monoliths running on EC2. Teams running containerized applications, orchestration systems, or VPCs as one application with no code changes.Hybrid Deployments
To configure a service mesh for applications deployed on-premises, we can use AWS App Mesh on AWS Outposts. AWS Outposts could be a fully managed service that extends AWS infrastructure, AWS services, APIs, and tools to virtually any connected site. With AWS App Mesh on Outposts, you'll provide consistent communication control logic. For services across AWS Outposts and AWS cloud to simplify hybrid application networking.There are various major challenges that come into the way while dealing with it which need to be taken care of with Agility. Click to explore about our, Top 6 Big Data Challenges and Solutions
Data Mesh vs Data Lake
Given below are the differences between Data Mesh and Data Lake.- The data lake is a storage repository. That holds a vast amount of raw data in its native format. The hierarchical data warehouse stores data in files or folders. Whereas the data lake uses a flat architecture to store data.
- The advantage of the data lake is that it is a Centralized, singular, schema-less data store with raw (as-is) data as well as massaged data.
- The Mechanism for fast ingestion of data with appropriate latency
- It helps to map data across various sources and give visibility and security to users
- Catalog to find and retrieve data
- Costing model of centralized service
- Ability to manage security, permissions, and data masking
- The main difference between data mesh and data lake is that it is decentralized ownership in which domain teams usually consider their data a byproduct that they don't own because a data lake is centralized ownership of that raw data.
How is Data Mesh different from Data Fabric?
- Data Fabric integrates data management across cloud and on-premises to accelerate digital transformation. It helps deliver consistent and integrated hybrid cloud data services that help data visibility and insights, data access and control, and data protection and security.
- Data Fabric and its difference is that Data fabric allows clear access of data and sharing of data across distributed computing systems by means of a data management framework that is single, secured, and controlled.
- But Data Mesh follows a metadata-driven approach and is a distributed data architecture supported by machine learning capabilities. It is a tailor-made distributed ecosystem with reusable data services, a centralized governance policy, and dynamic data pipelines.
Data Mesh vs. Centralised Platform: How it is different?
Data Mesh brings a shift from the Centralised platform in architecture, technical solutions, operations, and thought processes in the data engineering team. It changes how couples manage, analyze and use analytical data.
- Architectural Shift: Architecture shifts from collecting and managing data in monolithic data lakes/warehouses to connecting data through the mesh.
- Accountability and Ownership: It shifts Ownership/Accountability from a single central specialized team that works on data platform technologies to decentralized business domain-specific teams.
- Operational Shift: It shifts data governance from a top-down, centralized operational model to a confederate model with policies enforced on each mesh node.
- Thought Process Shift: It brings a change in thought process from thinking data as an asset to be collected to data as a product to be served.
Explore about Kafka vs pulsar : Pick the Right one for your Business
Solutions for Implementing Data Mesh
Below mentioned are Solutions for Implementing Serverless Data Mesh:
Amazon Web Services (AWS)
Aws provides various services to set up Data Mesh Architecture easily. Let’s discuss one solution for building the Data Mesh Architecture Platform.
Data Mesh Architecture using AWS Lake Formation and Glue
- AWS Glue: AWS Glue is a serverless service for ETL operations. It gives an easy path for data engineers to extract, clean, transform and load data. It provides code-based and visual interfaces to make data operations easy.
- AWS Lake: AWS Lake Formation is a managed service that can build, secure, and manage data lakes. Lake Formation automates various steps, including collecting, cleansing, transferring, and cataloging data to create a Data Lake. Lake Formation is pointed at data sources, and it crawls through attached sources and moves the data into the new Amazon Simple Storage Service (Amazon S3) data lake.
The workflow for data as a product includes the following steps:
- Data sources are hosted in the AWS Glue Data Catalog and enrolled with Lake Formation.
- Lake Formation Data Catalog entities will be created under the central governance account when a dataset is added as a product.
- The Central Catalog shares the catalog resources with domain accounts enabling required permissions via Lake Formation.
- Data changes made within the domain account are automatically transported to the central governance copy of the catalog.
- Lake Formation in the main domain account can define and enforce access permissions on these datasets for users to consume. Like Data Scientists and ML Engineers, user accounts can access data using their chosen tool, such as Athena.
Microsoft Azure
Azure cloud provides various services to support Data Mesh Architecture like Azure Data Lake Storage (ADLS), Azure Synapse, and Microsoft SQL Server.
- Azure Data Lake: It provides all the capabilities required to store data of any size, and shape and do all types of processing across platforms and languages.
- Azure Service Fabric Mesh: It is a fully managed service that allows you to build, deploy and manage applications consisting of polyglot services running within containers using a “serverless” approach. The service fabric platform has a built-in, highly available persistent state store. This service will be retiring this year (2021) though it will continue to support existing deployments.
Google Cloud Platform (GCP)
Anthos Service Mesh is a suite of tools that help monitor and manage a reliable service mesh on-premises or on Google Cloud. It is a powerful tool for managing and monitoring distributed applications.
Features of Google Cloud Platform
- Traffic Management
- Observability Insights
- Fine-grained security policies
- Failure recovery
- Easy Authentication and Encryption
- Load Balancing
Conclusion
A data mesh allows the organization to escape the analytical and consumptive confines of monolithic data architectures and connects siloed data. To enable machine learning and automated analytics at scale. It allows the company to be data-driven and give up data lakes and data warehouses. It replaces them with the power of data access, control, and connectivity.
- Discover here about Master Data Management in Manufacturing Industry
- Click to explore about Big Data Security and Management
