Introduction to M3DB
As the data load increases, a need to detect fraud out of that data also increases. This problem can be solved by tracking or analyzing that data in real time. For this problem, real-time databases are being developed at a very large scale and many companies are partying in by making them open-source. Real-time data is the need and Real-time query engines are the solutions. M3DB is developed by uber for their internal use and later they open sourced it under Apache.
A Time-series database specially designed to work with such data has a much better and faster approach to handling such data. Click to explore about, Time-series Databases in Real-Time Analytics
What is M3DB?
M3DB is Distributed Time series database inspired by Gorilla and Cassandra (tools by Facebook) that handles a large amount of data and obtain incremental results for different Matrics and written in Go Lang. As data at Uber is not particular to the same location and the same amount, so M3DB handles that with ease for solving Uber Use case. That uber case was not properly solved by other tools they were using before at Uber such as Graphite, Prometheus, etc.
Why M3DB is important?
- M3DB provides the facility to monitor time series data at a faster rate with M3TSZ compression support which is based on Facebook’s Gorilla TSZ compression but with some differences.
- If real-time matrics report needed to fetch then it can fetch data from active buffers at a fast rate as they are in-memory buffers.
- Storage can be extended to any value with namespaces defined for data retention time. (currently 26 hours, which is more than the previous system that provides it for 2 hours).
- Snapshotting, caching policies and deployment with Prometheus are some important points that make it different from others in processing and development way.
Database instability may cause the systems to behave unexpectedly. Click to explore about, Database Testing Types and its Best Tools
How does M3 Database (M3DB) work?
- M3 stack is used by uber for aggregating matrics and Time series database. In that stack, M3DB is placed with M3 coordinator.
- M3 coordinator is managed alongside Prometheus and acts as a read/write endpoint between Prometheus and M3DB. This provides long term storage goals.
- M3DB uses M3TSZ compression algorithm based on Facebook’s Gorilla with the different use case and provides a very scalable compression ratio of 1.45bytes/data point (as per Uber use case).
M3DB In-memory object layout is represented as a hierarchy
- DataBase: It will be only one for each M3DB process
- Namespaces: Each Database has multiple namespaces (equivalent to tables)
- Shards: Each namespace can have multiple shards based on horizontal partitions. They provide arbitrary distribution of time series data by using a murmur3 hash function.
- Series: Series is contained by Shards and it acts as time series data. There can be multiple series.
- Block: Compressed time series data is stored under these blocks that are part of some series. We can define blocks as named partitions according to time cap.
- M3DB has persistent storage apart of in-memory storage (Active -buffers) as it is the way to recover data. In persistent storage, data is in uncompressed form and only unflushed data is present. Fileset files and commit logs are used to achieve this functionality whereas caching policies also exist to improve performance in this case.
Write flow
There is a number of steps that were followed while writing data and the first step is called made to write. When a call to the writeBatchRaw endpoint is made by the client, It will have to contain the following information:- Namespace
- Series ID (byte blob)
- Timestamp
- Data
a GPU-powered real-time query engine that improves uber’s existing solutions too. Click to explore about, AresDB - GPU Accelerated Real Time Big Data Analytics Engine
Read Flow
Read request contains 3 components: Namespace, series ID, data. The data also gets reads from three places i.e. from active buffers, from the in-memory cache, and from disk respectively. There are three operations based on a cluster in M3DB i.e. Node Add, Node down and Node Remove. M3DB has some consistency levels maintained and defined for reading, write and connect. These are: For Write:- One – One node success
- The majority – Majority of nodes success
- All – All nodes success
- One – Single node read
- Majority – Majority nodes reads
- All – All nodes successfully read
- UnstrictMajority – Majority of nodes read without the restriction of failed read
- Any – Any number of nodes connection
- None – no node connection
- One – Single node connection
- The majority – Most of the node's connection
- All – All nodes connection
What are the benefits of M3DB?
- Reduce the cost of resource development, ingestion, and storage.
- M3 can process 500 million metrics per second with the persistence of up to 20 million aggregated metrics. Calculating this for 24 hours based structure it can process up to 45 TRILLION matrics a day.
- Most of the times, data to be accessed is available in Active buffers. That makes it faster to read from there.
- M3DB has one commit log for each database i.e. each namespace share one commit log. These commit logs are totally uncompressed and retain only that data which is not flushed to Persistent storage.
- It has multiple caching policies that let us define what data is needed to cache and on which level.
It is important to remember that Google sheet is a fantastic option for cases with small and vocational data dependencies. Click to explore about, How to use Google Sheets as a Database for HTML Pages?
How to implement M3DB?
M3 can be deployed by one of the following ways:- M3DB single node Deployment: Useful for standalone local implementation. Need to setup m3dbnode and m3coordinator instances.
- M3DB on Kubernetes: Useful when used on fast disks. Can be integrated with Prometheus
- M3DB manual Deployment: How a deployment or architecture looks like
- M3Query: By setting up M3query one can query M3DB easily as it is the part of M3QL and one can define aggregations and all by using M3Query.
How to integrate M3DB?
While Integrating M3DB it needs to set up some configurations. It can be integrated with one of the following and Grafana is used for querying results. Prometheus Prometheus provides the facility of matrics monitoring at large scale.- Define properties in the configuration file of M3 coordinator
- Merge it on Prometheus
- Query by using Grafana
- Define the Ingestion Pathway (using carbon plaintext protocol)
- Handle aggregations by using default defined functions or Graphite functions
Our solutions cater to diverse industries with a focus on serving ever-changing marketing needs. Click here to explore our End-to-End Cloud Managed Database Solutions
Conclusion
M3DB brings that handling of multi-million matrics per second with the persistence of some million aggregated matrics also. Snapshotting in M3DB is helps to create other compressed time series data in the form of fileSet structure but at a different location. These can help to remove the load from commit logs as they can be cleaned once snapshots of their data are created.
- Discover more about Local Storage vs Session Storage vs Cookie
- Click to explore Data Warehouse Design and its Architecture
