Skip to main content
XenonStack

Big Data Engineering

XenonStack Blog Data Reliability Engineering: Checks, SLOs, Tools

and | Published | Updated

Key takeaway

  • Learn how to run data reliability in production: the dimensions to measure, SLOs for data, pipeline controls and which tools still fit in 2027.

What is data reliability, and why does it matter to leaders?

Data reliability is the degree to which people and systems can depend on a dataset being complete, accurate, unique, current and consistent every time it is used, not only on the day it was checked. For a CTO or chief data officer, the practical test is simple: can a dashboard, model or AI agent act on this data today without a person double-checking it first?

This guide treats reliability as an operating practice with owners, targets and alerts. For the fundamentals and the benefits, read our companion post on data reliability benefits and framework; here we cover how to run it in production, which tools still fit in 2026, and when to stop adding checks.

What are the dimensions of reliable data?

Reliability is not one property. Treat each dimension below as something you measure, then agree a target for the datasets that feed decisions.

DimensionWhat to measureTypical failureWhere to check it
ValidityShare of values in the expected type, format and rangeA negative water volume, or a date stored as textAt ingestion
CompletenessShare of required fields and expected records presentMissing email addresses; a sensor that stops reportingAt ingestion and after each load
UniquenessShare of records with no duplicates on the business keyA pipeline restart that loads the same batch twiceAfter each load
FreshnessAge of the newest record against the expected cadenceA nightly job that silently stopsOn a schedule, with alerting
LineageWhether each field can be traced to its source and transformationsA metric changes and no one can say whyAcross the pipeline
Access integrityWho can change the data, and whether changes are loggedAn unreviewed edit alters a reported figureAt the storage layer

Validity and reliability are not the same thing. A dataset can pass every format check and still be unreliable because rows are missing or duplicated. For example, an employee file with correctly formatted records but several blank email fields is valid and still fails the mailing it was meant to support.

How do you turn data reliability into an SLO?

Site reliability engineering already gives a vocabulary for this. Google's SRE book defines a service level indicator (SLI) as "a carefully defined quantitative measure of some aspect of the level of service that is provided" and a service level objective (SLO) as "a target value or range of values for a service level that is measured by an SLI" (source: Google SRE book). Applied to data, the SLI might be the percentage of orders loaded with no null customer key, and the SLO the level that percentage must reach each day.

Start with the five to ten datasets that feed revenue, risk or regulatory reports. Give each one an owner, one to three SLIs and a target agreed with its consumers. Keep the number of SLOs small: an SLO nobody acts on is noise.

How do you make data reliable across the pipeline?

Check at ingestion

Reject or quarantine nulls, out-of-range values and malformed records as data enters, and ingest only what a use case needs. Catching a defect here is cheaper than repairing it after it has been loaded and used downstream.

Two-column diagram mapping four data defects (missing records, duplicate loads, invalid values, unexplained change) to controls (freshness alert, de-duplication, range validation, lineage capture).

Trace through transformation

Record lineage so that every change between source and destination is visible. OpenLineage describes itself as "an open platform for collection and analysis of data lineage" and offers a standard API for capturing events about datasets, jobs and runs. See our guide to data lineage tools for options.

Validate before storage

Map each record to the target schema and run constraint checks before it reaches the warehouse or lake. Our post on data validation methods covers the techniques in more detail.

Monitor continuously

Data that was reliable at load can degrade as sources, schemas and volumes change. Track freshness, volume and constraint pass rates over time, and alert the dataset owner when an SLO is at risk.

Which tools still fit a data reliability programme?

No single tool delivers reliability. Teams combine validation, lineage and monitoring components, and the open-source landscape has changed since many older guides were written.

ToolWhat it doesWhen it fitsCurrent status (read 11 Nov 2026)
Deequ (AWS Labs)Defines "unit tests for data" as constraints and runs them as Spark jobs, with profiling and anomaly detection on metricsSpark-based pipelinesActive repository; Deequ 2.x covers Spark 3.1 to 3.5 and 1.x covers Spark 3.0 and earlier
Apache AtlasA "data governance and metadata framework for Hadoop"Hadoop-centred estates that need a metadata catalogueListed on atlas.apache.org; check the project page for the release that suits your stack
OpenLineageOpen standard and client libraries for lineage eventsEstates with several schedulers, warehouses and enginesOpen specification with a technical steering committee
Apache GriffinBatch and streaming data quality measurementNot recommended for new workProject page states 'This project has retired'
Great Expectations (GX Core)Open-source Python library for declaring checks on dataPython-based pipelinesDocumentation live; see docs.greatexpectations.io

The change worth noting is Apache Griffin. Older guides, including the earlier version of this one, list it as a data quality tool. Its project page now says it has retired, so choose a maintained alternative for new work. See the Apache Griffin page for details.

Sources: Deequ repository, Apache Atlas, OpenLineage, Great Expectations documentation.

What does an unreliable pipeline look like in practice?

Consider an illustrative farm analytics pipeline. IoT sensors send readings, through a pipeline, to a database that feeds a yield forecast. Two ordinary events break it: a sensor goes offline and leaves gaps, and a pipeline restart loads a batch twice. The forecast then rests on incomplete and duplicated data, and nobody notices until the numbers look wrong.

StageControlExample rule
Sensor and collectionReliable hardware and a heartbeat checkAlert if a sensor sends no reading for the agreed interval
Stream filterRange validation on accepted valuesWater volume must be an integer between 0 and the tank capacity (2,000 in this example)
LoadSchema mapping and de-duplicationOne row per sensor and timestamp
DatabaseLineage and freshness monitoringAlert if the newest record is older than the expected cadence

Complete reliability cannot be achieved in one step. Apply controls where the defects actually occur, then widen coverage as the first controls prove their value.

What do reliable data and AI agents have in common?

As teams let AI agents act on enterprise data, unreliable inputs turn into unreliable actions. Two things help: a clear record of the context and policy behind each decision, and a way to ask questions of governed data and check the answer. ElixirData Context OS keeps governed context, versioned policy and decision traces in one decision layer above AI agents. AgentAnalyst lets teams ask questions of enterprise data in plain language and get real-time insights, and the Audit Agent assembles audit evidence for past decisions, although its full lineage covers only decisions made through the platform.

When is a reliability programme not worth it?

Not every dataset needs an SLO. A one-off analysis, a prototype or a low-impact internal report may need only basic validation. Reserve owners, targets and alerting for data that drives money, risk or compliance. Over-instrumenting low-value data adds alert fatigue and cost without improving decisions.

How should you sequence adoption?

StepActionOutcome
1List the datasets behind key decisions and name an owner for eachClear accountability
2Add validity, completeness and uniqueness checks at ingestionDefects stopped early
3Capture lineage for those datasetsFaster root-cause analysis
4Define one to three SLIs and an SLO per datasetShared definition of 'reliable'
5Add freshness and volume monitoring with alertsProblems found before consumers find them
6Review SLO misses monthly and retire checks that never fireA programme that stays lean

To map one data-dependent decision your agents or analysts make, and the context and policy it depends on, book a Context Workshop. See what happens in the workshop.

What should you plan for in 2027?

If your checks run as Spark jobs, the Spark release you depend on matters for the year ahead.

  • November 2027: the Apache Spark versioning policy states that the Spark 3.5 extended long term support period will end in November 2027.
  • The same page says that period covers security fixes only, so pipelines still on Spark 3.5 for Deequ checks should have an upgrade plan before then.
  • Keep each dataset owner, SLO and freshness alert under review as engines change, and retire checks that never fire.

Treat the upgrade as a reliability project in its own right and run your validity, completeness and uniqueness checks again afterwards.

FAQs on data reliability

What is the difference between data reliability and data validity?

Validity asks whether data is stored and formatted correctly. Reliability asks whether the data can be trusted: complete, unique, current and traceable. A dataset can be valid and still unreliable if records are missing or duplicated.

How do you measure data reliability?

Choose indicators for each dimension, such as the share of records passing validity checks, the share of required fields present, duplicate rate and the age of the newest record. Then set a target for each indicator on the datasets that matter most.

What is an SLO for data?

It applies the SRE idea of a service level objective: a target value or range for a service level measured by an indicator. For data, it might be the share of daily loads that finish with no null key by a set time.

Is Apache Griffin still maintained?

No. The Apache Griffin project page states that the project has retired, so use a maintained alternative for new work.

Which tools help with data reliability?

Deequ runs constraint checks as Spark jobs, OpenLineage standardises lineage events, and Great Expectations offers a Python library for data checks. Most teams combine several tools rather than relying on one.

Follow XenonStack on LinkedIn: XenonStack on LinkedIn for new guides on agentic AI, data and platform engineering.

Ready to put AI agents to work? Explore ElixirClaw, the agentic OS platform to build, orchestrate and govern AI agents.

Table of Contents

navdeep-singh-gill

Navdeep Singh Gill

Global CEO and Founder of XenonStack

LinkedIn profile

Navdeep Singh Gill is serving as Chief Executive Officer and Product Architect at XenonStack. He holds expertise in building SaaS Platform for Decentralised Big Data management and Governance, AI Marketplace for Operationalising and Scaling. His incredible experience in AI Technologies and Big Data Engineering thrills him to write about different use cases and its approach to solutions.

Get the latest articles in your inbox

Subscribe Now

Interested in solving your challenges with the XenonStack team?
Get Started

Tell us about you and your priorities so we can prepare the right conversation. It takes about 2 minutes.

Please Select your Industry
Banking
Fintech
Payment Providers
Wealth Management
Discrete Manufacturing
Semiconductor
Machinery Manufacturing / Automation
Appliances / Electrical / Electronics
Elevator Manufacturing
Defense & Space Manufacturing
Computers & Electronics / Industrial Machinery
Motor Vehicle Manufacturing
Food and Beverages
Distillery & Wines
Beverages
Shipping
Logistics
Mobility (EV / Public Transport)
Energy & Utilities
Hospitality
Digital Gaming Platforms
SportsTech with AI
Public Safety - Explosives
Public Safety - Firefighting
Public Safety - Surveillance
Public Safety - Others
Media Platforms
City Operations
Airlines & Aviation
Defense Warfare & Drones
Robotics Engineering
Drones Manufacturing
AI Labs for Colleges
AI MSP / Quantum / AGI Institutes
Retail Apparel and Fashion

Continue