Skip to main content
XenonStack

Enterprise Data Management

XenonStack Insights Apache Gobblin: Unified Data Ingestion Guide

| Published | Updated

Apache Gobblin: sources split into work units, extractors and converters, writer and quality checks, publisher makes data final

Key takeaway

  • Apache Gobblin is a distributed data integration framework. See its six components, job state, deployment modes and how it compares with Marmaray.

Apache Gobblin is a distributed data integration framework for ingesting, replicating, organising and managing the lifecycle of data, in both batch and streaming forms. The Apache Gobblin homepage describes it in those terms, and a January 2021 news item there records that it became a top-level Apache project.

This guide is for data engineers who need one framework to bring data from many sources into a lake or warehouse. It covers the six component interfaces, how a job runs, the deployment modes, how quality checks and publishing work, and where Gobblin sits against other options. For the concept behind it, start with our guide to data ingestion.

What is Apache Gobblin and what does it do?

Per its architecture documentation, Gobblin is built for extensibility: its constructs are pluggable through job configuration, while the runtime handles scheduling, retries, resource management, state, quality checks and publishing. In practice that means a team writes or configures the source-specific parts and reuses the common ingestion machinery, so many sources can run in the same execution framework instead of as many separate scripts. The same idea sits behind Uber Marmaray, which was inspired by Gobblin.

What are the components of Apache Gobblin?

The architecture page lists the following constructs.

ComponentWhat it does
SourcePartitions the ingestion into work units and creates an extractor for each one; modelled after Hadoop's InputFormat
ExtractorConnects to the data source and reads records one at a time
ConverterTransforms schemas and records and can be chained; the core construct for data transformation
Quality CheckerRuns at row level or task level; optional checkers are informational only, while mandatory checkers determine whether the check passes
WriterWrites records to a sink; the default writes Avro to HDFS
PublisherMakes extracted data final; the default moves each task's output into the job's final output directory

A work unit is a unit of work that also acts as a container for job configuration at runtime. Normally one task is created per work unit, although a multi work unit can wrap several.

How does a Gobblin job run?

A job ingests data from a source into a sink and may split into many tasks. According to the documentation, job state is persisted after each run so that incremental extraction can resume: the next run loads the previous job and task states to recover details such as watermarks, so it knows where to start. By default that state is stored as Hadoop SequenceFiles, one per job run. Data is published only if the commit policy allows it, and a cancelled job does not persist state or publish data.

Two-column diagram. Read and transform: source, extractor, converter, quality checker. Check and publish: writer, publisher, commit policy, job state.

This is why Gobblin suits incremental loads. A failed or cancelled run does not advance the position, so the next run can try again from the last good point. Related guidance is in our notes on building a data ingestion platform with Apache NiFi and real-time data integration.

How can you deploy Gobblin?

ModeHow it runsSource
StandaloneSingle node, with tasks running in a thread pool dedicated to each jobArchitecture documentation
Hadoop MapReduceCluster mode, with tasks running inside mappers that act purely as containersArchitecture documentation
Cluster and cloudThe homepage lists cluster or YARN and cloud execution among the supported modesGobblin homepage

The architecture page is older than the homepage: it says native YARN deployment was still being worked on, while the homepage now lists YARN and cloud modes. Check the current deployment guide on the project site before planning an installation. As a reference point, the homepage mentions Apache Gobblin 0.17.0, released in August 2023.

How does Gobblin compare with other ingestion tools?

QuestionGobblinMarmaray
Main purposeData integration: ingestion, replication, organisation and lifecycle managementIngestion into Hadoop and dispersal out of it
TransformationConverters transform schemas and recordsNo transformation capabilities, as Uber described it in 2018
StatusTop-level Apache projectRepository archived on 5 January 2024, per its GitHub page

For file formats on the sink side, see our guides to Apache Hudi and data serialisation in Hadoop. The extract, transform and load pattern that Gobblin automates is explained in our ETL glossary entry.

How do AI agents change data ingestion?

An ingestion framework decides which data is available, in what shape and how fresh it is. When agents rely on that data, errors in extraction or conversion become errors in decisions, so quality checks and run state matter more. Agents can help data teams by drafting source configurations, reading job state and quality results, and summarising failures, while people approve schema and sink changes. For the governed data layer behind such work, see ElixirData, XenonStack's Context OS.

What should you do first?

  1. List the sources you ingest today, with owners, formats and refresh needs.
  2. Run one source end to end in standalone mode and read the job and task state it writes.
  3. Mark which quality checks must be mandatory and which can be informational.
  4. Decide the commit policy for failed tasks before scaling out.
  5. Confirm the deployment mode against the current Gobblin documentation.

To map one decision your agents make, and the data and policy it depends on, book a Context Workshop. See what happens in the workshop.

Frequently asked questions

What is Apache Gobblin?

Apache Gobblin is a distributed data integration framework that simplifies data ingestion, replication, organisation and lifecycle management for batch and streaming data.

What are the main components of Gobblin?

Source, extractor, converter, quality checker, writer and publisher. The source splits work into work units, the extractor reads records, converters transform them, quality checkers test them, the writer writes to a sink and the publisher makes the data final.

How does Gobblin resume incremental loads?

It persists job and task state after each run, and the next run loads that state to recover details such as watermarks so it knows where to start.

What deployment modes does Gobblin support?

The architecture documentation describes standalone and Hadoop MapReduce modes, and the project homepage lists standalone, MapReduce, cluster or YARN, and cloud modes.

Is Apache Gobblin a top-level Apache project?

Yes. A January 2021 news item on the project homepage says Apache Gobblin became a top-level project.

Ready to put AI agents to work? See an ElixirClaw demo: the agentic OS to build, orchestrate and govern AI agents across your enterprise systems.

Table of Contents

Get the latest articles in your inbox

Subscribe Now

Interested in solving your challenges with the XenonStack team?
Get Started

Tell us about you and your priorities so we can prepare the right conversation. It takes about 2 minutes.

Please Select your Industry
Banking
Fintech
Payment Providers
Wealth Management
Discrete Manufacturing
Semiconductor
Machinery Manufacturing / Automation
Appliances / Electrical / Electronics
Elevator Manufacturing
Defense & Space Manufacturing
Computers & Electronics / Industrial Machinery
Motor Vehicle Manufacturing
Food and Beverages
Distillery & Wines
Beverages
Shipping
Logistics
Mobility (EV / Public Transport)
Energy & Utilities
Hospitality
Digital Gaming Platforms
SportsTech with AI
Public Safety - Explosives
Public Safety - Firefighting
Public Safety - Surveillance
Public Safety - Others
Media Platforms
City Operations
Airlines & Aviation
Defense Warfare & Drones
Robotics Engineering
Drones Manufacturing
AI Labs for Colleges
AI MSP / Quantum / AGI Institutes
Retail Apparel and Fashion

Continue