Skip to main content
XenonStack

Big Data Engineering

XenonStack Blog Data Ingestion: Batch, Streaming and CDC Guide

and | Published | Updated

Data Ingestion: Batch, Streaming and CDC Guide
9:36

Key takeaway

  • Data ingestion explained: batch, streaming and change data capture compared, a layered architecture, retired Sqoop, Kafka without ZooKeeper, first steps.

Data ingestion is the process of moving data from where it is created, such as databases, SaaS applications, logs, devices and event streams, into a platform where it can be stored, governed and analysed. It comes in three main forms: scheduled batch loads, continuous event streaming, and change data capture (CDC), which streams row-level inserts, updates and deletes from a database as they are committed.

This guide is for data engineers, architects and analytics leads who need to choose an ingestion pattern for each source, understand what has changed since the Hadoop era, and prepare data for AI agents. It covers the three patterns, a layered architecture, retired and changed tools, and a short first-steps list.

What is data ingestion?

Data ingestion is the first stage of any data pipeline architecture. It collects data that arrives at different speeds and in different formats, including structured tables, semi-structured JSON or XML, and unstructured files or logs, and delivers it to a landing zone, data lake, warehouse or lakehouse table.

Ingestion overlaps with ETL and ELT but is not the same thing. Ingestion is about reliable capture and delivery: a complete, ordered, traceable copy of the data in the platform. Transformation and modelling usually happen afterwards, closer to the consumer, which keeps pipelines easier to replay and audit. For the wider integration picture, see our guide to data integration tools and methods.

How do batch, streaming and CDC ingestion compare?

Each pattern suits a different kind of source and freshness requirement, and most enterprise platforms run all three side by side.

Two-column diagram. Source types: files and SaaS exports, operational databases, apps and devices, legacy Hadoop jobs. Patterns: batch load, change data capture, event streaming, migrate.
PatternWhat it isWhen it fitsTypical tools
BatchData is extracted and loaded in chunks on a schedule or when a file arrivesRegulatory reporting, historical loads, SaaS exportsManaged connectors, Apache NiFi, cloud copy services
StreamingProducers publish events to a durable log that consumers read continuouslyClickstreams, device telemetry, event-driven applications and agentsApache Kafka, Amazon Kinesis, Azure Event Hubs, Google Cloud Pub/Sub
Change data captureChanges are read from a database transaction log and streamed as they commitKeeping analytics or caches in sync with operational databasesDebezium, AWS DMS, Google Cloud Datastream

A useful rule: choose batch unless a decision genuinely depends on fresher data, and choose CDC over repeated full extracts when a relational source changes often. Our walkthrough of batch and real-time data ingestion with Apache NiFi shows both patterns in practice.

What does a data ingestion architecture look like?

A layered design gives each stage one job and keeps producers decoupled from consumers.

  • Sources: operational databases, SaaS applications, files, APIs, logs and devices.
  • Capture: connectors for SaaS and files, CDC connectors for databases, and producers that publish events.
  • Transport: a durable, replayable log that decouples producers from consumers.
  • Landing: object storage, ideally organised as open table formats rather than loose files.
  • Processing: stream and batch engines that clean, join and model the data.
  • Serving and governance: warehouses, query engines, catalogues, lineage, quality checks and access policies.

Storage format choices matter at file level too, and our note on data serialisation formats such as Avro and Parquet covers the basics. XenonStack's data engineering services cover building cloud-native pipelines.

What has changed in data ingestion since the Hadoop era?

Two changes affect architecture decisions directly, and both come from the Apache projects themselves.

ItemWhat changedWhat to do
Apache SqoopThe Apache Attic page says it was retired in June 2021 and the move to the Attic was completed in July 2021Do not start new work on it; plan a migration for existing jobs
Kafka and ZooKeeperThe Kafka 4.0 announcement of 18 March 2025 calls it the first major release to operate entirely without Apache ZooKeeper, running in KRaft mode by defaultCheck whether your clusters still depend on ZooKeeper before upgrading

Beyond those two, the direction is clear in practice: managed CDC and replication services are common across the major clouds, and many teams now land data in open table formats so several engines can read it. Verify the current status of any other tool on its own project page before you standardise on it. Our review of big data challenges and solutions shows how integration, security and cost recur as blockers.

What are the best practices for data ingestion?

  • Keep raw data immutable so any output can be traced and replayed.
  • Validate at the edge: check schema, nulls, ranges and volumes on arrival and quarantine failures instead of loading them silently.
  • Agree contracts: write down schema, quality and service expectations between producers and consumers for the feeds that matter most.
  • Design for retries: make loads idempotent and monitor lag and failed records.
  • Match freshness to need: streaming every source adds cost and operational burden.
  • Secure the path: encrypt in transit and at rest, use workload identities for connectors and mask personal data before it reaches broad-access zones.

When is real-time ingestion the wrong choice?

Streaming adds always-on infrastructure and harder failure handling. It is usually the wrong choice when the consumer is a daily report, when the source only offers file exports or rate-limited APIs, when the team cannot staff on-call support for brokers and stream jobs, or when data needs heavy reconciliation before anyone can trust it.

How do AI agents change data ingestion?

An agent acts on whatever the pipeline delivers, so a late file, a renamed column or an unlabelled source becomes a wrong decision, not only a wrong chart. That raises the bar on lineage, ownership and quality checks at ingestion time, and it means each feed an agent reads should have a named owner and a record of where it came from. ElixirData, XenonStack's Context OS, is built for governed data and decision context in this setting.

What should you do first?

  1. List each source with its owner, change rate and the decisions that depend on it.
  2. Assign batch, streaming or CDC to each source based on the freshness the decision needs.
  3. Choose a landing standard and a catalogue before building more pipelines.
  4. Find and plan the retirement of Sqoop jobs and ZooKeeper-dependent Kafka clusters.
  5. Add quality checks and contracts to the feeds that reach reports and agents.

To map one data-driven decision your agents should make, and the data and policy it depends on, book a Context Workshop. See what happens in the workshop.

Moving Forward with Data Ingestion

Ready to optimize your data ingestion process? Connect with our experts to explore the best strategies, tools, and technologies for streamlining your data flow, ensuring quality, and unlocking the full potential of your analytics capabilities.

Frequently asked questions

What is the difference between data ingestion and ETL?

Data ingestion focuses on capturing and delivering data reliably from sources into a platform. ETL and ELT add transformation and modelling, which many modern pipelines do after the data has landed.

What are the three main types of data ingestion?

Batch ingestion loads data on a schedule, streaming ingestion reads events continuously from a log such as Kafka, and change data capture streams row-level database changes as they are committed.

Is Apache Sqoop still supported?

No. The Apache Attic page says Sqoop was retired in June 2021 and the move to the Attic was completed in July 2021.

Does Apache Kafka still need ZooKeeper?

No. The Kafka 4.0 announcement calls it the first major release to operate entirely without ZooKeeper, running in KRaft mode by default.

When should I use change data capture instead of batch loads?

Use it when a relational source changes often and downstream systems need those changes without repeated full extracts or heavy queries on the production database.

Follow XenonStack on LinkedIn: XenonStack on LinkedIn for new guides on agentic AI, data and platform engineering.

Ready to put AI agents to work? Explore ElixirClaw, the agentic OS platform to build, orchestrate and govern AI agents.

Table of Contents

navdeep-singh-gill

Navdeep Singh Gill

Global CEO and Founder of XenonStack

LinkedIn profile

Navdeep Singh Gill is serving as Chief Executive Officer and Product Architect at XenonStack. He holds expertise in building SaaS Platform for Decentralised Big Data management and Governance, AI Marketplace for Operationalising and Scaling. His incredible experience in AI Technologies and Big Data Engineering thrills him to write about different use cases and its approach to solutions.

Get the latest articles in your inbox

Subscribe Now

Interested in solving your challenges with the XenonStack team?
Get Started

Tell us about you and your priorities so we can prepare the right conversation. It takes about 2 minutes.

Please Select your Industry
Banking
Fintech
Payment Providers
Wealth Management
Discrete Manufacturing
Semiconductor
Machinery Manufacturing / Automation
Appliances / Electrical / Electronics
Elevator Manufacturing
Defense & Space Manufacturing
Computers & Electronics / Industrial Machinery
Motor Vehicle Manufacturing
Food and Beverages
Distillery & Wines
Beverages
Shipping
Logistics
Mobility (EV / Public Transport)
Energy & Utilities
Hospitality
Digital Gaming Platforms
SportsTech with AI
Public Safety - Explosives
Public Safety - Firefighting
Public Safety - Surveillance
Public Safety - Others
Media Platforms
City Operations
Airlines & Aviation
Defense Warfare & Drones
Robotics Engineering
Drones Manufacturing
AI Labs for Colleges
AI MSP / Quantum / AGI Institutes
Retail Apparel and Fashion

Continue