Key takeaway
- Data ingestion explained: batch, streaming and change data capture compared, a layered architecture, retired Sqoop, Kafka without ZooKeeper, first steps.
Data ingestion is the process of moving data from where it is created, such as databases, SaaS applications, logs, devices and event streams, into a platform where it can be stored, governed and analysed. It comes in three main forms: scheduled batch loads, continuous event streaming, and change data capture (CDC), which streams row-level inserts, updates and deletes from a database as they are committed.
This guide is for data engineers, architects and analytics leads who need to choose an ingestion pattern for each source, understand what has changed since the Hadoop era, and prepare data for AI agents. It covers the three patterns, a layered architecture, retired and changed tools, and a short first-steps list.
What is data ingestion?
Data ingestion is the first stage of any data pipeline architecture. It collects data that arrives at different speeds and in different formats, including structured tables, semi-structured JSON or XML, and unstructured files or logs, and delivers it to a landing zone, data lake, warehouse or lakehouse table.
Ingestion overlaps with ETL and ELT but is not the same thing. Ingestion is about reliable capture and delivery: a complete, ordered, traceable copy of the data in the platform. Transformation and modelling usually happen afterwards, closer to the consumer, which keeps pipelines easier to replay and audit. For the wider integration picture, see our guide to data integration tools and methods.
How do batch, streaming and CDC ingestion compare?
Each pattern suits a different kind of source and freshness requirement, and most enterprise platforms run all three side by side.
| Pattern | What it is | When it fits | Typical tools |
|---|---|---|---|
| Batch | Data is extracted and loaded in chunks on a schedule or when a file arrives | Regulatory reporting, historical loads, SaaS exports | Managed connectors, Apache NiFi, cloud copy services |
| Streaming | Producers publish events to a durable log that consumers read continuously | Clickstreams, device telemetry, event-driven applications and agents | Apache Kafka, Amazon Kinesis, Azure Event Hubs, Google Cloud Pub/Sub |
| Change data capture | Changes are read from a database transaction log and streamed as they commit | Keeping analytics or caches in sync with operational databases | Debezium, AWS DMS, Google Cloud Datastream |
A useful rule: choose batch unless a decision genuinely depends on fresher data, and choose CDC over repeated full extracts when a relational source changes often. Our walkthrough of batch and real-time data ingestion with Apache NiFi shows both patterns in practice.
What does a data ingestion architecture look like?
A layered design gives each stage one job and keeps producers decoupled from consumers.
- Sources: operational databases, SaaS applications, files, APIs, logs and devices.
- Capture: connectors for SaaS and files, CDC connectors for databases, and producers that publish events.
- Transport: a durable, replayable log that decouples producers from consumers.
- Landing: object storage, ideally organised as open table formats rather than loose files.
- Processing: stream and batch engines that clean, join and model the data.
- Serving and governance: warehouses, query engines, catalogues, lineage, quality checks and access policies.
Storage format choices matter at file level too, and our note on data serialisation formats such as Avro and Parquet covers the basics. XenonStack's data engineering services cover building cloud-native pipelines.
What has changed in data ingestion since the Hadoop era?
Two changes affect architecture decisions directly, and both come from the Apache projects themselves.
| Item | What changed | What to do |
|---|---|---|
| Apache Sqoop | The Apache Attic page says it was retired in June 2021 and the move to the Attic was completed in July 2021 | Do not start new work on it; plan a migration for existing jobs |
| Kafka and ZooKeeper | The Kafka 4.0 announcement of 18 March 2025 calls it the first major release to operate entirely without Apache ZooKeeper, running in KRaft mode by default | Check whether your clusters still depend on ZooKeeper before upgrading |
Beyond those two, the direction is clear in practice: managed CDC and replication services are common across the major clouds, and many teams now land data in open table formats so several engines can read it. Verify the current status of any other tool on its own project page before you standardise on it. Our review of big data challenges and solutions shows how integration, security and cost recur as blockers.
What are the best practices for data ingestion?
- Keep raw data immutable so any output can be traced and replayed.
- Validate at the edge: check schema, nulls, ranges and volumes on arrival and quarantine failures instead of loading them silently.
- Agree contracts: write down schema, quality and service expectations between producers and consumers for the feeds that matter most.
- Design for retries: make loads idempotent and monitor lag and failed records.
- Match freshness to need: streaming every source adds cost and operational burden.
- Secure the path: encrypt in transit and at rest, use workload identities for connectors and mask personal data before it reaches broad-access zones.
When is real-time ingestion the wrong choice?
Streaming adds always-on infrastructure and harder failure handling. It is usually the wrong choice when the consumer is a daily report, when the source only offers file exports or rate-limited APIs, when the team cannot staff on-call support for brokers and stream jobs, or when data needs heavy reconciliation before anyone can trust it.
How do AI agents change data ingestion?
An agent acts on whatever the pipeline delivers, so a late file, a renamed column or an unlabelled source becomes a wrong decision, not only a wrong chart. That raises the bar on lineage, ownership and quality checks at ingestion time, and it means each feed an agent reads should have a named owner and a record of where it came from. ElixirData, XenonStack's Context OS, is built for governed data and decision context in this setting.
What should you do first?
- List each source with its owner, change rate and the decisions that depend on it.
- Assign batch, streaming or CDC to each source based on the freshness the decision needs.
- Choose a landing standard and a catalogue before building more pipelines.
- Find and plan the retirement of Sqoop jobs and ZooKeeper-dependent Kafka clusters.
- Add quality checks and contracts to the feeds that reach reports and agents.
To map one data-driven decision your agents should make, and the data and policy it depends on, book a Context Workshop. See what happens in the workshop.
Moving Forward with Data Ingestion
Ready to optimize your data ingestion process? Connect with our experts to explore the best strategies, tools, and technologies for streamlining your data flow, ensuring quality, and unlocking the full potential of your analytics capabilities.
Frequently asked questions
What is the difference between data ingestion and ETL?
Data ingestion focuses on capturing and delivering data reliably from sources into a platform. ETL and ELT add transformation and modelling, which many modern pipelines do after the data has landed.
What are the three main types of data ingestion?
Batch ingestion loads data on a schedule, streaming ingestion reads events continuously from a log such as Kafka, and change data capture streams row-level database changes as they are committed.
Is Apache Sqoop still supported?
No. The Apache Attic page says Sqoop was retired in June 2021 and the move to the Attic was completed in July 2021.
Does Apache Kafka still need ZooKeeper?
No. The Kafka 4.0 announcement calls it the first major release to operate entirely without ZooKeeper, running in KRaft mode by default.
When should I use change data capture instead of batch loads?
Use it when a relational source changes often and downstream systems need those changes without repeated full extracts or heavy queries on the production database.
Follow XenonStack on LinkedIn: XenonStack on LinkedIn for new guides on agentic AI, data and platform engineering.
Ready to put AI agents to work? Explore ElixirClaw, the agentic OS platform to build, orchestrate and govern AI agents.
Explore the XenonStack platform family: ElixirData · ElixirClaw · ElixirHub · NexaStack · MetaSecure · Neural AI · AgentGRC · AgentSRE · AgentAnalyst · AgentInstruct · AgentRAI · AgentScrum · AgentSearch · AgentForce · AI Aviator · SAIF Aviator · CohortIQ · XenonStack AI · XenonStack UAE