Key takeaway
- Apache Flink 2.0 changes how streaming jobs are built. See what is new, what broke, and how to connect Kafka and Flink with the current connectors.
Apache Flink is an open-source engine for stateful computations over unbounded and bounded data streams, and Flink 2.0 is the first major release since Flink 1.0. Paired with Apache Kafka, which carries the events, it is a common way to run real-time pipelines for fraud checks, monitoring and analytics.
This guide is for data engineers and architects planning or upgrading streaming pipelines. It explains what Flink and Kafka each do, what changed in Flink 2.0, what the release removed, how to connect Kafka to Flink with the current connector API, and how AI agents change the picture. It builds on our guides to data processing, Apache Kafka architecture and Apache Flink managed services.
Why use stream processing instead of batch?
Batch processing works on a bounded set of data that has a defined start and end and can be fully ingested before computation, as the Flink architecture page puts it. Unbounded streams have no defined end, so events must be handled as they arrive. Stream processing suits cases where a delay changes the outcome, such as flagging a suspicious payment or reacting to a sensor reading. See also modern batch processing.
What do Kafka and Flink each do?
| Layer | Tool | Role |
|---|---|---|
| Ingestion | Apache NiFi, Apache StreamSets | Move and shape data from sources into the pipeline |
| Transport and storage | Apache Kafka, Apache Pulsar, NATS | Carry events between producers and consumers |
| Processing | Apache Flink, Apache Spark, Apache Beam, Apache Storm | Compute over the stream: filter, join, window, detect patterns |
| Delivery | Dashboards, alerts, reports and APIs | Give people and applications the result |
Kafka decouples producers from consumers, so many systems can read the same events. Flink consumes those events and keeps state. Task state stays in memory when possible and spills to on-disk structures when it grows. Flink periodically and asynchronously saves it to durable storage, and this checkpointing supports exactly-once state consistency after failures.
What changed in Apache Flink 2.0?
According to the Flink 2.0 release announcement, Apache Flink 2.0.0 was released on 24 March 2025. The main changes are below.
| Area | What the release says |
|---|---|
| Disaggregated state | State moves to distributed file systems instead of local disks, using an asynchronous execution model and the new ForSt state backend |
| Materialized tables | Support for schema and query updates, refresh jobs on YARN and Kubernetes, and integration with Apache Paimon |
| Adaptive batch execution | New adaptive broadcast join and automatic join skew optimisation |
| AI | Flink CDC 3.3 can call AI models inside Transform expressions |
| DataStream V2 API | New and experimental, and not recommended for production |
| Configuration and Java | config.yaml replaces flink-conf.yaml; Java 17 is the default and recommended version, Java 21 is supported |
What did Flink 2.0 remove, and what breaks?
The same announcement lists removed APIs: the DataSet API, the Scala DataStream and DataSet APIs, SourceFunction, SinkFunction, Sink V1, and TableSource and TableSink. Per-job deployment mode was removed in favour of application mode. Connectors built on removed APIs will not work on 2.x, and state compatibility between 1.x and 2.x is not guaranteed. Treat an upgrade as a migration project and rehearse it in a test cluster.
How do you connect Kafka and Flink today?
Older tutorials used FlinkKafkaConsumer and addSource. The Flink Kafka connector documentation says FlinkKafkaConsumer is deprecated and was to be removed with Flink 1.17 in favour of KafkaSource, and SourceFunction is gone in 2.0. Build jobs with the builder-based source instead.
KafkaSource<String> source = KafkaSource.<String>builder()
.setBootstrapServers(brokers)
.setTopics("input-topic")
.setGroupId("my-group")
.setStartingOffsets(OffsetsInitializer.earliest())
.setValueOnlyDeserializer(new SimpleStringSchema())
.build();
DataStream<String> stream =
env.fromSource(source, WatermarkStrategy.noWatermarks(), "Kafka Source");
For output, KafkaSink writes a stream to one or more Kafka topics. With the exactly-once delivery guarantee it writes messages in a Kafka transaction committed on a checkpoint, which requires a transactional ID prefix, and downstream readers must use the read committed isolation level. Both at-least-once and exactly-once require checkpointing to be enabled.
Kafka itself has changed. The Kafka 4.0 announcement calls it the first major release to operate entirely without Apache ZooKeeper, running in KRaft mode by default. A local setup that starts ZooKeeper first, as many older guides show, does not apply to Kafka 4.0.
Stateless or stateful: which do you need?
| Stateless | Stateful | |
|---|---|---|
| What it does | Evaluates each event on its own | Keeps context across events |
| Typical tasks | Filtering, simple transformations | Windows, joins, counts over time, anomaly detection |
| Example | Flag any single high-value payment | Flag a card used many times in a short window |
What is Flink CEP?
The FlinkCEP documentation describes it as a complex event processing library on top of Flink that detects event patterns in an unbounded stream. A pattern is a graph of simple patterns with conditions between them, and a time window can limit how long a match may take. In event time, the library buffers and sorts elements by timestamp and processes them when a watermark arrives, so late elements need handling. FlinkCEP is not part of the binary distribution, so add it to your project.
How do AI agents change stream processing?
Streams make the events available, and agents decide what to do with them. An agent can watch alerts from a Flink job, pull the related context, and propose or take an action within policy limits. That works only if the data is trustworthy and each action is traceable. ElixirData is XenonStack's Context OS for data and governance. Flink 2.0 also points the same way, with Flink CDC able to call AI models inside transformations.
What should you do first?
- Inventory every job, connector and state store, and mark anything built on removed APIs.
- Move Kafka reads to KafkaSource and writes to KafkaSink in a test environment.
- Decide which jobs need exactly-once delivery and enable checkpointing for them.
- Pick one use case, such as fraud rules, and measure its end-to-end delay.
To map one decision your agents make from streaming events, and the data and policy it depends on, book a Context Workshop. See what happens in the workshop.
Next Steps towards Data Streaming with Apache and Kafka
Talk to our experts about implementing data streaming with Apache Kafka to enhance real-time data processing. Apache Kafka helps organizations manage large-scale data flows, automate data pipelines, and improve efficiency across departments. Learn how it drives faster decision-making and better insights for decision-centric strategies.
Frequently asked questions
What is new in Apache Flink 2.0?
Released on 24 March 2025, Flink 2.0 adds disaggregated state management with the ForSt backend, updates to materialized tables, adaptive batch improvements and an experimental DataStream V2 API. It also removes several older APIs.
Is Flink 2.0 compatible with Flink 1.x jobs?
Not fully. Connectors built on removed APIs do not work on 2.x, and state compatibility between 1.x and 2.x is not guaranteed, so plan the upgrade as a migration.
Should I still use FlinkKafkaConsumer?
No. The Flink documentation says it is deprecated in favour of KafkaSource, and the SourceFunction interface it relied on was removed in Flink 2.0.
Does Kafka still need ZooKeeper?
Apache Kafka 4.0 is described as the first major release to operate entirely without ZooKeeper and runs in KRaft mode by default.
When do I need stateful processing?
Use it when a decision depends on earlier events, such as counts in a time window, joins between streams or pattern detection. Use stateless processing when each event can be judged on its own.
Ready to put AI agents to work? See an ElixirClaw demo: the agentic OS to build, orchestrate and govern AI agents across your enterprise systems.
Explore the XenonStack platform family: ElixirData · ElixirClaw · ElixirHub · NexaStack · MetaSecure · Neural AI · AgentGRC · AgentSRE · AgentAnalyst · AgentInstruct · AgentRAI · AgentScrum · AgentSearch · AgentForce · AI Aviator · SAIF Aviator · CohortIQ · XenonStack AI · XenonStack UAE