Skip to main content
XenonStack

Enterprise Data Management

XenonStack Insights Apache Flink 2.0 and Kafka: Real-Time Processing

| Published | Updated

Apache Flink 2.0 and Kafka: Real-Time Processing
9:30
Apache Flink 2.0 and Kafka: sources feed Kafka event streams, Flink processes them, and results drive decisions and alerts

Key takeaway

  • Apache Flink 2.0 changes how streaming jobs are built. See what is new, what broke, and how to connect Kafka and Flink with the current connectors.

Apache Flink is an open-source engine for stateful computations over unbounded and bounded data streams, and Flink 2.0 is the first major release since Flink 1.0. Paired with Apache Kafka, which carries the events, it is a common way to run real-time pipelines for fraud checks, monitoring and analytics.

This guide is for data engineers and architects planning or upgrading streaming pipelines. It explains what Flink and Kafka each do, what changed in Flink 2.0, what the release removed, how to connect Kafka to Flink with the current connector API, and how AI agents change the picture. It builds on our guides to data processing, Apache Kafka architecture and Apache Flink managed services.

Why use stream processing instead of batch?

Batch processing works on a bounded set of data that has a defined start and end and can be fully ingested before computation, as the Flink architecture page puts it. Unbounded streams have no defined end, so events must be handled as they arrive. Stream processing suits cases where a delay changes the outcome, such as flagging a suspicious payment or reacting to a sensor reading. See also modern batch processing.

What do Kafka and Flink each do?

LayerToolRole
IngestionApache NiFi, Apache StreamSetsMove and shape data from sources into the pipeline
Transport and storageApache Kafka, Apache Pulsar, NATSCarry events between producers and consumers
ProcessingApache Flink, Apache Spark, Apache Beam, Apache StormCompute over the stream: filter, join, window, detect patterns
DeliveryDashboards, alerts, reports and APIsGive people and applications the result

Kafka decouples producers from consumers, so many systems can read the same events. Flink consumes those events and keeps state. Task state stays in memory when possible and spills to on-disk structures when it grows. Flink periodically and asynchronously saves it to durable storage, and this checkpointing supports exactly-once state consistency after failures.

What changed in Apache Flink 2.0?

According to the Flink 2.0 release announcement, Apache Flink 2.0.0 was released on 24 March 2025. The main changes are below.

Two-column diagram. Kafka side: producers, topics, KRaft mode, KafkaSink. Flink side: KafkaSource, stateful jobs, checkpoints, results.
AreaWhat the release says
Disaggregated stateState moves to distributed file systems instead of local disks, using an asynchronous execution model and the new ForSt state backend
Materialized tablesSupport for schema and query updates, refresh jobs on YARN and Kubernetes, and integration with Apache Paimon
Adaptive batch executionNew adaptive broadcast join and automatic join skew optimisation
AIFlink CDC 3.3 can call AI models inside Transform expressions
DataStream V2 APINew and experimental, and not recommended for production
Configuration and Javaconfig.yaml replaces flink-conf.yaml; Java 17 is the default and recommended version, Java 21 is supported

What did Flink 2.0 remove, and what breaks?

The same announcement lists removed APIs: the DataSet API, the Scala DataStream and DataSet APIs, SourceFunction, SinkFunction, Sink V1, and TableSource and TableSink. Per-job deployment mode was removed in favour of application mode. Connectors built on removed APIs will not work on 2.x, and state compatibility between 1.x and 2.x is not guaranteed. Treat an upgrade as a migration project and rehearse it in a test cluster.

How do you connect Kafka and Flink today?

Older tutorials used FlinkKafkaConsumer and addSource. The Flink Kafka connector documentation says FlinkKafkaConsumer is deprecated and was to be removed with Flink 1.17 in favour of KafkaSource, and SourceFunction is gone in 2.0. Build jobs with the builder-based source instead.

KafkaSource<String> source = KafkaSource.<String>builder()
    .setBootstrapServers(brokers)
    .setTopics("input-topic")
    .setGroupId("my-group")
    .setStartingOffsets(OffsetsInitializer.earliest())
    .setValueOnlyDeserializer(new SimpleStringSchema())
    .build();

DataStream<String> stream =
    env.fromSource(source, WatermarkStrategy.noWatermarks(), "Kafka Source");

For output, KafkaSink writes a stream to one or more Kafka topics. With the exactly-once delivery guarantee it writes messages in a Kafka transaction committed on a checkpoint, which requires a transactional ID prefix, and downstream readers must use the read committed isolation level. Both at-least-once and exactly-once require checkpointing to be enabled.

Kafka itself has changed. The Kafka 4.0 announcement calls it the first major release to operate entirely without Apache ZooKeeper, running in KRaft mode by default. A local setup that starts ZooKeeper first, as many older guides show, does not apply to Kafka 4.0.

Stateless or stateful: which do you need?

StatelessStateful
What it doesEvaluates each event on its ownKeeps context across events
Typical tasksFiltering, simple transformationsWindows, joins, counts over time, anomaly detection
ExampleFlag any single high-value paymentFlag a card used many times in a short window

What is Flink CEP?

The FlinkCEP documentation describes it as a complex event processing library on top of Flink that detects event patterns in an unbounded stream. A pattern is a graph of simple patterns with conditions between them, and a time window can limit how long a match may take. In event time, the library buffers and sorts elements by timestamp and processes them when a watermark arrives, so late elements need handling. FlinkCEP is not part of the binary distribution, so add it to your project.

How do AI agents change stream processing?

Streams make the events available, and agents decide what to do with them. An agent can watch alerts from a Flink job, pull the related context, and propose or take an action within policy limits. That works only if the data is trustworthy and each action is traceable. ElixirData is XenonStack's Context OS for data and governance. Flink 2.0 also points the same way, with Flink CDC able to call AI models inside transformations.

What should you do first?

  1. Inventory every job, connector and state store, and mark anything built on removed APIs.
  2. Move Kafka reads to KafkaSource and writes to KafkaSink in a test environment.
  3. Decide which jobs need exactly-once delivery and enable checkpointing for them.
  4. Pick one use case, such as fraud rules, and measure its end-to-end delay.

To map one decision your agents make from streaming events, and the data and policy it depends on, book a Context Workshop. See what happens in the workshop.

Next Steps towards Data Streaming with Apache and Kafka

Talk to our experts about implementing data streaming with Apache Kafka to enhance real-time data processing. Apache Kafka helps organizations manage large-scale data flows, automate data pipelines, and improve efficiency across departments. Learn how it drives faster decision-making and better insights for decision-centric strategies.

Frequently asked questions

What is new in Apache Flink 2.0?

Released on 24 March 2025, Flink 2.0 adds disaggregated state management with the ForSt backend, updates to materialized tables, adaptive batch improvements and an experimental DataStream V2 API. It also removes several older APIs.

Is Flink 2.0 compatible with Flink 1.x jobs?

Not fully. Connectors built on removed APIs do not work on 2.x, and state compatibility between 1.x and 2.x is not guaranteed, so plan the upgrade as a migration.

Should I still use FlinkKafkaConsumer?

No. The Flink documentation says it is deprecated in favour of KafkaSource, and the SourceFunction interface it relied on was removed in Flink 2.0.

Does Kafka still need ZooKeeper?

Apache Kafka 4.0 is described as the first major release to operate entirely without ZooKeeper and runs in KRaft mode by default.

When do I need stateful processing?

Use it when a decision depends on earlier events, such as counts in a time window, joins between streams or pattern detection. Use stateless processing when each event can be judged on its own.

Ready to put AI agents to work? See an ElixirClaw demo: the agentic OS to build, orchestrate and govern AI agents across your enterprise systems.

Table of Contents

Get the latest articles in your inbox

Subscribe Now

Interested in solving your challenges with the XenonStack team?
Get Started

Tell us about you and your priorities so we can prepare the right conversation. It takes about 2 minutes.

Please Select your Industry
Banking
Fintech
Payment Providers
Wealth Management
Discrete Manufacturing
Semiconductor
Machinery Manufacturing / Automation
Appliances / Electrical / Electronics
Elevator Manufacturing
Defense & Space Manufacturing
Computers & Electronics / Industrial Machinery
Motor Vehicle Manufacturing
Food and Beverages
Distillery & Wines
Beverages
Shipping
Logistics
Mobility (EV / Public Transport)
Energy & Utilities
Hospitality
Digital Gaming Platforms
SportsTech with AI
Public Safety - Explosives
Public Safety - Firefighting
Public Safety - Surveillance
Public Safety - Others
Media Platforms
City Operations
Airlines & Aviation
Defense Warfare & Drones
Robotics Engineering
Drones Manufacturing
AI Labs for Colleges
AI MSP / Quantum / AGI Institutes
Retail Apparel and Fashion

Continue