Real-Time Data Infrastructure: How Apache Kafka, Flink

The Batch-to-Streaming Paradigm Shift

For decades, enterprise data processing followed a batch paradigm: collect data throughout the day, load it into a warehouse overnight, and analyze it the next morning. ETL (Extract, Transform, Load) pipelines ran on schedules — hourly, daily, weekly — feeding data warehouses that analysts queried during business hours. This model worked when business decisions operated on daily or weekly cycles and when the cost of real-time processing was prohibitive.

That model is breaking. Modern businesses require real-time fraud detection (Visa processes up to 83,000 transaction messages per second and analyzes over 500 data points per transaction to flag fraud in milliseconds), instant recommendation engines (Netflix personalizes its interface for 325+ million subscribers based on current session behavior), real-time pricing (Uber’s surge pricing adjusts every minute based on supply-demand dynamics), and IoT sensor processing (a modern factory generates terabytes of telemetry daily from thousands of sensors that must be analyzed immediately). The question is no longer “do we need real-time?” but “which workloads justify the complexity of real-time, and what infrastructure supports it?”

The answer, for an increasing portion of the technology industry, is stream processing — and the technology stack that has emerged to support it is anchored by Apache Kafka, Apache Flink, and a constellation of supporting tools. This infrastructure now processes trillions of events daily across more than 150,000 organizations, representing a fundamental shift in how data moves through enterprises. The event stream processing market reached $1.21 billion in 2025 and is projected to grow at 16% annually, reaching $2.94 billion by 2030.

Apache Kafka: The Central Nervous System

Apache Kafka, originally developed at LinkedIn in 2010 and open-sourced in 2011, has become the de facto standard for high-throughput distributed event streaming. Kafka’s core abstraction is elegant: a distributed append-only log organized into topics, where producers write events and consumers read them. Events are persisted durably and can be replayed, enabling decoupled microservices, event sourcing patterns, and real-time analytics pipelines.

The numbers are staggering. LinkedIn processes 7+ trillion messages per day through Kafka across over 100 clusters with more than 4,000 brokers, handling 100,000+ topics and 7 million partitions. Apple operates one of the largest Kafka deployments globally, handling multiple petabytes of data daily. Netflix, Uber, Airbnb, Goldman Sachs, and the New York Times all run Kafka at scale. The 2025 Confluent Data Streaming Report, surveying 4,175 IT leaders, found that over 80% of Fortune 500 companies use Kafka or Kafka-compatible systems, with 86% of respondents identifying data streaming as a top strategic priority. Kafka’s strength is its combination of high throughput (millions of events per second per cluster), low latency (single-digit millisecond end-to-end), durability (replicated across brokers with configurable consistency), and operational maturity (15+ years of production hardening).

The biggest architectural change in Kafka’s history arrived in March 2025 with the release of Apache Kafka 4.0. This version fully removed Apache ZooKeeper — the external coordination service that Kafka had depended on since its inception — replacing it entirely with KRaft (Kafka Raft), a consensus mechanism built into Kafka itself. KRaft manages metadata using Kafka’s own log, eliminating the operational complexity of maintaining a separate ZooKeeper ensemble. This simplifies deployment, enhances scalability, and streamlines cluster administration. Kafka 3.9 was the last release to support ZooKeeper; organizations running older versions must migrate to KRaft mode before upgrading to 4.0.

The IBM-Confluent Megadeal and the Commercial Ecosystem

Confluent, founded in 2014 by Kafka’s original creators (Jay Kreps, Neha Narkhede, Jun Rao), has been the commercial steward of the Kafka ecosystem. The company offers Confluent Cloud (fully managed Kafka as a service), Confluent Platform (self-managed enterprise distribution), and complementary products including Schema Registry, ksqlDB (SQL interface for Kafka streams), and connectors to 200+ data systems. Confluent’s trailing twelve-month revenue exceeded $1.1 billion as of September 2025, growing over 21% year-over-year, demonstrating the commercial viability of open-source streaming infrastructure.

On December 8, 2025, IBM announced a definitive agreement to acquire Confluent for $31 per share in cash — an enterprise value of approximately $11 billion. The deal, expected to close by mid-2026, is the largest acquisition in the data streaming ecosystem to date. IBM CEO Arvind Krishna stated that “real-time data is incredibly important to how an enterprise functions,” framing the acquisition as creating a unified platform combining IBM’s watsonx AI capabilities with Confluent’s real-time data streaming. Confluent will continue to operate as a distinct brand within IBM.

The IBM-Confluent deal signals a broader industry conviction: real-time data infrastructure is no longer a niche concern — it is a core enterprise requirement, particularly as organizations build AI systems that depend on fresh, contextual data. The 2025 Confluent Data Streaming Report found that 87% of IT leaders expect data streaming platforms to increasingly feed AI systems with real-time, contextual, and trustworthy data.

Stream Processing: Flink 2.0, Spark, and the SQL-Over-Streams Movement

Kafka handles data transport — getting events from point A to point B reliably and at scale. But processing those events in real-time — aggregating, filtering, joining, windowing, and transforming streams — requires a stream processing engine. This is where Apache Flink, Apache Spark Structured Streaming, and newer entrants like Materialize and RisingWave come in.

Apache Flink reached a major milestone in March 2025 with the release of Flink 2.0.0 — its first major version since Flink 1.0 launched nine years earlier. The release introduced disaggregated state management for efficient cloud-native resource utilization, materialized tables that let users focus on business logic without understanding streaming internals, and deep integration with Apache Paimon for streaming lakehouse architectures. Over 165 contributors completed 25 Flink Improvement Proposals for the 2.0 release.

The pace accelerated through 2025. Flink 2.1.0 (July 2025) added AI model management and ML_PREDICT, a table-valued function enabling real-time AI model inference directly within Flink SQL. Flink 2.2.0 (December 2025) extended this with large language model inference and VECTOR_SEARCH for real-time vector similarity lookups — capabilities that position Flink as not just a stream processor but a real-time Data + AI platform. In February 2026, the Flink community released Flink Agents 0.2.0, a new sub-project for building event-driven AI agents directly on Flink’s streaming runtime. Flink provides exactly-once processing semantics, event-time processing (handling out-of-order events based on when they occurred, not when they arrived), and sophisticated windowing operations. Alibaba, the largest Flink user, runs its fork called Blink to process billions of events per second during peak shopping events like Singles’ Day. Uber uses Flink for real-time marketplace analytics, and Stripe uses it for real-time fraud detection scoring.

Apache Spark Structured Streaming, part of the broader Spark ecosystem, offers stream processing with the advantage of a unified batch-and-streaming API. Teams already using Spark for batch analytics can extend to streaming without learning a new framework. Databricks, the commercial Spark company now valued at $134 billion after completing a $5 billion funding round in early 2026, has heavily invested in making Structured Streaming production-ready. However, Spark Structured Streaming operates on micro-batches (processing data in small intervals, typically 100ms-1s), making it slightly higher-latency than Flink’s true event-at-a-time processing. For use cases where sub-second latency is critical — payment fraud, real-time bidding — Flink is generally preferred.

Materialize, backed by over $100 million in venture funding, takes a different approach: it provides incrementally maintained materialized views over streaming data, accessible via standard PostgreSQL-compatible SQL. Rather than writing Flink jobs in Java or Scala, developers write SQL queries that automatically stay up to date as new data arrives. RisingWave, an open-source alternative with over 8,400 GitHub stars, offers a similar SQL-over-streams model. These tools represent a bet that stream processing can be democratized beyond the specialist engineers who currently build and maintain Flink pipelines.

When Streaming Is Necessary vs. Over-Engineered

The streaming infrastructure ecosystem is powerful, but it carries significant operational complexity. A production Kafka cluster requires careful partition management, consumer group coordination, schema evolution strategy, and monitoring. Flink jobs require checkpointing, state management, and handling of late-arriving data. The engineering teams that build and maintain streaming pipelines need specialized skills — senior data engineers with Kafka and Flink expertise command $150,000-350,000+ in total compensation at major US technology companies, with base salaries typically ranging from $100,000 to $170,000 and significant equity components at top-tier firms.

The honest assessment: most applications do not need real-time streaming. A batch pipeline that processes data every 5 minutes serves 90% of analytics use cases adequately. A webhook-based integration that processes events within seconds is sufficient for most operational triggers. The canonical use cases where streaming infrastructure is genuinely justified include: financial fraud detection (millisecond decisions on transaction approval), real-time personalization at scale (millions of concurrent users), IoT telemetry processing (thousands of sensors generating continuous data), and operational monitoring (detecting infrastructure anomalies in real-time).

The anti-pattern — and it is common — is adopting Kafka and Flink because they are “modern” when a PostgreSQL database with a cron job would suffice. Martin Kleppmann, author of “Designing Data-Intensive Applications,” has noted that the complexity budget of an organization is finite, and spending it on streaming infrastructure that delivers marginal value over batch processing is a net negative. The decision framework should be: What is the business cost of latency? If processing data in 5 minutes instead of 5 seconds has no measurable business impact, batch wins on simplicity, cost, and operational burden.

The Operational Reality and Future Direction

Running streaming infrastructure at scale is an operational discipline. Kafka clusters require monitoring of consumer lag (how far behind a consumer is from the latest data), broker health (disk utilization, ISR — in-sync replicas — count), and partition balance. Flink applications require monitoring of checkpoint durations, backpressure (when downstream operators cannot keep up with upstream throughput), and state size growth. The observability stack for streaming — typically Prometheus, Grafana, and Kafka-specific tools like Burrow or Conduktor — is itself a non-trivial infrastructure investment.

Managed services have reduced but not eliminated this operational burden. Confluent Cloud manages Kafka infrastructure, handling cluster provisioning, scaling, and patching. Amazon MSK (Managed Streaming for Apache Kafka) offers a similar service within the AWS ecosystem. For Flink, Amazon Managed Service for Apache Flink and Alibaba Cloud’s Realtime Compute provide managed execution environments. These services handle the infrastructure layer but still require application-level expertise — designing topics and schemas, writing processing logic, tuning consumer configurations, and handling data quality issues.

The trajectory of the streaming ecosystem points toward two converging themes: simplification and AI integration. Confluent’s vision of a “data streaming platform” — where Kafka is not just a message bus but a complete data integration and processing platform with built-in governance, lineage, and SQL interfaces — is now being supercharged by IBM’s enterprise reach and AI ambitions. The rise of SQL-over-streams tools (Materialize, RisingWave, ksqlDB) suggests that stream processing will become accessible to analysts, not just engineers. And the integration of streaming with AI/ML pipelines has moved from emerging frontier to active deployment: Flink 2.x now supports native AI model inference and vector search, real-time feature stores like Tecton and Feast are built on top of Kafka and Flink to power machine learning models, and the IBM-Confluent deal is explicitly framed around feeding generative AI systems with real-time enterprise data. The era of real-time data is not coming — for the organizations that need it, it has already arrived.

Frequently Asked Questions

Sources & Further Reading

• Apache Kafka 4.0: Default KRaft, Queues, Faster Rebalances — Confluent
• Apache Flink 2.0.0: A New Era of Real-Time Data Processing — Apache Flink
• IBM to Acquire Confluent to Create Smart Data Platform for Enterprise Generative AI — IBM Newsroom
• How LinkedIn Customizes Apache Kafka for 7 Trillion Messages Per Day — LinkedIn Engineering
• 2025 Data Streaming Report — Confluent
• Apache Flink 2.2.0: Advancing Real-Time Data + AI — Apache Flink
• Databricks Surpasses $5.4B Revenue Run-Rate, Growing 65% YoY — Databricks
• Top Trends for Data Streaming with Apache Kafka and Flink in 2026 — Kai Waehner
• Inside Visa’s Engine of Global Commerce — Visa
• Designing Data-Intensive Applications — Martin Kleppmann, O’Reilly