What is Apache Kafka?
Apache Kafka is an open-source event-streaming platform that lets systems publish and subscribe to continuous streams of events (records) — acting as a durable, high-throughput 'central nervous system' that many producers and consumers can share in real time.
At a glance
- A durable log of events that producers write to and consumers read from.
- Organized into topics, split into partitions for parallelism and ordering.
- Retains events so multiple consumers can read — and replay — the same stream.
- Built for very high throughput; portable across clouds and on-prem.
How it works
Producers publish events to named topics; consumers subscribe to those topics and process the events. Under the hood, each topic is split into partitions — an append-only, ordered log — which lets Kafka spread load across machines while preserving order within a partition.
Crucially, Kafka retains events for a configurable period (or effectively forever), so it's not a fire-and-forget queue. Many independent consumers can read the same stream at their own pace, and you can replay history — which is why Kafka is often used as a durable event log or system of record.
Why teams use it
Kafka decouples systems: instead of services calling each other directly, they publish and subscribe to shared streams. That makes real-time pipelines, event-driven architectures, and change-data-capture feeds far easier to build and scale. Its large ecosystem — Kafka Connect, Kafka Streams, ksqlDB, Schema Registry — covers ingestion, stream processing, and schema management.
Because it's open and portable, Kafka avoids cloud lock-in, though many teams run it via managed services (Amazon MSK, Confluent Cloud) to skip the operational burden.
Frequently asked questions
What is Apache Kafka used for?
Streaming events between systems in real time — powering event-driven architectures, real-time analytics, change-data-capture pipelines, and durable event logs that many services publish to and consume from.
Is Kafka a message queue?
It's more than a traditional queue. Kafka retains events and lets multiple consumers read (and replay) the same stream independently, so it acts as a durable, replayable event log rather than a fire-and-forget queue.
What are Kafka topics and partitions?
A topic is a named stream of events. Each topic is split into partitions — ordered, append-only logs — which let Kafka scale across machines while preserving event order within each partition.
Do I have to run Kafka myself?
No. You can self-manage it, or use managed Kafka such as Amazon MSK or Confluent Cloud to get the full Kafka API and ecosystem without operating the brokers yourself.
Putting Apache Kafka to work?
We help data & AI teams design and ship this in production. Tell us what you're building and we'll point you at the shortest path.
Talk to our team