Apache Kafka explained: how it works, the hard parts, and why teams pick it over Azure, AWS and Google
Apache Kafka is the backbone of modern real-time data — but should you use it, and over what? This explainer walks through the Kafka mental model (topics, partitions, offsets, brokers, consumer groups and replay), the genuine challenges of running it at scale, and a clear comparison against Azure Stream Analytics, AWS Kinesis, Google Pub/Sub and managed Kafka so you know when Kafka is the right call.
Explore another walkthrough
Demo 01
dbt + Medallion Architecture
Watch a messy orders.csv get progressively refined by dbt models — with lineage, tests, macros and incremental builds.
Demo 02
SQLMesh vs dbt
A standalone look at where SQLMesh diverges from dbt — stateful incrementals, built-in audits, column-level lineage and virtual environments — and the situations where it's the better choice.
Demo 03
Master Data Management
Watch the same customer — scattered across CRM, ERP, e-commerce and support with conflicting data — get standardized, matched, merged into a Golden Record, and synced back to every system.
Demo 04
Change Data Capture (CDC)
Watch inserts, updates and deletes from an operational orders table get captured and streamed into Bronze → Silver → Gold — with a side-by-side of log-based, query-based and trigger-based capture.
Demo 05
Banking
Submit a sample loan form and watch a workflow fire node by node — webhook, credit check, risk scoring, decision branch, core-banking update and notifications — replacing days of manual hand-offs.
Demo 06
AI Support
Submit a real support message and watch a live workflow use AI to classify it, set priority, route it to the right team and draft a reply — actually running on the backend, not a simulation.
Demo 07
Finance / AP
Step through a workflow that runs a 3-way match of a vendor invoice against the purchase order and goods receipt — pausing for a receiving check and approval before posting to the ERP.
Demo 09
Azure AI Agent Blueprint
An interactive, animated reference architecture: watch a user request flow through the Application, Context and Orchestration layers — gateway, Container Apps, RAG, memory, Azure AI Foundry and model fine-tuning.
Demo 10
AWS AI Agent Stack
An interactive, animated AWS reference architecture: watch a request flow through ALB and API Gateway into ECS Fargate agents, out to Bedrock Knowledge Bases and Kendra for RAG, DynamoDB/ElastiCache memory, and SageMaker + Bedrock for reasoning and fine-tuning.
Demo 11
Google Cloud AI Agent Playbook
An interactive, animated Google Cloud reference architecture: watch a request flow through Cloud Load Balancing and Apigee into Cloud Run / GKE agents, out to Vertex Vector Search and the RAG Engine, Firestore/Memorystore memory, and Vertex AI + Gemini for reasoning and tuning.
Demo 12
Data Platform Architecture
An interactive, animated data-platform architecture: watch data flow from operational sources through CDC, streaming and batch ingestion into a Bronze → Silver → Gold lakehouse on an open table format, then out to BI, SQL, ML and real-time serving.
Demo 13
Enterprise RAG · Security & Governance
Take RAG inside the company: sync SharePoint, Confluence and shared drives, filter retrieval by each user's permissions, redact sensitive data, and measure answer quality before rollout.
Demo 14
Data Quality & Observability
Catch bad data before it reaches your dashboards — automated quality checks, anomaly detection, lineage and alerting across the pipeline.
Demo 15
RAG · Retrieval-Augmented Generation
Watch a question flow through a Retrieval-Augmented Generation pipeline — documents chunked and embedded into a vector database offline, then a query embedded, top-k chunks retrieved, and the LLM answering grounded in real, cited text.
Demo 16
3NF vs. Data Vault
The old Inmon-vs-Kimball argument is back, and this time it's about the silver layer of the medallion architecture. Should silver be modeled in Inmon-style 3NF, or as a Data Vault 2.0 raw vault?