Complete architectural overview of Apache Kafka — origins, the M×N integration problem, the 4 pillars of Kafka's extreme speed (sequential I/O, page cache, zero-copy, batching), message anatomy, and the dumb broker / smart consumer paradigm.
Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, and scalable real-time data pipelines and streaming applications.
Comprehensive guide on Change Data Capture (CDC), detailing how it works, alternatives comparison, implementation patterns with Debezium and Spring, and deep dives for senior engineers.
A **consumer group** is a set of consumers that collectively consume a topic's partitions. Each partition is assigned to exactly one consumer within the group.
Deep dive into Kafka consumer lag — offset mechanics, lag calculation, root cause diagnosis, scaling strategies, and non-blocking retry topics (DLQ, multi-tier backoff, 2-phase consumer commits).
A comprehensive guide to the Dead Letter Queue (DLQ) pattern — covering poison pill handling, retry strategies, alternatives comparison, AWS SQS / Kafka / RabbitMQ implementations, and production deep dives for senior engineers.
A comprehensive guide to preventing duplicate message processing across Kafka, Kafka Streams, RabbitMQ, SQS, and Redis — covering EOS internals, Kafka Streams State Store vs Redis architecture, idempotent consumers, and production deduplication patterns for senior engineers.
A comprehensive guide to managing and verifying application configurations, environment variables priority, HashiCorp Vault secrets, Kafka topics, ACLs, schema registry compatibility, database migrations, and post-deploy health verification.
In-depth guide to Event-Driven Microservices, covering domain events, event types (notification vs carried-state vs sourcing), choreography sagas, transactional outbox pattern, schema evolution, ordering, idempotency, and Kafka setups.
Kafka uses a hash of the message key to determine partition assignment. Understanding this mechanism is essential for ordering guarantees, avoiding hot partitions, and designing correct partition keys.
Without idempotence, network retries create duplicate messages on Kafka brokers. Enabling idempotence guarantees exactly-once delivery per producer session.
**Q1: What are the three layers required for end-to-end exactly-once in Kafka?**
**Q1: Explain Kafka's architecture in 2 minutes.**
**Q1: Walk me through what happens when a producer calls `send()`.**
Kafka Access Control Lists (ACLs) for fine-grained authorization. Covers KRaft ACL storage, resource patterns, OAuth/OPA/RBAC integration, and at-scale management.
Configure Kafka authentication with SASL/PLAIN, SCRAM-SHA-512, GSSAPI (Kerberos), mTLS, and OAuth 2.0. Understand how KRaft mode affects credential management.
A complete guide to Kafka brokers — what they are, how storage works, partition leadership, replication, ISR, KRaft vs ZooKeeper, log compaction, performance internals, and production monitoring. Beginner through senior depth.
**Kafka Connect** is a framework for **reliably moving data between Kafka and external systems** (databases, file systems, cloud services) without writing.
Lightweight per-record transformations in Kafka Connect. Covers built-in SMTs, conditional predicates, custom SMT development, and when to use SMTs vs stream processing frameworks.
A **consumer** reads messages from Kafka topics. Unlike traditional queues (push-based), Kafka consumers **pull** messages at their own pace. This gives.
The six primitives of Kafka data governance — schema policy, topic ownership, access control, encryption/masking, audit/lineage, and data quality. Why brokers don't provide governance and how to build it.
A complete guide to Kafka exactly-once semantics — delivery guarantees, idempotent producer, transactions, read_committed consumers, Kafka Streams EOS, zombie producer fencing, two-phase commit internals, and production patterns.
How Kafka log compaction preserves the latest value per key, enabling state stores, CDC changelog topics, and materialized views. Covers cleaner internals, tombstones, tiered storage, and KTable integration.
MirrorMaker 2 cross-cluster replication for disaster recovery, active-active, and hub-and-spoke. Covers architecture, offset translation, failover, Kubernetes deployment, and monitoring.
Deep dive into Kafka partitioning strategies — key-based, round-robin, sticky, custom partitioners, hot key mitigation, partition count sizing, and ordering trade-offs.
End-to-end Kafka performance tuning — producer batching, broker I/O, consumer throughput, JVM and OS tuning, compression selection, tiered storage, and benchmarking methodology.
A **producer** is a client application that publishes (writes) messages to Kafka topics. It is responsible for:
Combined guide to Kafka producers and consumers — internal architecture, delivery semantics, serialization, offset management, error handling, and production patterns.
End-to-end Kafka security guide covering authentication, ACLs, TLS 1.3 encryption, Zero Trust architecture, network isolation, credential management, and monitoring security events.
A comprehensive guide to Kafka Streams: from core concepts and internal architecture to stateful processing, failure recovery, exactly-once semantics, windowing, joins, interactive queries, and production system design patterns.
A deep-dive into maximizing Kafka throughput — covering Linux Zero-Copy sendfile(), OS Page Cache mechanics, compression algorithms, producer batching, consumer fetch tuning, and kernel network optimizations.
A topic is a named, durable stream of messages in Kafka — the logical category or feed where producers write and consumers read.
Comprehensive system design comparison of Apache Kafka vs RabbitMQ vs AWS SQS. Covers event streams vs message queues, Claim Check pattern, Web Crawler SQS vs Kafka trade-offs, and single-broker sizing heuristics.
Comprehensive guide comparing Apache Kafka's legacy ZooKeeper architecture with the modern KRaft (Kafka Raft) metadata mode — covering internal mechanics, failure scenarios, Strimzi Kubernetes deployment, migration strategies, and production deep dives for senior engineers.
Kafka guarantees **total ordering within a partition**. Messages written to the same partition are always consumed in the exact order they were produced.
Guide to asynchronous messaging systems including Kafka, RabbitMQ, SQS, event sourcing, pub/sub patterns, consumer groups, ordering guarantees, and exactly-once semantics.
Consumer lag is the most important consumer metric:
Deep-dive into the Confluent Parallel Consumer model (now unmaintained), its internals, ordering modes, offset bitmap mechanics, and practical migration paths — Spring Boot virtual threads, manual executor dispatch, and the upcoming Kafka Share Groups (KIP-932).
A partition is an ordered, immutable sequence of records within a topic — the fundamental unit of parallelism, replication, and storage scaling in Kafka.
Deep-dive into Kafka Connect rebalance storm internals — why they happen, the two rebalance protocols, incremental cooperative mechanics, task assignment algorithms, Spring Boot monitoring, and a production-grade rolling restart runbook.
Kafka guarantees ordering within a partition, but single-threaded processing limits throughput. This guide covers four patterns for achieving high throughput while preserving per-key ordering.
The `acks` configuration controls how many broker acknowledgements the producer requires before considering a send successful, trading off throughput, latency, and durability.
Idempotence protects against duplicates within a session, but it doesn't help when:
A comprehensive guide to the Raft Consensus Algorithm — covering leader election, log replication, safety guarantees, and how it is implemented in Apache Kafka's KRaft metadata mode.
Patterns for delivering real-time data to clients including WebSockets, Server-Sent Events, long polling, short polling, and push notification architectures.
Deep dive into landmark real-world system designs — Shopify Pod cell-based architecture, Twitter hybrid timeline fan-out, Netflix 7-year microservices evolution, Facebook TAO social graph datastore, Uber DOMA & Schemaless, Slack Flannel edge caching, and LinkedIn Kafka origin.
The replication factor defines how many copies of each partition exist across the cluster to guarantee fault tolerance and high availability.
Partitions are the unit of parallelism in Kafka. Scaling them is critical for throughput but can break ordering for keyed topics. This guide covers the mechanics, risks, and migration strategies.
Deep-dive into high write throughput techniques — sharding, partitioning, WAL internals, LSM trees, async pipelines, batching, backpressure, idempotency, and distributed transactions — with production Java/Spring code and failure mode analysis.
Deep-dive into Confluent Schema Registry — Avro wire format internals, subject naming strategies, compatibility mode mechanics, schema evolution patterns, code generation, topic reset with schema changes, and production Spring Boot configuration.
A complete guide to the Transactional Outbox Pattern — from the Dual-Write problem for beginners to CDC vs polling internals, at-least-once guarantees, ordering semantics, and production monitoring for senior engineers.
A detailed collection of real interview questions and answers from a Walmart Java Developer interview. Ideal for candidates with 3+ years of experience, covering DSA, Core Java, System Design, Spring Boot, and Kafka.
