kafka
mqDrawing the line: we're staying on RabbitMQ. Kafka suited us feature-wise, but its operation and deployment are noticeably heavier, and our loads and the risks of broken event chains just don't justify the cost. If a genuine event streaming task ever shows up — we'll be back.
Apache Kafka is a distributed event streaming platform (colloquially, a "distributed commit log"), and it can
- publish/subscribe to event streams
- store them reliably
- process streams as they arrive.
Key entities:
Brokers → topics → partitions → offsets
- A Kafka cluster consists of brokers (servers).
- Data is written to topics. Each topic is a partitioned append-only log (an ordered immutable sequence of records that keeps getting appended to at the end).
- Within a partition, every record gets an offset — a sequential number.
- Kafka keeps events according to a retention policy (time/size): events can stay available even after being read, which gives you replay and "read it again".
Consumer groups
Kafka "glues together" queues and pub/sub via consumer groups:
- Within one group, each partition is read by exactly one consumer (parallelism = number of partitions).
- Different consumer groups can read the same topic independently (multi-subscription).
- Ordering is guaranteed only within a partition, so choosing the partitioning key is critical
After a few trial runs on the MVPs of our first services, the verdict: it's a powerful tool and it fits us on every count, but it comes with a noticeable operational and deployment burden. The key point is that our risk of broken event chains is fairly low. And the projected load on our system just isn't high enough to justify Kafka.