Petr Volobuev | Technology RadarPetr Volobuev | Technology Radar

Outbox

pattern
Using

The pattern made it to production. We picked RabbitMQ over Kafka, which means we didn't get durable event storage and out-of-the-box replay — and yet there are areas where message delivery has to be guaranteed and processing idempotent. Outbox plugged exactly that hole.

We didn't write the implementation ourselves: we took the transactional outbox from MassTransit on top of RabbitMQ — the event is written in the same transaction as the domain changes, a background relay carries it to the broker, and an inbox on the consumer side provides deduplication for idempotency.

For some good reading on the topic — an article on Habr.

Using

The Outbox pattern is a way to reliably publish events/messages from a service so that changing data in the DB and sending the message happen atomically (within a single transaction), without two-phase commits and without "lost" events.

Why you need it

The classic problem: in a request handler you

  1. write changes to the DB
  2. publish an event to a broker (Kafka/RabbitMQ/…)

If you crash between (1) and (2) — the data is already in the DB, but the event never left. If it's the other way around (the event went out but the transaction rolled back) — the receiver sees an event about something that "never happened". Outbox eliminates both scenarios.

The idea of the pattern

Instead of going "straight to the broker", the service writes the event to an outbox table in the same transaction as the domain changes:

  • Transaction:

    • UPDATE/INSERT on the domain tables
    • INSERT INTO outbox (...) VALUES (...)

  • A separate background process (publisher/relay) reads the outbox and sends to the broker.

  • After a successful send, it marks the record as sent (or deletes it).

Bottom line: if there's a change in the DB — there's guaranteed to be an event in the outbox, and sooner or later it will be published.

A typical Outbox data schema

The minimally useful fields:

  • Id (UUID/ULID)
  • OccurredAt (when the event "happened")
  • Type (event/contract name)
  • Payload (JSON/Protobuf/base64)
  • Headers/Metadata (traceId, tenantId, correlationId, schemaVersion)
  • Status (Pending/Sent/Failed)
  • RetryCount, NextAttemptAt, LastError
  • PartitionKey/AggregateId (for per-entity ordering)

How to "pick up" messages from the outbox

Two popular approaches:

1) Polling

The publisher periodically runs a query:

  • select N "pending" rows (often ORDER BY occurred_at)
  • lock them (FOR UPDATE SKIP LOCKED in Postgres) — so multiple workers don't grab the same ones
  • send to the broker
  • mark as Sent

Pros: simpler, portable. Cons: latency, load on the DB.

2) CDC (Change Data Capture)

The service still writes to the outbox, but publishing is done by reading the WAL/binlog (Debezium and friends), then shipping to Kafka/… Pros: low latency and less polling load. Cons: trickier to operate.

Guarantees and important properties

  • Outbox gives you at-least-once delivery: an event may be sent more than once.

  • Which is why consumers must be idempotent:

    • keep "processed message ids"
    • use upserts keyed by event/aggregate
    • dedupe on the consumer side

Exactly-once in distributed systems is almost always achieved only "by design" (idempotency + dedup), not through transport magic.

Message ordering

If you need ordering per aggregate (e.g. OrderId):

  • store an AggregateId/PartitionKey
  • send to Kafka partitioned by that key (then ordering is preserved within the key)
  • with polling - select/publish sequentially per key (or serialize on the publisher)

Global ordering is usually unnecessary and expensive.

When Outbox is especially needed

  • microservices + a message broker
  • business events must "never get lost"
  • you can't (or won't) drag in 2PC/Distributed Transactions
  • you want predictable reliability and traceability