The transactional outbox turns a dual write into a single commit

The idea

A service that saves state and then announces it has a dual write problem. The database commit and the broker publish are separate operations, and no transaction spans both.

Either order fails. Commit first, then crash before publishing, and the event is lost while the row exists. Publish first, then have the transaction roll back, and consumers act on something that never happened.

The outbox pattern removes the second write from the critical path. The event is inserted as a row into an outbox table inside the same transaction as the state change. Both commit or neither does. A separate relay reads committed outbox rows, publishes them, and marks them sent. Delivery is now tied to the commit, and the guarantee becomes at-least-once.

An example

An order service needs downstream billing and fulfilment to learn about new orders.

• Begin a transaction.
• Insert the order into orders.
• Insert a row into outbox with an event id, the aggregate id, an event type, a versioned payload and published_at set to null.
• Commit.
• The relay selects unpublished rows in id order, publishes each to the broker, then sets published_at.

Now suppose the relay publishes an event and crashes before updating the row. On restart it publishes the same event again. Nothing was lost, but billing sees the order twice. That duplicate is the price of the pattern, and the consumer has to absorb it using the event id.

Where it breaks

It is at-least-once, never exactly-once. A consumer that is not idempotent will double charge, double ship or double count.

Ordering is subtler than it looks. Sequence values are assigned at insert time but transactions commit in a different order. A relay that tracks a high-water mark like id > last_seen can skip a row with a lower id that commits after a higher one, and that event is silently never sent.

The table grows with every write. Without cleanup it becomes a source of bloat and slow scans.

Polling adds latency, which matters when a workflow waits on the event.

It is the wrong tool when the caller needs an answer from the external system before committing, such as authorising a payment. The outbox gives eventual delivery, not a synchronous result.

In practice

I would start with these defaults:

• Track delivery per row with published_at, not with a high-water mark. If ordering by commit matters, read the write-ahead log through change data capture instead of polling.
• Run multiple relay workers with FOR UPDATE SKIP LOCKED so they do not fight over rows.
• Use the aggregate id as the broker partition key so events for one entity stay ordered.
• Give consumers an inbox: record processed event ids in the same transaction as their own effect, and skip ids already seen.
• Store a deliberate, versioned event contract in the payload, not a copy of the table row, so schema changes do not leak to consumers.
• Delete or archive published rows on a schedule.
• Alert on the age of the oldest unpublished row. It is the single number that tells you the relay has stalled.

When a state change and its announcement must agree, commit both in one transaction, let a relay deliver the announcement, and make every consumer idempotent on the event id.