Outbox and a choreographed saga: four services closing one payment

A payment goes through four services: record the payment, check for fraud, move the balance, notify. If the third one fails, how do you undo the first two? There is no BEGIN covering all four.

I wrote a reference implementation in Go to work through that scenario properly, with one Postgres per service and Kafka as the event backbone. What follows are the decisions that survived building it.

Before the saga there is a smaller problem

Every service that changes state and tells someone about it performs two writes, into two different systems. It commits to the database, then publishes to the broker. There is no shared transaction between them, and every ordering has a failure window:

database commit ok   →  Kafka publish fails    (state with no event)
Kafka publish ok     →  crash before commit    (event with no state)

This is the dual write. The symptom shows up far from the cause: an order that exists in the database and never reached the next service, or a consumer reacting to a payment that was never persisted. At low volume it takes a while to surface, and when it does it arrives as data divergence, not as an error.

Transactional outbox: a single write

The way out is to stop publishing directly. The event becomes a row in a table of your own database, written in the same transaction as the business state:

BEGIN;
  INSERT INTO payments (id, amount, from_account, to_account, status)
       VALUES ($1, $2, $3, $4, 'CREATED');

  INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload)
       VALUES ($5, 'payment', $1, 'payment.received.v1', $6);
COMMIT;

Either both inserts exist or neither does. The broker leaves the critical path, and the event now has exactly the same durability as the data it describes.

The table itself is unremarkable. The index is the part that matters:

CREATE TABLE outbox (
    id             UUID PRIMARY KEY,
    aggregate_type TEXT NOT NULL,
    aggregate_id   UUID NOT NULL,
    event_type     TEXT NOT NULL,
    payload        JSONB NOT NULL,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at   TIMESTAMPTZ
);

CREATE INDEX idx_outbox_unpublished
    ON outbox (created_at)
    WHERE published_at IS NULL;

The partial index covers pending rows only. That is the difference between scanning a table that grows forever and scanning a queue that, under normal conditions, holds almost nothing.

The relay publishes later, at least once

A separate process reads what has not been published yet and pushes it to Kafka:

SELECT id, aggregate_type, aggregate_id, event_type, payload
  FROM outbox
 WHERE published_at IS NULL
 ORDER BY created_at ASC
   FOR UPDATE SKIP LOCKED
 LIMIT 100;

SKIP LOCKED is what lets several relay replicas run without coordination: each one grabs a different batch instead of waiting on another’s lock. Once published, it sets published_at = now().

Note the order: publish first, mark second. If the process dies between the two, the event goes out again on the next cycle. The guarantee is at-least-once, and there is no fixing that on the producer side. Marking before publishing only trades duplicates for loss, which is far worse.

The inbox closes the loop on the other side

If delivery repeats, the consumer has to tolerate repetition. Each service records what it already processed and ignores the rest:

INSERT INTO processed_events (event_id, consumer)
     VALUES ($1, $2)
ON CONFLICT DO NOTHING;

Zero rows affected means that event already went through, and the consumer ends the transaction without touching anything. The detail that makes it work is the composite primary key, (event_id, consumer): dedup is per consumer, not global. The same payment.settled.v1 has to be processed by the notification service and by the payment service, once each.

The dedup insert, the state change and the next event all go in one transaction:

BEGIN
  INSERT INTO processed_events ... ON CONFLICT DO NOTHING   -- already processed? stop here
  <apply the state change>
  INSERT INTO outbox (next event)
COMMIT

At-least-once delivery plus idempotent consumption gives an exactly-once effect. Worth being precise: exactly-once delivery does not exist over a network, what exists is an effect applied exactly once.

Choreography: nobody is in charge

With reliable transport in place, the flow can be spread out. Two classic options: orchestration, with a central service calling the others, or choreography, where each service only knows which event it consumes and which one it emits.

Choreography was the choice here:

Service	Consumes	Emits	State
payment	settled, rejected, failed	payment.received.v1	payment status machine
fraud	payment.received.v1	approved, rejected	risk decision
ledger	payment.approved.v1	settled, failed	accounts and entries
notification	settled, rejected, failed	nothing, it is terminal	notifications

Topic name equals event type, Kafka key equals the payment_id. The key is what buys ordering per payment: everything for a given payment lands on the same partition, and ordering across different payments does not matter.

Compensation with no coordinator

The interesting case is not the happy path. When the ledger has insufficient funds it does not report to an orchestrator, it emits payment.failed.v1. Whoever started the saga consumes that event and rolls its own state back to FAILED. The same goes for payment.rejected.v1, emitted by fraud.

It is a distributed rollback, written as a reaction rather than as a command.

And here sits the distinction that usually gets skipped: compensation is not rollback. A database can undo because nothing was visible before the commit. A saga has already exposed every intermediate step to the rest of the world, so compensating means applying a new operation in the opposite direction, one that can fail too and has to be idempotent too. If the step already caused an external effect, an email, a charge, a webhook, no compensation erases it. The best you can do is emit the correction.

What choreography charges you

Removing the orchestrator does not make the system simpler, it moves where the complexity lives.

The flow stops existing in written form. No file describes the sequence of the four steps, it only emerges from who listens to what. Reading one service’s code does not tell the whole story, which is why instrumentation stops being optional: the project propagates OpenTelemetry traces across every hop precisely so the whole saga shows up in one place.

Then come the things nobody handles for you: the outbox table grows and needs pruning, so does processed_events, the payload becomes a public contract the moment another service reads that JSON, and the HTTP edge needs idempotency of its own, here a Redis SETNX on x-idempotency-key, otherwise a client retrying the POST creates two payments before any event exists.

Running it

Brings up everything, including Postgres, Redis, Redpanda, the OTel collector and Jaeger:

make up

A regular payment, which fraud approves roughly 70% of the time:

curl -X POST http://localhost:8080/payment \
  -H 'Content-Type: application/json' \
  -H 'x-idempotency-key: 11111111-1111-1111-1111-111111111111' \
  -d '{"amount": 1500, "from": "alice", "to": "bob"}'

And an amount high enough for the ledger to fail settling, which is the compensation path:

curl -X POST http://localhost:8080/payment \
  -H 'x-idempotency-key: 33333333-3333-3333-3333-333333333333' \
  -d '{"amount": 999999999, "from": "carol", "to": "bob"}'

After that, Jaeger shows the payment crossing all four services, and the final status in Postgres tells how the saga ended.

The code lives at mathehluiz/outbox-saga-choreography.