Proof of Concept Bakery

How it’s built

Why a paid order is never lost and never doubled

When a visitor pays for an order, six other objects have to hear about it, and none of them can share a transaction with the order. This is how each hears once in effect, though the message may travel twice, what happens when it cannot travel, and where the retrying stops.

A paid order, and who has to hear

Two cardamom buns and a country loaf, paid by test card. Stripe’s webhook reaches the Billing object, which records PaymentSucceeded; that event reaches the Order, which records OrderPaid. As built on 2026-11-06, the routing table incrates/app/src/routing.rs sends OrderPaid to six subscribers: the day’s Schedule, which turns the hold into a reservation; the Kitchen, which moves the card to confirmed; the Crm; the order’s Customer; the Inbox, which writes the receipt; and the day’s Dispatch, which confirms a delivery’s stop.

Each is a Durable Object with storage of its own, and no transaction spans two objects. So the Order cannot record the payment and move the card in one step. It writes its own state and then tells the others, and everything below is about that word “then”.

What the Queue Lab shows, and what it cannot

The Queue Lab, as read on 2026-11-06, has three workers take orders from a broker and print oven tickets. Crash one after it prints and before it acknowledges, and another worker prints the same order again. Its page concludes that a queue “delivers at least once. It cannot promise exactly once”, and exactly once “is something the worker builds on top”, with a record of the ids already printed. Orders that keep failing go to a dead-letter queue after four tries.

Two things are outside it. It starts at the broker, so getting a message there with the write that caused it is not drawn. And it says it writes the record and the ticket “together, in one atomic step”, which a real printer could not join. In the bakery the first is the hard part, and the second is true: every ticket is a row.

Two steps, and the gap between them

The Order’s write and the send to the Queue are two operations on two systems, with a moment between them whichever comes first. Send first and fail before the commit, and the board shows a paid card for an unpaid order. Commit first and fail before the send, and the kitchen never hears.

The bakery commits first and treats the send as something that may happen more than once. In one synchronous transaction (workers/api/src/objects/stream.rs) the Order appends OrderPaid to its events table, copies the envelope into itsoutbox table, writes its new state and records the command’s key. That is Chris Richardson’s transactional outbox: the message is stored with the change, so it exists exactly when the change does. After the commit the object sets an outbox timer 30 seconds out, arms its alarm for it, and, while the visitor already has the answer, sends up to 100 outbox rows to thebakery-events Queue in one batch, deleting the rows the Queue accepted.

The gap is still there, but the only failure left in it is a duplicate. If the isolate stops after the Queue took the batch and before the delete, the rows go again. If two commands reach one object in quick succession, the second send can read rows the first has not yet deleted. Richardson’s page says as much: the relay “might publish a message more than once”.

What happens to a row the Queue refused

As the code reads on 2026-11-06: when the batch send fails, every row stays, each row’sattempts count goes up by one (nothing reads it yet), and the object logsrelay_error. Two things send the rows again. The outbox timer fires 30 seconds after the commit, and the alarm sends what is left, setting the timer another 30 seconds out while rows remain. And the object’s next command sends the whole outbox after its own commit, oldest row first. The 15-minute cron is not involved: it retries unfinished wipes and never reads an outbox.

Each alarm run that arms the alarm again is counted by the loop guard,REARM_GUARD in stream.rs: 100 a UTC day for one object. At one every 30 seconds, a Queue that refuses everything for 50 minutes uses up the day; the 101st re-arm is refused and logged alarm_guard, and the rows wait for the object’s next command, which may always arm, or for the wipe at 10:00 UTC. Past the guard a row waits; it is never dropped.

At least once, applied once

Cloudflare’s Queues documentation, read on 2026-11-06, promises delivery at least once and rarely more, and recommends a unique id per message as the key to drop repeats. The bakery adds repeats of its own: its consumer, workers/api/src/queue.rs, acknowledges a message only when every subscriber answered 2xx, so when the Kitchen fails,OrderPaid goes again to the five that already applied it.

So the receiving end remembers: Hohpe and Woolf’s idempotent receiver. Each subscriber stores an applied event’s id in the same write as the change: the Schedule and the Dispatch in a seen table, the Kitchen in a set inside the board’s one row, the Order in itscommands_seen. A second delivery finds the id and changes nothing. At-least-once delivery and a receiver that remembers is the only exactly once there is, and it holds here for the reason the lab gives as its simplification: every effect the bakery has is a write to an object it owns, or a call to Stripe with the order’s id as the key.

Where the retries stop

The consumer takes batches of up to 10 and allows 3 retries, with the dead-letter queuebakery-events-dlq (workers/api/wrangler.jsonc). Cloudflare’s page on dead-letter queues, read on 2026-11-06, says a message at the retry limit with no such queue is deleted permanently. The bakery’s is read by the same handler, which writes each message, with the last error recorded for it, into the owning Sandbox’s dead_letters, keyed by the event’s id, until a redrive or the wipe at 10:00 UTC.

A redrive sends the stored envelope to the Queue again with the same id and deletes the row after the send, so a failure between the two can send it twice and never lose it, and the subscribers’ records make the repeat harmless.

What it costs

Try it

Sign in, open the console, and under Faults tick “Kitchen refuses deliveries”. Order from the menu and pay with the test card. The order is paid, and after three retries the Dead letters list holds the Kitchen’s events, each with the error “Kitchen answered 503”. Untick the fault and redrive the OrderPlaced first: an OrderPaid redriven before it finds no card, is refused with 409, and can come back as a dead letter. The card appears once.

Sources

Back to the Queue principle on How it’s built, to the decision that chose it, ADR 0004, or to the lab.