How it’s built

Proof of Concept Bakery is the made-up bakery that ihaveasegway.com’s lab simulates, built as a real business system: ordering, pickup and delivery scheduling, catering, billing through Stripe’s sandbox, and a CRM with campaigns. Every backend is Rust on Cloudflare Workers, its state is in SQLite-backed Durable Objects, every visitor gets a private sandbox that is wiped every night, and every decision is written down in public.

A working bakery you can order from in a minute, built to show the principles of the lab running for real: Rust on Cloudflare, a private copy for every visitor, and nothing in it real.

Each section below takes one principle in the same order: the problem the bakery has, the technique chosen, how it works here and what it costs, where it runs, and what you can do to see it. Then it links the lab page that explains the principle on its own, where there is one, and the decision record (ADR) that chose it. The tour on every screen links back here.

The five lab principles

Each of these has a lab page on ihaveasegway.com, where a simulation in your browser shows the principle on its own. Here it runs for real.

Transactional outbox, idempotent consumer and the dead-letter queue

The problem: A paid order has to reach the kitchen board, the CRM and the inbox, which are other objects. A write and a message sent after it can come apart: the write commits and the message is lost, or the message goes and the write is undone.
The technique: A transactional outbox, a queue that delivers at least once, consumers that skip what they have already applied, and a dead-letter queue for what keeps failing.
How it works here: Each object writes its events, and a copy of them in its outbox, in the same transaction as its state, then sends the outbox to the Queue: a row may go twice, but once committed it is never lost. Each subscriber records the ids of the events it has applied in the same transaction as the change it makes, so a second delivery changes nothing; without that, every retry would put a second card on the board. A delivery still failing after three retries is parked as a dead letter, not dropped, until someone fixes the cause and redrives it.
Where it runs: The outbox table of every aggregate object; the bakery-events Queue; each subscriber’s record of applied events; the dead-letter queue and the Sandbox’s dead letters.
See it yourself: Turn on the kitchen fault in the console, place an order, and watch the retries, the dead letter and a redrive that puts the card on the board once. The console.

Cache invalidation by version

The problem: Nearly every page view asks for the same menu, and most of the time it has not changed. Fetching the whole menu from the Catalog object each time costs a request and its body, and a cache that expires on a timer serves an old price until it does.
The technique: Cache keys that carry the menu’s version, and an ETag for the browser.
How it works here: Publishing the menu bumps its version and emits MenuPublished. The cache is keyed by sandbox and version, so the first read of a new version misses once and fills it, and an old version is never served again because nothing asks for its key: nothing has to be found and deleted. A client that sends the ETag back gets a 304 and no body. The cost is a version check on every read, one miss per publish in each data centre that serves the sandbox, and a new price nobody sees until it is published.
Where it runs: The Catalog object’s menu version and ETag; the Cache API, keyed by sandbox and version; the MenuPublished event.
See it yourself: Change a price in the console, publish it, and watch the version, the ETag and the Bakery-Cache header change on the menu. The menu.

Rebuilding derived data from a log

The problem: The CRM’s customer list, timelines, segments and campaign stats are derived from events. A bug in how they are derived, or a new view of them, means computing them again from history while orders keep arriving.
The technique: Projections rebuilt from an append-only log, in batches, behind a cursor.
How it works here: The CRM object keeps every event that changed its projections in a log. A rebuild empties the projections, sets a cursor at the start of the log and replays 200 events at a time, one alarm a batch, while commands keep working; the result is the same bytes it was before. That only holds because applying an event is idempotent and ordered per stream: a late event with a lower sequence number is dropped, and a projection that appended on every replay would write each order’s timeline line twice.
Where it runs: The Crm object: its log, its projections and its rebuild cursor.
See it yourself: Press Rebuild projections in the console, watch the cursor move, and compare the stats before and after. The console.

Vehicle routing, and computing where it is free

The problem: On a Saturday the bakery has 63 deliveries and four vans. Dealing the stops out as they come drives hours more than it needs to, and a good plan takes real computation, which on a server is CPU the bakery pays for.
The technique: Two planners and one record: a savings heuristic on the server, VROOM in the visitor’s browser, and the Dispatch object as the judge.
How it works here: The server plans with the Clarke–Wright savings heuristic, fast and good but not exact. VROOM, compiled to WebAssembly, plans the same stops in your browser and looks harder for a shorter plan, at no cost to the server. Neither is trusted: the Dispatch object works out any plan’s minutes from its own travel-time matrix, refuses one that misses a stop, and publishes one as the record. The times are free-flow, with no traffic.
Where it runs: VROOM in a Web Worker on the dispatch screen; Clarke–Wright and the publish check in the Dispatch object, one per sandbox and day; RoutePublished.
See it yourself: Plan Saturday’s stops both ways on the dispatch screen, compare the two, and publish one. Dispatch.

Consensus, and when a single writer needs none

The problem: Two tabs race for the last place in a pickup slot. If the slot’s count lived in several copies, the copies would have to agree on who got it, and making copies agree is what consensus protocols such as Raft are for.
The technique: A single writer: one Durable Object owns each day’s places, so there is only one copy to agree with.
How it works here: The Schedule object for a sandbox and a day takes every hold for that day one at a time, and its storage commits with the code that decided, so the second of two racing holds sees the first and is refused, naming the next free slot. The bakery replicates nothing itself, so it elects no leader and counts no quorum. What that gives up, and where the bakery pays for it, is the essay linked below.
Where it runs: The Schedule object, one per sandbox and business day; the SlotHeld event and the slot-full refusal.
See it yourself: Set a pickup slot’s capacity to one in the console, then check out that slot from two tabs: one order is held and the other is refused. The menu.

Principles with no lab page yet

The bakery needed these too. Each is explained here and decided in an ADR, and each has a step in the tour of the screen it runs on.

Idempotency keys

The problem: A phone loses its connection just after Place order. The visitor presses it again, and the first request may have arrived after all. Without care, that is two orders.
The technique: An Idempotency-Key on every command, remembered by the object that runs it.
How it works here: Each attempt carries a new key, and its retries the same one. The object records the key and its answer in the same transaction as the command’s events, so a repeat gets the first answer back with Idempotent-Replayed: true, and the same key with a different body is refused. The key has to be stored with the write: checked in one place and written in another, a crash between the two would let the repeat through.
Where it runs: The commands_seen table in every aggregate object; the Idempotency-Key header on every POST.
See it yourself: Copy the tour’s curl for placing an order and run it twice: the second answer is the first, replayed. The checkout.

Read more: no lab page yet · ADR 0004 · Why pressing the button twice never orders twice

Idempotency keys on payments and webhooks

The problem: Stripe is a second system with its own state. A request to it can time out after it succeeded, and its webhooks arrive at least once, sometimes late and out of order.
The technique: A key built from the sandbox and the order’s id as the idempotency key at Stripe, and each webhook event’s id as the record that it was applied.
How it works here: The Billing object asks Stripe for a payment with a key built from the sandbox and the order’s id, so asking twice gets the same one. The order is paid only when Stripe’s signed webhook says so, and Billing keeps the id of every webhook event it has applied, so a redelivered one changes nothing. Everything runs in Stripe’s test mode: no money moves.
Where it runs: The Billing object, one per sandbox: payments, refunds and the webhook events it applied; PaymentSucceeded and OrderPaid.
See it yourself: Pay for an order with the test card 4242 4242 4242 4242, then find the payment once in the console’s ledger. The checkout.

Read more: no lab page yet · ADR 0007 · Why pressing the button twice never orders twice

Sagas by alarm

The problem: A catering job runs for weeks: an inquiry, a quote, an acceptance, a deposit, the kitchen, the delivery and the balance. No transaction can last that long, or span the objects and Stripe it touches.
The technique: A saga: each step is one object’s transaction, and the next is started by an event or a timer.
How it works here: The CateringEvent object moves through its states one transaction at a time, and keeps its reminders, the quote’s expiry and the deposit’s lapse as rows in a timers table behind one alarm. A step that cannot go on is undone by a step of its own, such as releasing the day’s place or refunding the deposit, never by a rollback. The saga only moves forward, so a command out of order is refused with 409.
Where it runs: The CateringEvent object, one per inquiry, and its timers; the Schedule’s catering places; Billing’s deposit and balance invoice.
See it yourself: Run a catering job end to end, and advance the clock in the console to see each reminder fire. Catering.

Read more: no lab page yet · ADR 0004 · Why a catering job is a saga that only moves forward

Leases and TTLs

The problem: An order placed and never paid would hold its place in a slot for good, and a sandbox nobody comes back to would keep its data for good.
The technique: Leases: every hold and every sandbox has an end, written down when it starts.
How it works here: Placing an order holds its places for 15 minutes of bakery time, and if it is not paid, the hold’s timer releases them. Every sandbox is deleted at 10:00 UTC, and every object in it arms a 36-hour backstop on its first write, so one the wipe misses deletes itself. The cost is the payment that lands after its hold lapsed: the Schedule takes it back even past capacity, because the money already moved.
Where it runs: The Schedule’s hold timers; the Sandbox’s wipe alarm; every object’s 36-hour backstop.
See it yourself: Let a checkout lapse, or advance the clock past it, and watch the place come back; the console says when your sandbox will be wiped. The checkout.

Read more: no lab page yet · ADR 0005 · Why every hold and every sandbox has an end

Admission control

The problem: Every sandbox costs requests and storage, and the bakery pays for each one. A script could make thousands of them.
The technique: Admission control: a check that you are a person, rate limits, and daily budgets that refuse work before it costs anything.
How it works here: Making a sandbox or an account needs a Turnstile pass. The Control object counts the day’s sandboxes and refuses past its caps, giving guests a smaller share, while commands and API tokens have rate limits of their own and simulated traffic its own minutes a day. A refusal is cheap and says when to come back; the cost is that on a busy day a visitor can be turned away.
Where it runs: Turnstile on sign-in; the Control object’s daily ledger; the API Worker’s rate limits.
See it yourself: Ask GET /v1/status: it shows today’s sandboxes, guests and demo accounts against their caps. Sign in.

Read more: no lab page yet · ADR 0005 · Why a refusal is the cheapest thing the bakery does

CQRS: aggregates write, the CRM reads

The problem: Staff want a customer’s whole history, the segments and the campaign stats, which cut across hundreds of orders. Each order is its own object, and asking every one of them would be slow and expensive.
The technique: Command and query responsibility segregation: the aggregates take the writes, and the CRM keeps read models built from their events.
How it works here: An order, a customer or a catering event decides its own changes and publishes them as events; the CRM object consumes them and keeps the views the console reads. The two are joined only by the Queue, so a read trails a write by a moment, and the screens say so rather than hide it. A change of consent is a command to the Customer object, never an edit to the list.
Where it runs: The Order, Customer and CateringEvent objects write; the Crm object reads them through the Queue.
See it yourself: Place an order, then open its customer in the console and watch the order appear on their timeline a moment later. The console.

Read more: no lab page yet · ADR 0003 · Why the CRM trails the orders, and why you can watch it

Load you can watch: lag between writes and projections

The problem: A quiet demo hides how the system behaves under load: the board filling up, the customer list and the activity feed trailing the orders, the retries and dead letters in the console.
The technique: Simulated traffic: made-up customers who order through the same checks as a visitor.
How it works here: The Simulator object places orders on its own alarm, through the same API and objects as yours, so the Queue, the board and the CRM carry real load. It runs only while the console is open, at a capped rate and for capped minutes a day, and stops by itself once the console has been closed for 90 seconds. What you watch is lag: every write succeeds at once, and the views catch up.
Where it runs: The Simulator object, one per sandbox; the bakery-events Queue; the CRM’s projections.
See it yourself: Start traffic in the console, then watch the board fill and the timelines trail the orders. The console.

Read more: no lab page yet · ADR 0004 · Why the CRM trails the orders, and why you can watch it

Time travel with a logical clock

The problem: Most of what a bakery does happens later: a hold lapses in 15 minutes, a reminder fires in a week, tomorrow’s bake list starts at midnight. A visitor cannot wait for any of it.
The technique: A logical clock: business time is the wall clock plus an offset you can move.
How it works here: The Sandbox keeps the offset and hands it to every object, and each object works out its timers in business time and sets its alarm again when the clock moves. Each click carries its own Idempotency-Key, so a repeated request moves the clock once. The platform’s alarms, the nightly wipe, cookies and Stripe’s timestamps keep the wall clock: the offset moves what the bakery decides, not what the platform does.
Where it runs: The Sandbox’s clock offset; every object’s timers table; the ClockAdvanced event.
See it yourself: Advance a day in the console and watch expiries, reminders and tomorrow’s bake list fire in order. The console.

Read more: no lab page yet · ADR 0004 · Why the bakery’s clock is yours to move

How the demo itself is run

What makes it safe to hand a stranger a working bakery: their own copy, nothing personal in it, and an API they can call from a terminal.

A private sandbox per visitor, wiped nightly

The problem: A demo that lets anyone change prices, cancel orders and send campaigns needs every visitor to have a bakery of their own, and none of it to outlive the night.
The technique: A sandbox per visitor: its own set of objects, born from a seed and deleted every night.
How it works here: Every object’s name carries its sandbox’s id, and every one is born through the Sandbox, which keeps a register of them, so one sandbox cannot reach another’s data and the wipe knows everything there is to delete. Two sandboxes from the same seed and birth date are the same byte for byte. Every sandbox is deleted at 10:00 UTC.
Where it runs: The Sandbox object and its register of children; the seed; SandboxSeeded and SandboxWiped.
See it yourself: Open the console to see your sandbox’s id, its seed and when it will be wiped; Wipe now deletes it on the spot. The console.

Read more: no lab page yet · ADR 0005 · Why nothing here is real, by construction

No personal data, by construction

The problem: A demo visited by strangers must not become a store of their names, emails or cards.
The technique: Not asking: the system has nowhere to put personal data.
How it works here: Sign-in keeps only a keyed hash of the number Google or GitHub gives for your account, and no email, name or picture. The customers, venues, addresses and cards are made up; the only free text is an order note and a catering brief, each checked when it is written; and the inbox exists because the bakery has no address to send mail to.
Where it runs: The Account object’s keyed hash; the checks on the note and the brief; the Inbox object.
See it yourself: Read the privacy page, then GET /v1/session: its answer is the whole of what the bakery knows about you. Privacy.

Read more: no lab page yet · ADR 0005 · Why nothing here is real, by construction

An API as a first-class client, from one contract

The problem: A demo you can only click proves the screens work. An engineer wants to call it from a terminal and see the same thing happen.
The technique: One contract, written in Rust, from which the route table, the TypeScript types and the API reference are generated.
How it works here: The console is a client of the same public API as your terminal: every one of its actions is a route in the OpenAPI document, which is generated from the contract and checked against the Worker’s route table. A signed-in sandbox can issue a bearer token that lasts 24 hours, dies with the sandbox and can be rotated, and every tour step shows its request as a curl to copy.
Where it runs: crates/contracts; the API Worker’s route table; the reference at /docs/api/.
See it yourself: Issue a token in the console, copy any tour step’s curl and run it. The API reference.

Read more: no lab page yet · ADR 0013 · Why the console is just another client

A breaker that reads the platform’s meter

The problem: A loop that no visitor started, such as an alarm that re-arms itself, can spend the account’s money in hours, and the bakery’s own counters do not see it. Cloudflare’s budget alerts arrive a day later.
The technique: A circuit breaker driven by the platform’s own meter: a cron job reads the last complete 15-minute slot of usage, prices it, and trips in steps.
How it works here: Every 15 minutes the API Worker asks Cloudflare’s GraphQL Analytics API for five meters, prices them at the overage rates and hands the reads to the Control object, which decides with a pure function in the same transaction that raises the flag. A window over a line sets read-only, and a window past four times the dollar line sets off and enables a pre-built WAF rule. The state never goes down by itself: a person resets it, because an automatic reset would let a loop start again.
Where it runs: The 15-minute cron in the API Worker; the breaker_state table in the Control object; the Rulesets API; the Breaker line on the status page.
See it yourself: Open the status page and read the Breaker line, then ask GET /v1/status for the breaker’s last run and the meters it could not read. The status page.

Read more: no lab page yet · ADR 0010 · Why the bakery can switch itself off before the bill does