Why the CRM can be thrown away and rebuilt

The bakery’s customer list, timelines and campaign statistics can be emptied and computed again from a log while orders keep arriving, and the result is the same bytes as before. This is what has to be true for that to work, how the rebuild is paced, and what the lab shows of it.

The problem

On Monday morning someone finds that the rule counting a customer’s orders was wrong. Every customer row, timeline and campaign statistic derived with it is wrong too, and none can be edited into the right number; each has to be worked out again from what happened. Orders are arriving all the while.

What the lab shows, and what it cannot

The Backfill Lab, as read on 2026-11-06, is a week of a nightly job counting a made-up bakery’s orders. A load fails halfway, and each day’s report is checked against the source. Its result is that a run which replaces its day’s rows in one step can run any number of times and leave the same report, while one that appends leaves 275 where the source has 179. That is the first half of this essay.

The lab replays nothing from its event log, has no cursor or batch size, and has no object answering other requests meanwhile. It cannot show an alarm, a transaction in one object’s database, or a campaign send that has to wait. The bakery adds those.

What makes a projection disposable

The Crm object keeps two things apart. The projection is the customer list, timelines, activity feed, lead pipeline and campaign statistics, held as one stored row. The log is a table of every event that changed the projection, in the order applied. A projection is disposable only if folding the log again gives the same projection. Two rules make that so.

Applying an event is idempotent. The projection remembers the id of every event it has applied, and a second delivery is answered duplicate and changes nothing. Every application adds a line to a customer’s timeline, so a redelivered event would write its line twice, and the Queue delivers at least once. The count of a customer’s orders would survive, because it is worked out afresh from a table of orders on each change. The timeline and the bytes would not.

Applying an event is ordered per stream. For the streams of one order, one catering inquiry or one customer, the projection keeps the newest sequence number it has applied and ignores any event not newer. The Crm receives only some of each stream, so gaps are not waited for, but a late event with a lower number is answered stale. It is neither applied nor logged. That is what lets a replay reproduce the projection: the log is in applied order, and applying an entry depends only on what the projection holds at that point. Without the rule a late event would be applied and logged, and a timeline could move backwards, with an order shown as placed after it was shown as ready.

Streams that carry many things, such as the inbox, need no sequence rule: their events add to sets and time-sorted lists, which come out the same in any order.

How the rebuild runs

A rebuild starts from the console’s Rebuild projections button, or the rebuild route of the API. The object keeps the projection as it stood, called the shadow, resets the live one to the roster, sets a cursor at the start of the log and arms an alarm. Each alarm replays the next 200 log entries in one transaction, saves the cursor and sets the next alarm 300 milliseconds on. The batch size is REBUILD_BATCH in crates/domain/src/crm.rs, and the gap is REBUILD_GAP_MS in the object.

The batches keep the object usable. Cloudflare’s documentation said on 2026-11-06 that Durable Objects are single-threaded and cooperatively multi-tasked, like code in a browser. The bakery’s reading, in ADR 0012, is that long work in one piece holds an object’s other requests behind it. Between runs of 200 entries the object answers them. Each run is also its own alarm, and an alarm handler had 15 minutes of wall time on 2026-11-06, which a run of 200 entries does not approach.

While a rebuild runs, four things hold.

Reads answer the partial projection and say so with the header Projection-State: rebuilding.
A delivery is decided against the shadow, and if applied is appended to the log without touching the partial projection. The replay reaches it in its turn, and the cursor’s total grows to match.
A campaign send waits. It reads the projection to choose its recipients, and a partial projection would choose the wrong ones. The alarm logs sends_deferred and the send goes on the first alarm after the rebuild.
A second rebuild is refused with 409 rebuild. It is the one command that cannot run during a rebuild. The same key again is a replay.

The proof is the bytes

At the end the object appends ProjectionRebuildFinished with the count of entries and the milliseconds taken, drops the shadow, and logs rebuild_finished with identical: whether the new projection’s serialized form equals the shadow’s, character for character. On 2026-11-05 the roadmap records the smoke script reading the customers, a timeline, the segments, the campaigns, the leads, the activity feed and the stored projection row before and after a rebuild of a 1,033-entry log, in batches of 200, and finding them byte-identical, with identical: true in the Worker’s log. The seed’s history is 722 closed-order events for seed 42 (docs/roadmap.md, PR 8, as of 2026-11-05). A property test of 256 cases delivers events in any order, with duplicates, and compares the projection with a fold of the log it applied.

That shows the replay agrees with what it replaced, not that the rule is right: a bug in the rule is reproduced faithfully.

What it costs

A rebuild reads each log entry once: 1,033 rows for the log above, in 6 runs, five of 200 and one of 33 (as of 2026-11-05). Each run also reads and writes the one row holding the projection. The bakery has not measured the rows Cloudflare bills for a rebuild, and the pricing page does not say whether a query that skips earlier rows counts them, so 1,033 is the least. For scale, on 2026-11-06 the paid plan included 25 billion rows read a month, and 1,033 is about four millionths of one percent of that. The wait is at least five gaps of 300 ms, 1.5 seconds, before the work itself.

The bound is the loop guard. An object that re-arms its own alarm more than 100 times in a UTC day stops arming until the wipe and logs alarm_guard with rearm-cap (REARM_GUARD in stream.rs; docs/operations.md, as of 2026-11-05). Each run is one re-arm, so at 200 entries a run the guard allows about 20,000 entries a day, less the outbox’s own re-arms. A longer log would, as the code reads, stall the rebuild partway until the wipe. A sandbox, wiped nightly, does not come near it.

The cost to a reader is the window: during a rebuild the customer list is partial, and the API says so with a header.

What to try

Open the Backfill Lab, choose Overwrite the day, fail a load, and rerun it; then choose Append rows and see the total pass the source. Then press Rebuild projections in the console, watch the cursor climb in steps of 200, read the stats before and after, and try to start a second rebuild while the first runs. The summary on How it’s built says the same in short.

Sources

Durable Objects alarms, Cloudflare’s documentation, read on 2026-11-06: one alarm at a time for an object, at-least-once execution, retries with exponential backoff from 2 seconds, up to 6.
What are Durable Objects?, Cloudflare’s documentation, read on 2026-11-06: an object is single-threaded and cooperatively multi-tasked.
Durable Objects limits, Cloudflare’s documentation, read on 2026-11-06: an alarm handler’s wall time of 15 minutes, and 30 seconds of CPU a request by default.
Durable Objects pricing, Cloudflare’s documentation, read on 2026-11-06: 25 billion rows read a month included on the paid plan, then $0.001 per million.
ADR 0003, SQLite-backed Durable Objects as the system of record, the bakery’s decision record, read on 2026-11-06: projections live in their own objects and can be dropped and rebuilt from a log.
Backfill a week of reports, the Backfill Lab on ihaveasegway.com, read on 2026-11-06: a nightly job backfilled in your browser, with a failed load and a rerun to try.

Back to the rebuild principle on How it’s built, or to the lab.