Generated from the test run · 24 October 2026, 16:45
Every line below comes from a test driven against the live fleet over real HTTP and asserted by its effect. Nothing here is written by hand — this page can only claim what was actually measured.
The suite reports 86 of 90 documented features passing. That figure counts only what has a check — the 1 cutover gaps below are counted here so an absent feature cannot hide by never being tested. The headline above uses ALL verdict rows as its denominator (86 fully passing of 93) — 2 degraded row(s) answer correctly but miss their stated target, and they count against the page, not out of it.
What the test foundB5 ownership gate: stranger register HTTP 403 (want 403 FORBIDDEN)
MeaningThe door exists and did the wrong thing.
What the test foundB7 pre-rotate whoami dead (HTTP 401) — cannot test rotate without a working key
MeaningThe door exists and did the wrong thing.
What the test foundSSE open before write did not receive data: with key. out=event: message
MeaningThe door exists and did the wrong thing.
What the test foundB8 cutover — HTTP 401: {"ok":false,"error":"unknown key","code":"UNKNOWN_KEY"}
Meaningv1 has this and v2 does not. Blocks the cutover.
What the test foundcould not register a job to delete (000) — untestable
MeaningCould not be proven this run — NOT a pass.
What the test foundpair de->fi (Europe-Europe informational keyed GET; does not bind the 10s ceiling); 2995ms inside ceiling, past 2000ms target
MeaningAnswered correctly but missed a stated target. A real measurement, not a pass and not a failure.
What the test foundpair fi->ash (Europe-Singapore keyed GET); worst-of-3 7467ms (7467, 3470, 2500) — inside the 10000ms maximum but past the 2000ms target. pair fi->ash (Europe-Singapore keyed GET); worst-of-3 7467ms (7467, 3470, 2500); poll 200ms; target 2000ms; ceiling 10000ms.
MeaningAnswered correctly but missed a stated target. A real measurement, not a pass and not a failure.
What the test foundregistered, AND tools/call wrote a record that the ordinary door reads back; unknown tool refused
MeaningDriven through real HTTP and asserted by effect.
What the test foundauth/me returned public_id
MeaningDriven through real HTTP and asserted by effect.
What the test foundKEY whoami 200 on peer 46.225.15.249 (not 89.167.52.85)
MeaningDriven through real HTTP and asserted by effect.
What the test foundsuite KEY (pre-dating this binary) still resolves on 89.167.52.85
MeaningDriven through real HTTP and asserted by effect.
What the test foundaccepted receipt on target after 1s of a 30s wait — NO deliver tick called (fan-out is after-send by design)
MeaningDriven through real HTTP and asserted by effect.
What the test foundgrant recorded and listed (LEDGER only — no charge path built)
MeaningDriven through real HTTP and asserted by effect.
What the test foundidentical bytes → same sha256; different → different (h=f026ae1c178f2827585083e89b5757ede99ebd981b6dd63290f4d9d04dc11fb6)
MeaningDriven through real HTTP and asserted by effect.
What the test foundupload produced listable/queryable metadata
MeaningDriven through real HTTP and asserted by effect.
What the test foundfilter payload._blob.type found png upload
MeaningDriven through real HTTP and asserted by effect.
What the test found206 with full size in Content-Range: Content-Range: bytes 0-3/28
MeaningDriven through real HTTP and asserted by effect.
What the test foundqueue reached "failed" for an unreachable endpoint — honest, not "delivered"
MeaningDriven through real HTTP and asserted by effect.
What the test founddated record found by route scan (bare GET returns an empty set + a hint — door-only would pass on nothing)
MeaningDriven through real HTTP and asserted by effect.
What the test foundlisted for sale, order minted with a Revolut hosted payment link (payment itself not exercised by the suite; live purchase verified 2026-09-01)
MeaningDriven through real HTTP and asserted by effect.
What the test foundqueue reached "retrying" for an unreachable URL — honest, not "delivered"
MeaningDriven through real HTTP and asserted by effect.
What the test foundACAO present on 200 (health) and on the error class (entity 404)
MeaningDriven through real HTTP and asserted by effect.
What the test foundACAO present on the held text/event-stream head
MeaningDriven through real HTTP and asserted by effect.
What the test foundkeyless OPTIONS /v2/drop 204; Allow-Methods POST, OPTIONS; Allow-Headers has X-Ocean-Key; no Allow-Credentials
MeaningDriven through real HTTP and asserted by effect.
What the test found3 incrs = value 3 across 3 shards (append-only)
MeaningDriven through real HTTP and asserted by effect.
What the test foundREAL routes: MIRRORED cc/log/io-cc: batch w/ash/s/34569/b/10 r2 436B | b2 436B, b2 landed 0s after r2 (ceiling 1800s);MIRRORED cc/log/io-v1-cc: batch w/de/s/24843/b/15 r2 623B | b2 623B, b2 landed 1s after r2 (ceiling 1800s);DISKCACH
MeaningDriven through real HTTP and asserted by effect.
What the test foundno starvation signature (tombs=2415 handled=0 deadline=false)
MeaningDriven through real HTTP and asserted by effect.
What the test foundexport contained the record in all 12 samples across 3 min (no fold-window hole)
MeaningDriven through real HTTP and asserted by effect.
What the test foundwrite receipt includes drop_id
MeaningDriven through real HTTP and asserted by effect.
What the test foundqueue reached "sent" for an undeliverable address — honest, not "delivered"
MeaningDriven through real HTTP and asserted by effect.
What the test foundREAL corpus via the listen door: cc/log/* 20 events, no duplicate keys, complete flag present (route stamping is the /v2/query door's promise, tracked as its own gap). Seeded fresh-prefix note: 0 rows in 30s = new-route fold lag, claim-leg class, exempt per 0bbde9f ruling
MeaningDriven through real HTTP and asserted by effect.
What the test foundfound by field=colour
MeaningDriven through real HTTP and asserted by effect.
What the test foundPUT+GET byte-identical and Content-Type image/png round-trip
MeaningDriven through real HTTP and asserted by effect.
What the test foundenvelope carries next_cursor (object envelope, not the v1 bare array)
MeaningDriven through real HTTP and asserted by effect.
What the test foundsave x:1 readable; find refused with scope — dsl_save slice
MeaningDriven through real HTTP and asserted by effect.
What the test foundkeyless GET was 200 before erase and 404 after — public copy purged
MeaningDriven through real HTTP and asserted by effect.
What the test foundbatch FOUND with erase-me-ft1792851422 before erase after 2s wait (check started 14:27:36Z); secret gone from w/fi/s/24460/b/27 after 16s
MeaningDriven through real HTTP and asserted by effect.
What the test foundpublished copy stopped serving after erase (HTTP 404)
MeaningDriven through real HTTP and asserted by effect.
What the test foundpeer 46.225.15.249 (took neither the write nor the grant) served the written bytes 1s after the grant reached it — inside the 10s ceiling
MeaningDriven through real HTTP and asserted by effect.
What the test foundDELETE 200 and the page is 404 anonymously — the published copy was purged
MeaningDriven through real HTTP and asserted by effect.
What the test foundgrant reports published=1 skipped=0 for the one _body key
MeaningDriven through real HTTP and asserted by effect.
What the test foundbodyless route: share applied, backfill finished with nothing_publishable:true (rule-first status, not a 422)
MeaningDriven through real HTTP and asserted by effect.
What the test foundsubkey PUT @view 403, doomed2.html still served with the same bytes, @view still on the rule
MeaningDriven through real HTTP and asserted by effect.
What the test foundanonymous GET returned the written bytes exactly
MeaningDriven through real HTTP and asserted by effect.
What the test foundworked (200), revoked (200), then refused (401)
MeaningDriven through real HTTP and asserted by effect.
What the test foundbacklink resolved
MeaningDriven through real HTTP and asserted by effect.
What the test foundRFC 9728 protected-resource AND authorization-server metadata served
MeaningDriven through real HTTP and asserted by effect.
What the test found3 approvals from one key counted ONCE, threshold not reached
MeaningDriven through real HTTP and asserted by effect.
What the test foundunauthenticated write refused 401 and no key minted
MeaningDriven through real HTTP and asserted by effect.
What the test foundunregistered key refused 401 — a key exists only because an account issued it
MeaningDriven through real HTTP and asserted by effect.
What the test foundowner recorded on claim
MeaningDriven through real HTTP and asserted by effect.
What the test foundover-limit returns PAYLOAD_TOO_LARGE
MeaningDriven through real HTTP and asserted by effect.
What the test foundenumeration on peer 46.225.15.249 includes the record, not just the keyed read
MeaningDriven through real HTTP and asserted by effect.
What the test foundforeign key refused (401)
MeaningDriven through real HTTP and asserted by effect.
What the test foundrecord written, erased, and no longer readable
MeaningDriven through real HTTP and asserted by effect.
What the test foundB10 wire+sandbox: 17+25=42 and health.runner true
MeaningDriven through real HTTP and asserted by effect.
What the test found@view applied AND served after compaction
MeaningDriven through real HTTP and asserted by effect.
What the test foundgroup_by team summed correctly (a=6) — grouping actually ran
MeaningDriven through real HTTP and asserted by effect.
What the test found$ne excluded red, kept blue
MeaningDriven through real HTTP and asserted by effect.
What the test foundfrom timestamp accepted
MeaningDriven through real HTTP and asserted by effect.
What the test foundleft join attached remote fields; unmatched row kept; 2 resolves for 2 distinct ids
MeaningDriven through real HTTP and asserted by effect.
What the test foundlast N accepted
MeaningDriven through real HTTP and asserted by effect.
What the test foundreason=ok prefix with /* matched sub-route in 1s after child was keyed-got, expanded on peer 46.225.15.249
MeaningDriven through real HTTP and asserted by effect.
What the test foundwritten and read back
MeaningDriven through real HTTP and asserted by effect.
What the test foundattach door wired: garbage token refused 401 INVALID_TOKEN, nothing minted
MeaningDriven through real HTTP and asserted by effect.
What the test foundpager healthy over 40 page(s) at limit=200 (7850 unique routes, every page added rows, cursor never repeated). Did not reach mine_complete=true within the 40-page bound — that is account size, not a stalled pager; the defect cases are checked separately and none fired.
MeaningDriven through real HTTP and asserted by effect.
What the test foundempty segment rejected HTTP 400
MeaningDriven through real HTTP and asserted by effect.
What the test found19+23 computed to 42 by the sandbox — real execution, not an echo
MeaningDriven through real HTTP and asserted by effect.
What the test foundecho returned the marker; off-list op rejected 400 (allowlist holds)
MeaningDriven through real HTTP and asserted by effect.
What the test foundevent returned when polling from before the write, absent when polling from after
MeaningDriven through real HTTP and asserted by effect.
What the test foundfired (owned_here=true window=~96s same-node)
MeaningDriven through real HTTP and asserted by effect.
What the test founddeep string and entity key both searchable
MeaningDriven through real HTTP and asserted by effect.
What the test foundorder flipped after term removed from hi (was hi → lo)
MeaningDriven through real HTTP and asserted by effect.
What the test foundBM25-order zzz_short first + score present non-constant (score=0.3424)
MeaningDriven through real HTTP and asserted by effect.
What the test found202, identical body by construction
MeaningDriven through real HTTP and asserted by effect.
What the test foundtype refused at define; presence validate ok; missing city named
MeaningDriven through real HTTP and asserted by effect.
What the test foundSELECT n omits tag (not lazy *); key,n DESC LIMIT1→k3; empty+precedence — slice6
MeaningDriven through real HTTP and asserted by effect.
What the test foundfrom=0 accepted
MeaningDriven through real HTTP and asserted by effect.
What the test foundfrom=ISO accepted
MeaningDriven through real HTTP and asserted by effect.
What the test foundfrom=latest accepted
MeaningDriven through real HTTP and asserted by effect.
What the test foundlast=N accepted on listen
MeaningDriven through real HTTP and asserted by effect.
What the test foundlisten ft1792851422/sw/*?last=20 returned rank+live+events/a keys in 181640ms (convergence time, NOT a target; observed band 22.8s-291s, liveness bound 900000ms = 2 folds + margin; visibility is position-in-pass, so this number is where the prefix sat in the pass) window last=20 on isolated prefix, expanded on peer 46.225.15.249; polls=55
MeaningDriven through real HTTP and asserted by effect.
What the test foundfilter/sort/limit/select effect-true; join refused with scope — B4 s1
MeaningDriven through real HTTP and asserted by effect.
What the test foundminted key wrote (200), revoked key rejected (401)
MeaningDriven through real HTTP and asserted by effect.
What the test foundsum(amount)=10
MeaningDriven through real HTTP and asserted by effect.
What the test foundtemplate stored and returned with its body intact
MeaningDriven through real HTTP and asserted by effect.
What the test foundenrolled already, so transition unprovable — but confirm REFUSED both a malformed (401) and a well-formed (401) wrong code, and status did not move
MeaningDriven through real HTTP and asserted by effect.
What the test foundnumeric code 400 naming leading zeros; 5 wrong codes 401 INVALID_CODE; 6th 429 — attempt cap holds by effect (happy path: proven by hand 2026-10-12, not by this row)
MeaningDriven through real HTTP and asserted by effect.
What the test foundcurrency defined and credited, visible in balances
MeaningDriven through real HTTP and asserted by effect.
What the test foundmarked route, text query embedded (mode="mode":"hybrid", total_vectors>0), fruit doc first
MeaningDriven through real HTTP and asserted by effect.
What the test foundcosine ranked d1 first: {"ok":true,"route":"ft1792851422/vec","mode":"vector","index_type":"kvec_stream","vector_s
MeaningDriven through real HTTP and asserted by effect.
What the test foundboth versions returned ("returned":2)
MeaningDriven through real HTTP and asserted by effect.
What the test foundwhoami returned public_id
MeaningDriven through real HTTP and asserted by effect.
Everything above is measured. This section is not. It is the plan for a new v2 dashboard, put here so Niklái can mark it up before anything is built.
It is a new dashboard rather than a rewiring of the old one, on his call: customers still have v1 routes and apps, so the v1 dashboard keeps working against v1 while this grows beside it, and v2 takes over everywhere once migration completes.
Live now: view2.infiniteocean.io/dash/index.html
12 of 12 panels
are BUILT — not "the doors exist", but the dashboard actually calls them, read out of
its own source at generation time. Every panel below has every door it needs; what is left
is screens. Which doors exist is parsed from api.zig’s
registry as of the last commit when this page is generated — not the working tree, so
work in progress cannot show as ready. A door landing tomorrow moves a panel without anyone
editing the plan. The panels and the questions are mine.
Magic link in, identity out. CORRECTED 12 Aug, after building it: this list used to name /v2/auth/consume, which is a NODE-INTERNAL door requiring the fleet key — a browser must never call it, so the panel could never have counted as built. The browser-facing pair is /v2/auth/request then /v2/auth/redeem?token=. A measurable that cannot go green is a gate that always fails.
doors that exist/v2/auth/request, /v2/auth/redeem, /v2/whoami
in the dashboardyes — this panel calls those doors today
Handle, public id, and the addresses that can sign in. A lost KEY is recoverable — rotation preserves the public id, so a magic link restores ownership. A lost ACCOUNT is not, which is why this panel says plainly when only one address can sign in. Binding a second proves control first ({email, token}) and detaching refuses the last one.
doors that exist/v2/auth/me, /v2/auth/emails
in the dashboardyes — this panel calls those doors today
Niklái’s ruling: an agent gets a key because the ACCOUNT issues it, so the user pays and is responsible. The agent generates its keypair locally and hands over only the public half. Panel shows label, key id, created, and revoke.
doors that exist/v2/agent-keys
in the dashboardyes — this panel calls those doors today
Scheduled jobs, webhooks and triggers. The cron list was blind behind its own tombstones until yesterday; it lists correctly now.
doors that exist/v2/cron, /v2/hooks, /v2/triggers
in the dashboardyes — this panel calls those doors today
The core view: what is in a route, and edit a record. Query gives filter and sort; history gives versions.
doors that exist/v2/list/, /v2/entity/, /v2/drop, /v2/query
in the dashboardyes — this panel calls those doors today
Custom domains, and publishing a route as a site via the @view share.
doors that exist/v2/domains
in the dashboardyes — this panel calls those doors today
Blobs live under b/{route}/id, so they inherit the route’s ACL and the same sharing control. Upload, list, public toggle.
doors that exist/v2/blob/
in the dashboardyes — this panel calls those doors today
DECIDED and being built — Niklái, 12 Aug: “a list/index that gets added to/edited on every share/create. This should live on R2/B2.” Two routes carry it: __cfg__/rd-owner/{pid} for routes you own and __cfg__/rd-shared/{pid} for routes shared with you, one small drop per route, keyed by the route name. Per-route rows rather than one rewritten list per person, because a single blob is a read-modify-write: two shares landing at once and one silently wins — the manifest lost-update bug we have already had — and the rewrite costs more the more routes you own. This way a share is one append, an unshare is an ordinary tombstone, and reading your routes is O(a page). Written at the single choke point (putRuleLocked) so create, grant, revoke, expiry and ownership transfer are all caught in one place and a future grant path cannot forget — and emitted after the ACL lock is released, never across an R2 round trip.
doors that exist/v2/routes
in the dashboardyes — this panel calls those doors today
Full-text over a route. Vector search is NOT in v2 — v1 held every route vector in RAM and that shape does not ship.
doors that exist/v2/search/, /v2/index/
in the dashboardyes — this panel calls those doors today
Niklái: “a unified way to share things and manage sharing, so it always looks the same.” This is already ONE mechanism underneath — files and media live under routes, so the same ACL governs them. Four doors: see how a thing is shared, transfer owner, add a share, remove one. Build the control once and reuse it on every object.
doors that exist/v2/access/
in the dashboardyes — this panel calls those doors today
Doors built and rolled 12 Aug. Buying access grants a SHARE with an expiry, so cancellation needs nothing to happen — the clock ends access. It cannot take money yet, and says so: with Revolut unwired, POST /v2/checkout answers 503 CHECKOUT_UNCONFIGURED and an unsigned webhook is refused. That is deliberate — the test-only fallback would otherwise have returned a cheerful 200 with a checkout URL that takes nothing. SETTLEMENT DECIDED — Niklái, 12 Aug: “we’ll start by keeping the money for all the ones that sell on the platform and then they will invoice us for their share.” So there is no payout rail to build, no split, no seller KYC. What is needed instead is a per-seller statement and a way to mark an invoice settled; the ledger already records gross, cut and seller-net on every order at confirmation.
doors that exist/v2/checkout, /v2/billing
in the dashboardyes — this panel calls those doors today
Distinct actives per day and signups per month, plus per-route usage already recorded on every write. Allowlisted by pid and FAILS CLOSED — platform-wide numbers are not something every customer should read. Deliberately NOT here: a live feed of everything happening, which is the one panel whose cost grows with success.
doors that exist/v2/metrics, /v2/health
in the dashboardyes — this panel calls those doors today
Niklái’s rule 8 — scales, or it does not ship: bounded work per unit of NEW data. These are paths whose cost grows with how much has ever been stored rather than with how much arrives. A feature list can read 74 of 76 while these are open; that is why they are on the same page. Hand-written — the feature suite does not measure them. Every number below was measured on the live fleet and is dated.
AND THE DENOMINATOR IS NOT ESTABLISHED. Fourteen is what an audit found; nobody has shown it is what EXISTS. It was produced by reading paths already suspected, and on 15 Aug that method demonstrably missed one: GET /v2/listen/{route}/* takes 8–10 seconds on a live node — against 0.86s for the exact route and 0.97s for the query wildcard doing the same expansion. It was missed because the query variant was audited and the listen variant sharing its shape was not. The audit names four request-path doors out of 109 registered ones; listen, export, list and routes appear nowhere in it. So “all fourteen closed” and “it scales” are not the same claim, and this page will not print the first as if it were the second. Whether the sweep was systematic or targeted is an open question put to the agent that ran it. It has since answered: targeted.
THE DENOMINATOR NOW EXISTS, AND IT IS 28 — NOT 14. A separate read-only pass over the registry on 16 Aug counted it rather than estimating it: of 110 registered doors, 36 can expand, scan, list or walk — touch more than one object to answer one request — and of those, 28 have a cost set by what is stored rather than by what is asked for: 19 data-volume doors plus 9 fleet-catalogue ones. The fourteen rows below are therefore a sample of 28, and stopping here leaves 16 named enumerators unexamined — among them list, export, history, summarize, calendar, metrics, and both the listen and broadcast polls. The agent that proposed this pass guessed “a dozen or so”; the count is nearly three times that. The two paths we found by accident are on that list. The listen wildcard and the stream wildcard were not bad luck — they sat inside a population nobody had enumerated, which is a fact about the method rather than about those two doors.
AND THE 28 MAY NOT BE 28 PIECES OF WORK. The same pass found two shared primitives underneath many of these doors: read.enumerate, which merges uncompacted plus tip on every page, and deliver.storeEnumLive, which is O(all entries). If that holds, fixing two primitives closes a block of rows at once — and, less comfortably, any row closed at the door while those primitives stand is a cap rather than a close, which is rule 2 and would make some of the fourteen below weaker than they read. Which doors reduce to which primitive is the question now open. One honest limit on all of the above: it is classification read from the registry and the code, not measurement — the 28 are candidates, and some will prove bounded on inspection.
AND THE ENGINE HAS NOW BEEN MEASURED — IT IS BOUNDED BY WHAT IS STORED AND NOT BY WHAT IS UNFOLDED. Rather than audit fifteen doors, we measured the one thing under them, at two scales. Twenty times more stored keys produced identical read counts — 3 cold, 1 warm, either way — and flat time, 29µs against 36µs. So a listing does not get more expensive because a route has more history, which is the thing rule 8 actually asks. But ten times more un-compacted data took it from 23µs to 1,944µs. Cost tracks the compaction backlog. That single result joins the two halves of this page: if listings are O(un-folded), then fold health is not a separate correctness errand — it is the performance of fifteen doors, and the compaction repair in rows 7 and 9 is the scaling work. Stated with the caveats the measuring agent volunteered rather than had extracted: it was an in-process fixture, not the live fleet, so it measures the algorithm and not the deployment; and nothing was measured above four thousand keys, which is not a scale test. The index header may still grow with block count at sizes nobody has tried.
AND A GREEN ON THIS PAGE CAN BE A GREEN AT ONE ARBITRARY SCALE. The feature suite measures each door once, against whatever fixture the run happened to build. For a door whose cost grows with what is STORED, that verdict depends on a size that is incidental rather than chosen. The wildcard listen is the worked example: the same door PASSES at one child route and TIMES OUT at thirty, so a pass there would mean the fixture was small that night, not that the door was bounded. On 15 Aug three separate measurements were reported as evidence and none of them could have produced the failing answer — a 404 timed at 0.23s, an ACL rejection timed at 0.0004s, and a one-child fixture that met its target. Each was caught by whoever measured next, never by whoever measured first.
16 of 16 closed. 0 have nobody working on them.
Every negative answer re-parses a tip and walks every ref. The fail-open half is CLOSED (a transient read no longer answers “not erased”). The growth half is NOT: measured 749 refs at 08:30, 764 at 12:00, 920 by 15:00, and the walk rate doubled in the same window. The keyed replacement is built and the cutover door is committed and shut.
Now VERIFIED, and the answer is that the cost is not there rather than merely smaller. It was 321 of 321 entries tombstoned — 100% dead work, walked every 5 seconds. Measured 2026-08-17 on the live fleet: __deliver__/pending/idx is 377 bytes and 3 keys, all exact tombstones, last_entries=5 and last_tombs=5 on all five nodes, and six hours of enumerate-done lines on de ranging 5 to 122 — never thousands. The tomb horizon is being written. Decision: do NOT rewrite the enumerator. The bad shape is still in the code; the honest reason not to act is that the current cost is five keys. Watch for re-growth rather than building a walker for a number that is not there.
THIS ROW SAID “nobody has built one” AND THAT WAS FALSE ABOUT THIS TREE. The due-time index exists: c406c44 “cron: due-set so FIND is not O(all keys) every five seconds”, committed 2026-08-16 and an ancestor of HEAD, rolled. A row claiming work is unstarted when it is already shipped is the mirror of the un-switched-replacement failure this project has a freeze about — it invites someone to build a second one. It cost nothing only because the agent assigned to it read the tree instead of the row. The original numbers were real: 323 entries re-fetched every ~22s, 3,609 GETs returning SEVEN BYTES in 15 minutes, ~1.7M R2 requests/day with no customers, and R2 reads fell 291→191/min. THE ACTUAL RESIDUAL IS A CORRECTNESS BUG, not a cost one: dueSetUpsert is a read-modify-write — loadCoveringDueSet, edit, saveDueSet — with no If-Match anywhere in cron.zig. Two nodes upserting concurrently lose one update. And the failure does not self-heal, because the two cases are asymmetric: a MISSING artefact is a no-op and “the next tick pays the one scan”, but a STALE one looks valid, so nothing rescans. A lost update is a scheduled job that silently never fires. Found by io-lab-grok-cc reading the file; confirmed here at cron.zig:180-205.
A single GET parsed ALL local uncompacted batches regardless of route — one node hit 10,929 batches per read when compaction fell behind. Now skipped by route before any fetch.
Publication waited behind every other dirty route: 12 workers draining a 191-deep queue is ~9.5s with nothing wrong, against Niklái’s 10s ceiling. A just-written route now claims next.
The same prefix that was 202,731 objects and 249.7 seconds on 10 Aug. The LIST is deleted; it reads pointers instead.
THE ROW TEXT CITED THE WRONG FUNCTION, and correcting it matters more than closing it. The 10,929-bodies-parsed-every-pass figure comes from compact.zig:1732-1794, which is the rebuild/erase path and LISTs w/ — not the cadence local arm at ~2266-2453, which already has a route-list cache. Live backlogs measured 2026-08-17: de 24, fi 26, ash 142, hil 36, sin 22 uncompacted, local_r2=0 every pass on every node. Pass time is R2 round trips, not a RAM parse of 142 batches. Still open: ash still puts ~128 local bodies through the mem path each pass, and cache-hit versus due-reopen needs a counter nobody has added — so the residual is unmeasured, not absent.
A PUT costs the same at any size, and Class A operations are 5% of our R2 calls and 43% of the bill. Small dirty routes no longer fold every 10 seconds.
THE MOST SERIOUS OPEN ROW, and it is worse than the title says: the invisibility is PERMANENT, not a window. When a batch is marked compacted its tip ref is dropped; if the owner has not folded, the route leaves every candidate set and no future fold will ever pick it up. Proven on one route end to end — written 02:32, still unreadable on all five nodes eight hours later, bytes confirmed safe in R2 at w/fi/s/579/b/6, its tip reading {"batches":[]} and its idx 404. 112 of 4,475 tip-bearing routes are in that state (owners sin 46, hil 30, ash 24, fi 12, de 0 — against a uniform population, so de=0 is signal). Names are test artefacts, no cc/ routes, but that is a statement about names and not about bytes. Two separate fixes needed: the gate that stops NEW cases is built and closed by default, and the repair engine for existing ones is built and not yet run. Durability holds throughout — this is readability, not loss.
CLOSED 2026-08-18, and what it cost is part of the answer. The strand is gone: a batch now marks only when every route on it is in this pass’s folded set, and the whole foreign-mark path went with it — grace window, presence probe, and FOREIGN_PROBE_BUDGET, 316 lines out, 33 in. The old mechanism marked on an index exists, which is not this batch was folded; that inference was the defect. A second fix followed, because closing the strand made mixed-owner batches unmarkable anywhere: a sealed batch now belongs to exactly one owner, chosen at append by the same ownership function the fold uses. Measured after the roll: mixed=0 on all five nodes.
ONE CONSEQUENCE IS STILL OPEN AND IT IS NOT COSMETIC. A writer holds batches owned by other nodes that it can never retire — only the owner folds them, and only the writer can remove its own tip ref. Measured: de 714, ash 1,682 and rising ~20/min, and every compaction pass walks that set. Three designs died on the way to a fix: a watermark cannot exist because a writer’s sequence numbers scatter across owners, and a batch whose records all lose the timestamp compare writes no index at all. The surviving design adds a new durable object class to the core, which is a decision for Niklái rather than a change to make quietly. Until then this row is fixed in the sense that matters — no customer’s data goes permanently unfindable — and honest about the bookkeeping that leaks behind it.
MEASURED AND CLOSED 2026-08-18. The row’s premise is refuted by an order of magnitude: one proven-open SSE stream costs 0.32 tip fetches/s, not the 4/s this row claimed. If 4/s were right, three 20-second windows with a stream open would have shown ~240 extra fetches; they showed 19. holdSse already polls conditionally once per SSE_PEER_NOTICE_MS (2s) with an ETag, so unchanged polls are 304s, and it already reads localGen — the row said it never does. GETTING THIS NUMBER TOOK THREE FAILED ATTEMPTS AND EACH FAILURE IS THE POINT. Measuring from ops.r2_get gave +7.37/s in one window and −5.30/s in an alternating run — a negative cost is impossible, so that instrument could not answer at all: compaction noise (~15 GET/s) dwarfs the effect. tip_fetch_attempts counts exactly the call in question and was already incremented in tip.zig, but was unreachable from outside until it was exposed in /v2/health. Then the probe itself was wrong — it hit /v2/stream/{route}, which is a 404, and SSE requires the route be marked streaming first; the “stream” was an 86-byte error body. The control that caught it was itself buggy and counted every failed round as proven. Only a pre-flight asserting Content-Type: text/event-stream made the run mean anything. A number from a probe that never connected is not a small error, it is a fabricated one.
CLOSED 2026-08-18. Niklái: “Change the promise. We need indexes.” A route larger than the 200×50 page budget used to return sum-of-the-first-10,000 with truncated:true. A truncated LIST is a partial answer that can be resumed; a truncated SUM is simply wrong, and no caller can do anything correct with it. It is now refused with 409 AGGREGATE_NEEDS_INDEX — 409 and not 400 because the query is VALID and answers once the route is folded. THE PART THAT NEARLY SHIPPED A REGRESSION: scan_complete goes false for TWO reasons. The page budget stopping mid-route means rows were never seen. The enumerate failing to CERTIFY means every row WAS seen and the fold simply has not covered the route yet — measured, a 3-drop route reports complete:false until compaction folds it. Refusing on scan_complete would have 409’d every aggregate on freshly written data. A separate budget_exhausted flag is the only thing that refuses. Verified live: a fresh 3-row route still answers total:3, HTTP 200.
Closed 16 Aug, and closed by deletion rather than by a smaller number, which is the distinction this whole section exists to make. Ranking needed two things from the corpus: how many documents there are, and their average length. It got them by scanning up to fifty thousand per-document length records on the way to answering a search. Both now come from a small stats object maintained when documents are indexed, and an individual document’s length is a direct keyed read of that one document. Nothing walks the corpus. The cost of a search stops tracking how much has ever been indexed and starts tracking the query — which is bounded work per unit of new data, not a cheaper version of unbounded work. Search tests pass 7 of 7.
Now a keyed GET when idle.
THROWN 2026-08-17, and verified in both clouds: id/gdpr_erased/seed reads {"gen":1,"rows":298} from R2 and B2. 298 marks are durable. It had never opened, and the reason was arithmetic rather than configuration: the walk needs every one of 298 marks written to BOTH clouds, B2 returns 500 for a few percent of PUTs, and the walk restarted from zero each time — so it had to win ~298 consecutive coin flips. Four attempts aborted at four different keys. Making it resume was not enough on its own: it then oscillated, 119→19→231→201→2→197 durable across six runs, because inside a B2 failure cluster the resume read fails too, so marks that ARE durable read as absent and the walk halted near the start — each run measured the cluster it landed in, not the work it had done. Finishing the walk and refusing at the END converged in three attempts. Fail-closed is unchanged: no seed object is written unless every mark is confirmed in both clouds. The guarantee was never “stop at the first problem”, it was “do not open the door on an incomplete set”.
CLOSED 2026-08-18. Niklái, after asking whether the platform would still be truthful: “Let’s do it.” A wildcard listen now requires last=N, served from the __pfx__ index in O(N). since_ts across every descendant is refused with 400 WILDCARD_NEEDS_LAST. THE WITHDRAWN DOOR WAS THE LEAST TRUTHFUL ONE: ~1s at one child route, 14.5s at thirty, and one call in three hit the 15s ceiling — at which point it returned what it had gathered rather than what exists. A call that silently hands back a subset is a worse lie than one that refuses and says how to ask. THE INDEX CANNOT SERVE THE OTHER SHAPE, which is why this is a promise change and not a wiring fix: pfx encodes keys as maxInt(u64) − ts, so the artefact is stored NEWEST FIRST. NOTHING BECAME UNREACHABLE, and that was checked before agreeing rather than asserted after: /v2/{route}?cursor= paginates any route of any size, listen?after_key= pages one route, and /v2/query reports honestly (measured matched=7629 returned=5). Verified live on all three arms. Still open, and named before the decision rather than after: walking child routes needs their names, and GET /v2/routes is O(routes) and unmeasured.
Not found by the audit — found by a dead subsystem, which is the point of putting it here. Every node publishes a shared “tip” listing batches not yet folded. Folding removed an entry locally and then the republish copied it straight back from the cloud: the publisher merged in every cloud entry it did not have locally, and the entry it had just removed qualified. Removals could never reach the cloud. The tip was append-only by construction and had reached 907,380 bytes and 1,587 entries against a designed alarm threshold of 64. What it cost, and none of it looked like a scaling problem: delivery was dead on two of five nodes — hil ran its last pass six seconds after boot and none in the following hour, so webhooks, browser notifications and email were silently not running; GET /v2/routes took 87 seconds on one node and never answered on another; and de had restarted twice on its own. Three unrelated-looking symptoms, one cause. Why nothing caught it: the test named “fold removes entries” asserts only the local board, and its stub returns not-found so the republish path never executes — a test covering exactly the half that worked. And health reported the delivery tick as armed while zero passes ran, so a scrape saw a healthy node. Fixed: a node publishes its own board plus only its peers’ entries. Measured on the fleet after the roll: 907,380 bytes down to 356. The new test fails on the old code, which was checked both ways before it shipped.
This section is written by hand — the only part of the page that is, and it is labelled because everything else is generated. A correction can only ever make the picture worse, which is why it is allowed to bypass the generator; a claim of success never is.
v2 could never prove a key was unknown. It answered “unknown key — not authenticated” while having no idea whether the key existed — a guess that looked like a determination, on every node, since it shipped. It only appeared correct because refusing is the safe default. The cause, measured: Backblaze returns HTTP 500 on a ranged GET of a missing object — R2 returns 404 either way, B2 returns 404 unranged and 500 ranged — and the binary always sends a Range header. The classifier treats 500 as transient, which is correct (a real outage is also a 500), so absence was unprovable forever. Fixed by retrying the miss unranged rather than by calling 500 an absence: mapping 500 to not-found would have inverted the failure — proving absence during an outage, and for the erasure check that means treating an erased route as live. What it unblocked: three suite rows that could not previously be measured at all now pass on the real assertion — foreign key refused (401); worked, revoked, then refused (403); minted key wrote, revoked key rejected (403).
The key-rotation check returned 503 “store overloaded — old key left intact”. That message was false: the rotation completed its revoke, failed before publishing the successor, and reported that nothing had changed. Every credential on the fleet stopped working, twice — the second time because the suite was left running during the investigation, which proved the mechanism by accident. Recovery needed a fresh key minted through the sign-in flow. Same disease as two other doors closed today: the erasure check reported every failure as “not erased”, and the identity lookup reported every failure as “unknown key”. A doubt reported as a fact, three times — and this third one is destructive, because the thing it lied about was the credential.
The deploy script detected the authentication failure correctly and aborted with “the product does not work”. I read the wrong line of its output, saw the servers returning 200, and shipped anyway — twice. The gate could not say which failure it had caught, and a gate whose message names an unverified cause gets overridden by the person it is protecting. It now distinguishes a rejected key from an unread store, retries the transient case to a deadline, and names which one it hit. It caught a real fault within minutes of the change.
A node was OOM-killed every ~37 seconds, 272 restarts, unexplained for weeks. parseTip copied every batch key out of a parsed tip — about 97,376 tiny strings — onto the raw page allocator, and each rounded up to a full 4 KB page. ~53 MB of string data became ~389 MB resident. Why it hid: every instrument counts requested bytes — the app reported 80 MB while the process held 900. Page rounding is invisible to all of them. The fix DELETES the copy rather than moving it to a better allocator. Result: zero restarts fleet-wide, RssAnon 33–45 MB where it was stepping to 500.
The erasure check answered false — “not erased” — on a transient read, so a hiccup during compaction could leave an erased route’s history in place. Live on all five nodes, all day. It is now three-state: a transient read skips the batch and aborts one fold rather than guessing. This did not require the cutover. Price of the trade, asked for before it shipped: an hour of transient reads stops erasure reclaim for that hour; existing tombstones hold; nothing rematerialises.
Closing the fail-open did not make the check cheap. Every negative answer still re-parses a tip and walks every ref, and the erasure log only grows: 749 refs at 08:30, 764 at 12:00, 920 by 15:00 — and the walk rate doubled. Some growth is this suite erasing test routes; the cost curve is not. The keyed replacement is built and the cutover door is committed and shut.
The compactor cannot see segment-form batch objects. compact.zig:454 (and the orphan-repair fallback at :845) do parseInt(u64, ...) catch continue on the object name; w/{node}/s/{seg}/b/{n} has no seq at that position, so the parse fails and the object is skipped silently. The only segment-aware channel is a 64-deep ring, so any node writing more than ~64 batches faster than the fold gathers them orphans the excess permanently, by construction. Measured: 1005 rows written with every receipt counted (1005×HTTP 200, 127 batches durable, zero upload failures) enumerate as 661; the 344 absent keys form 18 contiguous ranges; every missing byte is still in R2. Durability held — the index is short.
A key absent from the index can still return HTTP 200, because getEntry serves the uncompacted merge and the tip before the index. It goes hard 404 only when those batches compact away. Verified three independent ways, including a raw SigV4 read of the index object by hand: cc/log/io-strategy-cc/idx is 1574 bytes and contains 2 keys against a census of 142, while 141 of 142 still read 200. So a soft hole passes the feature suite, passes the migration flip gate, passes a customer’s own read — and rots on its own schedule. Every number on this page is a point-in-time readability measurement and none of them can see this class. The only instrument that can is a direct index-versus-batches comparison.
FIXED AND VERIFIED: v1’s origin was reachable directly from the internet on hil, sin and ash — anyone could hit the customer-facing edge by IP, bypassing Cloudflare. de and fi were correctly locked down, so the right configuration already existed and three machines never got it. Now closed on all five, verified five ways: external :3003/:3004 refused everywhere, peer-to-peer still 200, local serving unaffected, api.infiniteocean.io untouched at 166 ms — and a boot-restore unit added, because iptables-save had written a file nothing was loading and the rules would have vanished on the next reboot while looking applied. RETRACTED, BOTH MINE: “+218 MB inside one tryTip” is dead — that function’s fetch and walk never ran; the memory moved on other threads in the same window, read through a process-wide gauge. And “compaction carries it” is void, not failed: I changed a node’s config while my own measurement ran against it, so the compaction-disabled process actually lived 13 minutes and was ended by me, while the 968 MB death was 30 minutes into normal configuration. Both the claim and my retraction of it were wrong. WHAT THE NIGHT ACTUALLY PRODUCED: fifteen candidate causes eliminated by measurement and none confirmed; the first honest idle baseline v2 has ever had (~20 MB at rest, with its remaining occupants named); and two process fixes that outlive the bug — a run is void if config changed mid-run, and a kill-count must be paired with a restart-count because zero kills since this process started is vacuously true on a process that just restarted. The memory bug is unsolved and has no date. Nodes now die in hours rather than seconds and recover through restarts; nothing loses data. The open question for cutover is not “does it work” but “can it stay up unattended”.
Fixed and verified: four memory defects in the fold and sweep paths, each disarm-verified; the pending-queue collector proven working (30,094 tombstones collected, delivery passes 886× lighter fleet-wide); the upload path taken off the critical path (8.5–17s → 1.8s per batch, fi’s drain 4.5h → 11min); an identity-takeover bug closed (an @handle could be stolen on a transient read failure — v2 only, v1 fails closed). ash, which had never uploaded a single batch, drained all 10,929 to both clouds and now runs production config with no diagnostic gates. Recovered on their own once folding was healthy, touching none of their code: fast lookups, summaries, backlinks — and GDPR byte-erasure, which had been hiding data rather than deleting it. THE NUMBER THAT NOW MATTERS: a drop written on one node takes 32–110 seconds to become readable on another, against Niklái’s rule of 10 seconds maximum, 2 seconds target. Diagnosed: the tip publisher is a single serial thread, and the fix that made uploads fast moved that cost here. Fix agreed (fresh-first ordering in the existing pool), not yet built. That is the cutover blocker now — it is the platform’s core promise, write anywhere and read anywhere. And the lesson that outranks any single fix: three times today a test passed while production failed, because the harness did not share the one property under test — a memory scope tested under a parent that frees while production used one that cannot; a sweep tested over a backend that cannot leak; a flush tested with a queue of one while production carries many. Each time the code was right and the test was blind.
The entry beneath this one says three of five nodes had completed zero folds in seven days and that the tomb GC had never run in seven days. Both were measured by grepping for log lines that did not exist for most of that window. fold done route= shipped in f20d589 at 11:21 today — roughly two hours before I counted its absence across a week. deliver pending tomb GC shipped in 931b9e4 at 23:34 yesterday, so its longest possible silence was 14 hours, not seven days. Counting the absence of a marker proves nothing about the thing it marks, which is a rule already written down in this project and walked into anyway, on a headline claim. What survives, on evidence that does not depend on a log line: nodes were OOM-killed roughly every 25s while a compaction pass measurably takes ~700s, so they could not have completed one — that is arithmetic. And hil’s recovery is read from its health endpoint, not its journal: uncompacted 155→0, candidates 10,390→24, 27 passes in 40 minutes, alive ~2h. The doom loop and the recovery are real. Two of the numbers I used to argue for them were not.
Measured across all five nodes at 10:45Z, with journal retention checked first so “ever” means six to seven days of continuous journal rather than a rotation artifact. In that window: the deliver pending tomb GC has run zero times on every node; the pending route has been folded zero times on every node; and ash, hil and fi have completed zero folds of any route at all — ash across 1,152 service starts, hil across 518. The loop: a compaction pass on de takes 400–3,164 seconds (measured wall_ms=3164009). A node OOM-killed every ~25s can never finish one, so its uncompacted batches grow, so every keyed read merges more batches, so it dies sooner. ash sits at 8,261 uncompacted; sin, outside the loop, at 8. The cost, on the node this page has been calling healthy: de walks entries=30128 tombstones to find five real jobs — wall_ms=113112 rss_mb=669, every pass, under a 1 GiB cap. Root cause: the tomb GC lives inside foldWholeBase2, so it only runs when compaction folds the pending route — which has never once been a candidate. The repair is gated behind an operation that never happens on the route it repairs, so it cannot bootstrap. Two fixes in flight. v1 has served normally throughout and no customer has been affected — this blocks the cutover, it is not an incident.
Why the collector never ran, exactly: compaction partitions folds by hash, and wyhash("__deliver__/pending") % 5 = 3 — computed independently, and de reports "index":1 from its own health endpoint, confirming the mapping. Node 3 is hil, and hil had completed zero folds in seven days across 518 restarts. de and sin skipping that route is the partition working correctly, not a bug — an earlier reading of this page blamed “never a candidate” from measurements taken on de, which is the wrong node to ask. One dead node was starving all five. And the first fix for it did nothing. A per-entry memory scope was gated, disarm-verified (19,840,500 bytes retained disarmed, TestUnexpectedResult) and rolled — and production was unchanged. Zig’s ArenaAllocator.free is a no-op for any buffer that is not the last allocation, and the new scope’s parent was the previous fix’s scope — so it freed into something that cannot free. The test used a really-freeing parent; the single property under test was the one the harness did not share with production. A green test and a dying node were both telling the truth about different programs. What is confirmed by measurement: with the cron tick disabled as a time-boxed diagnostic, hil has run 20+ minutes with zero kills, memory cycling 24→467→24 MB, and has completed 8 folds — its first in seven days. Recovery ETA from measurement, not hope: 10,390 candidate batches at 300 per pass and 706s per pass (98.8% of which is gather, not folding) ≈ 7 hours. Also linked: the GDPR byte-erasure failure below is the same root cause — scrubbing rides on a fold completing (compact.zig:1556, free-ride redact), so with folds starved, erasure hides data rather than deleting it.
Tonight’s migration wrote 33,025 records in 52 bulk requests — exactly the burst that overruns the ring. On de, cc/log/io-memory-cc returns 404 for 29 of 32 evenly-spaced keys and cc/log/invo-cc for 13 of 32 in seven contiguous runs. Reads stay on v1 until a fleet-wide index audit is clean — not until the flip gate passes, because the flip gate cannot see soft holes.
Niklái’s bar is all features must be there — no cutover with gaps. These are the conditions for putting real users on v2 — v1 keeps running beside it, untouched, and user-driven migration comes after (Niklái, 2026-08-19). Each condition names what measures it.
8 of 11 met, 1 unmeasured. An unmeasured criterion is a hole in the plan, not a pass. 1 have no automated check behind them and say so — they cannot flip on their own, which is deliberate: a line that can only ever read green is not a measurement.
Niklái, 16 Aug: “All indexes have to be done properly. Stored on R2/B2 and cached on the nodes disks.” Amended by Niklái, 28 Aug (“index only”): the per-route index is derived from the batches and is no longer mirrored to B2 — its whole-object copy grew with the route (13.6 MB on cc/log/io-cc, ~25 s of every pass on hil). What must be in both clouds is the data: the batches. This check measures the newest batch of real routes in R2 and B2. Kept true by a check that can fail. Its own output:PASS: newest batches of real routes present in BOTH clouds with equal size; disk block cache present (idx is derived — not mirrored by decision 2026-08-28)
Scope honestly stated: this measures the per-route index on two hot real routes plus the block cache on de. Not yet measured by any check: the prefix index, the aggregate group table, search’s corpus/length objects, and whether a warm read is served with zero cloud round trips (the index header path). Niklái’s ruling says all indexes; this check keeps the core one honest and the rest is named, not assumed.
how this is measuredscripts/v2-check-b2idx-real.sh — rclone stats of the NEWEST BATCH of real routes in both clouds (equal size, B2 landing inside the drift ceiling); the per-route idx is rebuilt from batches and no longer mirrored (Niklái, 2026-08-28) + the node block cache, every run
Erasure must delete, not hide. The read path 404ing is not evidence — only a direct R2/B2 read of the batch object is. The check writes a known secret, proves it is greppable in the object before the erase (an instrument that only ever says absent is not measuring), then erases through POST /v2/gdpr/erase and reads the same object door-free. PASS only when the secret is gone from every cloud that held it, the erase record reads history_purged, and bytes_gone:true was not written while the bytes were still there. Kept true by a check that can fail. Its own output:PASS positive control: secret greppable in R2 w/de/s/25066/b/16 before erase
PASS positive control: marker zxqer2026102414433467632 greppable in sidx w/de/s/25066/b/17 before erase
PASS read_mark saw THIS erase id 9a92e0aed98f-cd8a98fd8287 (MARK_GONE_FALSE via=id/gdpr_erased/d2b0ccb566b7bee786ecd4acafa750727c88294ce70d3527aba8d38d807104c9)
PASS bytes absent from R2 and B2; history_purged; bytes_gone not stamped ahead of the rewrite
how this is measuredscripts/v2-check-erasure.sh — write a known secret, prove it greppable in the batch object, POST /v2/gdpr/erase, rclone-cat the same object from R2 and B2; PASS only when bytes are absent, history_purged, and bytes_gone was not stamped ahead of the rewrite
3 broken, 1 missing (cutover gap) — counted from the run this page is generated from
how this is measuredthe suite run this page is generated from
1 feature(s) could not be proven this run (1 skipped). Untested is not passing — a skipped check and a passing one look identical in a summary
how this is measuredthe SKIP + BLOCK counts from the same run
all 12 panels call real doors
how this is measuredpanels parsed from the dashboard’s own source
Three ways an enumeration can lie while answering 200, each checked: a wildcard returning the same row twice, a row that does not name its route, and truncated that cannot say whether the scan saw everything. The property was verified by hand at 400dd16 — duplicates gone (19 rows, 19 distinct, was 8 for 4), every row carrying its route, truncation honest at 1000 of 1686 above the cap and false on a settled route. A fixture green is bounded by its own size, so the newest-N of the real ~56k-key cc/log must ALSO be recent per the wildcard listen door (scripts/v2-check-bigroute.sh) — the stricter verdict wins. A hand measurement is not what holds this line. The check could not run (exit 2): Writing 1005 more rows would not answer the question and would load the node.. Untested is not passing.
how this is measuredscripts/v2-check-enumeration.sh + big-route recency (v2-check-bigroute.sh), stricter verdict wins
Niklái’s rule is 10 seconds maximum, 2 seconds target — a design constraint, not an aspiration, because CF geo-routing makes write-here/read-there the ORDINARY case. First measurement, 2026-08-13, across four ordered node pairs: an exact-route read landed at 2.5–6.9s and a wildcard at 6.4–10.2s — one of four over the ceiling (sin→de, 10247ms) and all eight past the 2s target. The clouds are not the cause: raw PUTs measure R2 204ms and B2 311ms at p50 from de, against a seal-to-durable p50 of 2229ms on de and 8780ms on hil at the time, so the latency was ours. Re-measured 2026-08-27 from journal lines: seal-to-R2-durable is now p50 789ms over 540 batches (the door no longer awaits the second cloud); the remaining exact-path residual is the tip flusher after durable, and the wildcard trail is the route catalog shard flushed behind newer data. The wildcard trails the exact route by ~4s every time — that gap is the route catalog reaching the peer, and it is what the suite has been reporting as a missing wildcard feature. The rows were never absent. They were late. Measured in the run this page is generated from: pair fi->ash (Europe-Singapore keyed GET); worst-of-3 7467ms (7467, 3470, 2500) — inside the 10000ms maximum but past the 2000ms target. pair fi->ash (Europe-Singapore keyed GET); worst-of-3 7467ms (7467, 3470, 2500); poll 200ms; target 2000ms; ceiling 10000ms.. Past the 2s target but inside the ceiling — the ceiling is the contract. Standing cross-pair check: Kept true by a check that can fail. Its own output: PASS exact route visible at 5920ms — inside the ceiling, over the 2000ms target
PASS wildcard visible at 5948ms — inside the ceiling, over the 2000ms target
CROSS-NODE VISIBILITY: PASS exact=5920ms wild=5948ms
how this is measuredthe suite’s own peer-visibility row from this run, corroborated by scripts/v2-check-visibility.sh — the stricter verdict wins
Proven end to end on the fleet: the account registers an agent’s public key, the agent signs a request with no account key at all and is accepted, and the stored record carries agent:{key_id,label} taken from the verified key rather than anything the writer claimed. Revoking on one node had a peer refuse within 2 seconds — the cache TTL is 60s, so that is eviction propagating, not an expiry. Control: the same key was accepted on that peer moments before. A top-level agent field is now reserved and refused on every write, so the stamp cannot be shadowed by a duplicate key.
how this is measuredon JSON doors, end-to-end on the fleet: register, sign, stamp, revoke, peer refusal timed — a one-time proof, no automated check yet. NOT proven for spooled blob PUT: signing covers sha256(body) and the spooled body is on disk, so an agent-signed large upload would verify sha256("") and 401. No client sends X-Agent-Sig today; tracked, not shipped over
A lost KEY is recoverable and always was — rotation preserves the public id, so a magic link restores ownership. The unrecoverable case is a lost ACCOUNT: while exactly one mailbox is bound, that mailbox dying orphans the pid forever. Both halves hold, measured this run: the committed dashboard lets an account add a second address and warns while only one is bound, and the attach door refused a garbage token with 401 INVALID_TOKEN — wired end to end, nothing minted. Suite: attach door wired: garbage token refused 401 INVALID_TOKEN, nothing minted
how this is measureddashboard committed source (panel + warning) plus the suite’s Recovery second email row
Bounded means: a fixed 22-unit workload (10 writes, 10 reads, 2 queries) costs at most 50 cloud operations per unit, however much is already stored — cost tracks the work sent, never the size of the store. v1 measured 213 R2 GETs per second on de, 94% of it agents polling cc/log — ~18.4M a day, with the node at load 61. v2 removes the cause rather than capping it: values ride with the index entries, so enumeration does zero per-body resolves. That claim needs a standing number, not an argument, and the transport now counts every GET/LIST/PUT/DELETE attempt per cloud at the choke point and reports them in /v2/health — the check reads the total around the workload and the by_module deltas beside it, so a breach NAMES the subsystem that spent the ops (R2 and B2 reported separately: different bills, never summed). Kept true by a check that can fail. Its own output: PASS 10.0 ops/unit within ceiling 50
how this is measuredscripts/v2-check-cost.sh against /v2/health op counters, by_module deltas naming the spender
cc/log on v2 is the migration’s own proof, on a corpus we own and can break: 13,225 drops (counted with list+cursor, not find), dual-write shipped, reads still on v1. Three effects, all asserted by the check: a cc-log write reports v2:true, the same record reads back byte-identical through GET /v2/entity/cc/log/<source>/<key>, and the reader (cc-query) resolves through a v2 door. The third is false today by design decision — reads stay on v1 until the fleet-wide index audit is clean (cc/log is holed on de, see corrections) — so a red here is the flip not having happened, not a broken instrument. Kept true by a check that can fail. Its own output: PASS dual-write by effect: cc-log's own response says v2:true (cc/log/io-cc @system-20261024144309-c966)
PASS read-back by effect: content byte-identical through /v2/entity/cc/log/io-cc/system-20261024144309-c966 on 46.225.15.249
PASS reads flipped: cc-query resolves through a v2 door
PASS census control: CONTROL_OK instrument can return 200 and 404
CC/LOG DOGFOOD: PASS
how this is measuredscripts/v2-check-dogfood.sh: write via cc-log, read back through the v2 entity door, reader flip asserted