Blog · July 23, 2026
A migration deployed clean to production. The alarm that fired said CRITICAL anyway.
The hourly prod-deploy lane picked up a migration, applied it, deployed the edge functions, and finished. Every step in the run succeeded. Twenty seconds later the lane paged Matt with a CRITICAL alert saying production might be half-deployed.
Production was not half-deployed. It was fully deployed and working. The alert was wrong — and it was wrong in a way that was going to keep being wrong, on a schedule, until someone noticed the pattern instead of the individual page.
The convention that created the gap
Ledgenter's own migration convention splits a schema change into two pull requests on purpose: the migration lands first, the regenerated types.generated.ts and contract-snapshot.json that describe it land in a follow-up PR once dev has the migration applied and the types can be regenerated against something real. It's the same convention documented in an earlier post about the cronos loop's token boundary — the scoped credential this loop runs on can't reach the Supabase management API that regenerating those files requires, so a second, later step closes it out.
That convention is deliberately fine with a short window where production has more than the committed record describes. The prod-deploy lane's own verify step wasn't fine with it. It diffed live prod's schema against the committed files with a plain diff -u and failed on ANY difference — including the expected, temporary, entirely-benign one where prod is a superset of what's committed. A migration-only PR merges, the lane deploys it within the hour, and the very next run of that lane's post-deploy check sees prod ahead of committed and reads that as "prod may be behind or half-deployed" — backwards from what actually happened.
This had already fired once for real: a migration merged at 14:40 CT, deployed cleanly, and paged CRITICAL for roughly twenty minutes until the follow-up types PR closed the gap and the next hourly check went quiet again. Nothing was ever actually broken. The alert didn't know the difference.
Fixing the check, not the convention
The convention is the right one to keep — a scoped credential that can't reach the management API shouldn't block shipping the migration on the credential that can. So the fix had to be in the verify step: teach it to tell "prod is missing something committed asserts" (a real failure) apart from "prod has more than committed says" (the expected mid-convention state).
contract-snapshot.mjs gained a --check-superset mode: a structured diff over the same functions/relations/columns data the exact-match mode already parses, but asserting one direction only — every function, relation, and column the committed snapshot claims must exist on live, unchanged. Anything committed says should be there and isn't, or is there but different, still fails loud. Anything live has that committed doesn't know about yet — a new function, a new column — is reported as a note, not an error. The plain-text types.generated.ts diff got the same treatment: a line removed from live (something committed expects that's now missing) fails; a line only added is a note.
The regular CI gates — the ones that run on every pull request against dev — didn't change at all. They stay exact-match on purpose: a PR is exactly the moment you want to know if your local state has drifted from what's live, additions included. Only the prod lane's post-deploy verify, the one built specifically to survive the two-PR convention it's checking against, got the superset logic.
Why the distinction has to be structural
The easy version of this fix is "just page less" — widen the alert threshold, add a delay before paging, require two consecutive failures. All of those make the false positive quieter without making it false any less often, and all of them make a real failure slower to surface too. An unattended loop that runs every hour and pages on failure only stays trustworthy if failure means something specific. The moment "CRITICAL" sometimes means "everything is fine, wait twenty minutes" is the moment a human learns to open the alert and check for themselves before believing it — which is the same failure mode as not alerting at all, just slower to notice.
The fix that actually holds is the one that makes the check able to state, in its own output, which of the two states it's looking at — not one that makes the check quieter about not knowing.
Filed as a feature request against ourselves the same day it happened, fixed and shipped the next loop cycle. Read more on how this loop finds and files its own friction: your AI agent just hit a bug in the tool it's using — where does that report go?