Blog · August 13, 2026

embedding_jobs has one row per note or decision waiting to be vectorized, and the drain that processes it has exactly one reason to close a job without embedding anything: the target is gone. A failed network read produced the identical shape as a deleted row, and the drain couldn't tell them apart.

Ledgenter embeds every knowledge note and decision for semantic search. The app-side write path fires an async POST to knowledge-embed right after the row lands, but that POST is fire-and-forget — a short-lived agent that exits before the request resolves leaves it dropped, and the row's embedding_jobs entry sits pending with nothing watching it. embedding-drain is the backstop: a pg_cron tick invokes it every two minutes, and it batch-processes whatever's ready, with attempts and backoff so a permanently-failing target can't spin forever.

Two outcomes, one read

For each claimed job, the drain reads the target row — title/context/rationale for a decision, body for a note — scoped to the job's tenant_id and target_id, via .maybeSingle(). There are exactly two things that call return can mean: the row doesn't exist (or exists and is soft-deleted), or something went wrong asking. Supabase's client surfaces both as data: null unless you also destructure error — and the drain wasn't.

const { data: row } = await admin
  .from(table)
  .select(selectCols)
  .eq("tenant_id", job.tenant_id)
  .eq("id", job.target_id)
  .maybeSingle();

Every line after this treated row === null as one thing: tombstone. Target gone, or soft-deleted — nothing to embed, close the job, move on. That's the correct call when it's true. It's the wrong call when row is null because a momentary DB hiccup or network blip meant the read itself failed, with the actual target sitting right there in the table, never asked about again.

Why "close the job" is the expensive branch

The drain's failure path — a thrown error inside the loop's try — is not a heavyweight fallback. It increments attempts, sets last_error, and computes the next run_after via backoff, capped at MAX_ATTEMPTS. That path exists specifically for exactly this kind of transient blip; a job that hits it gets retried, up to six times, before anyone needs to look at it. Treating a failed read as a tombstone skips that path entirely and goes straight to the outcome the retry logic was built to avoid: markDone, status: 'done', last_error: null. The job leaves the queue clean. Nothing was embedded. Nothing says so.

Compare that to knowledge-embed, the synchronous path this drain exists to backstop: when it fails outright, the reconciler still holds a pending job row and will try again on the next tick. The drain has no such second backstop behind it — it is the backstop. A job it silently closes doesn't fall through to anything else. The note or decision keeps existing, keeps its title, keeps showing up in lexical search — knowledge_search's trigram fallback doesn't care whether the embedding ever landed — so nothing about the user-facing product breaks in a way anyone would notice. What's gone is quieter: that one row is permanently absent from every semantic search, and there was never an error, an alert, or a retry to say so.

The fix doesn't touch the real tombstone path

const { data: row, error: readErr } = await admin
  .from(table)
  .select(selectCols)
  .eq("tenant_id", job.tenant_id)
  .eq("id", job.target_id)
  .maybeSingle();
if (readErr) throw new Error(`read_${readErr.message}`);

Destructure the error, check it first, throw if it's set. A genuine tombstone — a real null row with no read error — still falls through to markDone exactly as before; that branch is untouched and still correct. A failed read now takes the same path every other failure in this function already takes: reserveErr, storeErr, and the embed-provider call itself all throw into the same catch, which is precisely the attempts/backoff machinery this function was built around. The read was the one call in the whole loop that didn't.

Found the same way most of what ends up on this blog gets found: not a support ticket, not an incident, a defect-audit pass reading a function's error-handling branch by branch and asking, for every place a call's result gets used, whether every way that call can fail actually reaches the code written to handle failure — or whether one of them quietly took the other exit instead.

Start at ledgenter.com.