Retries, Dead Jobs and Replay
How outbox jobs retry, when they become dead, and how to find, replay or cancel them.
Click to expandThe admin jobs list in dark mode with synthetic failed jobs.
Outcomes
A handler returns what happened:
| Outcome | Effect |
|---|---|
ok() | Done. ok({ commit: (tx) => … }) writes your bookkeeping in the same transaction that marks the job succeeded. |
retry({ code, after? }) | Try again with backoff, or after after ms (e.g. a provider's Retry-After). |
fail({ code, message? }) | Permanent failure: dead, no more retries. Use it for errors retrying can't fix (a 4xx, a deleted record). |
cancel({ reason }) | The work is moot (the user deleted their account): cancelled, not an error. |
A thrown error becomes retry({ code: 'unexpected' }). Map known errors yourself with classify:
import { defineOutboxJob, fail, ok } from '@kit/outbox/server';import { z } from 'zod';import { isClientError } from '#/lib/my-provider.server.ts'; // yoursexport const syncOrganization = defineOutboxJob({ kind: 'app.sync-organization', payload: z.object({ organizationId: z.string(), day: z.string() }), timeoutMs: 60_000, retry: { maxAttempts: 10, // default 20 backoff: { baseMs: 30_000, maxMs: 3_600_000 }, // exponential, jittered perCode: { rate_limited: { maxAttempts: 50, maxDelayMs: 600_000 }, // throttling isn't failure timeout: 3, // three timeouts in a row: dead }, }, // undefined: the library's default for everything else (a thrown error retries as `unexpected`) classify: (error) => (isClientError(error) ? fail({ code: 'client_error' }) : undefined), async handle({ organizationId, day }, ctx) { // … return ok(); },});A job becomes dead when its attempts run out, on fail, or after repeated crashes. Dead jobs stay in outbox.job for 90 days.
What Gets Logged
The kit logger (Workers Logs) gets:
- Every job transition (
outbox transition):succeededatinfo(one line per job),retryingatwarn,deadaterror. - Failures:
outbox schedule_failed,outbox infra_error,outbox settle_failed,outbox phase_failed(a sweep step that couldn't run, e.g. no database) andoutbox dlq_messageaterror;outbox handler_threw,outbox lease_lost,outbox nudge_failedandoutbox message_rejectedatwarn.
Routine events (each enqueue, each nudge, each sweep) aren't logged. Alert on error entries whose message starts with outbox .
Errors are redacted before they're logged or stored: a failed query keeps its name, code and stack frames, never its SQL or parameters; stored error messages also have email addresses masked. A message is still yours to keep clean: don't put personal data in fail({ message }).
Finding Jobs
Platform admins can use /admin/jobs instead of a database console. It defaults to all statuses, newest first, 50 per page; choose Dead to focus on failures. Filter status, kind, organization id and UTC creation days; the next-page link preserves filters and creation order. Changing creation order restarts the cursor; arbitrary column sorting is not supported. outbox:list controls this page, per-kind pending/retrying backlog tiles and the dashboard’s seven-day dead-job health tile. Jobs due more than 15 minutes ago are marked aged. Failed summary reads hide their tiles; a failed list/detail read shows a failed state.
outbox:get opens /admin/jobs/[id]: attempt history, last error, terminal reason and raw metadata, rendered as capped text. Unknown, malformed or retained-away job ids answer 404. Payloads never leave the service projection.
Keep metadata ids-only. Admin detail shows metadata as stored; it is not redacted. Never put personal data, secrets or request bodies in metadata. Error text and metadata are never included in the admin's logs or audit events. See Jobs.
For local troubleshooting, query the table (locally with void db studio, in production with your database console):
-- dead and cancelled jobs, newest firstSELECT id, kind, status, dedupe_key, last_error_code, last_error_message, terminal_reason, finished_atFROM outbox.job WHERE status IN ('dead', 'cancelled') ORDER BY finished_at DESC LIMIT 50;-- what's waitingSELECT kind, status, count(*) FROM outbox.jobWHERE status IN ('pending', 'retrying', 'processing') GROUP BY 1, 2;last_error_code is the code of the last failed attempt (unexpected, timeout, your retry/fail codes). terminal_reason says why the job ended: a cancel({ reason })'s reason, or for a dead job exhausted (out of retries), failed (fail()) or crash_loop (repeated crashes).
From code, @makerkit/workers-outbox/inspect has bounded, read-only queries: getJob, listJobs, countJobs, countBacklog, findJobsByCorrelationId.
import { listJobs } from '@makerkit/workers-outbox/inspect';import { db } from 'void/db';const dead = await listJobs(db, { status: 'dead', limit: 50 });Replay and Cancel
From a job's admin detail:
outbox:replay: confirm and enter the emailed step-up code to replay dead/cancelled work with a fresh budget. Other states are refused as not replayable now. Handlers must be safe to run again.outbox:cancel: confirm with a separate code to cancel pending/retrying work. Processing is refused as in flight; finished jobs stay unchanged.
Codes are bound to the acting admin, job and action. The shared code-request cooldown may ask you to wait after another admin action. No GET mutates a job. Each POST authorizes independently; the library guards the actual status transition, so stale tabs cannot move a terminal job backwards except through an authorized replay.
Only successful transitions record kit.admin.outbox_job_replayed or kit.admin.outbox_job_cancelled, after the write commits. Metadata is { jobId, kind } only. Tenant jobs use organization scope; system jobs use the acting admin's user scope. Audit remains best effort, as for other admin actions.
Operator actions are on the outbox, from server code only (a script, an admin action behind your own authorization), never an open route:
import { getOutbox } from '#/lib/outbox.server.ts';const outbox = getOutbox();await outbox.replay(jobId); // dead or cancelled: run again with a fresh retry budgetawait outbox.retryNow(jobId); // pending or retrying: run nowawait outbox.cancel({ id: jobId }, 'user_deleted');await outbox.replay(jobId, { tenantKey: organizationId }); // only if it belongs to this organizationPass { tenantKey } whenever the action is on an organization's behalf: without it the action applies to any tenant's job. eraseTenant(organizationId) deletes every job of an organization, including running ones.
Common Failures
| Symptom | Cause | Fix |
|---|---|---|
A job stays pending | Its kind isn't in outboxJobs (removed or never listed) | List it again, or cancel its jobs |
| Jobs run up to 5 minutes late | No queue binding (the log says outbox: no queue binding), or nudges are lost | Check queues/outbox.ts exists and the deploy created the queue; the sweep runs them meanwhile |
OutboxEnqueueError (invalid_payload) | The payload fails its schema or is over 256 KiB | Fix the payload; store large data in R2 |
OutboxEnqueueError (dedupe_conflict) | The dedupeKey belongs to another kind | Prefix keys with the job (welcome:…) |
| A job runs twice | At-least-once delivery after a crash | Make the handler idempotent (Jobs) |