Retries, Dead Jobs and Replay

How outbox jobs retry, when they become dead, and how to find, replay or cancel them.

The admin jobs list in dark mode with synthetic failed jobs.Click to expand

The admin jobs list in dark mode with synthetic failed jobs.

Outcomes

A handler returns what happened:

OutcomeEffect
ok()Done. ok({ commit: (tx) => … }) writes your bookkeeping in the same transaction that marks the job succeeded.
retry({ code, after? })Try again with backoff, or after after ms (e.g. a provider's Retry-After).
fail({ code, message? })Permanent failure: dead, no more retries. Use it for errors retrying can't fix (a 4xx, a deleted record).
cancel({ reason })The work is moot (the user deleted their account): cancelled, not an error.

A thrown error becomes retry({ code: 'unexpected' }). Map known errors yourself with classify:

import { defineOutboxJob, fail, ok } from '@kit/outbox/server';
import { z } from 'zod';
import { isClientError } from '#/lib/my-provider.server.ts'; // yours
export const syncOrganization = defineOutboxJob({
kind: 'app.sync-organization',
payload: z.object({ organizationId: z.string(), day: z.string() }),
timeoutMs: 60_000,
retry: {
maxAttempts: 10, // default 20
backoff: { baseMs: 30_000, maxMs: 3_600_000 }, // exponential, jittered
perCode: {
rate_limited: { maxAttempts: 50, maxDelayMs: 600_000 }, // throttling isn't failure
timeout: 3, // three timeouts in a row: dead
},
},
// undefined: the library's default for everything else (a thrown error retries as `unexpected`)
classify: (error) => (isClientError(error) ? fail({ code: 'client_error' }) : undefined),
async handle({ organizationId, day }, ctx) {
// …
return ok();
},
});

A job becomes dead when its attempts run out, on fail, or after repeated crashes. Dead jobs stay in outbox.job for 90 days.

What Gets Logged

The kit logger (Workers Logs) gets:

  • Every job transition (outbox transition): succeeded at info (one line per job), retrying at warn, dead at error.
  • Failures: outbox schedule_failed, outbox infra_error, outbox settle_failed, outbox phase_failed (a sweep step that couldn't run, e.g. no database) and outbox dlq_message at error; outbox handler_threw, outbox lease_lost, outbox nudge_failed and outbox message_rejected at warn.

Routine events (each enqueue, each nudge, each sweep) aren't logged. Alert on error entries whose message starts with outbox .

Errors are redacted before they're logged or stored: a failed query keeps its name, code and stack frames, never its SQL or parameters; stored error messages also have email addresses masked. A message is still yours to keep clean: don't put personal data in fail({ message }).

Finding Jobs

Platform admins can use /admin/jobs instead of a database console. It defaults to all statuses, newest first, 50 per page; choose Dead to focus on failures. Filter status, kind, organization id and UTC creation days; the next-page link preserves filters and creation order. Changing creation order restarts the cursor; arbitrary column sorting is not supported. outbox:list controls this page, per-kind pending/retrying backlog tiles and the dashboard’s seven-day dead-job health tile. Jobs due more than 15 minutes ago are marked aged. Failed summary reads hide their tiles; a failed list/detail read shows a failed state.

outbox:get opens /admin/jobs/[id]: attempt history, last error, terminal reason and raw metadata, rendered as capped text. Unknown, malformed or retained-away job ids answer 404. Payloads never leave the service projection.

Keep metadata ids-only. Admin detail shows metadata as stored; it is not redacted. Never put personal data, secrets or request bodies in metadata. Error text and metadata are never included in the admin's logs or audit events. See Jobs.

For local troubleshooting, query the table (locally with void db studio, in production with your database console):

-- dead and cancelled jobs, newest first
SELECT id, kind, status, dedupe_key, last_error_code, last_error_message, terminal_reason, finished_at
FROM outbox.job WHERE status IN ('dead', 'cancelled') ORDER BY finished_at DESC LIMIT 50;
-- what's waiting
SELECT kind, status, count(*) FROM outbox.job
WHERE status IN ('pending', 'retrying', 'processing') GROUP BY 1, 2;

last_error_code is the code of the last failed attempt (unexpected, timeout, your retry/fail codes). terminal_reason says why the job ended: a cancel({ reason })'s reason, or for a dead job exhausted (out of retries), failed (fail()) or crash_loop (repeated crashes).

From code, @makerkit/workers-outbox/inspect has bounded, read-only queries: getJob, listJobs, countJobs, countBacklog, findJobsByCorrelationId.

import { listJobs } from '@makerkit/workers-outbox/inspect';
import { db } from 'void/db';
const dead = await listJobs(db, { status: 'dead', limit: 50 });

Replay and Cancel

From a job's admin detail:

  • outbox:replay: confirm and enter the emailed step-up code to replay dead/cancelled work with a fresh budget. Other states are refused as not replayable now. Handlers must be safe to run again.
  • outbox:cancel: confirm with a separate code to cancel pending/retrying work. Processing is refused as in flight; finished jobs stay unchanged.

Codes are bound to the acting admin, job and action. The shared code-request cooldown may ask you to wait after another admin action. No GET mutates a job. Each POST authorizes independently; the library guards the actual status transition, so stale tabs cannot move a terminal job backwards except through an authorized replay.

Only successful transitions record kit.admin.outbox_job_replayed or kit.admin.outbox_job_cancelled, after the write commits. Metadata is { jobId, kind } only. Tenant jobs use organization scope; system jobs use the acting admin's user scope. Audit remains best effort, as for other admin actions.

Operator actions are on the outbox, from server code only (a script, an admin action behind your own authorization), never an open route:

import { getOutbox } from '#/lib/outbox.server.ts';
const outbox = getOutbox();
await outbox.replay(jobId); // dead or cancelled: run again with a fresh retry budget
await outbox.retryNow(jobId); // pending or retrying: run now
await outbox.cancel({ id: jobId }, 'user_deleted');
await outbox.replay(jobId, { tenantKey: organizationId }); // only if it belongs to this organization

Pass { tenantKey } whenever the action is on an organization's behalf: without it the action applies to any tenant's job. eraseTenant(organizationId) deletes every job of an organization, including running ones.

Common Failures

SymptomCauseFix
A job stays pendingIts kind isn't in outboxJobs (removed or never listed)List it again, or cancel its jobs
Jobs run up to 5 minutes lateNo queue binding (the log says outbox: no queue binding), or nudges are lostCheck queues/outbox.ts exists and the deploy created the queue; the sweep runs them meanwhile
OutboxEnqueueError (invalid_payload)The payload fails its schema or is over 256 KiBFix the payload; store large data in R2
OutboxEnqueueError (dedupe_conflict)The dedupeKey belongs to another kindPrefix keys with the job (welcome:…)
A job runs twiceAt-least-once delivery after a crashMake the handler idempotent (Jobs)