# Retries, Dead Jobs and Replay

> How outbox jobs retry, when they become dead, and how to find, replay or cancel them.

*Canonical: https://makerkit.dev/docs/void-cloudflare/outbox/failures*

---

{% img src="/assets/images/docs/void-cloudflare/admin-jobs.png" width="1440" height="900" alt="The admin jobs list in dark mode with synthetic failed jobs." /%}

The admin jobs list in dark mode with synthetic failed jobs.

## Outcomes

A handler returns what happened:

| Outcome | Effect |
|---|---|
| `ok()` | Done. `ok({ commit: (tx) => … })` writes your bookkeeping in the same transaction that marks the job succeeded. |
| `retry({ code, after? })` | Try again with backoff, or after `after` ms (e.g. a provider's `Retry-After`). |
| `fail({ code, message? })` | Permanent failure: dead, no more retries. Use it for errors retrying can't fix (a 4xx, a deleted record). |
| `cancel({ reason })` | The work is moot (the user deleted their account): cancelled, not an error. |

A thrown error becomes `retry({ code: 'unexpected' })`. Map known errors yourself with `classify`:

```ts
import { defineOutboxJob, fail, ok } from '@kit/outbox/server';
import { z } from 'zod';

import { isClientError } from '#/lib/my-provider.server.ts'; // yours

export const syncOrganization = defineOutboxJob({
  kind: 'app.sync-organization',
  payload: z.object({ organizationId: z.string(), day: z.string() }),
  timeoutMs: 60_000,
  retry: {
    maxAttempts: 10, // default 20
    backoff: { baseMs: 30_000, maxMs: 3_600_000 }, // exponential, jittered
    perCode: {
      rate_limited: { maxAttempts: 50, maxDelayMs: 600_000 }, // throttling isn't failure
      timeout: 3, // three timeouts in a row: dead
    },
  },
  // undefined: the library's default for everything else (a thrown error retries as `unexpected`)
  classify: (error) => (isClientError(error) ? fail({ code: 'client_error' }) : undefined),
  async handle({ organizationId, day }, ctx) {
    // …
    return ok();
  },
});
```

A job becomes **dead** when its attempts run out, on `fail`, or after repeated crashes. Dead jobs stay in `outbox.job` for 90 days.

## What Gets Logged

The kit logger (Workers Logs) gets:

- **Every job transition** (`outbox transition`): `succeeded` at `info` (one line per job), `retrying` at `warn`, `dead` at `error`.
- **Failures:** `outbox schedule_failed`, `outbox infra_error`, `outbox settle_failed`, `outbox phase_failed` (a sweep step that couldn't run, e.g. no database) and `outbox dlq_message` at `error`; `outbox handler_threw`, `outbox lease_lost`, `outbox nudge_failed` and `outbox message_rejected` at `warn`.

Routine events (each enqueue, each nudge, each sweep) aren't logged. **Alert on `error` entries whose message starts with `outbox `.**

Errors are redacted before they're logged or stored: a failed query keeps its name, code and stack frames, never its SQL or parameters; stored error messages also have email addresses masked. A message is still yours to keep clean: don't put personal data in `fail({ message })`.

## Finding Jobs

Platform admins can use `/admin/jobs` instead of a database console. It defaults to all statuses, newest first, 50 per page; choose **Dead** to focus on failures. Filter status, kind, organization id and UTC creation days; the next-page link preserves filters and creation order. Changing creation order restarts the cursor; arbitrary column sorting is not supported. `outbox:list` controls this page, per-kind pending/retrying backlog tiles and the dashboard’s seven-day dead-job health tile. Jobs due more than 15 minutes ago are marked aged. Failed summary reads hide their tiles; a failed list/detail read shows a failed state.

`outbox:get` opens `/admin/jobs/[id]`: attempt history, last error, terminal reason and raw metadata, rendered as capped text. Unknown, malformed or retained-away job ids answer 404. Payloads never leave the service projection.

**Keep metadata ids-only.** Admin detail shows metadata as stored; it is not redacted. Never put personal data, secrets or request bodies in metadata. Error text and metadata are never included in the admin's logs or audit events. See [Jobs](./jobs).

For local troubleshooting, query the table (locally with `void db studio`, in production with your database console):

```sql
-- dead and cancelled jobs, newest first
SELECT id, kind, status, dedupe_key, last_error_code, last_error_message, terminal_reason, finished_at
FROM outbox.job WHERE status IN ('dead', 'cancelled') ORDER BY finished_at DESC LIMIT 50;

-- what's waiting
SELECT kind, status, count(*) FROM outbox.job
WHERE status IN ('pending', 'retrying', 'processing') GROUP BY 1, 2;
```

`last_error_code` is the code of the last failed attempt (`unexpected`, `timeout`, your `retry`/`fail` codes). `terminal_reason` says why the job ended: a `cancel({ reason })`'s reason, or for a dead job `exhausted` (out of retries), `failed` (`fail()`) or `crash_loop` (repeated crashes).

From code, `@makerkit/workers-outbox/inspect` has bounded, read-only queries: `getJob`, `listJobs`, `countJobs`, `countBacklog`, `findJobsByCorrelationId`.

```ts
import { listJobs } from '@makerkit/workers-outbox/inspect';
import { db } from 'void/db';

const dead = await listJobs(db, { status: 'dead', limit: 50 });
```

## Replay and Cancel

From a job's admin detail:

- `outbox:replay`: confirm and enter the emailed step-up code to replay **dead/cancelled** work with a fresh budget. Other states are refused as not replayable now. Handlers must be safe to run again.
- `outbox:cancel`: confirm with a separate code to cancel **pending/retrying** work. Processing is refused as in flight; finished jobs stay unchanged.

Codes are bound to the acting admin, job and action. The shared code-request cooldown may ask you to wait after another admin action. No GET mutates a job. Each POST authorizes independently; the library guards the actual status transition, so stale tabs cannot move a terminal job backwards except through an authorized replay.

Only successful transitions record `kit.admin.outbox_job_replayed` or `kit.admin.outbox_job_cancelled`, after the write commits. Metadata is `{ jobId, kind }` only. Tenant jobs use organization scope; system jobs use the acting admin's user scope. Audit remains best effort, as for other admin actions.

Operator actions are on the outbox, from server code only (a script, an admin action behind your own authorization), never an open route:

```ts
import { getOutbox } from '#/lib/outbox.server.ts';

const outbox = getOutbox();

await outbox.replay(jobId); // dead or cancelled: run again with a fresh retry budget
await outbox.retryNow(jobId); // pending or retrying: run now
await outbox.cancel({ id: jobId }, 'user_deleted');
await outbox.replay(jobId, { tenantKey: organizationId }); // only if it belongs to this organization
```

Pass `{ tenantKey }` whenever the action is on an organization's behalf: without it the action applies to any tenant's job. `eraseTenant(organizationId)` deletes every job of an organization, including running ones.

## Common Failures

| Symptom | Cause | Fix |
|---|---|---|
| A job stays `pending` | Its kind isn't in `outboxJobs` (removed or never listed) | List it again, or cancel its jobs |
| Jobs run up to 5 minutes late | No queue binding (the log says `outbox: no queue binding`), or nudges are lost | Check `queues/outbox.ts` exists and the deploy created the queue; the sweep runs them meanwhile |
| `OutboxEnqueueError (invalid_payload)` | The payload fails its schema or is over 256 KiB | Fix the payload; store large data in R2 |
| `OutboxEnqueueError (dedupe_conflict)` | The `dedupeKey` belongs to another kind | Prefix keys with the job (`welcome:…`) |
| A job runs twice | At-least-once delivery after a crash | Make the handler idempotent ([Jobs](./jobs)) |
