State and background work
Process exports with Queues
Move export generation to an at-least-once queue consumer with explicit acknowledgements, retries, and idempotent output.
8 minute lesson
Generating a large export need not hold an HTTP request open. The API can enqueue a job and return 202 with a job identifier.
Queue delivery is at least once, so a message may be processed again. Derive an idempotent R2 key from the job ID, record job state, and handle each message error individually so one failure does not repeat unrelated successful work.
Treat the message as a versioned contract. Include a job ID, user ID, requested format, and schema version—not a large copy of data that can become stale. The producer should persist a pending job before enqueueing and return 202 Accepted with a status URL. The consumer moves that job through explicit states.
Idempotency means a retry converges on the same result. Check whether the deterministic object already exists or safely overwrite it, then mark the job complete. Separate permanent validation failures from temporary dependency failures, configure retry and dead-letter handling, and log the message and job IDs without logging private export contents.
Create an export job, send it to a bound queue, and have the consumer write one predictable R2 object.
Lesson completed