When should work become a background job?
A web request should normally do only enough work to validate the command, record durable intent, and return a useful response. Email delivery, report generation, media processing, third-party synchronization, AI pipelines, and bulk imports are better candidates for a worker because their duration and dependencies are variable.
Keep work synchronous when
- The caller must know the final outcome before continuing.
- The operation is fast, local, and has predictable latency.
- Splitting the transaction would create more complexity than value.
A clean queue-backed architecture
API
Validate input and authorize the user.
Database
Record the business state and durable intent.
Queue
Deliver a small, versioned job payload.
Worker
Perform the task and record its outcome.
Put identifiers in the job payload instead of full mutable records. The worker can load current data, verify that the operation is still valid, and avoid leaking sensitive information into queue infrastructure.
{
"type": "invoice.generate.v1",
"jobId": "job_8d21",
"invoiceId": "inv_2048",
"requestedBy": "user_42",
"attempt": 1
}Avoid the dual-write trap
If an API commits database state and then fails before publishing a job, the business record exists but the work never runs. The transactional outbox pattern stores an event in the same database transaction. A relay later publishes that event and marks it delivered, making recovery possible without pretending two systems share a transaction.
Design every handler for duplicate delivery
Most production queues provide at-least-once delivery. A worker can finish the side effect and crash before acknowledging the message, so the same job may run again. Use a stable idempotency key and persist the completed outcome before acknowledging.
Idempotency
Repeated execution produces the same business result instead of duplicate charges, emails, or records.
Bounded retries
Retry transient failures with exponential backoff and jitter, but stop retrying permanent validation errors.
Dead-letter handling
Move exhausted jobs to an inspectable state with failure context and a controlled replay process.
Concurrency limits
Protect databases and vendor APIs by limiting global and per-tenant parallel work.
Make scheduling duplicate-safe
A recurring scheduler may run twice during failover or deployment. Treat the schedule as a request to enqueue a uniquely named occurrence—for example, combining the task name with its intended execution window—and reject duplicates atomically.
Operate the queue as a product dependency
Queue depth alone is not enough. Monitor the age of the oldest waiting job, enqueue rate, completion rate, processing duration, retry rate, dead-letter count, and time spent waiting versus working. Break metrics down by job type and deployment version.
Deploy and shut down workers safely
- Stop accepting new jobs when shutdown begins.
- Allow active handlers a bounded grace period.
- Release incomplete jobs so another healthy worker can retry them.
- Close queue, database, and telemetry connections after handlers settle.
Version job payloads and keep workers backward compatible during rolling deployments. Producers and consumers rarely change at exactly the same instant.
Production launch checklist
✓ Durable intent is recorded before work can be lost
✓ Every handler has a tested idempotency strategy
✓ Retryable and permanent failures are classified
✓ Backoff, retry ceilings, and dead-letter policies are defined
✓ Payloads are small, versioned, and free of unnecessary secrets
✓ Concurrency protects downstream services and tenant fairness
✓ Queue latency and oldest-job age have actionable alerts
✓ Workers drain safely during deployment and shutdown
Need a dependable Node.js backend?
Endurance Softwares builds scalable Node.js platforms, background processing systems, and observable cloud services designed for real production workloads.