Background Workers

Run long-lived processes that consume queues, process events, or handle async workloads, with no public endpoint.

What is a Background Worker?

A background worker is a service that runs continuously but is never exposed to the internet. It has no Ingress, no public URL, and no port binding requirement. Workers are ideal for:

• Queue consumers (BullMQ, Celery, Sidekiq, NATS, RabbitMQ)
• Event stream processors (Kafka, Redis Streams)
• Data sync and ETL pipelines
• Email / notification dispatchers
• Webhook fanout processors
• Machine learning inference workers

Build pipeline

Workers use the exact same build pipeline as web services. StackBlaze detects your runtime, runs the install and build commands, and packages the result into a Docker image. The only difference is the run configuration, workers are deployed as Kubernetes Deployments with no Service or Ingress.

This means you can share a monorepo between a web service and its worker, they build from the same code but run different start commands.

Restart policy

Workers must run continuously. If your worker process exits (due to an unhandled exception or crash), Kubernetes restarts it with an exponential backoff: 10s, 20s, 40s, up to a maximum of 5 minutes. After a pod runs successfully for 10 minutes, the backoff counter resets.

Tip

Design your worker to crash-fast on unrecoverable errors (bad config, missing env vars) rather than silently hanging. A fast crash + restart is healthier than a zombie process.

Environment variables

Workers have full access to the project's environment variables. Commonly you'll want:

Variable	Example
`REDIS_URL`	redis://cache.internal:6379
`DATABASE_URL`	postgresql://user:pass@postgres.internal:5432/mydb
`WORKER_CONCURRENCY`	5
`QUEUE_NAME`	email-dispatch

BullMQ + Redis example (Node.js)

BullMQ is a popular Node.js queue library backed by Redis. Here is a complete worker that processes jobs from an email queue:

worker.js

import { Worker } from 'bullmq'
import { createTransport } from 'nodemailer'

const redisConnection = {
  host: process.env.REDIS_HOST || 'cache.internal',
  port: parseInt(process.env.REDIS_PORT || '6379', 10),
}

const transporter = createTransport({
  host: process.env.SMTP_HOST,
  port: 587,
  auth: {
    user: process.env.SMTP_USER,
    pass: process.env.SMTP_PASS,
  },
})

const worker = new Worker(
  'email-dispatch',
  async (job) => {
    const { to, subject, html } = job.data
    console.log(`[worker] Sending email to ${to} (job ${job.id})`)
    await transporter.sendMail({ from: 'no-reply@acme.com', to, subject, html })
    console.log(`[worker] Done: job ${job.id}`)
  },
  {
    connection: redisConnection,
    concurrency: parseInt(process.env.WORKER_CONCURRENCY || '5', 10),
  }
)

worker.on('failed', (job, err) => {
  console.error(`[worker] Job ${job?.id} failed:`, err.message)
})

// Keep the process alive
process.on('SIGTERM', async () => {
  await worker.close()
  process.exit(0)
})

Corresponding web service (enqueue jobs)

api.js

import { Queue } from 'bullmq'

const emailQueue = new Queue('email-dispatch', {
  connection: {
    host: process.env.REDIS_HOST || 'cache.internal',
    port: 6379,
  },
})

// Enqueue from your API handler
app.post('/send-welcome-email', async (req, res) => {
  await emailQueue.add('welcome', {
    to: req.body.email,
    subject: 'Welcome to Acme!',
    html: '<h1>Welcome!</h1>',
  })
  res.json({ queued: true })
})

Note

Both the API and the worker connect to the same Redis instance using the internal hostname cache.internal. No public exposure of Redis is needed.

Celery + Redis example (Python)

worker.py

from celery import Celery
import os

redis_url = os.environ.get('REDIS_URL', 'redis://cache.internal:6379/0')

app = Celery('tasks', broker=redis_url, backend=redis_url)

@app.task
def process_order(order_id: str) -> dict:
    # heavy processing here
    print(f"Processing order {order_id}")
    return {"status": "processed", "order_id": order_id}

# Start command: celery -A worker worker --loglevel=info --concurrency=4

Scaling workers

Workers can be scaled horizontally just like web services, increase the replica count to process more jobs in parallel. Each replica runs an independent worker process and pulls from the same queue. Queue libraries like BullMQ and Celery handle concurrent access safely via Redis atomic operations.

terminal

stackblaze scale worker --replicas 4

Monitoring worker health

Workers don't have HTTP health checks (no port to probe). StackBlaze monitors the process exit code instead. You can view worker logs in real time from the dashboard or CLI: