Part 5: Background Workers¶
Add background job processing with BullMQ and Redis. Learn about portless services and the command override.
What you'll learn¶
- Adding a BullMQ worker that processes jobs from a Redis queue
- Configuring a portless service with a
commandoverride - Enabling managed Redis via
cache: enabled: true - Managing worker lifecycle with service commands
What you'll need¶
- The full-stack app from Part 4 deployed and working
Install BullMQ in the backend¶
Add a job queue to the API¶
Create backend/queue.js:
const { Queue } = require("bullmq");
const connection = {
host: process.env.REDIS_HOST || "localhost",
port: parseInt(process.env.REDIS_PORT || "6379", 10),
};
const todoQueue = new Queue("todo-notifications", { connection });
module.exports = { todoQueue, connection };
Update backend/index.js to enqueue a job when a todo is created:
const express = require("express");
const knex = require("knex")(require("./knexfile"));
const { todoQueue } = require("./queue");
const app = express();
app.use(express.json());
const PORT = process.env.PORT || 3000;
app.get("/health", (_req, res) => res.json({ status: "healthy" }));
app.get("/api/todos", async (_req, res) => {
const todos = await knex("todos").orderBy("created_at", "desc");
res.json(todos);
});
app.post("/api/todos", async (req, res) => {
const [todo] = await knex("todos").insert({ title: req.body.title }).returning("*");
await todoQueue.add("new-todo", { todoId: todo.id, title: todo.title });
res.status(201).json(todo);
});
app.put("/api/todos/:id", async (req, res) => {
const [todo] = await knex("todos")
.where({ id: req.params.id })
.update({ title: req.body.title, done: req.body.done })
.returning("*");
if (!todo) return res.sendStatus(404);
res.json(todo);
});
app.delete("/api/todos/:id", async (req, res) => {
await knex("todos").where({ id: req.params.id }).del();
res.sendStatus(204);
});
app.listen(PORT, () => console.log(`Listening on ${PORT}`));
Create the worker¶
Create backend/worker.js:
const { Worker } = require("bullmq");
const { connection } = require("./queue");
const worker = new Worker(
"todo-notifications",
async (job) => {
console.log(`Processing job ${job.id}: new todo "${job.data.title}"`);
// In a real app: send email, push notification, update search index, etc.
await new Promise((resolve) => setTimeout(resolve, 500));
console.log(`Job ${job.id} completed`);
},
{ connection }
);
worker.on("failed", (job, err) => {
console.error(`Job ${job?.id} failed:`, err.message);
});
console.log("Worker started, waiting for jobs...");
Same container, different entrypoint
The worker reuses the same Docker image as the backend. AppMecca runs it with a different command, so you do not need a separate Dockerfile.
Update appmecca.yaml¶
Add the worker as a portless service and enable Redis:
database:
enabled: true
redis:
enabled: true
containers:
- name: backend
dockerfile: backend/Dockerfile
context: backend/
- name: frontend
dockerfile: frontend/Dockerfile
context: frontend/
deploy:
- name: backend
container: backend
port: 3000
path: /api
health_check:
path: /health
- name: frontend
container: frontend
type: spa
command: 'nginx -g "daemon off;"' # serves the built bundle from a container
spa_root: /usr/share/nginx/html
port: 3000
health_check:
path: /
environment:
PUBLIC_API_URL: "${service:backend}"
- name: worker
container: backend
command: "node worker.js"
- name: scheduler
container: backend
command: "node scheduler.js"
No port on worker or scheduler
Omitting port makes this a portless service. AppMecca runs the container but gives it no public hostname, no route on the platform HTTPS router, and no health check. The worker communicates only through Redis.
Top-level cache: is the managed Redis
The cache: enabled: true block at the top level of appmecca.yaml turns on the managed Redis cache and injects REDIS_URL into every container. It defaults to off.
Do not confuse it with the per-service deploy[].cache: block, which configures edge caching for a web service and is an entirely different feature.
Running one instance of the scheduler
A scheduler that enqueues repeating jobs must not run on more than one host, or every tick fires twice. Add singleton: true to pin it to a single instance.
Optional: add a scheduler¶
Create backend/scheduler.js for recurring jobs:
const { Queue, QueueScheduler } = require("bullmq");
const { connection } = require("./queue");
const queue = new Queue("todo-notifications", { connection });
async function scheduleCleanup() {
await queue.add(
"cleanup-done-todos",
{},
{ repeat: { every: 60 * 60 * 1000 } } // every hour
);
console.log("Scheduled hourly cleanup job");
}
scheduleCleanup().catch(console.error);
Build and deploy¶
Manage worker services¶
These take the app name first and the service name second, both as positional arguments.
Restart a worker after a code change:
Stop and start individually:
Read worker output. The service is selected with --service:
Workers and deployments
Workers are redeployed alongside every other service during mecca app deploy. Use mecca app restart only for ad-hoc restarts between deployments.
Verify it works¶
Create a todo and watch the worker process the job:
curl -X POST https://todo-app-backend-yourgroup.dev.example.com/api/todos \
-H "Content-Type: application/json" \
-d '{"title": "Test worker"}'
mecca logs todo-app --type container --service worker --since 1m
You should see the "Processing job" and "Job completed" messages in the worker logs.
What's next¶
In Part 6: Branch Deployments, you will deploy a feature branch side-by-side with production and learn about branch-scoped commands.