Background Workers¶

This guide walks through deploying background worker services alongside a web server. We use the django-celery-tasks sample app -- a Django web application with a Celery worker for processing background tasks.

The Pattern¶

Background workers are services that process tasks without serving HTTP traffic. Common examples:

Task queues -- Celery, RQ, Dramatiq, BullMQ, Sidekiq
Scheduled jobs -- Celery Beat, cron-like processes
Event consumers -- Kafka consumers, Redis stream consumers

The key insight is that a worker uses the same Docker image as your web server but runs a different command. You don't need a separate Dockerfile or a separate container build -- just a different entry in the deploy section.

django-celery-tasks¶

appmecca.yaml¶

containers:
  - name: django-celery-app
    dockerfile: Dockerfile
    context: .

deploy:
  - name: web
    container: django-celery-app
    port: 8000
    health_check:
      path: /health/

  - name: worker
    container: django-celery-app
    command: "celery -A config worker -l info"

database:
  enabled: true

redis:
  enabled: true

There are three things to notice:

One container -- both services use the same django-celery-app image
Two services -- web has a port (gets a URL), worker does not (internal only)
Command override -- the worker service overrides the Dockerfile's CMD with the Celery command

The web service runs the default command from the Dockerfile (gunicorn). The worker service overrides it with celery -A config worker -l info.

Services with and without ports¶

This is the critical distinction:

Service	Has port?	Gets a URL?	What it does
`web`	Yes (8000)	Yes	Serves HTTP requests
`worker`	No	No	Processes background tasks

The worker service has no port field and no health_check. AppMecca starts the container but does not route any web traffic to it. It runs as a standalone process.

No health check for workers

Services without a port don't get platform health checks — there is no HTTP endpoint to probe. The container simply runs and AppMecca monitors that it stays alive via Docker. If the process crashes, Docker reports the container as stopped.

Why cache is enabled¶

Celery needs a message broker to pass tasks from the web server to the worker. In this example, we use Redis as the broker:

redis:
  enabled: true

AppMecca auto-injects the CELERY_BROKER_URL environment variable (along with REDIS_URL) when cache is enabled. The Django settings simply read it:

CELERY_BROKER_URL = os.environ.get("CELERY_BROKER_URL", "redis://localhost:6379/0")
CELERY_RESULT_BACKEND = CELERY_BROKER_URL

Dockerfile¶

The Dockerfile is the same as any Django app -- there is nothing worker-specific about it:

FROM python:3.12-slim

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

WORKDIR /app

RUN apt-get update && \
    apt-get install -y --no-install-recommends gcc libpq-dev && \
    rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

RUN chmod +x entrypoint.sh

EXPOSE 8000

CMD ["./entrypoint.sh"]

The default CMD runs the entrypoint script (which starts gunicorn). The worker service overrides this with the Celery command.

Entrypoint script¶

The entrypoint.sh handles database readiness and migrations, then starts gunicorn:

#!/bin/bash
set -e

echo "Waiting for database..."

RETRIES=10
DELAY=3

for i in $(seq 1 $RETRIES); do
    python -c "
import psycopg2, os
try:
    conn = psycopg2.connect(
        host=os.environ.get('DB_HOST', 'localhost'),
        port=int(os.environ.get('DB_PORT', '5432')),
        user=os.environ.get('DB_USER', 'postgres'),
        password=os.environ.get('DB_PASSWORD', ''),
        dbname=os.environ.get('DB_NAME', 'postgres'),
    )
    conn.close()
    print('Database is ready!')
except Exception as e:
    print(f'Database not ready: {e}')
    exit(1)
" && break

    echo "Attempt $i/$RETRIES failed. Retrying in ${DELAY}s..."
    sleep $DELAY
done

echo "Running migrations..."
python manage.py migrate --noinput

echo "Starting gunicorn..."
exec gunicorn config.wsgi:application \
    --bind 0.0.0.0:8000 \
    --workers 2 \
    --timeout 120

The worker skips this script

Because the worker service specifies a command override, it runs celery -A config worker -l info directly instead of entrypoint.sh. Only the web service runs the entrypoint. Migrations are handled by the web service on startup.

Deploy It¶

# Register
mecca register git@gitlab.com:appmecca-sample-apps/django-celery-tasks.git -e dev

# Build (one image used by both services)
mecca build trigger -e dev -g git@gitlab.com:appmecca-sample-apps/django-celery-tasks.git

# Deploy
mecca app deploy -e dev django-celery-tasks

Expected output

Deploying django-celery-tasks to dev...
Deployment queued: deploy-xxxxxxxx
Waiting for agents...
Container started: django-celery-tasks-web
Container started: django-celery-tasks-worker
Health check passed: /health/ -> 200 OK
Deployment complete.

Services:
  web:    https://django-celery-tasks-web.dev.appmecca.net (healthy)
  worker: running (no port)

Notice that only the web service gets a URL. The worker service runs without one. The service name is always part of the hostname — there is no bare {app}.{env}.{domain} form.

Viewing Worker Logs¶

Even though the worker has no URL, you can still view its logs. Select the service with --service:

mecca logs django-celery-tasks --type container --service worker -e dev

Expected output

[2025-01-15 10:23:01,234: INFO/MainProcess] celery@abc123 ready.
[2025-01-15 10:23:01,235: INFO/MainProcess] Connected to redis://10.0.10.50:6379/0
[2025-01-15 10:23:15,890: INFO/ForkPoolWorker-1] Task myapp.tasks.send_email[abc-123] succeeded in 0.234s

Adding More Workers¶

You can add as many worker services as needed. For example, adding a Celery Beat scheduler:

deploy:
  - name: web
    container: django-celery-app
    port: 8000
    health_check:
      path: /health/

  - name: worker
    container: django-celery-app
    command: "celery -A config worker -l info"

  - name: scheduler
    container: django-celery-app
    command: "celery -A config beat -l info"

All three services use the same Docker image. Only web gets a URL and health check. The worker and scheduler services run their respective commands as background processes.

Key Takeaways¶

Background workers use the same container image as your web server
Define them as separate services with a command override and no port
Services without a port are internal only -- no URL, no inbound traffic
Use mecca logs <app> --service <worker> to view worker output
If your workers need a message broker, set cache: enabled: true for Redis