Troubleshooting¶

This page covers the most common issues you may encounter when building, deploying, and managing applications on AppMecca. Problems are organized by symptom so you can find the solution quickly.

First, run `mecca debug`¶

Most "what's broken?" investigations start with the same handful of commands: mecca status, mecca logs, mecca queue status, mecca var list. The debug command bundles them into one:

mecca debug myapp

A single screen returns:

Last 5 deploys — each one's status and error_message, so you can see if the same failure has been recurring.
Per-service health — most recent check, response time, error.
Resolved URLs — the actual external URL each service is reachable at.
Container env drift — keys configured in mecca var list that are missing from the running container, or vice versa. A red DRIFT line is the strongest signal that a deploy hasn't picked up your latest var changes.
Configured env vars — values masked according to your API key scope (sensitive keys redacted as <redacted: sha256=…> even at admin scope by default).

If you need cleartext for a sensitive value, add --reveal (admin scope only; the use is audit-logged server-side):

mecca debug myapp --reveal

If you need to see what env the container is actually booting with (not what's configured), admin only:

mecca debug myapp -s web --container-env

When the API itself is unreachable, the admin variant has a fallback that reads the platform database and the platform secret store directly (platform operators only):

appmecca debug myapp -e dev --no-api

mecca debug --json returns the raw payload for scripting.

Quick Reference: Common Errors¶

Error / Symptom	Likely Cause	Fix
"No configuration for environment"	Typo in `-e` flag, stale `APPMECCA_ENV` shell var, or the environment was never configured on this machine	Run `mecca configure list` to see what is configured, then `mecca configure -e <env>` to add it
"App not found"	App not registered in this environment, or the app lives in a group other than the one this command resolved	Register with `mecca register <repo> -e <env>`; otherwise see App not found — but it's in `mecca app list`
"the group's slug changed while this app was being registered"	The group was renamed while you registered or built	Run the command again with `--group <new-slug>` — the error names the new slug
"another change to this group is in progress"	A rename, NS setup, delete or registration is briefly holding the group	Run the command again in a few seconds
"No builds found"	No image has been built yet	Run `mecca build trigger -e <env> -g <repo>`
"declared drive storage … exceeds the free allowance / storage grant"	App declares more drive storage (`storage.drives[].size_gb`) than the group is entitled to	Lower `size_gb` (or remove a drive) and redeploy, or ask your operator for a storage grant — see Deploy refused: storage
"Health check failed"	App not responding on the health check path	Verify endpoint returns HTTP 200; check logs
Deployment stuck in QUEUED	Nothing has picked the action up yet, or an admission gate is holding it	`mecca queue list -a <app> -e <env>` shows the reason; `mecca queue diagnose <action-id>` explains it
Build fails immediately	appmecca.yaml missing or invalid	Ensure `appmecca.yaml` exists in repo root
"Port mismatch"	appmecca.yaml port does not match app's listen port	Align `port` in appmecca.yaml with your app
Container starts then exits	App crashes on startup	Check `mecca logs <app> --type container --service <service> -e <env>`
502 Bad Gateway	Container not healthy yet, or crashed	Wait 30-60 seconds; check health check and logs
503 Service Unavailable	No healthy containers behind the platform router	Check `mecca app show <app> -e <env>` and health check configuration
DNS not resolving	Deployment not complete or DNS propagation delay	Wait 60 seconds; verify deployment completed
Branch URL not working	Branch not deployed or DNS not propagated	Run `mecca app show <app> -e <env>` to verify
Cache data leaking between branches	App not using `CACHE_PREFIX`	Use `CACHE_PREFIX` env var for all cache keys
A YAML key you set has no effect	Unknown keys in `appmecca.yaml` are silently ignored — no error, no warning	Check the spelling against Configuration; a typo'd key simply disappears
Node.js `SELF_SIGNED_CERT_IN_CHAIN`	`pg-connection-string` v3 treats `sslmode=require` as `verify-full` (full chain validation)	Strip `sslmode` from URL, set `ssl.rejectUnauthorized=false` programmatically — see Database Problems
Laravel `laravel.log` "Permission denied"	`artisan migrate` runs as root and creates root-owned files; Apache runs as `www-data`	Add `chown -R www-data:www-data storage bootstrap/cache` to entrypoint after artisan — see Database Problems
Spring Boot `FATAL: remaining connection slots are reserved`	HikariPool default of 10 connections × N replicas × M Spring apps exhausts the shared PostgreSQL `max_connections`	Cap `spring.datasource.hikari.maximum-pool-size=3` per service — see Database Problems

Build Failures¶

Build fails: "appmecca.yaml not found"¶

The build system looks for appmecca.yaml in the root of your repository. If it is missing, misnamed, or in a subdirectory, the build will fail.

Fix: Ensure the file is named exactly appmecca.yaml (not appmecca.yml, not .appmecca.yaml) and is committed to the repository root.

"Cannot access `<url>`" at `mecca register`¶

mecca register checks that it can reach the repository — git ls-remote, run with your own local Git credentials — before it registers anything. If the check fails you'll see a message like:

Cannot access https://gitlab.com/org/private-repo — repository is not
reachable with your current credentials (details below).

Fix with one of:
  (a) appmecca git-auth setup -e <env>   (admin; persistent per-env SSH key —
      register the public key on the Git provider after running this)
  (b) --link-git   (clone with your OWN linked Git account — link one in the
      dashboard first)

If the repo is public, double-check the URL and branch. Use --skip-check to
bypass this guard (CI pipelines with pre-validated URLs).

Underlying git error:
  fatal: could not read Username for 'https://gitlab.com': terminal prompts disabled

Nothing is registered — you pay the fix-up cost in seconds, not minutes into a build.

Fixes, in order of preference:

Your own linked account — re-run with --link-git (see below). With it, a failed local check is only a warning: the platform clones with your linked account, and its own git preflight is the check that counts.
The environment's SSH deploy key — an operator runs appmecca git-auth setup -e <env>, and appmecca git-auth show-key -e <env> prints the public key to register on the Git provider (it survives infra teardown). That is what the platform clones with when there is no binding. It does not change what your credentials can reach, so if the local check still fails, pass --skip-check.

If the repository is public, double-check the URL and branch. An HTTPS URL fails the check whenever git would have to prompt for a password, which it cannot do here — use the SSH URL. A check that timed out says nothing about your credentials: retry, or raise --precheck-timeout. --skip-check also skips the default-branch lookup, so pass -b if your repository's default branch is not main.

mecca app deploy <git-url> runs no local check. A repository the platform cannot read fails server-side instead, at the git preflight: the app is held at no_access and nothing builds until access is fixed.

Apps bound to your Git account need none of the above. An app bound to your linked GitHub/GitLab account has its git preflight and every build clone with your OAuth token (refreshed server-side automatically) — no deploy key, no manual setup. Link the provider account in the dashboard first — binding uses an account you already have, it does not create one. Then either:

Register from the CLI: mecca register <git-url> --link-git. The provider is worked out from the URL (GitHub and GitLab only), and the flag needs a signed-in session — an API key has no linked account to bind.
Register from the dashboard's repo picker. Picking the repository is what creates the binding.

Without a binding, a private repo cannot be cloned with the environment deploy key unless you have added that key to the repository, so the app quarantines at no_access on registration.

If preflight reports "can't access the repo with your linked account", the token was revoked or expired beyond refresh — reconnect the provider from the dashboard, then run mecca app recheck <app>. The env deploy key remains the credential for apps registered without a linked-account binding.

Build fails with "ERROR: Cannot clone `<url>`"¶

The BUILD row's error_message shows ERROR: Cannot clone <url> — repository requires authentication .... This is the build-service-side equivalent of the client-side pre-check above, for cases where:

No local check ran — mecca register was run with --skip-check, or the app came in through mecca app deploy <git-url>, which runs none.
The repo became inaccessible after registration (auth token expired, SSH key rotated).
The build service can no longer read the stored SSH key from the platform secret store (a platform-side permissions regression — raise it with your operator).

Fixes: the same as above — a linked-account binding, or the environment deploy key added to the repository.

Build fails: Dockerfile not found¶

The dockerfile path in your appmecca.yaml is relative to the repository root.

containers:
  - name: myapp
    dockerfile: backend/Dockerfile   # relative to repo root
    context: backend/

Fix: Verify the path is correct and the Dockerfile is committed. A common mistake is setting context: backend/ and then using dockerfile: Dockerfile -- this will look for Dockerfile in the repo root, not in backend/. The dockerfile path is always relative to the repo root, regardless of context.

dockerfile: must be written out explicitly

Pre-flight validation requires a dockerfile value on every container. Omitting the key fails with Container 'x': 'dockerfile' is required even though a bare Dockerfile might look like a safe default. Always state it.

Build fails: Docker build error¶

If the Docker build itself fails (dependency installation, compilation, etc.):

Check the build logs:
```
mecca build logs -e dev <build-id>
```
Try building locally first:
```
docker build -t test .
```
Common causes:
Missing system dependencies in the Dockerfile (e.g., libpq-dev for psycopg2)
.gitignore excluding files that the Dockerfile needs (e.g., a lib/ pattern ignoring frontend/src/lib/)
Network issues pulling base images

Check your .gitignore

AppMecca builds from a fresh git clone. If a file exists on your machine but is in .gitignore, it won't be in the build. Run git check-ignore -v <file> to see if a file is being ignored.

Build succeeds but image is wrong¶

If the build completes but the app doesn't work as expected:

Verify you pushed your latest code before triggering the build

Check which branch was built (defaults to main):

mecca build trigger -e dev -g <repo> -b feature-branch

All builds are fresh with no Docker layer caching, so stale layers are never the issue

Configuration Gotchas¶

These are the settings that fail quietly — no error, no warning, just behaviour you didn't ask for.

A key I set in `appmecca.yaml` has no effect¶

Unknown keys are silently ignored. There is no error and no warning — a misspelled or invented key simply vanishes. If a setting isn't doing anything, check the spelling against Configuration first.

The usual suspects:

replica instead of replicas
healthcheck instead of health_check
resource instead of resources
keys nested at the wrong level (static_site: is top-level or per-service; static: is top-level only)

`resources.memory` isn't limiting anything (or the container dies instantly)¶

memory is a Docker memory string and requires a unit suffix:

resources:
  memory: 512m     # ✅ 512 MB

512 → interpreted as 512 bytes, so the container is OOM-killed the moment it starts.
512mb → not a valid Docker memory string.
Valid suffixes are k, m, g.

resources.cpu is accepted by the schema but does nothing — no code path applies it. Don't rely on it to shape scheduling.

`type:` typo silently produces an ordinary container service¶

type: is not validated against a list of allowed values. type: statc or type: SPA does not raise an error — the service just falls through to the normal container path, so your static bundle never gets deployed and you get a container that may not even exist.

Valid values are spa, ssr, api, static. Check the spelling if a service is deploying the wrong way.

`type: spa` fails validation as "ambiguous"¶

A service with type: spa and neither command: nor a static_site: block is a hard error — the platform cannot tell whether you want a static deploy or a containerised SPA. Pick one:

# Static path — build the bundle, no container
deploy:
  - name: web
    type: static
    static_site:
      build_command: npm run build
      output_dir: dist
      spa_fallback: true

# Container path — drop `type: spa` entirely, or add a command
deploy:
  - name: web
    container: web
    port: 3000
    command: "node build/index.js"

PUBLIC_* env vars are passed to every container regardless, so type: spa is only needed for the static path's runtime config injection.

`type: static` rejected for "container-shaped fields"¶

A type: static service has no container, so it must not declare port, replicas greater than 1, drives, command or health_check. Declaring any of them is a hard error. A valid minimal static service:

deploy:
  - name: web
    type: static
    static_site:
      build_command: npm run build
      output_dir: dist

Custom domains in `appmecca.yaml` do nothing¶

domain: and domains: inside appmecca.yaml are ignored — the certificate lifecycle cannot be driven from config alone. Validation emits a warning, but the deploy still succeeds and your domain is never attached.

Use the CLI instead:

mecca domain add myapp www.example.com -s web -e dev
mecca domain verify myapp www.example.com -e dev
mecca domain list myapp -e dev

A cron job "never runs my command"¶

A cron job is a scheduled HTTP request to an endpoint in your app — not a command run inside a container. There is no container: or command: field. AppMecca calls the URL on schedule; your app does the work behind it.

cron:
  - name: warm-cache
    schedule: "*/15 * * * *"   # 5 fields
    url: /cron/warm-cache      # must start with /
    service: api               # required for multi-service apps
    method: POST               # GET or POST
    timeout: 30                # 5-300 seconds

If the job appears to do nothing, the endpoint is the place to look:

mecca cron history myapp -e dev
mecca cron trigger myapp warm-cache -e dev

`replicas` above 3 is rejected¶

replicas accepts 1-3. Each replica is placed on a different host, so 3 is the ceiling. If you need more capacity, raise resources.memory or split the workload across services.

Deployment Issues¶

`mecca queue logs` reports a BUILD as "queued" while it's running¶

mecca queue list and mecca queue status report a BUILD's live state from the deploy queue (build_in_progress while the agent is running the build). mecca queue logs used to read a different record that only finalises at build completion — so during the run window it reported status: queued and "Action hasn't started yet", contradicting what queue status said.

The API now cross-checks both sources and returns a queue_status field plus an explanatory message, so mecca queue logs tells you the build is still running and that the full output lands when it completes, instead of claiming the action never started.

If you still see the misleading "hasn't started yet" hint, the platform API in your environment is on an older version — ask your operator to update it. Live log streaming during a build is not available yet; today the full log appears after the build completes.

Use mecca build logs for build output

mecca queue logs <id> is the generic queue-row view. For a BUILD, prefer:

mecca build logs <build-id> -e dev

`${service:<name>}` template failures¶

If a deploy fails with Unresolved template variable(s) in service 'X': ${service:<name>}, or mecca build trigger refuses to queue with Service 'X': environment variable 'FOO' references unknown service '${service:<name>}', the error is structured and self-diagnosing — the two Known services sections tell you immediately which map the lookup hit:

Unresolved template variable(s) in service 'web':
  - ${service:svlete} (not in external map; closest match: 'svelte')

Known services:
  External (public HTTPS): api, frontend, web
  Internal (private DNS):  api, frontend, web

Common causes:

Typo — the per-token hint shows the closest-match when one exists (closest match: 'svelte'). Fix the name in appmecca.yaml and retry.
Wrong environment — the referenced service exists in another environment but not this one.
Service renamed — an earlier deploy[].name has changed; update every ${service:<old-name>} reference to match.
${service:<name>:internal} fails with internal map is empty — a platform issue, not a config one: the environment's private service-discovery DNS is not set up. Fall back to the external form ${service:<name>} until your operator fixes it, or raise it with them directly.
${service:<name>:internal} fails with not in internal map; service is known externally — the public URL exists but the internal DNS record was not created. Retry the deploy; if it persists, raise it with your operator.

:internal is safe to use again

The :internal variant used to go stale — the URL carried the host port the target held when your container started, and that port moved on the target's next deploy. Fixed in BUGS #680: a service's internal address is now permanent and the platform follows the container behind it. Use :internal when you want to stay in-VPC (~3 ms vs ~25 ms), and the plain ${service:<name>} form when you want the public route. See An internal service URL returns "Connection refused" below if you hit one.

Self-references (${service:api} inside the api service's own environment) are allowed and resolve to the service's own external URL. Use them when a container needs to know the public URL it's being served at (CSRF_TRUSTED_ORIGINS, canonical <meta> tags, etc.).

The check runs at config-validation time via validate_appmecca_yaml() on every mecca register, mecca build trigger, and appmecca config validate, so typos are caught before a BUILD is queued. A deploy-time fail-fast remains as a last-resort guard for configs that predate the validator.

Deploy marked failed — finding the reason¶

Start with mecca status <app>. When the most recent deploy is failed the command renders a red banner immediately below the Status line showing:

Reason: — the top-level error the agent recorded.
Per-service crash excerpts (first 3 of up to 30 lines).
A Full logs: mecca logs <app> --type container --service <svc> hint per crashed service.

If the banner shows a Reason: but no per-service crash section, the failure happened before a container started (infrastructure setup, image pull, routing setup). Follow the agent log for the deployment:

mecca queue logs <deployment-id> -e dev

If the banner isn't appearing for an app you know failed, the API may be on an older version — re-check after the platform's next publish that updates the API (appmecca publish <ref>).

If the app has more than one branch, add -b:

mecca status <app> -b dev -e dev

Without -b the app's production branch is used and the Pipeline and Incidents blocks cover every branch, so a main build and a dev build both appear.

An empty incident list means none are open

This copy has changed twice and the history is worth knowing. It began as "No active incidents", which during an outage read as reassurance while the truth was that nothing was tracked for tenant apps at all. It was then changed to say so plainly. Since 2026-08-15 tenant incidents ARE tracked (a producer opens one after three consecutive failed health checks and closes it after two successes), so an empty list once again genuinely means none are open — and both commands say that.

Your per-service health IS tracked, every 60 seconds: the Service table in mecca status <app> shows current liveness, memory and CPU, and

mecca status history <app> <service> -e dev

returns the recent check history — response times, status codes and the error text of any failed probe. That command previously returned nothing at all for tenant apps; it works now.

mecca status no longer says "not found" when the API is down

It used to report any failure — a 502 mid-deploy, a DNS failure, a timeout — as "App 'x' not found", and exit 0. That was most likely to happen precisely when you reached for it: during an API rolling deploy. It now distinguishes them, so Error (502) … the API may be restarting after a deployment means your app is fine and the control plane is briefly unavailable, while a genuine 404 quotes the server's own message naming the environment and the branch it looked in. A missing app now exits 1, not 0 — a script chaining mecca status <app> && … will stop rather than continue.

Deployment stuck in QUEUED¶

The deployment was written to the deploy queue but no agent has picked it up.

Check agent health:
```
appmecca agent status -e dev
```
If no agents are responding, verify the hosts are running:
```
appmecca infra status -e dev --live
```
Check agent logs for errors:
```
appmecca agent logs -e dev
```

Common causes:

The hosts are paused to save cost (appmecca infra resume -e dev to restart them)
The agent service crashed (check appmecca agent logs -e dev)
Network connectivity problems between the host and the platform control plane

The last three are operator territory — if you are a tenant, mecca queue list will show the action sitting in queued; raise it with your platform operator.

Deployment stuck in IN_PROGRESS¶

An agent claimed the deployment but hasn't completed it.

Check agent logs to see what the agent is doing:
```
appmecca agent logs -e dev --since 30m
```
Look for errors pulling the Docker image from the private image registry (authentication, network issues).
If the deployment is truly stuck, you can redeploy:
```
mecca app deploy myapp -e dev
```

Deploy refused: "declared drive storage … exceeds the free allowance"¶

Your app declares more drive storage (storage.drives[].size_gb in appmecca.yaml) than your group is entitled to. Drive storage is bounded:

size_gb must be an integer between 1 and 100 — 100 GB per drive is the hard platform ceiling for every tenant.
Each app gets a small free allowance (5 GB total across its drives by default). Below that, no action is needed.
Above the allowance your group needs a storage grant, and your group's total declared drive storage across all apps must fit inside it.

The 409 message names whether you exceeded the per-app allowance or the group grant. To resolve:

Reduce it — lower size_gb (or remove a drive) in appmecca.yaml and redeploy. A build is never blocked, so reducing the declaration always lands via mecca app deploy / a rebuild; the reduced value then deploys.
Switch to a bucket — declare storage.buckets: instead. Buckets are object storage, are self-service (no operator grant needed), and are far cheaper than drives for bulk data (images, uploads, exports). See Storage.
Request more — ask your platform operator to grant your group more drive storage.

Drives and buckets are per-branch

Each branch of an app gets its own drives and its own buckets. Data does not cross branches — a file written by dev is not visible to main.

When the refusal is caught at the build→deploy step, the reason shows in mecca queue list on the cancelled DEPLOY row.

Container starts then immediately exits¶

The container is starting but crashing before it can respond to health checks.

Check the container logs:

mecca logs myapp --type container --service web -e dev

Common causes:
Missing environment variables -- your app expects a variable that isn't set
Database not reachable -- if database: enabled: true, make sure your entrypoint waits for the database
Port mismatch -- your app listens on a different port than what's in appmecca.yaml
Syntax error in command override -- if using command: in appmecca.yaml, verify the command is correct
Out of memory -- if resources.memory is set too low the container is killed on startup. Note the value requires a unit suffix: 512m, 1g. A bare number like 512 is interpreted as bytes, and 512mb is not a valid Docker memory string at all.

resources.cpu does nothing

The schema accepts resources.cpu, but no code path applies it — setting it will not change how much CPU your container gets. Only resources.memory is enforced.

App deployed but returning 502 Bad Gateway¶

The platform router is sending traffic to your container, but the container is not responding correctly.

Check if the service is healthy:
```
mecca app show myapp -e dev
```
If the service shows as unhealthy, the health check is failing. See the next section.

If the service shows as healthy but you still get 502s, check that your app is binding to 0.0.0.0 (not 127.0.0.1 or localhost):

# Wrong -- only accepts connections from inside the container
gunicorn --bind 127.0.0.1:8000

# Correct -- accepts connections from the platform router
gunicorn --bind 0.0.0.0:8000

Health Check Failures¶

Health check path returns 404¶

The platform health check is hitting a path that your app doesn't serve.

Fix: Make sure the health_check.path in your appmecca.yaml matches an actual endpoint in your application:

health_check:
  path: /health    # Your app must respond to GET /health with a 200

Test it locally:

docker run -p 8000:8000 myapp
curl http://localhost:8000/health

Health check times out¶

The platform sends a request but doesn't get a response in time (default: 5 seconds).

Common causes:

Your app takes too long to start (e.g., loading a large ML model)
The health check endpoint does heavy work (database queries, external API calls)
The app is overloaded

Fix: Make the health check endpoint lightweight (just return 200), or increase the timeout:

health_check:
  path: /health
  timeout: 10
  interval: 30
  healthy_threshold: 2
  unhealthy_threshold: 5

Thresholds have a minimum of 2

healthy_threshold and unhealthy_threshold both accept 2-10. Setting either to 1 is rejected at validation time. The other bounds: interval 5-300, timeout 2-120, startup_timeout 30-600.

If your app is slow to boot (JVM warmup, loading a large model), raise startup_timeout rather than the per-check timeout — it controls how long the platform waits for the container to become healthy during a deploy:

health_check:
  path: /health
  startup_timeout: 300   # 30-600 seconds; default 120

Health check passes locally but fails on AppMecca¶

Differences between your local environment and AppMecca:

Port: Are you binding to the right port? The port in appmecca.yaml must match your app's listen port.
Host: Are you binding to 0.0.0.0? Binding to localhost or 127.0.0.1 won't work.
Path: Does your health check path include a trailing slash? Django apps often require /health/ not /health.
HTTPS: Your app should serve HTTP, not HTTPS. AppMecca terminates TLS for you at the platform router.

Database Problems¶

"Database not ready" in container logs¶

Your entrypoint script cannot connect to the managed PostgreSQL database.

Ensure your entrypoint includes a retry loop (see App with Database)
On a brand-new app the database is provisioned during the first deploy and may take a few minutes to become fully available
Verify database: enabled: true is in your appmecca.yaml

Never hard-code the database name

The database and user names are opaque hashes (e.g. am_<hash>_db, am_<hash>_user), not {app}_db / {app}_user. Always read the injected DATABASE_URL — anything that constructs a connection string from the app name will fail.

Migration fails on deploy¶

If python manage.py migrate fails in your entrypoint:

Check the specific error in the logs:

mecca logs myapp --type container --service web -e dev

Common causes:
Conflicting migrations (resolve locally, rebuild, redeploy)
Migration tries to access data that doesn't exist yet
Database permissions issue (rare with AppMecca-managed databases)

If you need to fix a bad migration:

# Back up current state first
mecca app backup myapp -e dev --description "before migration fix"
# Fix the migration in your code, rebuild, and redeploy

Restore fails¶

If mecca app restore fails:

Verify the bundle prefix is correct: mecca app backups myapp -e dev
Ensure no active connections are blocking the restore (try restarting the service first)
The --force flag is required: mecca app restore myapp <bundle-prefix> --force -e dev
On a multi-branch app, -b <branch> is required so the restore cannot land on the wrong branch's database
"backup not found … (it may have expired)": the automatic safety copy you picked has been removed. Pre-deploy copies may be removed after 14 days and pre-restore copies after 30; nothing was restored and your database was not touched. Pick a newer row, or one of your own bundles — take mecca app backup for a copy you keep

There are no mecca db commands

Database lifecycle is implicit: the database is provisioned on the first deploy when database: enabled: true, and dropped by mecca app delete <app>. To load an existing dump, use mecca app restore-upload <app> --file <dump> (add -b <branch> on an app with more than one branch) or the dashboard's Database tab — open your app, choose the file, confirm; to roll back to a platform bundle, use mecca app restore.

Node.js: `SELF_SIGNED_CERT_IN_CHAIN` / "self-signed certificate in certificate chain"¶

Your Node.js app using pg / node-postgres / knex crashes on connect with:

Error: self-signed certificate in certificate chain
    at /app/node_modules/pg-pool/index.js:45:11 { code: 'SELF_SIGNED_CERT_IN_CHAIN' }

Or logs this warning before crashing:

SECURITY WARNING: The SSL modes 'prefer', 'require', and 'verify-ca' are treated as aliases for 'verify-full'.

Cause: pg-connection-string v3 (shipped with pg v9+) changed sslmode=require in a connection string to imply verify-full — full CA-chain validation. AppMecca's injected DATABASE_URL ends with ?sslmode=require, and the managed PostgreSQL database presents a certificate from a private CA that Node.js does not trust out of the box. Setting ssl: { rejectUnauthorized: false } at the Pool level alone does not help — the connection-string parser overrides it.

Fix: Strip sslmode from the URL before passing it to pg and set ssl programmatically:

const { Pool } = require('pg');

const rawUrl = process.env.DATABASE_URL;
const connectionString = rawUrl
  ? rawUrl.replace(/[?&]sslmode=[^&]*/g, '').replace(/\?&/, '?').replace(/[?&]$/, '')
  : undefined;
const dbConfig = connectionString
  ? { connectionString, ssl: { rejectUnauthorized: false } }
  : {};
const pool = new Pool(dbConfig);

Same pattern works for knex:

// knexfile.js
const rawUrl = process.env.DATABASE_URL;
const connectionString = rawUrl
  ? rawUrl.replace(/[?&]sslmode=[^&]*/g, '').replace(/\?&/, '?').replace(/[?&]$/, '')
  : undefined;
module.exports = {
  client: 'pg',
  connection: connectionString
    ? { connectionString, ssl: { rejectUnauthorized: false } }
    : {},
};

Laravel / PHP-Apache: `storage/logs/laravel.log` "Permission denied"¶

Your Laravel app crash-loops with:

The stream or file "/var/www/html/storage/logs/laravel.log" could not be opened in append mode:
Failed to open stream: Permission denied
The exception occurred while attempting to log: The /var/www/html/bootstrap/cache
directory must be present and writable.

Cause: Your entrypoint.sh runs php artisan migrate --force as root (the container's default user), which creates cache files in storage/ and bootstrap/cache/ with root:root ownership. Apache then starts its workers as www-data, which cannot write to root-owned files — even if the Dockerfile pre-set chmod 775, the files created by artisan at container start are new and root-owned.

Fix: Re-chown after artisan finishes and before Apache starts. In entrypoint.sh:

#!/bin/sh
set -e

# ... wait for DB ...

php artisan migrate --force
# artisan runs as root and may create root-owned files in storage/ and bootstrap/cache/;
# restore www-data ownership before Apache workers try to write to them
chown -R www-data:www-data storage bootstrap/cache

exec apache2-foreground

Also ensure the Dockerfile pre-creates the log file and directories (so the first write doesn't need to create anything):

RUN mkdir -p storage/logs storage/framework/cache storage/framework/sessions storage/framework/views bootstrap/cache \
    && touch storage/logs/laravel.log \
    && chown -R www-data:www-data storage bootstrap/cache \
    && chmod -R 775 storage bootstrap/cache

Spring Boot / JVM: `FATAL: remaining connection slots are reserved`¶

Your Spring Boot app fails Flyway init or early DB connect with:

FATAL: remaining connection slots are reserved for roles with privileges of the "pg_use_reserved_connections" role

Cause: Spring Boot's HikariPool defaults to 10 connections per replica. A few Spring apps with multiple replicas can exhaust the shared PostgreSQL server's max_connections limit, especially during a deploy where both the old -blue and new -green containers are briefly running. Once the limit is hit, any app trying to establish a new connection — even a healthy new deploy — is denied and crashes.

Fix: Cap HikariPool per service to what the app actually needs. For most API services, 3 connections per replica is plenty:

# src/main/resources/application.properties
spring.datasource.hikari.maximum-pool-size=3
spring.datasource.hikari.minimum-idle=1

Then declare that budget in appmecca.yaml so the platform's admission gate can see the demand before the deploy rather than at crash time:

database:
  enabled: true
  max_connections: 3   # per replica; 1-50, default 5

Tune up only if you've measured genuine connection-pool exhaustion under real load. maximum-pool-size=3 across 3 replicas is 9 connections total for the app — which leaves plenty of room on the shared server.

Platform-side work in progress: fleet-level connection budgeting is being extended so this class of failure can't be caused by neighbour apps. Until that lands, cap HikariPool explicitly and declare database.max_connections.

App Not Found Errors¶

"App not registered"¶

You tried to deploy or check status of an app that hasn't been registered.

# Register the app first (this runs a repository pre-flight check — it does
# NOT queue a build)
mecca register git@github.com:yourorg/myapp.git -e dev

# Then build and deploy
mecca build trigger -e dev -g git@github.com:yourorg/myapp.git
mecca app deploy myapp -e dev

App not found — but it's in `mecca app list`¶

Every app belongs to exactly one group, and app-scoped commands (mecca var, mecca app show, mecca domain, mecca cron, and the rest) resolve the app name inside a single group — by default whichever group you last set with mecca use <slug>, or your personal group if you've never run that. If the app lives in a different one of your groups, that lookup misses and you get "App not found" even though the app is real and you can see it.

Running mecca app list with no --group shows every app across every group you belong to, with a Group column naming which one owns each app:

mecca app list -e dev

App           Slug          Group     Branch  Status   Health  URL
Saldo         saldo-website unomena   main    running  ok      https://...
Ticket Uno    go-ticket-uno gtu       main    running  ok      https://...

Once you know the group, pass it explicitly to the command that 404'd:

mecca var list saldo-website --group unomena -e dev

--group works before or after the subcommand (mecca --group unomena var list saldo-website is equivalent), and it always overrides whatever group a command would otherwise have defaulted to. See Groups & Teams for the full picture, including a few commands that already resolve an unambiguous name across your groups automatically, without --group — and refuse with a clear error naming the candidates when the same name exists in more than one of your groups, rather than guessing which one you meant.

"That name resolves onto an app another group already holds"¶

mecca register refused with a 409 naming a canonical app name.

Your app's canonical name is {group-slug}-{app-name}, and several things the platform derives from it — the S3 backup prefix most consequentially — collapse runs of hyphens. So two names that look different can resolve to the same place:

acme-widgets--3f9c1a2b      <- another group's app
acme-widgets-3f9c1a2b       <- yours; resolves to the SAME backup directory

Sharing that directory would let either app list and restore the other's database dumps, so the second claim is refused at registration.

To resolve: pick a different app name, or a different group slug. Anything that changes the canonical name so it no longer collapses onto the existing one works — the refusal names the canonical form, so compare against that rather than against what you typed.

⚠ An app that already holds a colliding name is never blocked. The check refuses only a new claim, so rebuilds, redeploys and new branches of an existing app keep working exactly as before.

App registered in wrong environment¶

Apps are registered per-environment. If you registered in dev but are deploying to staging, you need to register in staging too:

mecca register git@github.com:yourorg/myapp.git -e staging

Check where your app is registered:

mecca app list

Service Communication Issues¶

Inter-service template not resolving¶

If ${service:backend} appears as a literal string in your container instead of a URL:

The referenced service name must exactly match a name in the deploy section
The referenced service must have a port defined
You must deploy (not just build) for templates to resolve

Note that a service URL always includes the service name — there is no bare {app}.{env} hostname, even for a single-service app. If you were expecting one, that is the mismatch.

Connection refused between services¶

If one service cannot reach another:

Verify both services are running:
```
mecca app show myapp -e dev
```
For internal URLs (${service:name:internal}), both services must be in the same environment and the same branch — services on different branches are isolated and cannot reach each other
Check that the target service is actually listening on the expected port
If the target service was redeployed since the caller started, the :internal URL has gone stale — see An internal service URL returns "Connection refused"

Environment Variable Issues¶

Environment variable not available in container¶

Variables set with mecca var set require a restart or redeploy:

# Set the variable
mecca var set myapp SECRET_KEY=abc123 -e dev

# Restart to pick up the change
mecca app restart myapp web --force -e dev

A static service has no container, so it cannot be restarted, stopped or started; the platform refuses the action. Redeploy the app to republish it (mecca app deploy myapp -e dev).

var set writes to ALL branches by default

Without -b, mecca var set applies the value to every branch of the app. Narrow it explicitly when you only mean one:

mecca var set myapp SECRET_KEY=abc123 -b dev -e dev

Secrets in appmecca.yaml¶

If you accidentally committed secrets in the environment section of appmecca.yaml:

Remove them from the file
Set them securely: mecca var set myapp -e dev SECRET_KEY=abc123
Rotate the exposed credentials
Consider using git filter-branch or BFG Repo-Cleaner to remove from Git history

Branch Deployment Issues¶

Wrong branch deployed¶

If the wrong code is running, verify which branch is deployed:

mecca app show myapp -e dev -b feature/payments

Check the image tag -- it corresponds to the Git commit hash from the branch that was built.

Branch URL not working¶

Branch URLs use a sanitised branch name: it is lowercased, and every character other than a letter, digit or hyphen becomes a hyphen, with repeated hyphens collapsed. So feature/payments becomes feature-payments, and release/1.0.0 becomes release-1-0-0 — a dot never reaches the hostname, because it would split the name outside the platform's TLS certificate. Two branches that sanitise to the same name (for example release/1.0.0 and release/1-0-0) cannot both be registered; the second is refused with a 409.

A hostname's first label (everything before .{env}.appmecca.net) is limited to 63 characters. If a new branch would exceed that, mecca register refuses it with a 422 that tells you how long the branch name may be.

Like every AppMecca URL, branch URLs always carry the service name and your group slug:

https://{app}-{branch}-{service}-{group}.{env}.appmecca.net

# e.g. app "myapp", branch "feature/payments", service "web", group "acme":
https://myapp-feature-payments-web-acme.dev.appmecca.net

On the app's production branch the branch segment is dropped (myapp-web-acme.dev.appmecca.net). Use mecca app show <app> to see the resolved URL rather than assembling it by hand.

If the URL is not resolving:

Check that the branch deployment completed: mecca app show myapp -e dev -b feature/payments
List all branches to verify it's deployed: mecca app show myapp -e dev
Allow 30-60 seconds for DNS propagation after the first branch deploy

Branch using main's database¶

Each branch gets its own separate database — and its own drives and buckets. If your branch appears to share data with main, verify:

The deployment used the -b flag: mecca app deploy myapp -e dev -b feature/payments
The APPMECCA_BRANCH environment variable inside the container shows the correct branch (mecca debug myapp -b feature/payments shows the resolved env)

Cache data leaking between branches¶

Branches share the managed Redis cache but are isolated via CACHE_PREFIX. If you see cache data from another branch:

Verify your application uses the CACHE_PREFIX environment variable when constructing cache keys
Check CACHE_PREFIX inside the container -- it should be myapp:feature-payments: (not myapp:main:)
See Cache Configuration for code examples

Two different things are called "cache"

The top-level cache: block enables the managed Redis cache and is off unless you set enabled: true. The per-service deploy[].cache: block configures edge caching for that service's HTTP responses, and is on whenever the block is present. They are unrelated features — see Cache Configuration and Content Caching.

Cannot communicate between branches¶

Containers from different branches run on separate Docker networks and cannot communicate with each other. This is by design. If you need cross-branch communication, use the external HTTPS URLs.

Cleaning up old branches¶

Merged feature branches should be cleaned up to free ports, routing rules, and Docker networks:

# Undeploy (stops containers, preserves data)
mecca app undeploy myapp -e dev -b feature/payments --force

# Full cleanup (backs up then DROPS the database, removes drives, buckets,
# images and registry records for that branch)
mecca app delete myapp -e dev -b feature/payments

delete drops the database

mecca app delete takes a safety backup first and then drops that branch's database. Existing backup bundles are retained, so the delete is recoverable via mecca app restore — but the live data is gone. mecca app unregister is not a lighter alternative: it performs the same full purge.

See Branch Deployments for the complete guide.

Getting More Information¶

When reporting issues, gather this information (add -b <branch> for branch deployments):

# One-shot diagnostic dump (start here)
mecca debug myapp -e dev

# Service status + resolved URLs + deployed branches
mecca app show myapp -e dev

# Queued / in-flight actions for the app
mecca queue list -a myapp -e dev

# A specific action (takes the ACTION ID from `queue list`, not the app name)
mecca queue status <action-id> -e dev

# Recent container logs
mecca logs myapp --type container --service web -e dev

# What the deploy itself did
mecca logs myapp --type deploy -e dev

Platform operators can add fleet-level context:

appmecca agent status -e dev
appmecca agent logs -e dev --since 1h
appmecca infra status -e dev --live

Get more detail

mecca has no verbosity flag. For machine-readable output with the full payload, add --json to any command:

mecca queue status <action-id> -e dev --json

The -v / -vv flags belong to the admin CLI (appmecca -vv infra status).

Find deployment logs by type

To see only the logs from a deploy (not the running container), filter by type:

mecca logs myapp --type deploy        # what the deploy itself did
mecca logs myapp --type build         # the build service output
mecca logs myapp --type container --service web   # the app's own stdout
mecca logs myapp --type backup        # backup / restore operations

--type deploy is the fastest way to see why a deploy failed — it returns the agent's deploy steps without the noise of build or container output.

Empty container logs — what it means

When --type container returns nothing, the hint tells you which case you're in. "No container logs found" with a "try a wider --since" hint means the log stream exists and is simply quiet — widening the window may help. A hint that no log stream exists yet means nothing has deployed for this app on this branch (or the deploy is still running) — no time range will help; check mecca app show <app>. If the log pipeline itself is broken (a permissions or throttling problem) the command shows a yellow error, not silence.

CORS errors calling my own backend from my own frontend¶

Symptom — your frontend cannot call your backend, and the browser console blames CORS:

Access to fetch at 'https://myapp-backend-mygroup.qa.appmecca.net/api/...'
  from origin 'https://myapp-frontend-mygroup.qa.appmecca.net'
  has been blocked by CORS policy:
  No 'Access-Control-Allow-Origin' header is present on the requested resource.
... Response to preflight request doesn't pass access control check:
  Redirect is not allowed for a preflight request.

Your CORS config is almost certainly fine. The usual cause is that your backend is private, and the request never reaches your code.

New apps are private by default. A private app sits behind the auth gateway, which answers any unauthenticated request with a redirect to the login page. When your frontend calls your backend on a different hostname, the browser first sends a preflight OPTIONS request — and preflights are sent anonymously, with no cookies, by design. The gateway redirects it, the browser is not allowed to follow a redirect on a preflight, and it reports the failure as a missing CORS header. Your backend never ran, so it never got the chance to send one.

⚠ Logging in does not fix it. The session cookie is deliberately scoped to a single hostname, so a session on your frontend's host is never sent to your backend's host.

The fix — declare the backend public in appmecca.yaml:

deploy:
  - name: backend
    container: backend
    port: 8000
    visibility: public

visibility is per-service in yaml but resolves to one app-level flag (most-restrictive-wins), so this makes the whole app public — which a browser-facing frontend needs anyway.

⚠ An app that is already registered also needs a one-time flip. Visibility is set when the app is first registered and is deliberately not re-derived on rebuild, so editing the yaml alone will not change a running app:

mecca app publish <app>

The backend goes public within about 15 seconds at the load balancer, and is fully converged within about 90 (each gateway replica caches its routing for 60 s and expires on its own clock).

⚠ A static site takes minutes, not seconds, and the first time is the slowest. A private static app deliberately has no CloudFront distribution — it is served through the gateway instead, and standing an edge distribution up beside that would be a second, unauthenticated front door. So publishing a static app for the first time has to create one; the platform then waits for CloudFront to finish deploying before moving DNS, because moving it early serves 403s. Until that completes the app's URL returns 404. Nothing is wrong and no redeploy is needed — but if it is still 404ing after ten minutes, check mecca queue status for the reconcile.

⚠ An app that has never deployed has nothing to serve. Publishing it succeeds and the app really is public, but there is no bundle in storage yet, so the URL cannot answer until you deploy. mecca queue status shows the skip and names the remedy.

Can I keep the backend private? Not while the browser calls it directly on a separate hostname — that is a limitation of how CORS preflights and cookies work, not of the platform. Same-origin routing (serving the API under the frontend's own hostname, so there is no cross-origin request at all) is the durable answer and is planned.

An internal service URL returns "Connection refused"¶

Symptom — a call to ${service:name:internal} fails with [Errno 111] Connection refused (or the equivalent for your HTTP client), while the same service answers fine on its public ${service:name} URL.

This used to be a known platform bug and is now fixed. The internal URL carried the host port the target service happened to be on when your container started, and that port moved every time the target was redeployed — so the URL was correct on the target's first deploy and refused from its second onward. Since BUGS #680 a service's internal address is permanent: the platform keeps a listener on it and follows the service wherever its container moves.

If you still see this, it is worth reporting — but check these first:

Different branches. Two services can only reach each other internally when they are in the same environment and the same branch. This has always been true and is not a bug.
The target has never deployed successfully. An internal address only exists once the service has run at least once.
A very recent platform upgrade. Immediately after the fix rolled out, a service adopts its permanent address on its next ordinary deploy. If your target has not been deployed since, redeploy it once.

How to confirm it is not your code — compare the address your container was given against where the service actually is:

# The URL your container was handed (admin scope)
mecca debug <app> --container-env -s <calling-service>

# The live host port per running service (platform operators)
appmecca infra pool -e <env> --json

Unlike before, these two are now expected to differ: the URL carries the service's permanent address while the container runs on a temporary port behind it. A refusal means the listener is missing, not that the ports disagree — so raise it with your operator, who can check for port_relay_still_unserved_after_cleanup in the agent log.