Best Practices for Production Apps

This guide is for anyone taking an app from “it works” to “people depend on it”. It explains the setup we recommend, and the habits that keep an app up while you deploy, scale, change its database and recover from mistakes. Each section says what to do, why it matters, and where to read more.

If you’re in a hurry, jump to the checklist at the end.

The recommended setup

   GitHub / GitLab repo (main)
              │  every push
              ▼
   development   builds every push · Wait for CI on · linked to the repo
              │  promote (the same version, never rebuilt)
              ▼
   staging       promote-only
              │  promote
              ▼
   production    promote-only · Protected

In words:

1. Your code lives on GitHub or GitLab, and one Environment (here development) is linked to your main branch. Every push is built and deployed there automatically.
2. Wait for CI is on (CI is your automated tests, such as GitHub Actions or GitLab CI), so d2o waits for the commit’s checks. Configure a test job on the linked branch: if a commit has no checks, d2o deploys after 10 minutes. See Wait for CI.
3. Staging and production never build anything. You promote the exact version that already ran in development, so what you tested is byte-for-byte what you ship.
4. Production is Protected, so it can’t be deleted by accident and promoting into it asks you to confirm.

The rest of this guide explains each piece, and how to build your app so that deployments don’t interrupt anyone.

1. Deploy from GitHub or GitLab

Linking a repository is the recommended way to get code onto d2o:

• Your repo is the source of truth. What’s running is always a commit you can see on GitHub or GitLab, reviewed through your normal pull or merge requests.
• Tests gate deployments. With Wait for CI on, d2o waits for the commit’s checks (GitHub Actions, GitLab CI, or any other CI that reports a status) and refuses to deploy a commit whose checks failed.
• You see the result on the commit. Each push gets a d2o: <environment> check showing Building, Rolling out and then Live — or why it didn’t deploy — with a link to the deployment.
• Keep deployments tied to the linked repository. Leave Allow direct pushes off to refuse pushes straight to d2o’s git remote. Configure branch protection and required reviews on GitHub or GitLab to enforce your review policy.

To set it up:

1. An organization Owner or Admin connects GitHub or GitLab under the organization’s Settings → Integrations.
2. Link your build Environment (or an app — see the next section) to the branch you deploy, usually main.
3. Turn on Wait for CI if your repository runs tests.

If a push ever doesn’t start a deployment, Deploy latest deploys the branch’s current commit on demand.

Can’t use GitHub or GitLab? A CI job can push with a deployment token instead. It’s scoped to one Environment or app and belongs to the organization, not to a person.

See Deploy from GitHub or GitLab for every option.

2. One repository for the Environment, or one per app?

Every Environment can deploy in two ways, and you choose per app:

Rule of thumb: if your apps share code and should always run the same version, use the Environment repository. If they’re separate projects that change at different times, give each its own.

With the Environment repository, each app picks which process to run — web, worker, and so on — from your Procfile or from d2o-paas.yaml. See One remote for several apps.

3. Build once, then promote

Set up a pipeline of Environments in your project:

• One build Environment (for example development). It builds every push and is the one you link to GitHub or GitLab.
• Promote-only Environments after it (staging, production). Switch them to promote-only in the Environment’s Settings → General → Release source before you create apps in them; each app you then create there starts from the same-named app upstream.

Promoting copies the exact build that ran upstream — it never rebuilds — so a version that worked in staging is the same bytes in production. The pipeline board on the project page shows which version is in each stage and promotes with one click, or from the CLI:

paas promote web --from staging --to production -o acme -p shop

Things to know:

• Data and settings aren’t promoted. Each Environment has its own databases, buckets, secrets and environment variables. Set them up in every stage.
• Your app’s release command runs in each Environment you promote into, against that Environment’s database (see migrations).
• Link only the build Environment to GitHub or GitLab. Promote-only Environments can’t be linked, because they never build.
• Turn on Protected for production. Promoting into a protected Environment asks you to type its name, and it can’t be deleted.

See Pipelines, Promotion & Rollback.

4. Build your app the way the platform expects

Follow Designing Apps for d2o: keep durable state outside instances, read runtime configuration from the environment, validate required settings, bind to 0.0.0.0:$PORT, and log to stdout/stderr. Temporary scratch files are fine; they must not be the only copy of data you need to keep.

Version process selection, port, health checks, and release commands in d2o-paas.yaml. The tutorial-to-production guide identifies additional work needed before exposing the example apps to real users.

5. Zero-downtime deployments

When you deploy, d2o replaces your app’s instances one at a time:

1. It starts one new instance next to the old ones.
2. It waits until the new instance passes its health check.
3. Only then does it stop one old instance — and repeats until every instance runs the new version.

Old instances keep serving traffic the whole time. If the new version never becomes healthy, the old ones are never stopped and the platform rolls back for you. A few habits make this work.

Give every web app a health check

Set an HTTP path your app actually serves; give workers an HTTP or command check where practical. See health-check defaults and rollout waiting periods. A missing probe cannot establish that an app is ready to serve, and a singleton still stops before its replacement starts.

Keep the health check fast and shallow

Return a quick, unauthenticated success response once the process has started. Avoid cascading restarts when a shared dependency is briefly unavailable. The health-check guide explains ongoing checks and failure behavior.

Shut down gracefully

Handle SIGTERM, drain in-flight work, and exit within the 30-second window. Workers must safely retry interrupted jobs. Follow the shutdown and retry guidance.

A stuck deployment fails after five minutes

A deployment fails if five minutes pass without another new instance becoming healthy; a slow deployment that keeps making progress is never cut short. A new instance that keeps crashing — three restarts — fails the deployment right away, with the app’s last log lines as the reason. Follow a deployment in the dashboard, or with:

paas watch

See Deployments, Health Checks & Rollback.

6. High availability

Run at least two instances

An app with one instance has one point of failure. When it runs two or more:

• They’re spread across different servers, so if one server fails, the other instances keep serving.
• Platform maintenance moves one instance at a time, so the app stays up.

With a single instance, your app is unavailable while that instance moves — for a few minutes if its server fails. For anything your users depend on, run at least two:

paas scale 2 -o acme -p shop -e production -a web

Autoscale with a floor of two

Autoscaling adds instances when CPU use is high and removes them when it drops. For production, set the minimum to at least 2, so the app stays spread across servers even when traffic is quiet.

The Target CPU % is measured against each instance’s baseline of 0.1 CPU core (or its CPU limit, if that’s lower), not its CPU limit — so the default target of 70 adds instances once they average about 0.07 cores each.

Use singleton only when an app must never run twice

A singleton app runs at most one instance, and every deployment, restart or configuration change stops the old instance before starting the new one. That means a short outage every time — from a few seconds to a minute, depending on how fast your app stops and starts — and several minutes if its server fails.

Use it only when two copies running at once would cause a problem — a chat bot that holds one live session, or a scheduler that must not fire twice. Everything else should run normally.

Give instances enough memory

An instance that uses more memory than its Memory Limit is stopped and restarted (exit code 137). Watch the memory chart on the app’s Metrics and raise the limit before you get close to it. CPU works differently: an instance that uses too much is slowed down, not stopped.

Protect your data

Choose PostgreSQL with high availability when production data needs automatic failover. Attach services instead of copying credentials. Review the durability and recovery matrix for backup coverage, Redis/MongoDB limitations, object-storage tiers, and what deletion removes. Practise recovery; replication does not undo accidental deletion.

7. Change your database safely

Run migrations as a release command

Configure a release command for versioned migrations. It runs with the new code and target Environment’s configuration before rollout, while old code continues serving. A shared build runs it once for the build, not independently for every app. Follow the execution and retry rules in that guide; keep long backfills in background jobs.

Old and new code run at the same time

This is the most important rule for migrations. During a deployment, old and new instances serve traffic side by side for a while. And after a rollback, the old code runs against the database your new migration already changed — a rollback never undoes database changes.

So every migration must keep working with both the version before it and the version after it.

Do:

• Add before you remove. Add a new column or table in one deployment, and remove the old one in a later deployment, once no running code uses it.
• Make new columns optional (nullable, or with a default), so old code that doesn’t know about them can still insert rows.
• Widen before you narrow — loosen a constraint before the code relies on the looser rule; tighten it only after all code obeys it.

Don’t:

• Rename or drop a column in one step. Old instances still running would break immediately.
• Edit a migration that has already run. Write a new one instead.

Example: renaming name to full_name, safely. Remember that the release command runs before the new code starts, so each step must still work with the code from the step before:

1. Deployment 1 — add a nullable full_name column. The code writes both columns and reads name.
2. Deployment 2 — copy the old values into full_name (as a background job, if the table is large). The code still writes both, but now reads full_name.
3. Deployment 3 — make name nullable. The code stops using it.
4. Deployment 4 — drop name.

Make migrations safe to run twice

A release command can occasionally run more than once — for example if its server is restarted mid-run, or when you roll back by hand, which runs the older version’s release command again. Use your framework’s migration tool, which records what has already run, rather than raw SQL scripts.

For PostgreSQL, each deployment whose release command ran shows a data restore point — the time to ask support to restore to if a migration damages data. See Database migrations and rollbacks.

8. Rollbacks

Automatic rollback: keep it on

If a new version doesn’t become healthy, d2o puts the previous version back on its own. This is on by default. Keep it on.

It protects you during the deployment itself. It doesn’t help with:

• A build or release command that fails. Nothing changed yet, so the old version simply keeps running.
• A version that breaks after it went live. Roll back by hand (below).
• Configuration changes. Changing an environment variable or attaching a database restarts your app with the new settings, but it isn’t a tracked deployment: there’s no deployment number and no automatic rollback. A health check still keeps traffic away from instances that never become ready. Try configuration changes in staging first.

Rolling back by hand

How you roll back depends on the Environment:

• In a promote-only Environment (staging, production), click Rollback… in the app’s header and pick an earlier release, or run paas rollback web --to <deployment-id> (the ids are in paas deploys list --json). It switches straight back to a build that already ran there, so it’s fast — nothing is rebuilt. This is one more reason to promote into production rather than build there.
• In a build Environment, fix forward: push a fix, or git revert the bad commit and push. The branch decides what runs, so a revert is a rollback. (If the newest deployment failed, its deployment page also offers Roll back.)

What a rollback restores

Because the database stays as it is, a rollback is only safe if your migrations followed the rules above.

9. Configuration and secrets

Set plain configuration at the appropriate scope using the precedence rules. Keep each stage’s settings separate. Use secrets for credentials and attachments for managed services.

Variable changes and secret rotation roll out automatically; the secret lifecycle table explains the delete-and-recreate exception. Test configuration changes in staging because they do not have numbered-deployment automatic rollback.

10. Watch your app

• Follow every deployment. paas watch ends with exit code 0 only when the new version is live, so you can use it as a step in CI with an account API token. See CI verification.
• Read your logs in the app’s Logs tab or with paas logs --tail — they include every instance. See Logs.
• Check metrics after a deployment and when traffic changes: CPU, memory and, for web apps, requests and response times. See Metrics.
• Deployment problems are emailed to the organization owner: failed builds, failed or rolled-back deployments.

Production checklist

Before you send real users to an app, check that:

• The code deploys from a linked GitHub or GitLab repository, with Wait for CI on and direct pushes off.
• There’s a build Environment, and staging and production are promote-only. Production is Protected.
• The app listens on $PORT on 0.0.0.0, logs to stdout/stderr, and keeps durable state outside instances (local scratch files are temporary).
• It checks its required settings at startup and exits with a clear message if one is missing.
• Web apps have a fast, shallow health check path; workers have a health check command.
• The app handles SIGTERM and finishes in-flight work within 30 seconds.
• Production runs at least two instances (or autoscales with a minimum of two), and isn’t a singleton unless it must be.
• Memory limits leave headroom above normal use.
• Production data lives in PostgreSQL with High availability, attached rather than copied.
• Migrations run as the release command, are backward compatible, and are safe to run twice.
• Automatic rollback is on, and you know how to roll back by hand in each Environment.
• Credentials are in secrets, and each stage has its own settings and databases.

Next steps

• Deploy from GitHub or GitLab — connect your repository
• Pipelines, Promotion & Rollback — set up development, staging and production
• Deployments, Health Checks & Rollback — what happens during a deployment, in detail
• Scaling & Autoscaling — instances, CPU, memory and singleton apps
• d2o-paas.yaml Reference — keep your app’s settings in your repo