Rollback and deployment concurrency in GitHub Actions

Read this page in French

A rollback is not improvised on the day of the incident: it is prepared while everything is fine. This page shows how to go back to the previous version without replaying the pipeline, why cancel-in-progress is dangerous on a deployment job, and what the GitHub history lets you find when you have to decide fast.

What you will learn

• Tell apart a rollback from an emergency fix, and know which to apply
• Replay the previous digest without rebuilding or retesting
• Find the previous version in the deployment history
• Serialise the deployments with concurrency, without cancelling the one running
• Prepare what must exist before the incident for all of this to work

Rollback or fix: two different answers

Facing a regression in production, two paths exist, and confusing them costs time at the worst moment.

	Rollback	Emergency fix
Principle	Redeploy the previous, known-good version	Fix, test, deploy a new version
Duration	Minutes	A full cycle
Risk	Low, the artifact has already run	That of a normal deployment, under pressure
When	A functional regression, an outage	A security flaw, data corruption

Glissez pour voir

The default rule is rollback first, diagnose after. A direct fix is justified when going back is not safe, typically after an irreversible schema migration, or when the previous version carries the vulnerability you have just fixed.

The precondition: an artifact still available

A rollback is only possible if the previous version still exists, and if you can name it. That is the counterpart of the promotion chain: you do not redeploy a commit, you redeploy a digest.

Deployment N-1  ->  ghcr.io/my-org/my-app@sha256:3f1c...   (still in the registry)
Deployment N    ->  ghcr.io/my-org/my-app@sha256:9a47...   (the regression)
Rollback        ->  redeploy sha256:3f1c..., with no rebuild

Three conditions make that true, and each is checked before the incident:

• the previous image has not been deleted from the registry by an overly aggressive cleanup policy;
• the digest is findable, through the deployment history or the registry tags;
• the deployment workflow accepts a digest as an input, rather than only deploying main.

Artifact retention sets the depth of your rollback

Workflow artifacts expire after 90 days by default, and registry cleanup policies often delete untagged images. If your only copy of version N-1 is an expired artifact or a purged image, the rollback becomes a rebuild, with all the risks of a fresh deployment. Check what your policies actually keep.

Finding the previous digest

GitHub's deployment history keeps a trace of every deployment per environment. From the command line, the gh CLI queries it directly:

# The latest deployments of the production environment
gh api "repos/my-org/my-app/deployments?environment=production&per_page=10" \
  --jq '.[] | {id, sha, created_at, ref}'

To find the images available on the registry side, with their digests:

# The published versions of the container package
gh api "orgs/my-org/packages/container/my-app/versions" \
  --jq '.[] | {id, digest: .name, tags: .metadata.container.tags, created_at}'

Both commands are worth knowing before you need them. Typing them for the first time during an outage adds a problem to a problem.

The rollback workflow

There is no specific rollback workflow: it is the deployment parameterised by digest, called with the previous value. The same path, the same controls, the same approvals.

name: Deploy

on:
  workflow_dispatch:
    inputs:
      digest:
        description: "Digest to deploy (sha256:...), rollbacks included"
        required: true
        type: string

permissions: {}

concurrency:
  group: deploy-production
  cancel-in-progress: false   # we never cut a running deployment

jobs:
  deploy:
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    environment:
      name: production
      url: https://my-app.example.com
    permissions:
      contents: read
    env:
      IMAGE: ghcr.io/my-org/my-app
      DIGEST: ${{ inputs.digest }}
    steps:
      - name: Verify the provenance of the requested digest
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: gh attestation verify "oci://${IMAGE}@${DIGEST}" --owner my-org
      - name: Deploy
        run: ./deploy.sh "${IMAGE}@${DIGEST}"
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

Keeping the provenance verification in the rollback path is deliberate. It is precisely under pressure that you would accept deploying anything "just to get it working again", and precisely the moment an attacker wants you to.

Should you keep the approval on a rollback?

Yes, and it costs less than it looks: a single approval is enough, and the reviewers are already mobilised during an incident. Removing it creates an uncontrolled path to production, which will end up serving something other than a rollback. If the approval delay really is the problem, widen the list of reviewers, do not drop the rule.

Concurrency: serialise without cancelling

The concurrency block groups runs and decides the fate of overlapping ones. On a CI, you happily cancel the previous run: it carries an outdated commit. On a deployment, cancelling is dangerous.

# ✅ Deployment: queue them, do not cancel
concurrency:
  group: deploy-production
  cancel-in-progress: false

# ❌ On a deployment: cuts a production release in half
concurrency:
  group: deploy-production
  cancel-in-progress: true

A deployment interrupted mid-flight leaves the system in an intermediate state: half the instances on the new version, a migration applied without the matching code, a lock never released. That is exactly the situation the rollback will then have to untangle, starting from a state no version has ever known.

The group name deserves attention. A group per environment (deploy-production) serialises production releases while letting staging move forward in parallel:

concurrency:
  group: deploy-${{ github.event.inputs.environment || 'production' }}
  cancel-in-progress: false

On the CI side, by contrast, cancelling remains the right answer:

# CI: cancel stale runs of the same branch
concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

What has to be prepared

1. A deployment parameterisable by digest, not only triggered by a push to main.

2. A retention policy keeping at least the last few deployed versions, untagged images included.

3. The search commands for the previous digest, written somewhere reachable outside the tool that is down.

4. A concurrency group per environment, with cancel-in-progress: false.

5. A written decision on the cases where a rollback is forbidden, typically after an irreversible schema migration.

Key points

• Rollback first, diagnose after, except after an irreversible migration or when the running version fixes a flaw.
• A rollback redeploys a digest, not a commit: there is no rebuild, no new tests, therefore no new risk.
• The retention of the registry and the artifacts sets the real depth of your rollback; 90 days by default on the artifact side.
• The rollback workflow is the parameterised deployment, with the same provenance checks and the same approval.
• On a deployment, cancel-in-progress: false: a production release cut in half leaves a state no version has known.
• A concurrency group per environment serialises production without blocking staging.
• The commands that find the previous digest are tested before the incident, not during.

Next steps

• Lab: promoting an approved deployment: the rollback exercised for real on a repository, after a promotion that has been approved.
• Concurrency: the grouping mechanism in detail, including the cancel-in-progress a deployment must leave at false.
• Runners: an introduction: choosing and hardening the machines that run these deployments.

×

📖 Voir la documentation →