---
title: "Frontend CI/CD in the age of AI - Part 2: Deployments"
publishDate: 2026-08-05T00:00:00.000Z
excerpt: "Agents write code that passes every test and still breaks in production. Here is how to ship it to a slice of your users first, and how to take it back in seconds when it goes wrong."
category: "devops"
tags: ["javascript", "typescript", "ci-cd", "devops", "vercel", "cloudflare", "netlify", "deployment", "ai"]
canonical: https://neciudan.dev/ci-cd-in-the-age-of-ai-part-2-deployments
---

[Part 1](https://neciudan.dev/ci-cd-in-the-age-of-ai-part-1) covered the Continuous Integration part, where the goal was to run the pipeline less (cheaper) and run it faster, because agents produce more pull requests than a normal pipeline can absorb.

On the other hand, while AI has gotten better and better at coding and reviewing code, it still is missing context and product sense, and as a result, it makes mistakes. 

A lot of mistakes.

And this is an industry problem. If you look carefully, reliability is down across the board.

DORA's 2025 report says that teams with higher AI adoption show both higher delivery throughput and higher delivery instability, resulting in more changes failing in production and more unplanned work to fix them.

This makes our deployment part that much more important. 

We need to either deploy to a subset of our users to ensure it doesn't cause outages, and revert quickly if something goes wrong.

Unfortunately, that is not how we are doing things. 

Most of the companies I worked with typically use an all-or-nothing approach to deployments, and reverting often requires rebuilding the entire CI pipeline, which can take hours.

Here is how our ideal pipeline looks like 👇

A pull request merges into main only through a merge queue, where the full integration suite runs against the latest code. We do this because in the Part 1 article, we only checked what was touched or impacted by the PR code.

The merge produces a single build artifact, which we use for deployment and later serve in production without being rebuilt, so promoting and rolling back both come down to where the traffic is pointing.

That artifact goes out to a piece of your traffic, with a smoke test hitting it the moment it lands. Five minutes later, something looks for catastrophic movement, and an hour after that, a slower comparison against the previous release decides whether the artifact is ready for 100% traffic.

A bad result rolls it back, posts the offending pull request to Slack, opens a revert branch, and freezes merging until somebody closes the incident.

In this article, we are building this pipeline (or parts of it).

Let's go.

## Canaries 

British coal mines carried canaries underground until 1986.

A canary breathes through a one-way system with air sacs feeding the lungs, and it burns oxygen fast enough that carbon monoxide fills it well before anyone holding the cage feels a thing. 

A bird that stopped singing and swayed on its perch bought the crew a few minutes to climb out. Later cages came with a small oxygen bottle attached, so you could seal the bird in and revive it on the way up.

Deployments applied the same concept. A small, known group goes into the unverified place ahead of everybody else, and you watch them while the rest of your traffic carries on where it was.

Your five percent of traffic is the canary, and when you have a broken release, they are paying the price. 

(To mitigate for randomness and not affect important users, some companies apply this pattern to geo-location traffic, testing in low-impact countries)

## The workflow we start from

Most teams have something close to this, which builds the app and ships it in one go.

```yaml
# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: 'npm'
      - run: npm ci
      - run: npm run build
      - run: npx vercel deploy --prod --token="$VERCEL_TOKEN"
        env:
          VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
```

That last step is the only line that changes from platform to platform, and the moment it finishes, every user gets the new code.

Our pipeline, regardless of the platform we are deploying to, will use these concepts and I will refrence them all through the article:

- **deploy** ships the build and gives it a small slice of traffic
- **smoke** hits the new version before anybody else does with some tests
- **soak** waits, then reads your error rate split by version
- **promote** gives the build all the traffic
- **abort** removes the traffic from the build and gives it to the previous version
- **hold** posts to Slack when the numbers are too thin to call
- **watch-15** and **watch-60** keep checking after promotion
- **rollback** puts the previous build back when they find something

### The common setup

Every job below starts the same way:

```yaml
# .github/actions/setup/action.yml
name: Setup
description: Node and dependencies
runs:
  using: composite
  steps:
    - uses: actions/setup-node@v4
      with:
        node-version: 22
        cache: 'npm'
    - run: npm ci
      shell: bash
```

Composite actions need `shell:` on every `run` step, and the action lives in your repository, so `actions/checkout` has to run before you can call it.

### The environments

The soak-and-watch jobs need to pause. 

If you use a step that does `sleep 1800` it will work but it will also occupy a runner for half an hour, billed to you on every merge.

GitHub environments do it for free. A job targeting an environment with a wait timer sits waiting before it starts, outside a runner.

In the Settings dashboard, then Environments, then New environment, tick Wait timer, and enter the minutes, with 43200 as the ceiling. All four at once from the CLI:

```bash
REPO=my-org/my-app

gh api -X PUT "repos/$REPO/environments/canary-soak" -F wait_timer=60
gh api -X PUT "repos/$REPO/environments/watch-15min" -F wait_timer=15
gh api -X PUT "repos/$REPO/environments/watch-45min" -F wait_timer=45
gh api -X PUT "repos/$REPO/environments/geo-soak"    -F wait_timer=60
```

### Telling Sentry which release is which

The soak job compares two releases (for bugs and issues), which only works if your app reports which one it is.

```js
// vite.config.ts
const release =
  process.env.VERCEL_GIT_COMMIT_SHA ??  // Vercel builds on its own machines
  process.env.COMMIT_REF ??             // so does Netlify
  process.env.GITHUB_SHA ??             // Cloudflare, Cloud Run, and ECS build in CI
  'local';

export default defineConfig({
  define: { 'import.meta.env.RELEASE': JSON.stringify(release) },
});
```

Vercel and Netlify build remotely, where `GITHUB_SHA` does not exist, so a config that reads only that variable tags every release as `local`, and the soak job compares two things that are not there. 

All three variables hold the full commit SHA, which is what `github.sha` gives the workflow.

```js
Sentry.init({
  dsn: import.meta.env.VITE_SENTRY_DSN,
  release: import.meta.env.RELEASE,
});
```

Without this, every query returns the same blended number for both versions, and the check can never tell them apart.

### The script that decides

We neeed to compare releases and for that we need to run a script.

The script can return `hold`, `promote`, and `abort`. At five percent traffic on  Tuesday early afternoon, half an hour might produce forty sessions, and forty sessions say nothing about whether your crash rate moved. 

Promoting on numbers that thin is guessing, and aborting throws away a deploy that was probably fine.

```js
// scripts/canary-health.mjs
import { appendFileSync } from 'node:fs';

const { SENTRY_TOKEN, SENTRY_ORG, SENTRY_PROJECT_ID,
        CANARY_RELEASE, BASELINE_RELEASE,
        MIN_SESSIONS = 200 } = process.env;

const MAX_DROP_POINTS = 0.5;   // crash-free rate, in percentage points

// Sentry declares this metric as `crash_free_rate@ratio`, so the API hands
// back 0..1 even though the UI shows you 99.98%. Converting on the value
// rather than assuming the scale costs nothing and survives Sentry changing
// its mind.
const points = (rate) =>
  rate === null || rate === undefined || Number.isNaN(rate)
    ? null
    : rate <= 1 ? rate * 100 : rate;

async function health(release) {
  if (!release) return { rate: null, sessions: 0 };

  const url = new URL(`https://sentry.io/api/0/organizations/${SENTRY_ORG}/sessions/`);
  url.searchParams.append('field', 'crash_free_rate(session)');
  url.searchParams.append('field', 'sum(session)');
  url.searchParams.set('project', SENTRY_PROJECT_ID);
  url.searchParams.set('statsPeriod', '1h');
  // query= rather than groupBy=release, which has a habit of
  // returning zeroed buckets for individual releases
  url.searchParams.set('query', `release:"${release}"`);

  const response = await fetch(url, {
    headers: { Authorization: `Bearer ${SENTRY_TOKEN}` },
    // Without this, a hung Sentry holds the runner to the six-hour job ceiling
    signal: AbortSignal.timeout(20_000),
  });
  if (!response.ok) throw new Error(`Sentry returned ${response.status}`);

  const totals = (await response.json()).groups?.[0]?.totals;
  if (!totals) return { rate: null, sessions: 0 };

  return {
    rate: totals['crash_free_rate(session)'] ?? null,
    sessions: totals['sum(session)'] ?? 0,
  };
}

function decide(decision, why) {
  console.log(`${decision}: ${why}`);
  // Running this by hand to debug a stuck rollout must not crash
  if (process.env.GITHUB_OUTPUT) {
    appendFileSync(process.env.GITHUB_OUTPUT, `decision=${decision}\n`);
  }
  process.exit(0);
}

try {
  const canary = await health(CANARY_RELEASE);
  const baseline = await health(BASELINE_RELEASE);

  if (canary.sessions < Number(MIN_SESSIONS)) {
    decide('hold', `only ${canary.sessions} sessions on the canary so far`);
  }

  const canaryRate = points(canary.rate);
  const baselineRate = points(baseline.rate);

  // `undefined` slips past a `=== null` check, makes the subtraction NaN, and
  // NaN fails every comparison — which reads as "no drop" and promotes a
  // release nobody measured. A baseline with no sessions is not a baseline.
  if (canaryRate === null || baselineRate === null || baseline.sessions === 0) {
    decide('hold', 'no session data for one of the two releases');
  }

  // Round before comparing, so the number in the log is the number
  // the decision was made on
  const drop = Number((baselineRate - canaryRate).toFixed(2));
  if (drop > MAX_DROP_POINTS) {
    decide('abort', `crash-free rate down ${drop.toFixed(2)} points`);
  }

  decide('promote', `crash-free rate within ${MAX_DROP_POINTS} points`);
} catch (error) {
  // Sentry being unreachable is not evidence that your deploy is bad
  decide('hold', `health check failed: ${error.message}`);
}
```

A hold parks the canary where it is, with production still mostly on the old build, which is a safe place to leave something while you look at it yourself.

But be aware that the endpoint buckets by the hour and refuses a window shorter than one, so a 30-minute soak reads 60 minutes of data, including traffic from before your deploy existed. 

Let it soak for an hour, or swap the query for a count of issues first seen since the deployment timestamp.

Until this point everything can be reused regardless of deployment platform. Let's go into specifics. You can skip to your platform of choice or the end where we build something independend. 

## Vercel

We need to change two settings to get canary releases to work on Vercel.

**Skew Protection.** 
Go to Settings, then Advanced, then switch it on and set Maximum Age. Having two versions live at once means that one browser can load HTML from the old version and JavaScript from the new, resulting in a blank screen. 

This pins each session to whichever deployment it first loaded.

**The stages.** Rolling Releases needs a Pro or Enterprise plan, and the percentages are fixed before the rollout as configuration.

```bash
vercel rolling-release configure --cfg '{
  "enabled": true,
  "advancementType": "manual-approval",
  "canaryResponseHeader": true,
  "stages": [
    { "targetPercentage": 5 },
    { "targetPercentage": 25 },
    { "targetPercentage": 60 },
    { "targetPercentage": 100 }
  ]
}'
```

### Deploying to five percent instead of everyone

The baseline workflow ended at `vercel deploy --prod`. Two lines change that into a canary.

```yaml
# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

# One rollout at a time. Two mergers of racing produce two canaries.
# GitHub keeps only one *pending* run per group, so a third merge arriving
# mid-rollout cancels the second. One at a time is guaranteed; every merge
# reaching production is not.
concurrency:
  group: production-deploy
  cancel-in-progress: false

permissions:
  contents: read

env:
  VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      url: ${{ steps.deploy.outputs.url }}
      baseline: ${{ steps.baseline.outputs.id }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup

      # What production is running right now, so the soak job has
      # something to compare against. The whole rollingRelease object is
      # null until a deployment has happened, so your first ever run gets
      # "null" here and the health check holds rather than promoting.
      # There is a sibling canaryDeployment.id if you would rather name
      # the canary that way than by commit SHA.
      - id: baseline
        run: |
          ID=$(npx vercel rolling-release fetch --token="$VERCEL_TOKEN" \
            | jq -r '.rollingRelease.currentDeployment.id')
          echo "id=$ID" >> "$GITHUB_OUTPUT"

      - id: deploy
        run: |
          URL=$(npx vercel deploy --prod --token="$VERCEL_TOKEN")
          echo "url=$URL" >> "$GITHUB_OUTPUT"
          # A start that fails leaves the deployment live on all the traffic,
          # with no smoke test run and no abort job watching it
          if ! npx vercel rolling-release start --dpl="$URL" --token="$VERCEL_TOKEN" --yes; then
            npx vercel rolling-release abort --dpl="$URL" --token="$VERCEL_TOKEN" --yes || true
            exit 1
          fi
```

`rolling-release start` is doing all the work in that step. It leaves the deployment sitting at stage zero on five percent, where, without it, the deployment would have taken everything.

That job doesn't include `npm run build`, because Vercel builds on its own infrastructure when you run `vercel deploy`.

### Hitting it before your users do

```yaml
  smoke:
    needs: deploy
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test e2e/smoke --project=chromium
        env:
          PLAYWRIGHT_BASE_URL: ${{ needs.deploy.outputs.url }}
```

### Waiting, then deciding

```yaml
  soak:
    needs: [deploy, smoke]
    runs-on: ubuntu-latest
    environment: canary-soak         # the 60-minute wait lives here
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          SENTRY_TOKEN: ${{ secrets.SENTRY_TOKEN }}
          SENTRY_ORG: my-org
          SENTRY_PROJECT_ID: '4504000000000000'
          CANARY_RELEASE: ${{ github.sha }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}
```

### Taking the rest, or giving it back

```yaml
  promote:
    needs: [deploy, soak]
    if: needs.soak.outputs.decision == 'promote'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      # complete skips the remaining stages and takes 100%
      - run: npx vercel rolling-release complete --dpl='${{ needs.deploy.outputs.url }}' --token="$VERCEL_TOKEN" --yes

  abort:
    needs: [deploy, smoke, soak]
    if: >
      always() && (
        needs.smoke.result == 'failure' ||
        needs.soak.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx vercel rolling-release abort --dpl='${{ needs.deploy.outputs.url }}' --token="$VERCEL_TOKEN" --yes
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Canary aborted. Production is back on the previous build.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"
```

A `hold` decision matches neither condition, so the canary stays at five percent and nothing else runs. 

The full file below adds a third job on `decision == 'hold'` that posts to Slack, because otherwise you find rollouts parked there days later.

### Rolling back after promotion

A release that behaves at 5% for an hour can still fail at full traffic.

We want to make sure we can always do a fast and safe rollback.

```yaml
  watch-15:
    needs: [deploy, promote]
    runs-on: ubuntu-latest
    environment: watch-15min
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          SENTRY_TOKEN: ${{ secrets.SENTRY_TOKEN }}
          SENTRY_ORG: my-org
          SENTRY_PROJECT_ID: '4504000000000000'
          CANARY_RELEASE: ${{ github.sha }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  watch-60:
    needs: [deploy, promote, watch-15]
    if: needs.watch-15.outputs.decision != 'abort'
    runs-on: ubuntu-latest
    environment: watch-45min    # runs after watch-15, so 15 + 45 = an hour in
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          SENTRY_TOKEN: ${{ secrets.SENTRY_TOKEN }}
          SENTRY_ORG: my-org
          SENTRY_PROJECT_ID: '4504000000000000'
          CANARY_RELEASE: ${{ github.sha }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  rollback:
    needs: [deploy, watch-15, watch-60]
    if: >
      always() && (
        needs.watch-15.outputs.decision == 'abort' ||
        needs.watch-60.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx vercel rollback '${{ needs.deploy.outputs.baseline }}' --token="$VERCEL_TOKEN"
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Rolled back after promotion.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"
```

The previous build never went anywhere, so `rollback` points the domain at something that already exists and finishes in seconds.

Sessions you pinned stay where they are, though. 

Skew Protection keeps them loading assets from the deployment they landed on until Maximum Age expires, so a rollback catches everybody arriving fresh while open tabs sit put.

### The whole file

Every fragment above, plus the `hold` job the soak decision needs, in one file.

```yaml
# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

concurrency:
  group: production-deploy
  cancel-in-progress: false

permissions:
  contents: read

env:
  VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
  SENTRY_TOKEN: ${{ secrets.SENTRY_TOKEN }}
  SENTRY_ORG: my-org
  SENTRY_PROJECT_ID: '4504000000000000'

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      url: ${{ steps.deploy.outputs.url }}
      baseline: ${{ steps.baseline.outputs.id }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: baseline
        run: |
          ID=$(npx vercel rolling-release fetch --token="$VERCEL_TOKEN" \
            | jq -r '.rollingRelease.currentDeployment.id')
          echo "id=$ID" >> "$GITHUB_OUTPUT"
      - id: deploy
        run: |
          URL=$(npx vercel deploy --prod --token="$VERCEL_TOKEN")
          echo "url=$URL" >> "$GITHUB_OUTPUT"
          # A start that fails leaves the deployment live on all the traffic,
          # with no smoke test run and no abort job watching it
          if ! npx vercel rolling-release start --dpl="$URL" --token="$VERCEL_TOKEN" --yes; then
            npx vercel rolling-release abort --dpl="$URL" --token="$VERCEL_TOKEN" --yes || true
            exit 1
          fi

  smoke:
    needs: deploy
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test e2e/smoke --project=chromium
        env:
          PLAYWRIGHT_BASE_URL: ${{ needs.deploy.outputs.url }}

  soak:
    needs: [deploy, smoke]
    runs-on: ubuntu-latest
    environment: canary-soak
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          CANARY_RELEASE: ${{ github.sha }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  promote:
    needs: [deploy, soak]
    if: needs.soak.outputs.decision == 'promote'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx vercel rolling-release complete --dpl='${{ needs.deploy.outputs.url }}' --token="$VERCEL_TOKEN" --yes

  hold:
    needs: soak
    if: needs.soak.outputs.decision == 'hold'
    runs-on: ubuntu-latest
    steps:
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Canary parked at 5%. Not enough data to decide.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"

  abort:
    needs: [deploy, smoke, soak]
    if: >
      always() && (
        needs.smoke.result == 'failure' ||
        needs.soak.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx vercel rolling-release abort --dpl='${{ needs.deploy.outputs.url }}' --token="$VERCEL_TOKEN" --yes
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Canary aborted. Production is back on the previous build.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"

  watch-15:
    needs: [deploy, promote]
    runs-on: ubuntu-latest
    environment: watch-15min
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          CANARY_RELEASE: ${{ github.sha }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  watch-60:
    needs: [deploy, promote, watch-15]
    if: needs.watch-15.outputs.decision != 'abort'
    runs-on: ubuntu-latest
    environment: watch-45min    # runs after watch-15, so 15 + 45 = an hour in
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          CANARY_RELEASE: ${{ github.sha }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  rollback:
    needs: [deploy, watch-15, watch-60]
    if: >
      always() && (
        needs.watch-15.outputs.decision == 'abort' ||
        needs.watch-60.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx vercel rollback '${{ needs.deploy.outputs.baseline }}' --token="$VERCEL_TOKEN"
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Rolled back after promotion.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"
```

### Rolling out to one country

Rolling Releases takes a percentage and nothing else, so a country-scoped rollout means routing in middleware and a config value your workflow can move.

Create the store once from the dashboard, under Storage, then Edge Config, then Create, and connect it to your project on the same screen. Vercel injects an `EDGE_CONFIG` environment variable for you.

```ts
// middleware.ts
import { NextResponse, type NextRequest } from 'next/server';
import { geolocation } from '@vercel/functions';
import { get } from '@vercel/edge-config';

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
};

export async function middleware(request: NextRequest) {
  const rollout = await get<{
    countries: string[];
    percentage: number;
    canaryHost: string;
  }>('canaryRollout');

  const { country } = geolocation(request);
  if (!rollout?.countries?.includes(country ?? '')) {
    return NextResponse.next();
  }

  // Returning visitors keep whatever they were given
  const existing = request.cookies.get('canary')?.value;
  const assigned =
    existing ?? (Math.random() * 100 < rollout.percentage ? 'on' : 'off');

  const url = new URL(request.url);
  url.host = rollout.canaryHost;

  const response =
    assigned === 'on' ? NextResponse.rewrite(url) : NextResponse.next();

  response.cookies.set('canary', assigned, { maxAge: 86400, path: '/' });
  return response;
}
```

The rollout then gets its own workflow, triggered by hand, with the same soak-and-decide loop pointing to the config value rather than a platform percentage.

```yaml
# .github/workflows/geo-rollout.yml
name: Geo rollout
on:
  workflow_dispatch:
    inputs:
      countries:
        description: 'Comma-separated ISO codes, for example PT, IE'
        required: true

concurrency:
  group: geo-rollout
  cancel-in-progress: false

jobs:
  start:
    runs-on: ubuntu-latest
    outputs:
      baseline: ${{ steps.baseline.outputs.id }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup

      # Without this, the health check compares against nothing, the
      # arithmetic gives NaN, and every rollout promotes itself
      - id: baseline
        run: |
          ID=$(npx vercel rolling-release fetch --token="$VERCEL_TOKEN" \
            | jq -r '.rollingRelease.currentDeployment.id')
          echo "id=$ID" >> "$GITHUB_OUTPUT"

      - name: 10% of the named countries
        run: |
          COUNTRIES=$(echo "${{ inputs.countries }}" | jq -R 'split(",")')
          npx vercel edge-config items add canaryRollout --value \
            "$(jq -nc --argjson c "$COUNTRIES" \
               '{countries:$c,percentage:10,canaryHost:"canary.example.com"}')"

  soak-10:
    needs: start
    runs-on: ubuntu-latest
    environment: geo-soak
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          SENTRY_TOKEN: ${{ secrets.SENTRY_TOKEN }}
          SENTRY_ORG: my-org
          SENTRY_PROJECT_ID: '4504000000000000'
          CANARY_RELEASE: ${{ github.sha }}
          BASELINE_RELEASE: ${{ needs.start.outputs.baseline }}
          MIN_SESSIONS: 40      # ten percent of Portugal is a small number

  widen:
    needs: soak-10
    if: needs.soak-10.outputs.decision == 'promote'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: |
          COUNTRIES=$(echo "${{ inputs.countries }}" | jq -R 'split(",")')
          npx vercel edge-config items add canaryRollout --value \
            "$(jq -nc --argjson c "$COUNTRIES" \
               '{countries:$c,percentage:50,canaryHost:"canary.example.com"}')"

  stop:
    needs: soak-10
    if: always() && needs.soak-10.outputs.decision == 'abort'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx vercel edge-config items add canaryRollout --value '{"percentage":0}'
```

Keep that `MIN_SESSIONS` override. Ten percent of Portugal produces far fewer sessions than ten percent of everything, and the default threshold would return `hold` on every run.

## Netlify

Netlify has a built-in canary called Split Testing, and you can only use it from the Netlify Dashboard.

Turn on branch deploys under Site configuration > Build and deploy > Branches and deploy contexts, and set Branch deploys to "All" or name your release branch. 

Push a candidate there, and Netlify builds it alongside main. 

Split Testing then lives under Site configuration > Split Testing, where you pick two branches and assign percentages.

The percentages sit behind a form with no CLI and no documentation in the API, so a workflow has no way to touch a test once someone has started one.

**So the pipeline does the split itself**, in an edge function reading numbers the CLI can set.

The obvious place to put those numbers is an environment variable, and it does not work. Netlify is explicit that "changes to environment variables for edge functions require a build and deploy to take effect" — each deploy captures the values as they were at deploy time. A workflow that sets `CANARY_PERCENT` would never move the running function, and, much worse, the abort job setting it back to zero would not either.

So the numbers live in [Netlify Blobs](https://docs.netlify.com/build/data-and-storage/netlify-blobs/) instead, which is Netlify's runtime store: readable from an edge function, writable from CI, and no deploy in between. It is the same shape as Vercel's Edge Config and Cloudflare's KV.

```js
// netlify/edge-functions/canary.js
import { getStore } from '@netlify/blobs';

export default async (request, context) => {
  // Blobs default to eventual consistency, which propagates within 60
  // seconds. An abort cannot wait 60 seconds, so ask for strong.
  const store = getStore({ name: 'canary', consistency: 'strong' });

  // A missing or malformed key sends everyone to the stable version
  const rollout = await store.get('rollout', { type: 'json' });
  if (!rollout?.percentage || !rollout.origin) return;

  // Countries stay empty for a plain percentage rollout
  const countries = rollout.countries ?? [];
  const country = context.geo?.country?.code;
  if (countries.length && !countries.includes(country)) return;

  // Returning visitors keep whatever they were given
  const existing = context.cookies.get('canary');
  const assigned =
    existing ?? (Math.random() * 100 < rollout.percentage ? 'on' : 'off');

  context.cookies.set({ name: 'canary', value: assigned, path: '/' });

  if (assigned === 'off') return;
  return context.rewrite(rollout.origin + new URL(request.url).pathname);
};

export const config = { path: '/*' };
```

That one file covers both the percentage rollout and the country rollout, since leaving `countries` empty skips the geo check. 

This file lives in `netlify/edge-functions/`.

Every command below writes that one key with `netlify blobs:set`, which takes effect on the next request.

### Deploying to five percent instead of everyone

```yaml
# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

concurrency:
  group: production-deploy
  cancel-in-progress: false

permissions:
  contents: read

env:
  NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
  NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      url: ${{ steps.deploy.outputs.url }}
      id: ${{ steps.deploy.outputs.id }}
      previous: ${{ steps.previous.outputs.id }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup

      - id: previous
        run: |
          PREV=$(netlify api listSiteDeploys --data '{"site_id":"'"$NETLIFY_SITE_ID"'"}' \
            | jq -r '[.[] | select(.state=="ready" and .context=="production")][0].id')
          echo "id=$PREV" >> "$GITHUB_OUTPUT"

      # Netlify deploys what you hand it, so the build happens here
      - run: npm run build

      - id: deploy
        run: |
          OUT=$(netlify deploy --dir=dist --json)
          echo "url=$(echo "$OUT" | jq -r '.deploy_url')" >> "$GITHUB_OUTPUT"
          echo "id=$(echo "$OUT" | jq -r '.deploy_id')" >> "$GITHUB_OUTPUT"

      - name: Send it 5% of traffic
        run: |
          netlify blobs:set canary rollout \
            "$(jq -nc --arg o '${{ steps.deploy.outputs.url }}' \
               '{origin:$o,percentage:5}')"
```

The `smoke` and `soak` jobs are the ones from the Vercel section, with `PLAYWRIGHT_BASE_URL` set to the draft URL and `BASELINE_RELEASE` set to `needs.deploy.outputs.previous`.

### Taking the rest, or giving it back

```yaml
  promote:
    needs: [deploy, soak]
    if: needs.soak.outputs.decision == 'promote'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      # Publish the draft that is already up there. Rebuilding here
      # would ship different bytes from the ones you just tested.
      - run: |
          netlify api restoreSiteDeploy --data '{
            "site_id": "'"$NETLIFY_SITE_ID"'",
            "deploy_id": "${{ needs.deploy.outputs.id }}"
          }'
          netlify blobs:set canary rollout '{"percentage":0}'

  abort:
    needs: [deploy, smoke, soak]
    if: >
      always() && (
        needs.smoke.result == 'failure' ||
        needs.soak.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: netlify blobs:set canary rollout '{"percentage":0}'
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Canary aborted. Split closed, production unchanged.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"
```

### Rolling back after promotion

Same two watch jobs as Vercel, with one step swapped. Deployments on Netlify are immutable, so the previous one remains available for republishing.

```yaml
  rollback:
    needs: [deploy, watch-15, watch-60]
    if: >
      always() && (
        needs.watch-15.outputs.decision == 'abort' ||
        needs.watch-60.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: |
          netlify api restoreSiteDeploy --data '{
            "site_id": "'"$NETLIFY_SITE_ID"'",
            "deploy_id": "${{ needs.deploy.outputs.previous }}"
          }'
```

Everybody moves at once here. Your edge function returns early when `CANARY_PERCENT` is zero, so a visitor holding a `canary=on` cookie stops being routed anywhere the moment the abort or promote step runs.

While you investigate, "Lock to stop auto publishing" on that deploy in the dashboard stops the next merge, which would put the bad version straight back.

### The whole file

```yaml
# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

concurrency:
  group: production-deploy
  cancel-in-progress: false

permissions:
  contents: read

env:
  NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
  NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
  SENTRY_TOKEN: ${{ secrets.SENTRY_TOKEN }}
  SENTRY_ORG: my-org
  SENTRY_PROJECT_ID: '4504000000000000'

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      url: ${{ steps.deploy.outputs.url }}
      id: ${{ steps.deploy.outputs.id }}
      baseline: ${{ steps.baseline.outputs.id }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npm run build
      - id: baseline
        run: |
          ID=$(netlify api listSiteDeploys --data '{"site_id":"'"$NETLIFY_SITE_ID"'"}' \
            | jq -r '[.[] | select(.state=="ready" and .context=="production")][0].id')
          echo "id=$ID" >> "$GITHUB_OUTPUT"
      - id: deploy
        run: |
          OUT=$(netlify deploy --dir=dist --json)
          echo "url=$(echo "$OUT" | jq -r '.deploy_url')" >> "$GITHUB_OUTPUT"
          echo "id=$(echo "$OUT" | jq -r '.deploy_id')" >> "$GITHUB_OUTPUT"
      - name: Send it 5% of traffic
        run: |
          netlify blobs:set canary rollout \
            "$(jq -nc --arg o '${{ steps.deploy.outputs.url }}' \
               '{origin:$o,percentage:5}')"

  smoke:
    needs: deploy
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test e2e/smoke --project=chromium
        env:
          PLAYWRIGHT_BASE_URL: ${{ needs.deploy.outputs.url }}

  soak:
    needs: [deploy, smoke]
    runs-on: ubuntu-latest
    environment: canary-soak
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          CANARY_RELEASE: ${{ github.sha }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  promote:
    needs: [deploy, soak]
    if: needs.soak.outputs.decision == 'promote'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: |
          netlify api restoreSiteDeploy --data '{
            "site_id": "'"$NETLIFY_SITE_ID"'",
            "deploy_id": "${{ needs.deploy.outputs.id }}"
          }'
          netlify blobs:set canary rollout '{"percentage":0}'

  hold:
    needs: soak
    if: needs.soak.outputs.decision == 'hold'
    runs-on: ubuntu-latest
    steps:
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Canary parked at 5%. Not enough data to decide.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"

  abort:
    needs: [deploy, smoke, soak]
    if: >
      always() && (
        needs.smoke.result == 'failure' ||
        needs.soak.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: netlify blobs:set canary rollout '{"percentage":0}'
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Canary aborted. Production is back on the previous build.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"

  watch-15:
    needs: [deploy, promote]
    runs-on: ubuntu-latest
    environment: watch-15min
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          CANARY_RELEASE: ${{ github.sha }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  watch-60:
    needs: [deploy, promote, watch-15]
    if: needs.watch-15.outputs.decision != 'abort'
    runs-on: ubuntu-latest
    environment: watch-45min    # runs after watch-15, so 15 + 45 = an hour in
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          CANARY_RELEASE: ${{ github.sha }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  rollback:
    needs: [deploy, watch-15, watch-60]
    if: >
      always() && (
        needs.watch-15.outputs.decision == 'abort' ||
        needs.watch-60.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: |
          netlify api restoreSiteDeploy --data '{
            "site_id": "'"$NETLIFY_SITE_ID"'",
            "deploy_id": "${{ needs.deploy.outputs.baseline }}"
          }'
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Rolled back after promotion.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"
```

### Rolling out to one country

The edge function above already reads `CANARY_COUNTRIES`, so the geo workflow is would be:

```yaml
  start:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: |
          COUNTRIES=$(echo "${{ inputs.countries }}" | jq -R 'split(",")')
          netlify blobs:set canary rollout \
            "$(jq -nc --argjson c "$COUNTRIES" --arg o "$CANARY_ORIGIN" \
               '{countries:$c,percentage:10,origin:$o}')"

  widen:
    needs: soak-10
    if: needs.soak-10.outputs.decision == 'promote'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      # Widen without touching the other keys
      - run: |
          CURRENT=$(netlify blobs:get canary rollout)
          netlify blobs:set canary rollout \
            "$(jq -c '.percentage = 50' <<<"$CURRENT")"

  stop:
    needs: soak-10
    if: always() && needs.soak-10.outputs.decision == 'abort'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: netlify blobs:set canary rollout '{"percentage":0}'
```

## Cloudflare

In the words of Tejas Kumar: "I goon to Cloudflare"

Cloudflare splits traffic between Worker versions, and it keeps uploading a version separate from deploying one, which makes it pleasant to automate.

```bash
WRANGLER_OUTPUT_FILE_PATH=out.ndjson npx wrangler versions upload
```

That returns an ID and a preview URL, and the version carries zero production traffic until you say otherwise. Your smoke tests then run against the preview before any split exists.

To have it working properly you need a header `Cloudflare-Workers-Version-Key`, so that every request with the same value goes to the same version. 

In the dashboard, open your zone, then Rules, then Transform Rules, then Modify Request Header.

```
When incoming requests match:   (http.host eq "app.example.com")

Then:  Set dynamic  →  Header name:  Cloudflare-Workers-Version-Key
                       Value:        http.request.cookies["sid"][0]
```

Swap `sid` for your session cookie and set that cookie in the first response, since visitors arriving without one fall back to per-request splitting.

The same rule as Terraform, if your zone lives in code:

```hcl
resource "cloudflare_ruleset" "version_affinity" {
  zone_id = var.zone_id
  kind    = "zone"
  phase   = "http_request_late_transform"

  rules {
    action     = "rewrite"
    expression = "(http.host eq \"app.example.com\")"
    action_parameters {
      headers {
        name       = "Cloudflare-Workers-Version-Key"
        operation  = "set"
        expression = "http.request.cookies[\"sid\"][0]"
      }
    }
  }
}
```

One more binding, so your metrics have a version to group by. The runtime hands you the ID, which means Cloudflare skips the build-time Sentry release step from earlier.

```jsonc
// wrangler.jsonc
{
  "version_metadata": { "binding": "CF_VERSION_METADATA" }
}
```

```ts
// run `npx wrangler types` after adding the binding so env is typed
export default {
  async fetch(request: Request, env: Env) {
    const { id: versionId } = env.CF_VERSION_METADATA;
    // attach versionId to your Sentry scope and your analytics events
  },
};
```

### Deploying to five percent instead of everyone

```yaml
# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

concurrency:
  group: production-deploy
  cancel-in-progress: false

permissions:
  contents: read

env:
  CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
  CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      version: ${{ steps.upload.outputs.id }}
      preview: ${{ steps.upload.outputs.preview }}
      previous: ${{ steps.previous.outputs.id }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npm run build

      - id: previous
        run: |
          PREV=$(npx wrangler deployments list --json \
            | jq -r '.[0].versions[0].version_id')
          echo "id=$PREV" >> "$GITHUB_OUTPUT"

      # Run this once locally and read the JSON before trusting either
      # path. An empty preview URL sends Playwright to localhost and
      # passes for the wrong reason.
      - id: upload
        env:
          WRANGLER_OUTPUT_FILE_PATH: ${{ runner.temp }}/wrangler.ndjson
        run: |
          npx wrangler versions upload
          # ND-JSON, one object per line. The version-upload entry has both.
          ENTRY=$(jq -sc '[.[] | select(.type=="version-upload")] | last' \
            "$WRANGLER_OUTPUT_FILE_PATH")
          echo "id=$(jq -r '.version_id' <<<"$ENTRY")" >> "$GITHUB_OUTPUT"
          echo "preview=$(jq -r '.preview_url' <<<"$ENTRY")" >> "$GITHUB_OUTPUT"

      - name: Send it 5% of traffic
        run: |
          npx wrangler versions deploy \
            "${{ steps.upload.outputs.id }}@5%" \
            "${{ steps.previous.outputs.id }}@95%" --yes
```

Point `PLAYWRIGHT_BASE_URL` in the smoke job at `needs.deploy.outputs.preview`, and set `CANARY_RELEASE` in the soak job to `needs.deploy.outputs.version`.

### Taking the rest, or giving it back

```yaml
  promote:
    needs: [deploy, soak]
    if: needs.soak.outputs.decision == 'promote'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      # A single version at 100% ends the split
      - run: npx wrangler versions deploy "${{ needs.deploy.outputs.version }}@100%" --yes

  abort:
    needs: [deploy, smoke, soak]
    if: >
      always() && (
        needs.smoke.result == 'failure' ||
        needs.soak.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx wrangler versions deploy "${{ needs.deploy.outputs.previous }}@100%" --yes
```

### Rolling back after promotion

```yaml
  rollback:
    needs: [deploy, watch-15, watch-60]
    if: >
      always() && (
        needs.watch-15.outputs.decision == 'abort' ||
        needs.watch-60.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx wrangler rollback --message "post-promotion health check" --yes
```

It reaches the edge in seconds, and a split deployment collapses onto the version you picked.

Version affinity stops applying at that point, since one version is now serving everything, so the sessions you pinned move along with everybody else. 

### The whole file

```yaml
# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

concurrency:
  group: production-deploy
  cancel-in-progress: false

permissions:
  contents: read

env:
  CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
  CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
  SENTRY_TOKEN: ${{ secrets.SENTRY_TOKEN }}
  SENTRY_ORG: my-org
  SENTRY_PROJECT_ID: '4504000000000000'

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      url: ${{ steps.upload.outputs.preview }}
      version: ${{ steps.upload.outputs.id }}
      baseline: ${{ steps.baseline.outputs.id }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npm run build
      - id: baseline
        run: |
          ID=$(npx wrangler deployments list --json \
            | jq -r '.[0].versions[0].version_id')
          echo "id=$ID" >> "$GITHUB_OUTPUT"
      - id: upload
        env:
          WRANGLER_OUTPUT_FILE_PATH: ${{ runner.temp }}/wrangler.ndjson
        run: |
          npx wrangler versions upload
          # ND-JSON, one object per line. The version-upload entry has both.
          ENTRY=$(jq -sc '[.[] | select(.type=="version-upload")] | last' \
            "$WRANGLER_OUTPUT_FILE_PATH")
          echo "id=$(jq -r '.version_id' <<<"$ENTRY")" >> "$GITHUB_OUTPUT"
          echo "preview=$(jq -r '.preview_url' <<<"$ENTRY")" >> "$GITHUB_OUTPUT"
      - name: Send it 5% of traffic
        run: |
          npx wrangler versions deploy \
            "${{ steps.upload.outputs.id }}@5%" \
            "${{ steps.baseline.outputs.id }}@95%" --yes

  smoke:
    needs: deploy
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test e2e/smoke --project=chromium
        env:
          PLAYWRIGHT_BASE_URL: ${{ needs.deploy.outputs.url }}

  soak:
    needs: [deploy, smoke]
    runs-on: ubuntu-latest
    environment: canary-soak
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          CANARY_RELEASE: ${{ needs.deploy.outputs.version }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  promote:
    needs: [deploy, soak]
    if: needs.soak.outputs.decision == 'promote'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx wrangler versions deploy "${{ needs.deploy.outputs.version }}@100%" --yes

  hold:
    needs: soak
    if: needs.soak.outputs.decision == 'hold'
    runs-on: ubuntu-latest
    steps:
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Canary parked at 5%. Not enough data to decide.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"

  abort:
    needs: [deploy, smoke, soak]
    if: >
      always() && (
        needs.smoke.result == 'failure' ||
        needs.soak.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx wrangler versions deploy "${{ needs.deploy.outputs.baseline }}@100%" --yes
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Canary aborted. Production is back on the previous build.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"

  watch-15:
    needs: [deploy, promote]
    runs-on: ubuntu-latest
    environment: watch-15min
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          CANARY_RELEASE: ${{ needs.deploy.outputs.version }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  watch-60:
    needs: [deploy, promote, watch-15]
    if: needs.watch-15.outputs.decision != 'abort'
    runs-on: ubuntu-latest
    environment: watch-45min    # runs after watch-15, so 15 + 45 = an hour in
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          CANARY_RELEASE: ${{ needs.deploy.outputs.version }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  rollback:
    needs: [deploy, watch-15, watch-60]
    if: >
      always() && (
        needs.watch-15.outputs.decision == 'abort' ||
        needs.watch-60.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx wrangler rollback --message "post-promotion health check" --yes
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Rolled back after promotion.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"
```

### Rolling out to one country

The Worker reads its numbers from a KV namespace, so create one and bind it first.

```bash
npx wrangler kv namespace create CONFIG
```

```jsonc
// wrangler.jsonc
{
  "kv_namespaces": [
    { "binding": "CONFIG", "id": "0f2ac74b498b48028cb68387c421e279" }
  ]
}
```

```js
export default {
  async fetch(request, env) {
    // cacheTtl keeps this off the hot path; 60s is fast enough to abort
    const rollout = await env.CONFIG.get('canary', {
      type: 'json',
      cacheTtl: 60,
    });

    // A missing or malformed key sends everyone to the stable version
    if (!rollout?.countries?.length) return fetch(request);

    const country = request.cf?.country;
    if (!country || !rollout.countries.includes(country)) {
      return fetch(request);
    }

    const cookie = request.headers.get('cookie') ?? '';
    const existing = cookie.match(/canary=(on|off)/)?.[1];
    const assigned =
      existing ?? (Math.random() * 100 < rollout.percentage ? 'on' : 'off');

    let upstream;
    if (assigned === 'on') {
      const url = new URL(request.url);
      url.hostname = rollout.canaryHost;
      upstream = await fetch(new Request(url, request));
    } else {
      upstream = await fetch(request);
    }

    const response = new Response(upstream.body, upstream);
    response.headers.append(
      'set-cookie',
      `canary=${assigned}; Path=/; Max-Age=86400; SameSite=Lax`
    );
    return response;
  },
};
```

Keep the guard on a missing key. A Worker that throws when somebody deletes that KV entry takes the whole site down with it.


```yaml
      - run: |
          COUNTRIES=$(echo "${{ inputs.countries }}" | jq -R 'split(",")')
          npx wrangler kv key put --binding CONFIG --remote canary \
            "$(jq -nc --argjson c "$COUNTRIES" \
               '{countries:$c,percentage:10,canaryHost:"canary.example.com"}')"
```

## Deploying it yourself on GCP or AWS

Putting the app in a container means nobody hands you a traffic splitter anymore, so you configure the one your platform already has.

### Cloud Run

A Cloud Run service keeps every revision you have ever deployed, and each deployment is split across them. Deploying with `--no-traffic --tag canary` ships a revision that serves nobody and gets its own URL, which is the same preview step Cloudflare gives you.

```bash
gcloud run deploy my-app \
  --image gcr.io/my-project/my-app:$SHA \
  --region us-central1 --no-traffic --tag canary
# reachable at https://canary---my-app-xxxxx.a.run.app
```

Traffic then moves by tag.

```bash
gcloud run services update-traffic my-app --region us-central1 --to-tags canary=5
gcloud run services update-traffic my-app --region us-central1 --to-tags canary=50
gcloud run services update-traffic my-app --region us-central1 --to-tags canary=100
```

Rolling back names a revision instead of a tag, which is why the workflow captures the current one before deploying anything.

```bash
gcloud run services update-traffic my-app --region us-central1 \
  --to-revisions my-app-00042-abc=100
```

```bash
gcloud run services update my-app --region us-central1 --session-affinity
```

Use Workload Identity Federation for the credentials. It lets the workflow assume a service account with no JSON key sitting in GitHub, which is why every job below asks for `id-token: write`.

```yaml
# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

concurrency:
  group: production-deploy
  cancel-in-progress: false

env:
  SERVICE: my-app
  REGION: us-central1
  SENTRY_TOKEN: ${{ secrets.SENTRY_TOKEN }}
  SENTRY_ORG: my-org
  SENTRY_PROJECT_ID: '4504000000000000'

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write        # Workload Identity Federation, no JSON keys
    outputs:
      preview: ${{ steps.deploy.outputs.preview }}
      revision: ${{ steps.deploy.outputs.revision }}
      baseline: ${{ steps.baseline.outputs.revision }}
    steps:
      - uses: actions/checkout@v4
      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.GCP_WIF_PROVIDER }}
          service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
      - uses: google-github-actions/setup-gcloud@v2

      # The entry carrying the most traffic, rather than whichever one
      # happens to sit first in the list
      - id: baseline
        run: |
          REV=$(gcloud run services describe "$SERVICE" --region="$REGION" --format=json \
            | jq -r '[.status.traffic[] | select(.percent > 0)] | max_by(.percent) | .revisionName')
          echo "revision=$REV" >> "$GITHUB_OUTPUT"

      - run: |
          gcloud builds submit --tag "gcr.io/$GCP_PROJECT/$SERVICE:${{ github.sha }}"
        env:
          GCP_PROJECT: ${{ secrets.GCP_PROJECT }}

      # --no-traffic ships it with zero users. --tag gives it a URL.
      - id: deploy
        run: |
          gcloud run deploy "$SERVICE" \
            --image "gcr.io/${{ secrets.GCP_PROJECT }}/$SERVICE:${{ github.sha }}" \
            --region "$REGION" --no-traffic --tag canary
          URL=$(gcloud run services describe "$SERVICE" --region="$REGION" --format=json \
            | jq -r '.status.traffic[] | select(.tag=="canary") | .url')
          REV=$(gcloud run services describe "$SERVICE" --region="$REGION" \
            --format='value(status.latestCreatedRevisionName)')
          echo "preview=$URL" >> "$GITHUB_OUTPUT"
          echo "revision=$REV" >> "$GITHUB_OUTPUT"

      - name: Send it 5% of traffic
        run: gcloud run services update-traffic "$SERVICE" --region="$REGION" --to-tags canary=5

  smoke:
    needs: deploy
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test e2e/smoke --project=chromium
        env:
          PLAYWRIGHT_BASE_URL: ${{ needs.deploy.outputs.preview }}

  soak:
    needs: [deploy, smoke]
    runs-on: ubuntu-latest
    environment: canary-soak
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          CANARY_RELEASE: ${{ github.sha }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  promote:
    needs: [deploy, soak]
    if: needs.soak.outputs.decision == 'promote'
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.GCP_WIF_PROVIDER }}
          service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
      - uses: google-github-actions/setup-gcloud@v2
      - run: gcloud run services update-traffic "$SERVICE" --region="$REGION" --to-tags canary=100

  hold:
    needs: soak
    if: needs.soak.outputs.decision == 'hold'
    runs-on: ubuntu-latest
    steps:
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Canary parked at 5%. Not enough data to decide.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"

  abort:
    needs: [deploy, smoke, soak]
    if: >
      always() && (
        needs.smoke.result == 'failure' ||
        needs.soak.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.GCP_WIF_PROVIDER }}
          service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
      - uses: google-github-actions/setup-gcloud@v2
      - run: |
          gcloud run services update-traffic "$SERVICE" --region="$REGION" \
            --to-revisions="${{ needs.deploy.outputs.baseline }}=100"
      - run: |
          curl -sS -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"Canary aborted. All traffic back on the previous revision.\"}" \
            "${{ secrets.SLACK_WEBHOOK }}"

  watch-15:
    needs: [deploy, promote]
    runs-on: ubuntu-latest
    environment: watch-15min
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          CANARY_RELEASE: ${{ github.sha }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  watch-60:
    needs: [deploy, promote, watch-15]
    if: needs.watch-15.outputs.decision != 'abort'
    runs-on: ubuntu-latest
    environment: watch-45min    # runs after watch-15, so 15 + 45 = an hour in
    outputs:
      decision: ${{ steps.check.outputs.decision }}
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - id: check
        run: node scripts/canary-health.mjs
        env:
          CANARY_RELEASE: ${{ github.sha }}
          BASELINE_RELEASE: ${{ needs.deploy.outputs.baseline }}

  rollback:
    needs: [deploy, watch-15, watch-60]
    if: >
      always() && (
        needs.watch-15.outputs.decision == 'abort' ||
        needs.watch-60.outputs.decision == 'abort'
      )
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.GCP_WIF_PROVIDER }}
          service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
      - uses: google-github-actions/setup-gcloud@v2
      - run: |
          gcloud run services update-traffic "$SERVICE" --region="$REGION" \
            --to-revisions="${{ needs.deploy.outputs.baseline }}=100"
```

### ECS behind an Application Load Balancer

In ECS, containers use the load balancer split, where a listener rule forwards traffic to two target groups with weights.

```bash
aws elbv2 modify-listener --listener-arn "$LISTENER" --default-actions '[{
  "Type": "forward",
  "ForwardConfig": {
    "TargetGroups": [
      {"TargetGroupArn": "'"$BLUE"'",  "Weight": 95},
      {"TargetGroupArn": "'"$GREEN"'", "Weight": 5}
    ],
    "TargetGroupStickinessConfig": {"Enabled": true, "DurationSeconds": 3600}
  }
}]'
```

Deploy registers a task definition and updates the green service. The ramp is that `modify-listener` call with new weights, promoting set green to 100, and both abort and rollback set blue back to 100.

```yaml
      - name: Send it 5% of traffic
        run: |
          aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE_GREEN" \
            --task-definition "${{ steps.taskdef.outputs.arn }}" --force-new-deployment
          aws ecs wait services-stable --cluster "$CLUSTER" --services "$SERVICE_GREEN"
          ./scripts/set-weights.sh 95 5

      # promote
      - run: ./scripts/set-weights.sh 0 100

      # abort and rollback are the same call, the other way round
      - run: ./scripts/set-weights.sh 100 0
```

```bash
#!/usr/bin/env bash
# scripts/set-weights.sh <blue-weight> <green-weight>
set -euo pipefail

aws elbv2 modify-listener --listener-arn "$LISTENER" --default-actions "$(cat <<JSON
[{
  "Type": "forward",
  "ForwardConfig": {
    "TargetGroups": [
      {"TargetGroupArn": "$BLUE",  "Weight": $1},
      {"TargetGroupArn": "$GREEN", "Weight": $2}
    ],
    "TargetGroupStickinessConfig": {"Enabled": true, "DurationSeconds": 3600}
  }
}]
JSON
)"
```

### Country rollouts on either

Neither platform reads geography on its own, so the country check goes in the CDN sitting in front.

On AWS, that is a CloudFront Function reading `CloudFront-Viewer-Country`, which CloudFront adds when you enable it in the cache policy. The function is the one from the Cloudflare section with the header swapped for `request.headers['cloudfront-viewer-country'].value`.

On GCP, the external Application Load Balancer can add a custom request header populated from `{client_region}`, which your container then reads like any other header. Configure it under the backend service, then route to it in the app or in a small proxy.

## All the code

Everything above lives in [github.com/Cst2989/canary-deploys](https://github.com/Cst2989/canary-deploys), one folder per platform.

```
scripts/canary-health.mjs     the decision: promote | abort | hold
scripts/__tests__/            28 tests over every branch of it
setup/environments.sh         creates the four wait-timer environments
platforms/vercel/             deploy.yml, geo-rollout.yml, middleware.ts
platforms/netlify/            deploy.yml, edge-functions/canary.js
platforms/cloudflare/         deploy.yml, worker-geo.js, version-affinity.tf
platforms/cloud-run/          deploy.yml
platforms/ecs/                set-weights.sh and the job fragments
```

Copy `scripts/`, `setup/` and the one `platforms/<yours>/` folder you need. The `deploy.yml` files go in `.github/workflows/`.

The change detection from [part 1](https://neciudan.dev/ci-cd-in-the-age-of-ai-part-1) is in [change-detection](https://github.com/Cst2989/change-detection), packaged as [affected-ci](https://github.com/Cst2989/affected-ci).