Visual testing in CI/CD pipelines

A visual check that runs after deployment is a monitoring tool. A visual check that runs on a pull request is a testing tool. The difference in value is enormous, and it is decided entirely by where in the pipeline you put it.

This piece is about the pipeline design: the stage, the environment, the budget, and the blocking policy. If you are still choosing what to run, start with what visual regression testing is.

Run against a preview deployment, not inside the build

The instinct is to build the app inside the CI job, start a server, and screenshot localhost. It works, and it is the wrong shape for most teams.

• The CI runner has to reproduce production's data, environment variables, and service dependencies — a constant source of drift.
• Build and boot time is charged to every visual run.
• Nothing a human can open exists afterwards, so a failing diff cannot be inspected in a real browser.

If your platform already produces a per-pull-request preview URL, point the visual run at that. The environment is the one reviewers look at, the app is built once, and a surprising diff can be investigated by opening the page.

name: Visual review

on: pull_request

jobs:
  visual:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Wait for the platform's preview deployment to be ready,
      # then hand its URL to the visual run.
      - name: Await preview deployment
        id: preview
        uses: patrickedqvist/wait-for-vercel-preview@v1.3.2
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          max_timeout: 600

      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium

      - run: npx playwright test --project=visual
        env:
          BASE_URL: ${{ steps.preview.outputs.url }}

      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: visual-report
          path: playwright-report/
          retention-days: 7

Budget: keep the whole check under five minutes

Pull request feedback competes with a developer's attention span. Past roughly five minutes people context-switch, and a check they have stopped waiting for is a check they will merge past.

Capture only what the change can affect

This is the largest available saving and the one most setups skip. A copy change on the pricing page does not require recapturing the dashboard. Map changed source files to the routes that import them, directly or transitively, and recapture that set.

Hand-rolling that mapping is real work — it needs the module graph, route definitions, and an honest answer about global CSS. The fallback heuristics, in descending order of quality: recapture routes whose files changed, recapture a fixed high-traffic set on every run, and recapture everything only on the main branch nightly.

Shard across runners

strategy:
  fail-fast: false
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: npx playwright test --shard=${{ matrix.shard }}/4

fail-fast: false matters. With the default, one shard failing cancels the rest and the reviewer sees a partial picture of what changed.

Cut the matrix honestly

Routes × viewports × browsers multiplies fast. Three viewports and three browsers is nine captures per route, and cross-browser visual differences are rarer than the matrix implies now that engines have converged. One engine on every pull request and a broader matrix nightly is the usual right answer.

Should a visual diff block the merge?

Not on day one, and the sequencing matters more than the eventual policy.

1. 01Weeks 1–2: non-blocking. The check posts results and cannot fail the build. Use this window to eliminate noise — see fixing flaky visual tests.
2. 02Weeks 3–4: blocking on a trusted subset. Pick the routes that are now stable and revenue-critical. Everything else stays advisory.
3. 03Ongoing: expand the blocking set as routes prove themselves quiet.

Turning on blocking while the suite is still noisy is how visual testing gets a reputation inside a company that it never recovers from — the decisions a manager actually owns here are in visual testing for engineering leaders. The first time a flaky diff blocks an urgent fix, someone adds a bypass label, and the bypass label becomes the process.

Put the result where review already happens

A green or red dot in a checks list is not a review surface. The useful integration posts a pull request comment containing the changed pages, before and after images, and a summary of what moved — so a designer or product manager can weigh in without cloning the repository.

One comment, updated in place on each push. A bot that appends a new comment per run trains everyone to collapse it.

Keep main's baseline current

Baselines drift. A pull request approved against a stale baseline compares against a state of the world that no longer exists, and differences accumulate until the report is meaningless.

• Recapture the baseline on every merge to the default branch, not on a schedule that is only loosely related to when the product changes.
• Record which commit produced each baseline image so a surprising diff can be traced.
• Re-record after dependency upgrades that touch rendering — browser versions, font packages, CSS frameworks.

Secrets and authenticated routes

Visual runs on protected routes need credentials, and pull requests from forks do not get your secrets — correctly so. The workable patterns are a dedicated test account with the narrowest possible permissions on non-fork pull requests, and a marketing-only subset for forks. Never run an authenticated visual job with credentials supplied by an untrusted pull request.