Managing visual test baselines

The baseline is the approved picture of what your interface is supposed to look like. Everything a visual suite tells you is relative to it, which makes baseline management the part that quietly decides whether the suite means anything a year in.

Where baselines should live

Committing images is the default because it is the path of least resistance, and it is the one that ages worst. A hundred routes × two viewports × two platforms is four hundred PNGs, re-committed on every intentional redesign. Repositories in the hundreds of megabytes are common, and clone times degrade for everyone including CI.

The platform trap

This catches nearly every team once. Screenshots rendered on macOS do not match screenshots rendered on Linux — font hinting, antialiasing, and emoji all differ. Playwright handles this by naming snapshots per platform, so you get both pricing-chromium-darwin.png and pricing-chromium-linux.png.

That works, and it has a subtle failure mode: a developer who updates a baseline locally has updated the darwin image, not the linux one CI compares against. The change looks approved and CI still fails.

The fix is to record baselines only inside the container CI uses, and to make that the only sanctioned way to do it:

{
  "scripts": {
    "visual:update": "docker run --rm --network host -v \"$PWD\":/work -w /work mcr.microsoft.com/playwright:v1.56.0-noble npx playwright test --update-snapshots"
  }
}

Then delete the darwin baselines and add them to .gitignore. One platform, one set of images, no ambiguity about which one is authoritative.

Who approves a baseline

Approving a baseline is the moment a visual change becomes permanent — including, if you get it wrong, a regression. It deserves the same seriousness as merging code.

• Never let a run auto-approve its own changes. A --update-snapshots flag in CI on the feature branch makes the entire suite decorative.
• Require a record of who approved what. Committed baselines get this free from Git. Hosted tools should provide an audit trail; if one does not, that is a real gap.
• Approve alongside the code that caused the change, not in a separate cleanup pull request where nobody remembers the context.
• Treat a mass approval as a review smell. Nobody meaningfully reviews four hundred images; they click approve.

The redesign problem

Eventually a font change, a token update, or a header redesign legitimately changes every page. Handling it badly is how teams lose trust in the suite.

1. 01Isolate it. Land the visual change and the baseline update in their own pull request, touching nothing else. A redesign mixed with a feature makes both unreviewable.
2. 02Review a sample deliberately. Pick ten representative pages — widest, narrowest, most data-dense, most text-heavy — and look at those properly rather than skimming all four hundred.
3. 03Check the pages nobody thinks about. Error states, empty states, and the settings screens are where a global change breaks something specific.
4. 04Record the reason in the commit message. Six months later, "updated baselines" tells nobody whether the change was intended.
5. 05Re-baseline from main after merge, so branches opened before the redesign do not all report the same four hundred diffs.

Drift: the slow failure

Baselines go stale when they stop being recaptured as the product changes. A pull request compared against a three-month-old baseline reports differences that are all legitimate merges, buried among which might be one real regression. Reviewers learn to approve in bulk, and the suite becomes theatre.

• Recapture on every merge to the default branch, not on a weekly schedule loosely related to when things change.
• Record which commit produced each baseline, so a surprising diff can be traced to a specific change.
• Re-record deliberately after dependency upgrades that touch rendering — browser versions, font packages, CSS frameworks — in their own commit.
• Watch the approval rate. If almost every diff is being approved, the suite is reporting noise rather than findings, and something upstream needs fixing.

That last metric is the most useful health check available. A suite where 95% of diffs are approved is not protecting anything — it is generating work. The causes are usually determinism problems rather than baseline problems; how to fix flaky visual tests covers them in the order worth addressing.