Visual regression

Two folders of screenshots, compared shot by shot, ordered worst-first, with one number a pipeline can gate on. Command-line only — and that is a consequence of what it needs, not an omission.

Click to enlarge

The one invocation#

twinscope baseline/ current/ --engine visual

Screenshots pair on their path relative to the folder you gave, so baseline/pages/checkout.png compares against current/pages/checkout.png however deep the tree is. A shot present on one side only is reported as added or removed rather than as a difference.

Command-line only, by construction#

This engine needs two capabilities at once: it has to list directories and it has to decode images. No single process in the desktop app has both.

Handing the window a whole baseline set as bytes over the app's internal bridge is exactly what the standing rule against moving big inputs across it forbids — a four-hundred-shot set is not a payload, it is a directory.

So in the desktop app this engine declines, and says where it does run rather than failing obscurely:

Click to enlarge

Compare as folders is offered beside it as a button, since the folder engine can still tell you which screenshots changed at all — just not by how much.

Why you have to ask for it#

The engine's detection priority is 0, which means detection never picks it. Two folders of screenshots and two folders of source code are indistinguishable from outside, so TwinScope does not guess: a plain twinscope baseline/ current/ runs the folder engine, which hashes files and finds renames, and --engine visual decodes every pair.

Naming it is one word, and it saves accidentally decoding a repository.

Two budgets, and neither is zero#

Note that the two engine budgets and the build gate are separate things. perImagePercent decides what the report calls a regression; --max-diff decides whether the build fails. The number the gate reads is worst difference — the highest percentage across the whole set, so the run gates on its worst image.

The sample set#

16-visual-regression/ holds three shots, chosen to show all three outcomes at once:

$ twinscope 16-visual-regression/baseline 16-visual-regression/current --engine visual
baseline → current
Visual regression

+0 added -0 removed ~2 modified
images: 3 · worst difference: 0.84% · over budget: 1

Two shots are reported as modified and only one is over budget. login.png is the whole point of the example: four pixels out of 120,000 is what a font renderer does on a Tuesday, and a suite that failed on it would teach its owners to ignore it.

Gating a build#

twinscope baseline/ current/ --engine visual --max-diff 0.1
# FAIL  difference 0.84% > allowed 0.1%
# exit 1

A threshold takes over exit code 1. Without one, exit 1 means "these two differ", which for a visual run is nearly always true and rarely a failure; with one, it means "these differ by more than you allowed".

See Continuous integration for the workflow — including --github, which writes annotations and a job summary naming every threshold with a tick or a cross — and the command line for the rest of the flags.

Details worth knowing#

Mismatched sizes compare on the union. A screenshot that grew is a regression, not an error: both images are padded out to the larger width and height and compared across the whole area, exactly as the image engine does. The pixel maths is literally the same code, so a fix to one fixes both.

One unreadable screenshot does not fail the run. It is listed as its own row with the reason, and the other 399 still carry the answer. A run that threw away 399 useful comparisons because one file was truncated would be worse than one that reports the truncation.

Percentages are this engine's own arithmetic, computed from the differing pixel count over the compared area, so a 4000 × 4000 pair and a 40 × 40 pair produce comparable numbers.

Symbolic links are skipped during the walk, and the walk stops at 24 levels deep.

Images are scaled down past 4096 pixels on their longest side before comparison. On the command line that downscale is nearest-neighbour, deliberately: an interpolating resample invents intermediate colours and inflates the number of pixels reported as different, which is the one number this engine exists to report.

No adapters to install#

TwinScope compares directories of images; it does not take screenshots. Whatever Playwright, Cypress or Storycap already writes is what gets compared — there is no adapter to add and no TwinScope-shaped configuration for your test suite.

Accepting a new baseline is a file copy you run yourself. TwinScope will not overwrite one of its own inputs.