API contracts and captures

Two shapes, one engine, and a verdict rather than a tree of keys. An OpenAPI pair answers "will a consumer break". A HAR pair answers "what did the API actually do differently", with the capture's own noise set aside and counted.

Click to enlarge

Recognised by shape, before the extension map#

This is the unusual part, and it is deliberate. A HAR and an OpenAPI document are both .json — an extension tells you nothing. So detection looks at the content before it looks at the name, in the same way a package.json is recognised by its basename rather than by .json.

The tell has to survive the sniff that produced the text, which reads only the first 8 KB of a file. A truncated head cannot be parsed, so the test is a substring match rather than a parse:

Both sides have to be the same shape. A HAR against an OpenAPI document is refused by name, with Compare as JSON offered as a button rather than left as an idea.

OpenAPI: a compatibility verdict#

A structural diff of two OpenAPI files reports that a required array gained an entry. That is true and it is not the question. What a reader wants to know is whether clients sending that request now fail validation — same bytes, different answer.

So the engine reduces each document to what compatibility actually depends on: for every operation, the request shape (body fields and parameters, since a newly required query parameter breaks a caller exactly as a body field does) and the shape of its first 2xx response, which is the contract a consumer reads. A changed error body matters less and is reported by the body comparison rather than as a verdict.

The rules#

Every finding carries the rule that produced it, because a verdict nobody can audit is a guess with a badge on it.

The rules are deliberately conservative. Where compatibility depends on something the document does not state, the change is reported as notable rather than asserted as safe — and compatible findings are listed too, because "compatible" is not "uninteresting" and a tool that hides additions teaches people to stop looking.

The sample pair#

08-api/contract/ in the sample set is two versions of an Orders API. Four operations on each side, and the comparison reports 3 breaking against 2 compatible:

What it will not do#

• External $refs are not followed. Following one means reading a file you did not hand to the comparison. Local #/components/… refs are resolved, and a cycle — legal in OpenAPI, fatal to a naive walk — terminates instead of recursing.
• An unresolvable ref degrades to an unknown type rather than throwing. Half a contract read is worth more than a parse error.
• Schemas are flattened to a depth of 12. Past that, a document is defending itself against being read.

HAR: pairing a capture against its own noise#

Two recordings of the same session never agree on much. Timestamps differ, cookies differ, and the order of concurrent requests is whatever the browser felt like. Most of the work here is therefore pairing, so that the comparison describes what changed about the API rather than what changed about the capture.

Entries pair on METHOD plus the path, with the query string reduced to its sorted parameter names:

GET /v1/orders?limit=20&cursor=abc     ← before
GET /v1/orders?limit=20&cursor=xyz     ← after

both key on: GET /v1/orders?cursor,limit

Pairing on the raw URL would report every cache-busted request as removed-and-added. A key seen three times before and twice after yields three pairs, the last with no after side — the honest reading, since which of the three calls stopped happening is not knowable from a capture.

Volatile headers are suppressed and counted#

A date header differing is not news, and reporting it once per entry buries everything that is. Sixteen header names are ignored by default, and every suppression is counted and stated in the notes rather than silently dropped:

date            age              etag             set-cookie
cookie          content-length   request-id       x-request-id
x-amzn-requestid  x-correlation-id  traceparent    x-runtime
server-timing   expires          last-modified    authorization

Header names are lower-cased when the HAR is read, since casing is not meaningful and two captures rarely agree on it.

Verdicts from a capture#

A capture can prove two things break, and only two, so those are the only two it claims:

Everything else a capture shows is a change, and calling it breaking would be a guess. The contract path is where verdicts are earned.

Response bodies are compared structurally when both sides are JSON — the same comparison the JSON engine does, normalisation rules and all. When only one side has a JSON body, or neither does but they differ, the row says so instead of pretending to a structural answer. Request bodies are off by default; the toolbar toggle turns them on, and the notes say when they were skipped.

The sample pair#

08-api/capture/ is two recordings of the same two calls. The comparison reports 2 entries, 1 breaking, 8 body changes and 2 suppressed header differences:

• GET /v1/orders — paired despite the changed cursor value. Its body differs, and a new cache-control header arrives; the changed date and x-request-id are the two suppressions.
• GET /v1/orders/o_1 — 200 became 404. That is the breaking one.

In the app#

Change navigation walks findings for a contract pair and entries for a capture, so ⌥↑/⌥↓ and the n/m badge always agree on what there is to step through.

Gating a build on it#

--fail-on-breaking turns a breaking finding into a failed build. It reads the same breaking count the verdict is built from:

twinscope api.v1.json api.v2.json --fail-on-breaking
# FAIL  3 breaking changes
# exit 1

A threshold takes over exit code 1: without one, exit 1 means "these two differ", which is usually true and rarely a build failure; with one it means "these differ by more than you allowed". A comparison that reports no breaking count at all — anything other than an API pair — fails --fail-on-breaking rather than quietly passing, because a pipeline asking a question the comparison cannot answer is a mistake worth surfacing.

See Continuous integration for the workflow, and the command line for the rest of the flags.

Nothing is fetched#

Both sides are files you chose. No request is made to any host named in a HAR, no $ref is followed off disk, and comparing two live URLs is a decision that has not been made — so it is not quietly half-built.