Runtime validation

Three layers, and only one of them tests behaviour

Documentation checking tends to get discussed as one activity. It is three, and they catch different failures.

• Linting checks the text — broken links, malformed code fences, style rules. It knows nothing about the software.
• Static comparison checks existence — does this endpoint appear in the code, does this flag exist in the parser. It catches documentation describing things that are gone.
• Runtime validation checks behaviour — call the endpoint, run the command, compare what came back against what was promised.

The first two can both pass on documentation that is wrong. An endpoint that exists and accepts the documented parameters can still return a different shape than the docs describe, or a different status code, or silently ignore a flag that was deprecated two releases ago.

What only execution finds

The failures that survive static checking are the expensive ones, because they look correct:

• A parameter that is still accepted but no longer does anything
• A response field renamed, where the old name is absent rather than erroring
• A status code changed from 400 to 422
• A documented default that no longer matches the actual default
• An example that worked against last quarter's data and returns empty now

None of these break a link check. All of them break the reader.

Why it matters more now

A human reading documentation applies judgement. The response looks different from the example, so they hedge, retry, or go and read the source. The documentation was wrong and the cost was ten minutes of confusion.

An agent does not hedge. It takes the documented contract as fact and writes code against it, confidently and at volume. Documentation errors that used to cost a reader some irritation now compile into defects. This is the shift that makes execution worth the setup cost — documentation became a runtime dependency without anyone deciding it should.

What it costs to run

Runtime validation is not free, and the constraints are real:

• It needs a reachable environment — a staging deployment or a sandbox, with credentials.
• Not every example is safe to execute. Anything that deletes, charges, or sends needs to be marked and skipped, or pointed at a sandbox that tolerates it.
• Some examples cannot be validated at all — anything requiring state the validator cannot create. Being honest about that subset is more useful than pretending coverage is total.

A validator that quietly skips what it cannot run, and says so, is trustworthy. One that reports a clean pass over a filtered subset is not.