Make every implementation test the others

Conformance tests check the cases you already know. Differential fuzzing searches for cases you did not think to write down: Polyform generates one input, runs it through every implementation of the same function, and flags different results.

polyform fuzz
polyform fuzz --function decode_filename --cases 10000 --seed 42

What Polyform does locally

1. Passes a reproducible seed and case number to the project’s language-native fuzz harness.
2. Runs the exact same JSON-serializable input through every registered implementation.
3. Compares normalized success values and documented errors without assuming the majority is correct.
4. Shrinks a disagreement to a smaller input while preserving the disagreement.
5. Saves the input, every implementation ID and result, seed, and case number under fuzz/counterexamples/FUNCTION.
6. Exits unsuccessfully so the disagreement cannot be overlooked in polyform check.

The harness is generated with the tests

The test-author agent creates a small adapter for the project’s language and implementation registry. The adapter generates inputs and calls implementations; Polyform owns comparison, shrinking, persistence, and reproducibility. Everything runs on the developer’s computer.

[fuzz]
command = ["cargo", "run", "--quiet", "--bin", "polyform-fuzz-harness"]
cases = 100
seed = 1
run_on_check = true
corpus_dir = "fuzz/counterexamples"

Harness protocol

The command reads one JSON request from standard input and writes one JSON response to standard output. A generated case receives operation, function, seed, and case. A shrink evaluation receives operation: "evaluate" and an input that must be used unchanged.

{
  "valid": true,
  "input": { "bytes": [255, 46], "flags": 2048 },
  "outcomes": [
    { "implementation": "decode.strict", "result": { "error": "InvalidEncoding" } },
    { "implementation": "decode.lossy", "result": { "ok": "�." } }
  ]
}

A disagreement is evidence, not a vote

Polyform does not declare the majority result correct. Several agents can repeat the same misconception or share a faulty dependency. A developer or coding agent compares the minimized case with the public specification, fixes the outlier or the specification, and promotes the case into the permanent conformance suite.