# Experiment results and interpretation

Polyform's demo produces an experiment report from observed assignments and
telemetry rather than embedding a predetermined success table in this document.
Run `./demo.sh`; the terminal prints the report location and the dashboard shows
the same population. Keeping the measured values in a generated artifact makes
the experiment reproducible while avoiding claims based on stale fixture data.

## Questions measured

The experiment compares homogeneous, random-diversified, and
balanced-diversified assignment and records:

- clients and executions exposed to each implementation;
- total failures and affected-client fraction;
- per-implementation failure rate and relative risk;
- supported pairwise associations;
- ranking of the controlled faulty implementations without disclosing ground
  truth to the analyzer;
- number of clients recomposed after quarantine; and
- failure rate before versus after quarantine.

## Expected qualitative result

A homogeneous release provides no within-population counterfactual: if everyone
uses the same variant, telemetry can show a release problem but cannot separate
that implementation from its peers. A diversified population limits the
affected fraction to clients assigned the faulty variant and provides exposed
and unexposed groups. With sufficient balanced observations, the faulty variant
should rise above alternatives in failure rate and relative risk. A fault that
requires two otherwise-safe variants should appear in pairwise analysis.

After quarantine, new assignments exclude the suspect and affected clients move
to a new composition generation. Workloads in the post-quarantine epoch should
therefore show a lower failure rate. The dashboard reports both periods and the
number of replacements rather than claiming that quarantine rewrites already
observed history.

## Reading the statistics responsibly

Relative risk is easy to inspect but can be unstable with few exposures. The
implementation table therefore includes raw support, and pairwise findings use
minimum-support filtering. Association does not prove that a variant is the
root cause: workload mix, assignment imbalance, correlated selections, and
interactions can confound it. Balanced assignment reduces—without eliminating—
those problems. The controlled experiment can compare the ranking against a
separate ground-truth file after analysis is complete.

## Verified reference run

On 2026-08-28, the default deterministic demo used 72 virtual clients per
cohort and four executions per client. It observed:

| Assignment | Clients exposed to a controlled fault | Failure rate |
|---|---:|---:|
| Homogeneous | 100.0% | 75.7% |
| Random diversified | 25.0% | 17.7% |
| Balanced diversified | 26.4% | 18.8% |

The two controlled faulty implementations ranked first and second in the
statistical suspect table before ground truth was consulted. Quarantining them caused 127
composition replacements (a client exposed to both can be replaced twice) and
reduced the fixed post-quarantine cohort from 16.3% failures to 0.0%. The run
also registered real Rust processes, assigned them through FastAPI, and
exercised the actual long-comment and deep-path ZIP fault triggers before and
after quarantine. Exact current values are regenerated in
`artifacts/demo-results.json`; small differences are expected when population,
seed, or workload settings change.

The release contains 45 accepted implementations: five for each of nine spec
functions. That produces 1,953,125 safe compositions. Before intervention, the
experiment adds one controlled fault to each of two functions for 2,812,500
eligible experimental compositions. After both are quarantined, the dashboard
reports the 1,953,125 compositions that remain eligible for assignment.

## Reproducing and extending

Use a fixed simulator seed for a directly repeatable run, then vary the seed,
population size, workload frequency, base fault rate, assignment policy, and
interaction trigger. Useful next experiments include confidence intervals,
regularized logistic regression with interaction terms, online sequential
testing, drift across releases, and quarantine policies that require a minimum
effect size as well as a score threshold.
