ConstellOps is tested by an independent conformance and benchmark harness: a separate code base, run by a separate team, that only ever sees tagged releases. Its job is to find out what is true, including when that is bad news for the product.
Independent by construction
The harness never imports a branch. It installs a tagged release, the way an operator would, and a capability only counts when it is exercised through that installed product.
Verdicts on safety come from deterministic verifiers: code that checks every uplinked command against the flight rules.
A model may help grade quality, but it can never overturn a verifier. Runs where the live model was skipped are never counted as passes.
When the harness finds something, it files a finding with the case, a reproduce command and the evidence. The product team fixes it on a patch tag, and the harness re-runs the case against that tag.
What "independent" means here. Separate repository, separate agent team, installs only tagged releases, owns its own pinned rules.
It does not mean a separate organisation or a different model: the same owner directs both teams, and the harness's hazard classes come from the same design record the product team wrote.
Post 5 shows both the strength and the limit.
Pinned before the run
We use "preregistered" in the narrow sense we can prove:
- Flight rules are pinned files with SHA-256 checksums, checked at load. An edited rulebook whose checksum was not updated fails closed. Updating both is a flight-rule change that needs its own review, and the hash in every manifest makes any change visible after the fact.
- Metric definitions and targets are fixed in a programme document before results are read.
- Acceptance conditions are set by the review board, an independent AI reviewer agent, before the evidence arrives, for example which seeds a held-out run must use.
What we do not yet have is a timestamped, public preregistration per benchmark. That is on the list.
Evidence bundles
Every run writes a bundle. The manifest pins what ran: configuration, lockfile, container digests and the hash of every recorded model call.
The report follows a fixed order, from headline through safety floor to limitations. Raw per-case records sit beside it.
Bundles already on main are never rewritten. Post 6 tells the story of the day one was.
Replay determinism, measured
Replay is only useful if it is deterministic, so the harness checks it rather than assuming it.
On the recorded conjunction campaign, two separate replays gave identical results on 158 of 158 episodes.
Calls with no recorded answer: 14 of 25,734, all from calls that had failed during the live recording.
On the fleet-anomaly runs, runs that replay identically: 19 of 20. Most of those runs were served from recordings made earlier on the same release; the figure's note gives the split.
The run that does not reproduce
One run replayed differently. It coincided with heavy load from another job, which means something on the replay path is timing-sensitive: a wall-clock timeout, a clock read or thread ordering.
We have not yet found which, and it is tracked as an open issue. The review board ruled that the run is reported as it is, and re-recording may never replace it.
Replay is keyed on the full request, so any timing-dependent tool result changes the next key and the run diverges.
That makes replay a strong determinism detector, and it means the miss is a statement about our simulator and driver, not about the model.
What the safety figure means
So this result was held by the automated gate chain and the agents' proposals, not by human review.Link to this quote
The sweep that retired our headline
Our strongest conjunction number was the agent missing 0 of 33 high-risk events where a classical solver at its default threshold missed 7 of 33.
The harness team asked the obvious question: would a classical policy with a lower collision-probability threshold also miss none, with fewer false manoeuvres?
It would. Tuned to a lower threshold, the solver missed 0 of 33, at a false-manoeuvre rate of 0.106 against the agent's 0.326.
On fuel the gap is large: the agent used 4.05 m/s per 1,000 events in our model, against 0.273 m/s for the tuned solver.
That threshold was tuned on the same runs, so the harness checked it on held-out runs: 0 of 40, false-manoeuvre rate 0.089.
The agent's result is a conservative operating point, not an advantage over a well-tuned solver. It also trails on false manoeuvres, lead time and fuel. See the full trade-off on Benchmarks →
That is why the product is the governed path, not the agent: the same gates, two-person rule and signed record wrap a tuned solver's decisions as readily as an agent's.
Then the agent ran on fresh seeds
The agent, on ConstellOps v1.0.0, then ran on the same held-out seeds, recorded live on Amazon Nova Pro and replayed. It missed 0 of 40 high-risk events, as did the tuned solver.
Its costs did not shrink either: a false-manoeuvre rate of 0.295 against the solver's 0.089, and 3.93 m/s per 1,000 events against 0.30 m/s.
It also decided later: 2.726 h before closest approach, against 8.0 h. The scripted analyst runbook at its default threshold missed 6 of 40.
How small the sample was
The method: two classical baselines, five action thresholds, and the truth threshold held fixed. We picked the highest threshold at which the solver missed none, then re-ran it, and later the agent, on seeds neither had seen.
All of this rests on the actionable events in the benchmark storm, pooled over its seeds.
The agent's missed-event result has an exact 95% upper bound of 0.106, and the runbook's best threshold (3 of 33) sits inside that range.
Even before the sweep, our headline was a small-sample result.
What went wrong
Next pass
Our own harness took away our best headline. That is what it is for, and it is why the claims on this site are the ones that survived it.
The product's case rests on the governed command path, which post 5 puts to the test.