Engineering blog Post 04

A harness that is allowed to say no

Vivek Raja6 min readPart 4 of 9, published as one series

  • harness
  • benchmarks
  • evidence
  • replay
Independent by constructionpipeline
Taggedreleasenever a branchPinnedscenariorules hashedRunlive or replayVerifiersdeterministicEvidencebundlemanifest ·reportGo / no-goAI review boarda model judge can never overturn a verifier

ConstellOps is tested by an independent conformance and benchmark harness: a separate code base, run by a separate team, that only ever sees tagged releases. Its job is to find out what is true, including when that is bad news for the product.

Independent by construction

The harness never imports a branch. It installs a tagged release, the way an operator would, and a capability only counts when it is exercised through that installed product.

Verdicts on safety come from deterministic verifiers: code that checks every uplinked command against the flight rules.

A model may help grade quality, but it can never overturn a verifier. Runs where the live model was skipped are never counted as passes.

When the harness finds something, it files a finding with the case, a reproduce command and the evidence. The product team fixes it on a patch tag, and the harness re-runs the case against that tag.

What "independent" means here. Separate repository, separate agent team, installs only tagged releases, owns its own pinned rules.

It does not mean a separate organisation or a different model: the same owner directs both teams, and the harness's hazard classes come from the same design record the product team wrote.

Post 5 shows both the strength and the limit.

Pinned before the run

We use "preregistered" in the narrow sense we can prove:

  • Flight rules are pinned files with SHA-256 checksums, checked at load. An edited rulebook whose checksum was not updated fails closed. Updating both is a flight-rule change that needs its own review, and the hash in every manifest makes any change visible after the fact.
  • Metric definitions and targets are fixed in a programme document before results are read.
  • Acceptance conditions are set by the review board, an independent AI reviewer agent, before the evidence arrives, for example which seeds a held-out run must use.

What we do not yet have is a timestamped, public preregistration per benchmark. That is on the list.

Evidence bundles

Every run writes a bundle. The manifest pins what ran: configuration, lockfile, container digests and the hash of every recorded model call.

The report follows a fixed order, from headline through safety floor to limitations. Raw per-case records sit beside it.

Bundles already on main are never rewritten. Post 6 tells the story of the day one was.

What an evidence bundle pinsdata layout
manifest.jsonconfig · lockfile · containers · cassettes (SHA-256)what ran, exactlyreport.mdheadline → safety floor → quality → reproduce → limitsfixed section ordermetrics.jsonmachine-readable figuresfeeds the claims registerraw/per-case records · re-verification · replay passnever rewrittenbilling-verify.txtmodel spend cross-checkcredits only

Replay determinism, measured

Replay is only useful if it is deterministic, so the harness checks it rather than assuming it.

On the recorded conjunction campaign, two separate replays gave identical results on 158 of 158 episodes.

Calls with no recorded answer: 14 of 25,734, all from calls that had failed during the live recording.

On the fleet-anomaly runs, runs that replay identically: 19 of 20. Most of those runs were served from recordings made earlier on the same release; the figure's note gives the split.

The run that does not reproduce

One run replayed differently. It coincided with heavy load from another job, which means something on the replay path is timing-sensitive: a wall-clock timeout, a clock read or thread ordering.

We have not yet found which, and it is tracked as an open issue. The review board ruled that the run is reported as it is, and re-recording may never replace it.

Replay is keyed on the full request, so any timing-dependent tool result changes the next key and the run diverges.

That makes replay a strong determinism detector, and it means the miss is a statement about our simulator and driver, not about the model.

What the safety figure means

So this result was held by the automated gate chain and the agents' proposals, not by human review.
Link to this quote

The sweep that retired our headline

Our strongest conjunction number was the agent missing 0 of 33 high-risk events where a classical solver at its default threshold missed 7 of 33.

The harness team asked the obvious question: would a classical policy with a lower collision-probability threshold also miss none, with fewer false manoeuvres?

It would. Tuned to a lower threshold, the solver missed 0 of 33, at a false-manoeuvre rate of 0.106 against the agent's 0.326.

On fuel the gap is large: the agent used 4.05 m/s per 1,000 events in our model, against 0.273 m/s for the tuned solver.

That threshold was tuned on the same runs, so the harness checked it on held-out runs: 0 of 40, false-manoeuvre rate 0.089.

The threshold sweep that retired our headlinechart
High-risk events missed (lower is better)ConstellOps agent0 of 33Solver, default threshold7 of 33Solver, tuned threshold0 of 33Runbook, best threshold3 of 33False-manoeuvre rate (lower is better)ConstellOps agent0.326Solver, default threshold0.009Solver, tuned threshold0.106Runbook, best threshold0.051
Missed of the actionable events: agent 0 of 33, solver at default 7 of 33, solver tuned 0 of 33, runbook best 3 of 33. False-manoeuvre rate: agent 0.326, solver at default 0.009, solver tuned 0.106, runbook best 0.051.

The agent's result is a conservative operating point, not an advantage over a well-tuned solver. It also trails on false manoeuvres, lead time and fuel. See the full trade-off on Benchmarks →

That is why the product is the governed path, not the agent: the same gates, two-person rule and signed record wrap a tuned solver's decisions as readily as an agent's.

Then the agent ran on fresh seeds

The agent, on ConstellOps v1.0.0, then ran on the same held-out seeds, recorded live on Amazon Nova Pro and replayed. It missed 0 of 40 high-risk events, as did the tuned solver.

Its costs did not shrink either: a false-manoeuvre rate of 0.295 against the solver's 0.089, and 3.93 m/s per 1,000 events against 0.30 m/s.

It also decided later: 2.726 h before closest approach, against 8.0 h. The scripted analyst runbook at its default threshold missed 6 of 40.

How small the sample was

The method: two classical baselines, five action thresholds, and the truth threshold held fixed. We picked the highest threshold at which the solver missed none, then re-ran it, and later the agent, on seeds neither had seen.

All of this rests on the actionable events in the benchmark storm, pooled over its seeds.

The agent's missed-event result has an exact 95% upper bound of 0.106, and the runbook's best threshold (3 of 33) sits inside that range.

Even before the sweep, our headline was a small-sample result.

What went wrong

Next pass

Our own harness took away our best headline. That is what it is for, and it is why the claims on this site are the ones that survived it.

The product's case rests on the governed command path, which post 5 puts to the test.

§ · NEXT PASS

Request a briefing.

A 45-minute session with the evidence behind any figure. Operators can send a week of their queue.

This address is used only to reply to you. The site sets no cookies and loads no trackers.

Briefing requests

Sources and labels for every figure on this page (26)

Why two release numbers: the conjunction and safety figures were measured on release v1.0.0rc3 by replaying the model calls recorded during the live campaign on v1.0.0rc2. The demonstration clips were captured on v1.0.0rc4; v1.0.0 ships identical code.

  1. 158 of 158 Two replays of the recorded campaign gave identical results: 158 of 158 episodes

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 158 seeds · baseline: target 100% · scenario: conjunction storm, CDM prompt injection, product-native storm (158 episodes)

  2. 14 of 25,734 Model calls with no recorded answer: 14 of 25,734 (3 seeds, kept in n)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 25,734 model-call lookups over 158 seeds (3 seeds affected, kept in n) · scenario: conjunction storm, CDM prompt injection, product-native storm (158 episodes)

  3. 19 of 20 Runs that replay identically: 19 of 20 (misses the 100% target)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 8 runs recorded live, 12 replayed · n: 20 seeds · 95% range: 0.751–0.999 · scenario: common-mode anomaly from a bad software build

  4. 0 of 1,952 Unsafe commands in the scored runs of the released code, with an operator that approved everything: 0 of 1,952

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3 (conjunction replay) and v1.0.0rc4 (other workflows); v1.0.0 ships the same code as rc4 · n: 1,952 commands sent to spacecraft: 738 conjunction (v1.0.0rc3 replay of recorded model responses) and 1,214 on v1.0.0rc4 recorded live (fleet anomaly 777, rollout 306, space weather 131); the incident workflow sends no command to spacecraft · 95% range: 0–0.002 · baseline: target 0 (safety floor) · scenario: scored runs of the released code; in every run the operator approved every proposal the gates left pending (an automated stand-in for conjunction and fleet anomaly, the product's scripted operator for the others), so the result was held by the automated gate chain and the agents' proposals, not by human review

  5. 1,281 of 2,434 Earlier build's two-person defect: 1,281 of 2,434 commands unsafe under the stricter rules (found by the separate benchmark checks, fixed in rc4)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc2 (pre-fix); fixed in v1.0.0rc4 · n: 2,434 commands in the rc2 staged-rollout recording · 95% range: 0.506–0.546 · baseline: target 0 (safety floor) · scenario: an earlier build had a defect: two software commands needed only one approver; the separate benchmark checks found it and rc4 fixed it

  6. 0 of 33 Missed high-risk conjunctions: 0 of 33

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds (33 actionable events pooled) · exact 95% upper bound: 0.106 (about 10.6%) · baseline: scripted threshold runbook 7/33; classical solver-only policy 7/33 · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  7. 7 of 33 Classical solver-only policy: 7 of 33 missed

    mode: Simulated · n: 60 seeds (33 actionable events pooled) · 95% range: 0.088–0.355 · row: this row is the classical solver-only baseline · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  8. 0 of 33 Classical solver tuned to a lower threshold: 0 of 33 missed

    mode: Simulated · n: 60 seeds (33 actionable events pooled) · 95% range: 0–0.106 · row: classical solver-only policy at a lower Pc threshold, 2e-5 · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  9. 0.106 Classical solver tuned to a lower threshold: false-manoeuvre rate 0.106

    mode: Simulated · n: 60 seeds (567 below-threshold events pooled) · 95% range: 0.08–0.133 · row: classical solver-only policy at a lower Pc threshold, 2e-5 · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  10. 0.326 False-manoeuvre rate: 0.326 (higher than both baselines)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds (567 below-threshold events pooled) · 95% range: 0.288–0.368 · baseline: scripted threshold runbook 0.019; classical solver-only policy 0.009 · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  11. 4.05 m/s Delta-v spent: 4.05 m/s per 1000 events (higher than both baselines)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds · 95% range: 3.558–4.55 · baseline: scripted threshold runbook 0.119; classical solver-only policy 0.139 · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  12. 0.273 m/s Classical solver tuned to a lower threshold: 0.273 m/s per 1000 events

    mode: Simulated · n: 60 seeds · 95% range: 0.216–0.329 · row: classical solver-only policy at a lower Pc threshold, 2e-5 · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  13. 0 of 40 Tuned classical solver on held-out runs: 0 of 40 missed

    mode: Simulated · n: 60 held-out seeds 60-119 (40 actionable events pooled) · 95% range: 0–0.088 · row: classical solver-only policy at Pc 2e-5, threshold chosen on seeds 0-59, checked on held-out seeds · scenario: held-out confirmation, same storm generator, seeds 60-119

  14. 0.089 Tuned classical solver on held-out runs: false-manoeuvre rate 0.089

    mode: Simulated · n: 60 held-out seeds 60-119 (560 below-threshold events pooled) · 95% range: 0.067–0.113 · row: classical solver-only policy at Pc 2e-5 on held-out seeds · scenario: held-out confirmation, same storm generator, seeds 60-119

  15. 3 of 33 Scripted runbook, best swept threshold (Pc 5e-5): 3 of 33 missed

    mode: Simulated · n: 60 seeds (33 actionable events pooled) · 95% range: 0–0.2 · row: scripted threshold runbook at its best swept threshold, Pc 5e-5; analyst capacity binds · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  16. 0.009 Classical solver-only policy: 0.009

    mode: Simulated · n: 60 seeds · 95% range: 0.002–0.017 · row: this row is the classical solver-only baseline · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  17. 0.051 Scripted runbook, best swept threshold (Pc 5e-5): false-manoeuvre rate 0.051

    mode: Simulated · n: 60 seeds (567 below-threshold events pooled) · 95% range: 0.032–0.071 · row: scripted threshold runbook at its best swept threshold, Pc 5e-5 · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  18. 0 of 40 ConstellOps on held-out runs: 0 of 40 high-risk missed

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 (40 high-risk events pooled) · 95% range: 0–0.088 · baseline: classical solver-only policy at Pc 2e-5 (threshold chosen on seeds 0-59) and scripted analyst runbook at its default threshold (Pc 1e-4), same held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  19. 0.295 ConstellOps on held-out runs: false-manoeuvre rate 0.295 (165/560)

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 (560 below-threshold events pooled) · 95% range: 0.262–0.329 · baseline: classical solver-only policy at Pc 2e-5 (threshold chosen on seeds 0-59) and scripted analyst runbook at its default threshold (Pc 1e-4), same held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  20. 3.93 m/s ConstellOps on held-out runs: 3.93 m/s per 1000 events

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 · 95% range: 3.458–4.4 · baseline: classical solver-only policy at Pc 2e-5 (threshold chosen on seeds 0-59) and scripted analyst runbook at its default threshold (Pc 1e-4), same held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  21. 0.30 m/s Tuned classical solver on held-out runs: 0.30 m/s per 1000 events

    mode: Simulated · n: 60 held-out seeds 60-119 · 95% range: 0.242–0.345 · row: classical solver-only policy at Pc 2e-5, threshold chosen on seeds 0-59, held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  22. 2.726 h ConstellOps on held-out runs: 2.7 h before closest approach (median)

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 · 95% range: 2.401–3.09 · baseline: classical solver-only policy at Pc 2e-5 (threshold chosen on seeds 0-59) and scripted analyst runbook at its default threshold (Pc 1e-4), same held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  23. 8.0 h Tuned classical solver on held-out runs: 8.0 h before closest approach (fixed by the policy)

    mode: Simulated · n: 60 held-out seeds 60-119 · 95% range: no spread (the same value on every seed) · row: classical solver-only policy at Pc 2e-5; lead time fixed by the policy · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  24. 6 of 40 Scripted analyst runbook at its default threshold, held-out runs: 6 of 40 missed

    mode: Simulated · n: 60 held-out seeds 60-119 (40 high-risk events pooled) · 95% range: 0.05–0.27 · row: scripted analyst runbook at its default threshold, Pc 1e-4, held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  25. 0.106 Missed high-risk conjunctions, exact 95% upper bound: 0.106 (about 10.6%)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds (33 actionable events pooled) · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59