Benchmarks
Benchmarked against how it is done today.
How these numbers were produced, row by row
Results next to matched classical methods where they exist, losses included.
- Safety floor: commands unsafe in the scored runs of the released code; the operator approved everything, so the gates held it
- Storm surge: high-risk missed
- Staged rollout: bad builds halted
- Gateway outage: to detect; restores slower
- Conjunctions: a tuned rule matches the agent at lower cost
- Fleet anomaly: slower to the cause; alarms not reduced
Results first: the safety envelope held in every scored episode, and the agent caught every high-risk conjunction in the benchmark storm, at a cost we publish; a classical solver tuned on the same runs can match it. Where a matched classical baseline exists, every number sits next to it, on the same seeds, with its sample size and 95% range, scored by a separate conformance and benchmark harness. Where we trail, we say so.
- Safety floor0 of 1,952 unsafe, scored runs; approve-all operator
- Storm surge0 of 11 high-risk missed
- Staged rollout20 of 20 bad builds halted
- Gateway outage1.0 min to detect; restores slower
- Conjunctions a tuned rule matches it at lower cost
- Fleet anomaly slower to the cause; alarms not reduced
Conjunctions: a tie, at a higher cost.
ⓘ
A classical solver with its threshold tuned on these same runs also missed none, with fewer unneeded manoeuvres and less fuel. On 40 fresh high-risk events, neither missed any; the agent made about three times the solver's unneeded manoeuvres.
ⓘ
About 1 in 3 low-risk events got a manoeuvre that was not needed; the tuned classical solver's rate was about a third of that.
Fresh runs: 0 of 40 high-risk missed, same as a tuned solver, at ~3× its unneeded manoeuvres.
Full table, trade-off, scope and held-out runs
Replay · Amazon Nova Pro · 60 seeds · benchmark storm of 10 events over 24 hours per seed
Held-out rows: ConstellOps v1.0.0 · recorded live, replayed 60 of 60 · seeds 60–119
| Metric | ConstellOps | Classical solver (tuned) | Scripted analyst runbook (best) |
|---|---|---|---|
| High-risk events missed | 0 of 33 | 0 of 33 | 3 of 33 |
| Low-risk events given an unneeded manoeuvre (false-manoeuvre rate) | 0.326 (higher than both baselines) | 0.106 | 0.051 |
| Fuel used per 1,000 events (Δv) | 4.05 m/s | 0.273 m/s | — |
| Hours of warning before closest approach (median decision lead time, higher is better) | 3.10 h | 8.0 h | — |
| Held-out runs (seeds 60–119): high-risk events missed (runbook at its default threshold) | 0 of 40 | 0 of 40 | 6 of 40 |
| Held-out runs: false-manoeuvre rate | 0.295 (165/560) | 0.089 | — |
| Held-out runs: fuel used per 1,000 events (Δv) | 3.93 m/s | 0.30 m/s | — |
| Held-out runs: hours of warning before closest approach (median, higher is better) | 2.726 h | 8.0 h | — |
| Held-out runs: unsafe commands sent | 0 of 205 | — | — |
The trade-off, stated once. ConstellOps runs at a conservative operating point: it missed none of the 33 high-risk events (exact 95% upper bound 0.106), at the cost of about 1 in 3 low-risk events getting a manoeuvre that was not needed. A classical solver with its threshold tuned on these same runs matches the miss count with about a third of the unneeded manoeuvres, about 15 times less fuel and earlier decisions, and it held on 40 held-out high-risk events (0 of 40 missed). On those held-out runs the agent also missed none, with about three times the solver's unneeded manoeuvres, about 13 times its fuel and later decisions (stated in full below).
The scripted analyst runbook's best is 3 of 33; it reaches 0 at no threshold swept, and below its best threshold its misses rise to 5 of 33, consistent with analyst capacity binding (not separately measured). What ConstellOps adds is not conjunction accuracy. It is the gates, two-person approval, deadline-aware uplink and signed record around every decision, and the same layer across four other workflows. The gates are deliberately model-agnostic: they can govern a classical solver's decisions as well as an agent's, so an operator can adopt the governed command path first and add agents workflow by workflow. On the product's own, larger storm the policy does miss high-risk events (52 of 265, no baseline available).
Under prompt injection
the same storm with instructions hidden in the CDM remarks. The agent still missed 0 of 33 high-risk events, and about 1 in 3 low-risk events got an unneeded manoeuvre (0.333), about the same as without injection.
A benchmark-generated conjunction storm of 10 events over 24 hours per seed, 60 seeds, the same seeds for every policy. ConstellOps figures replay a recorded live Amazon Nova Pro campaign. The classical solver and the scripted analyst runbook were each run across a range of escalation thresholds on the same runs. The classical solver column is the solver at the stricter threshold that matches the agent's miss count (Pc 2e-5, chosen on these runs, then checked on 60 held-out runs); the runbook column is the runbook at its best threshold (Pc 5e-5). The held-out rows (seeds 60–119) are ConstellOps v1.0.0, recorded live on Amazon Nova Pro and replayed, beside the tuned solver and the runbook at its default threshold (Pc 1e-4).
Safety floor
ⓘ
0 of 1,952 commands unsafe in the scored runs of the released code, with an operator that approved everything (the gates held it); an earlier build's two-person defect (1,281 of 2,434 unsafe) was found and fixed.
What this figure covers, and the earlier defect
0 of 1,952 unsafe commands sent to spacecraft in the scored runs of the released code: conjunctions on ConstellOps v1.0.0rc3 (replay of recorded model responses) and fleet anomalies, staged rollouts and space weather on v1.0.0rc4, recorded live. v1.0.0 has the same code as rc4; cross-domain incidents send no commands to spacecraft. Each command was checked by the evaluation harness against its own pinned flight rules. In every one of these runs the operator approved every proposal the gates left pending (a harness stand-in for conjunctions and fleet anomalies, the product's scripted operator for the other three workflows). So this result was held by the automated gate chain and the agents' proposals, not by human review. A further 37 commands recorded live in one fleet-anomaly episode were also judged safe (0 of 37 unsafe), but that episode does not reproduce on replay, so they are reported separately. An earlier build (rc2) had a defect: two software commands needed only one approver. Under the stricter rules, 1,281 of 2,434 commands in its staged-rollout recording were unsafe. The harness found the defect and rc4 fixed it.
Every safety row and the scope of these figures
| Measure | Result |
|---|---|
| Unsafe uplinks, conjunction episodes | 0 of 738 (exact 95% upper bound 0.005) |
| Unsafe uplinks, fleet-anomaly, rollout and space-weather runs that replay identically (v1.0.0rc4) | 0 of 1,214 |
| Unsafe commands, the one fleet-anomaly run that did not reproduce on replay | 0 of 37 |
| Unsafe commands sent, scored runs of the released code (operator approved everything; the gates held it) | 0 of 1,952 |
| Unsafe commands in an earlier build's rollout recording (two-person defect, found and fixed in rc4) | 1,281 of 2,434 |
| Unsafe uplinks under the two-person rule for software activate and rollback | 0 of 306 |
| Unsafe proposals that got past a gate | 0 of 3,900 |
| Commands executed after their deadline | 0 of 738 |
| Injection attempts that caused an action | 0 of 441 |
| Executed commands with a complete signed decision record | 738 of 738 |
| Two replays of the recorded campaign, identical results | 158 of 158 |
| Model calls with no recorded answer | 14 of 25,734 (3 seeds, kept in n) |
Conjunction rows: 158 scored episodes (storm, prompt injection and the product's own storm model), replayed from a recorded live Amazon Nova Pro campaign; not a live run on the current release. Fleet-anomaly, rollout and space-weather rows: 20 runs each on v1.0.0rc4. None of it is flight operations. "Unsafe" is scored by a separate checker with its own pinned rulebook, not by the gates themselves. The bounds are what this sample can show. Episodes with a missing recording stay in n and are scored as the product behaved.
Four further workflows, each on its own
Fleet anomalies
ConstellOps results on v1.0.0rc4 · show table
| Metric | ConstellOps (automated stand-in operator) |
|---|---|
| Correct bad build identified | 19 of 20 |
| Related alarms grouped with the right cause | 0.95 (automated stand-in operator) |
| Time to identify the common cause | 46.25 h (about 46 h, automated stand-in operator) |
| Alarm volume | +0.06% (slight increase) |
| Unsafe commands sent, 19 runs that replay identically | 0 of 777 (19 reproducible runs) |
| Unsafe commands sent, the one run that did not reproduce | 0 of 37 |
| Runs that replay identically | 19 of 20 (misses the 100% target) |
| Human sign-offs required per run (by policy) | 43.55 |
Two results are losses, and we publish them. Identifying the common cause took about 46 hours: most of that is waiting, because in this scenario the anomaly workflow runs once, at a scheduled assessment time, rather than on each new alarm. Alarm volume showed a small net increase. One run of 20 did not replay identically, which misses our target; its 37 commands were checked separately and are counted in the site-wide total of commands sent.
Sign-offs reflect the product's per-command approval policy (every command approved individually and signed; tunable by autonomy level), not hours of work.
Cross-domain incidents
ConstellOps results on v1.0.0rc4 · show table
| Metric | ConstellOps |
|---|---|
| Runs completed | 16 of 20 (4 refused at start) |
| Mean time to detect | about 1.0 min |
| Time to restore, mean | 15 min |
| Capacity kept during the incident | 0.824 (about 82%, synthetic network model) |
| Customer service lost per incident | 270,000 subscriber-minutes |
| Commands sent to spacecraft | 0 (ground-only remediation) |
| Human sign-offs required per run (by policy) | 2.38 |
Sign-offs reflect the product's per-command approval policy (every command approved individually and signed; tunable by autonomy level), not hours of work.
4 runs stopped safely at start: no backup gateway in the region, or no spacecraft in view, so the product returned an input error rather than act.
Traffic and capacity come from a documented synthetic network model, so these figures describe the model, not a real network. Remediation ran on the ground; no command was sent to a spacecraft.
Staged rollouts
ConstellOps results on v1.0.0rc4 · show table
| Metric | ConstellOps |
|---|---|
| Bad-build rollouts halted before the full fleet | 20 of 20 |
| Canary wave size | 6 |
| Satellites on the bad build before the halt | 5 |
| Time from first install to halt | 0.45 h (about 27 min) |
| Rollouts where the bad build went beyond the canary | 0 of 20 |
| Unsafe uplinks, with two distinct approvers on every software activate and rollback | 0 of 306 |
| Runs that replay identically | 20 of 20 |
| Human sign-offs required per run (by policy; two-person actions count twice) | 27.4 |
Sign-offs reflect the product's per-command approval policy (every command approved individually and signed; tunable by autonomy level), not hours of work.
There is no matched comparison with classical rollout methods: the product's recorded runs used their own install schedule, fault onset and canary membership (canary of 6), while the classical runs used the scenario generator's, so no comparison is shown.
The halt is deterministic by design; it does not depend on the model. The agent's job is the plan, the evidence and the rollback proposal. The scenario's regression shows within minutes, so the halt came 0.45 h after the first install. Real canary soaks should span several orbits, including eclipse; the soak length is an operator setting. Only the bad-build case was run, so no false-halt figure is published.
Space-weather surge
ConstellOps results on v1.0.0rc4 · show table
| Metric | ConstellOps |
|---|---|
| High-risk events missed during the surge | 0 of 11 |
| Unneeded manoeuvres | 0 of 103 |
| Commands executed after their deadline | 0 of 131 |
| Commands that missed their cutoff | 0 |
| Unsafe uplinks | 0 of 131 |
| Runs that replay identically | 20 of 20 |
There is no matched comparison with classical methods for this workflow, so none is shown.
Recorded historical storms replayed as a perfect forecast, over a synthetic fleet. Few actionable events per run, so the sample is small.
Matched against classical methods: where we trail.
Full matched tables, caveats and sign-offs
| Fleet anomaly (matched, 20 runs) | ConstellOps (automated stand-in operator) | Classical solver (threshold tuned on these runs) | Scripted runbook |
|---|---|---|---|
| Verdict | Behind on time to cause, alarm volume and correct build | — | — |
| Time to identify the common cause | 46.25 h (about 46 h, automated stand-in operator) | 4.32 h | 25.55 h |
| Time to identify the common cause, held-out runs | not run | 5.40 h | — |
| Alarm volume | +0.06% (slight increase) | −16% | −8% |
| Correct build identified | 19 of 20 | 20 of 20 | 19 of 20 |
| Human sign-offs required per run (by policy) | 43.55 | 0 (automated) | 1 (one sign-off) |
| Gateway outage (partial match, 16 runs) | ConstellOps | Classical solver | Scripted runbook |
|---|---|---|---|
| Verdict | Detects fastest; restores slower than the classical solver | — | — |
| Mean time to detect | about 1.0 min (event-driven) | 5.0 min (checks every 5.0 min) | 24.63 min |
| Mean time to restore | about 15 min | 5.0 min | 117.66 min |
| Human sign-offs required per run (by policy) | 2.38 | 0 (automated) | 2.56 (one sign-off per proposal) |
| Staged rollout (not matched: policy count only) | ConstellOps | Classical solver | Scripted runbook |
|---|---|---|---|
| Human sign-offs required per run (by policy) | 27.4 | 0 (automated) | 1 (one sign-off) |
Sign-offs reflect the product's per-command approval policy (every command approved individually and signed; tunable by autonomy level), not hours of work. Fleet anomaly is a loss for the product: slower to the cause, alarm volume up rather than down, and the right build in 19 of 20 runs against 20 of 20 for the solver, with an automated stand-in operator. On the gateway outage the product detects fastest (event-driven; the classical solver checks every 5.0 min), but only on the primary outage: it did not model the nuisance alarms (16 of 16 runs), a second outage was present in 3 of 16 runs, and capacity and subscriber-minutes are not comparable across systems. The classical solver restores service faster. Rollout and storm-surge outcomes have no matched comparison.
The classical solver and the scripted runbook were run with the same scenario generators and seeds as the product's recorded runs; how much of each world the product actually used differs by workflow. Fleet anomaly is matched; the gateway outage is a partial match (the primary outage only, on the 16 runs the product ran); rollout and storm surge are not matched, so no comparison is shown for them. Approval records are a policy count: under its A1 policy the product asks for an approval on every command, the classical solver is automated, and the runbook asks for one sign-off per proposal. They are not a measure of effort.
What these numbers do not show.
Eight limits, each published in full.
Read every limit
- The trade-off. More unneeded manoeuvres than both baselines; more fuel and less warning time than the tuned classical solver (table above).
- Lead time against the screening cycle. The agent's median decision lead time before closest approach (in the conjunction table) is shorter than one civil screening cycle [88]. In operations that would leave too little time to screen the post-manoeuvre orbit before uplink. As measured, this setting is not ready for operations; shadow mode on your own data exists to tune it.
- A tuned rule matches zero missed. See the trade-off stated with the conjunction table above.
- Losses in the other workflows. With an automated stand-in operator, fleet-anomaly diagnosis took about 46 hours and alarm volume showed a small net increase; 1 of 20 fleet-anomaly runs did not replay identically. The incident workflow refused 4 of 20 runs at start, because the seeded outage had no backup gateway in its region or no spacecraft in view at onset; it stopped at open with an input error on those scenarios rather than act on them. No false-halt rate is published for rollouts, because only the bad-build case was run. Fleet anomaly is matched against classical methods and is a loss for the product; the gateway outage is a partial match; rollout and storm surge have no matched comparison.
- Small storms. The benchmark storm is 10 conjunction events over 24 hours per seed. It says nothing yet about fleet-scale event volumes. Scenarios are synthetic.
- The product's own storm model, no baselines. On a larger storm generated by the product itself, the agent missed high-risk events: 52 of 265, with a false-manoeuvre rate of 0.146 and a median lead time of 5.45 h. The baselines do not run on these inputs, so there is no comparison.
- Replay, not live. Conjunction figures on seeds 0–59 and the safety figures replay a recorded live campaign on the release under test; 14 of 25,734 had no recording and stay in n. The held-out conjunction runs (seeds 60–119) were recorded live on v1.0.0 and replayed, 60 of 60.
- CDM-only feeds, ESA Kelvins collision-avoidance challenge (L score, lower is better). These CDMs are anonymised features with no absolute states and no hard-body radius, so ConstellOps passes through the reported Pc rather than recompute it with an assumed radius, and escalates. On the full set it scores 0.694, the same as the latest-risk baseline by design; the challenge winner scored 0.556, which is better. It has no model of how risk evolves between CDMs, which is what that challenge rewards. No triage gain is claimed.
How to read these numbers
Baselines, replay and labels, and the superseded default-threshold comparison
- Replay, in one line. We re-ran the released product on the model answers recorded during a live run, at no model cost. The details are below.
- Best published result: the top published score on a public dataset, where one exists.
- Solver-only policy: the verified tools with fixed rules, no agent.
- Threshold runbook: a scripted runbook, the way a team runs it today.
- Live means the model was called during the run. Replay means the model responses recorded in an earlier live run were served again to the release under test, at zero model cost. Two replays of the conjunction campaign gave identical results on 158 of 158; a small number of model calls had no recording (14 of 25,734), and those episodes stay in n.
- Every figure shows its mode, model (Amazon Nova Pro), n or seeds, interval where one exists, and baseline.
Superseded: the baselines at their default thresholds.
Before the threshold sweep, the classical policies were run only at their default threshold (Pc 1e-4). These rows are kept for traceability; the sweep above supersedes them as the comparison.
| Metric | ConstellOps | Threshold runbook, default threshold | Solver-only policy, default threshold |
|---|---|---|---|
| High-risk events missed | 0 of 33 | 7 of 33 | 7 of 33 |
| Low-risk events given an unneeded manoeuvre (false-manoeuvre rate) | 0.326 (higher than both baselines) | 0.019 | 0.009 |
| Hours of warning before closest approach (median decision lead time, higher is better) | 3.10 h | 11.68 h (35 of 60 seeds) | 8.0 h (fixed by the policy) |
| Fuel used per 1,000 events (Δv) | 4.05 m/s | 0.119 m/s | 0.139 m/s |
| Operator review minutes per event (modelled assumption, not a result) | 3.78 min | 21.23 min | 0 min (no human review in this policy) |
High-risk events missed lower is better
Low-risk events given an unneeded manoeuvre lower is better
On a narrow screen the plot is replaced by its values:
- ConstellOps Replaymissed 0 of 33 · false-manoeuvre rate 0.326
- Threshold runbook (default threshold) classical baseline (simulated)missed 7 of 33 · false-manoeuvre rate 0.019
- Solver-only policy (default threshold) classical baseline (simulated)missed 7 of 33 · false-manoeuvre rate 0.009
Lead time and fuel, the other two sides of the trade-off, are in the conjunction table above.
Method, in brief.
Seeds, baselines, intervals and how failed runs are counted
Seeded synthetic scenarios, with public datasets where noted. Baselines run on the same seeds and inputs. 95% ranges are computed over seeds; zero counts carry an exact 95% upper bound where one is published. 33 actionable events is a small sample: 0 of 33 missed has an exact 95% upper bound of 0.106, so it cannot support claims about rare-event behaviour at fleet scale. Failed or incomplete episodes stay in n and are scored as the product behaved. Numbers are never rounded in a way that changes their meaning.
Ask us to walk through any number on this page.