Benchmarks

Benchmarked against how it is done today.

How these numbers were produced, row by row

Results next to matched classical methods where they exist, losses included.

  • Safety floor: commands unsafe in the scored runs of the released code; the operator approved everything, so the gates held it
  • Storm surge: high-risk missed
  • Staged rollout: bad builds halted
  • Gateway outage: to detect; restores slower
  • Conjunctions: a tuned rule matches the agent at lower cost
  • Fleet anomaly: slower to the cause; alarms not reduced

Results first: the safety envelope held in every scored episode, and the agent caught every high-risk conjunction in the benchmark storm, at a cost we publish; a classical solver tuned on the same runs can match it. Where a matched classical baseline exists, every number sits next to it, on the same seeds, with its sample size and 95% range, scored by a separate conformance and benchmark harness. Where we trail, we say so.

Results by workflow
  1. Safety floor0 of 1,952 unsafe, scored runs; approve-all operator
  2. Storm surge0 of 11 high-risk missed
  3. Staged rollout20 of 20 bad builds halted
  4. Gateway outage1.0 min to detect; restores slower
  5. Conjunctions a tuned rule matches it at lower cost
  6. Fleet anomaly slower to the cause; alarms not reduced
behindtieaheadSafety floor: Target heldTarget held (vs target)Storm surge: No comparisonNo comparisonStaged rollout: No comparisonNo comparisonGateway outage: SplitSplitConjunctions: TieTieFleet anomaly: BehindBehind

Conjunctions: a tie, at a higher cost.

0 of 33

high-risk missed
ⓘ

A classical solver with its threshold tuned on these same runs also missed none, with fewer unneeded manoeuvres and less fuel. On 40 fresh high-risk events, neither missed any; the agent made about three times the solver's unneeded manoeuvres.

How it was measured

0.326

unneeded-burn rate
ⓘ

About 1 in 3 low-risk events got a manoeuvre that was not needed; the tuned classical solver's rate was about a third of that.

How it was measured

Fresh runs: 0 of 40 high-risk missed, same as a tuned solver, at ~3× its unneeded manoeuvres.

Full table, trade-off, scope and held-out runs

Replay · Amazon Nova Pro · 60 seeds · benchmark storm of 10 events over 24 hours per seed

Held-out rows: ConstellOps v1.0.0 · recorded live, replayed 60 of 60 · seeds 60–119

Conjunction benchmark
MetricConstellOpsClassical solver (tuned)Scripted analyst runbook (best)
High-risk events missed0 of 330 of 333 of 33
Low-risk events given an unneeded manoeuvre (false-manoeuvre rate)0.326 (higher than both baselines)0.1060.051
Fuel used per 1,000 events (Δv)4.05 m/s0.273 m/s—
Hours of warning before closest approach (median decision lead time, higher is better)3.10 h8.0 h—
Held-out runs (seeds 60–119): high-risk events missed (runbook at its default threshold)0 of 400 of 406 of 40
Held-out runs: false-manoeuvre rate0.295 (165/560)0.089—
Held-out runs: fuel used per 1,000 events (Δv)3.93 m/s0.30 m/s—
Held-out runs: hours of warning before closest approach (median, higher is better)2.726 h8.0 h—
Held-out runs: unsafe commands sent0 of 205——

The trade-off, stated once. ConstellOps runs at a conservative operating point: it missed none of the 33 high-risk events (exact 95% upper bound 0.106), at the cost of about 1 in 3 low-risk events getting a manoeuvre that was not needed. A classical solver with its threshold tuned on these same runs matches the miss count with about a third of the unneeded manoeuvres, about 15 times less fuel and earlier decisions, and it held on 40 held-out high-risk events (0 of 40 missed). On those held-out runs the agent also missed none, with about three times the solver's unneeded manoeuvres, about 13 times its fuel and later decisions (stated in full below).

The scripted analyst runbook's best is 3 of 33; it reaches 0 at no threshold swept, and below its best threshold its misses rise to 5 of 33, consistent with analyst capacity binding (not separately measured). What ConstellOps adds is not conjunction accuracy. It is the gates, two-person approval, deadline-aware uplink and signed record around every decision, and the same layer across four other workflows. The gates are deliberately model-agnostic: they can govern a classical solver's decisions as well as an agent's, so an operator can adopt the governed command path first and add agents workflow by workflow. On the product's own, larger storm the policy does miss high-risk events (52 of 265, no baseline available).

Under prompt injection

the same storm with instructions hidden in the CDM remarks. The agent still missed 0 of 33 high-risk events, and about 1 in 3 low-risk events got an unneeded manoeuvre (0.333), about the same as without injection.

A benchmark-generated conjunction storm of 10 events over 24 hours per seed, 60 seeds, the same seeds for every policy. ConstellOps figures replay a recorded live Amazon Nova Pro campaign. The classical solver and the scripted analyst runbook were each run across a range of escalation thresholds on the same runs. The classical solver column is the solver at the stricter threshold that matches the agent's miss count (Pc 2e-5, chosen on these runs, then checked on 60 held-out runs); the runbook column is the runbook at its best threshold (Pc 5e-5). The held-out rows (seeds 60–119) are ConstellOps v1.0.0, recorded live on Amazon Nova Pro and replayed, beside the tuned solver and the runbook at its default threshold (Pc 1e-4).

Safety floor

0 of 1,952

commands unsafe, scored runs
ⓘ

0 of 1,952 commands unsafe in the scored runs of the released code, with an operator that approved everything (the gates held it); an earlier build's two-person defect (1,281 of 2,434 unsafe) was found and fixed.

How it was measured

What this figure covers, and the earlier defect

0 of 1,952 unsafe commands sent to spacecraft in the scored runs of the released code: conjunctions on ConstellOps v1.0.0rc3 (replay of recorded model responses) and fleet anomalies, staged rollouts and space weather on v1.0.0rc4, recorded live. v1.0.0 has the same code as rc4; cross-domain incidents send no commands to spacecraft. Each command was checked by the evaluation harness against its own pinned flight rules. In every one of these runs the operator approved every proposal the gates left pending (a harness stand-in for conjunctions and fleet anomalies, the product's scripted operator for the other three workflows). So this result was held by the automated gate chain and the agents' proposals, not by human review. A further 37 commands recorded live in one fleet-anomaly episode were also judged safe (0 of 37 unsafe), but that episode does not reproduce on replay, so they are reported separately. An earlier build (rc2) had a defect: two software commands needed only one approver. Under the stricter rules, 1,281 of 2,434 commands in its staged-rollout recording were unsafe. The harness found the defect and rc4 fixed it.

Every safety row and the scope of these figures
Safety floor
MeasureResult
Unsafe uplinks, conjunction episodes0 of 738 (exact 95% upper bound 0.005)
Unsafe uplinks, fleet-anomaly, rollout and space-weather runs that replay identically (v1.0.0rc4)0 of 1,214
Unsafe commands, the one fleet-anomaly run that did not reproduce on replay0 of 37
Unsafe commands sent, scored runs of the released code (operator approved everything; the gates held it)0 of 1,952
Unsafe commands in an earlier build's rollout recording (two-person defect, found and fixed in rc4)1,281 of 2,434
Unsafe uplinks under the two-person rule for software activate and rollback0 of 306
Unsafe proposals that got past a gate0 of 3,900
Commands executed after their deadline0 of 738
Injection attempts that caused an action0 of 441
Executed commands with a complete signed decision record738 of 738
Two replays of the recorded campaign, identical results158 of 158
Model calls with no recorded answer14 of 25,734 (3 seeds, kept in n)

Conjunction rows: 158 scored episodes (storm, prompt injection and the product's own storm model), replayed from a recorded live Amazon Nova Pro campaign; not a live run on the current release. Fleet-anomaly, rollout and space-weather rows: 20 runs each on v1.0.0rc4. None of it is flight operations. "Unsafe" is scored by a separate checker with its own pinned rulebook, not by the gates themselves. The bounds are what this sample can show. Episodes with a missing recording stay in n and are scored as the product behaved.

Economics

Economics
MeasureResult
Model cost per 10-event storm episode, list price$0.468
Model cost to re-run the conjunction benchmark from recordings$0

Four further workflows, each on its own

Fleet anomalies

ConstellOps results on v1.0.0rc4 · show table
Fleet anomalies
MetricConstellOps (automated stand-in operator)
Correct bad build identified19 of 20
Related alarms grouped with the right cause0.95 (automated stand-in operator)
Time to identify the common cause46.25 h (about 46 h, automated stand-in operator)
Alarm volume+0.06% (slight increase)
Unsafe commands sent, 19 runs that replay identically0 of 777 (19 reproducible runs)
Unsafe commands sent, the one run that did not reproduce0 of 37
Runs that replay identically19 of 20 (misses the 100% target)
Human sign-offs required per run (by policy)43.55

Two results are losses, and we publish them. Identifying the common cause took about 46 hours: most of that is waiting, because in this scenario the anomaly workflow runs once, at a scheduled assessment time, rather than on each new alarm. Alarm volume showed a small net increase. One run of 20 did not replay identically, which misses our target; its 37 commands were checked separately and are counted in the site-wide total of commands sent.

Sign-offs reflect the product's per-command approval policy (every command approved individually and signed; tunable by autonomy level), not hours of work.

Fleet anomalies page →

Cross-domain incidents

ConstellOps results on v1.0.0rc4 · show table
Cross-domain incidents
MetricConstellOps
Runs completed16 of 20 (4 refused at start)
Mean time to detectabout 1.0 min
Time to restore, mean15 min
Capacity kept during the incident0.824 (about 82%, synthetic network model)
Customer service lost per incident270,000 subscriber-minutes
Commands sent to spacecraft0 (ground-only remediation)
Human sign-offs required per run (by policy)2.38

Sign-offs reflect the product's per-command approval policy (every command approved individually and signed; tunable by autonomy level), not hours of work.

4 runs stopped safely at start: no backup gateway in the region, or no spacecraft in view, so the product returned an input error rather than act.

Traffic and capacity come from a documented synthetic network model, so these figures describe the model, not a real network. Remediation ran on the ground; no command was sent to a spacecraft.

Cross-domain incidents page →

Staged rollouts

ConstellOps results on v1.0.0rc4 · show table
Staged rollouts
MetricConstellOps
Bad-build rollouts halted before the full fleet20 of 20
Canary wave size6
Satellites on the bad build before the halt5
Time from first install to halt0.45 h (about 27 min)
Rollouts where the bad build went beyond the canary0 of 20
Unsafe uplinks, with two distinct approvers on every software activate and rollback0 of 306
Runs that replay identically20 of 20
Human sign-offs required per run (by policy; two-person actions count twice)27.4

Sign-offs reflect the product's per-command approval policy (every command approved individually and signed; tunable by autonomy level), not hours of work.

There is no matched comparison with classical rollout methods: the product's recorded runs used their own install schedule, fault onset and canary membership (canary of 6), while the classical runs used the scenario generator's, so no comparison is shown.

The halt is deterministic by design; it does not depend on the model. The agent's job is the plan, the evidence and the rollback proposal. The scenario's regression shows within minutes, so the halt came 0.45 h after the first install. Real canary soaks should span several orbits, including eclipse; the soak length is an operator setting. Only the bad-build case was run, so no false-halt figure is published.

Staged rollouts page →

Space-weather surge

ConstellOps results on v1.0.0rc4 · show table
Space-weather surge
MetricConstellOps
High-risk events missed during the surge0 of 11
Unneeded manoeuvres0 of 103
Commands executed after their deadline0 of 131
Commands that missed their cutoff0
Unsafe uplinks0 of 131
Runs that replay identically20 of 20

There is no matched comparison with classical methods for this workflow, so none is shown.

Recorded historical storms replayed as a perfect forecast, over a synthetic fleet. Few actionable events per run, so the sample is small.

Space-weather surge page →

Matched against classical methods: where we trail.

46.25 h

fleet anomaly: to the cause (behind)
ⓘ

A tuned classical solver took 4.32 h; the scripted runbook 25.55 h.

How it was measured

1.0 min

outage: to detect (restores slower)
ⓘ

Event-driven; the classical solver checks every 5.0 min and restored service in 5.0 min, against about 15 min.

How it was measured

Full matched tables, caveats and sign-offs
Fleet anomaly (matched, 20 runs)
Fleet anomaly (matched, 20 runs)ConstellOps (automated stand-in operator)Classical solver (threshold tuned on these runs)Scripted runbook
VerdictBehind on time to cause, alarm volume and correct build——
Time to identify the common cause46.25 h (about 46 h, automated stand-in operator)4.32 h25.55 h
Time to identify the common cause, held-out runsnot run5.40 h—
Alarm volume+0.06% (slight increase)−16%−8%
Correct build identified19 of 2020 of 2019 of 20
Human sign-offs required per run (by policy)43.550 (automated)1 (one sign-off)
Gateway outage (partial match, 16 runs)
Gateway outage (partial match, 16 runs)ConstellOpsClassical solverScripted runbook
VerdictDetects fastest; restores slower than the classical solver——
Mean time to detectabout 1.0 min (event-driven)5.0 min (checks every 5.0 min)24.63 min
Mean time to restoreabout 15 min5.0 min117.66 min
Human sign-offs required per run (by policy)2.380 (automated)2.56 (one sign-off per proposal)
Staged rollout (not matched: policy count only)
Staged rollout (not matched: policy count only)ConstellOpsClassical solverScripted runbook
Human sign-offs required per run (by policy)27.40 (automated)1 (one sign-off)

Sign-offs reflect the product's per-command approval policy (every command approved individually and signed; tunable by autonomy level), not hours of work. Fleet anomaly is a loss for the product: slower to the cause, alarm volume up rather than down, and the right build in 19 of 20 runs against 20 of 20 for the solver, with an automated stand-in operator. On the gateway outage the product detects fastest (event-driven; the classical solver checks every 5.0 min), but only on the primary outage: it did not model the nuisance alarms (16 of 16 runs), a second outage was present in 3 of 16 runs, and capacity and subscriber-minutes are not comparable across systems. The classical solver restores service faster. Rollout and storm-surge outcomes have no matched comparison.

The classical solver and the scripted runbook were run with the same scenario generators and seeds as the product's recorded runs; how much of each world the product actually used differs by workflow. Fleet anomaly is matched; the gateway outage is a partial match (the primary outage only, on the 16 runs the product ran); rollout and storm surge are not matched, so no comparison is shown for them. Approval records are a policy count: under its A1 policy the product asks for an approval on every command, the classical solver is automated, and the runbook asks for one sign-off per proposal. They are not a measure of effort.

What these numbers do not show.

Eight limits, each published in full.

Read every limit
  • The trade-off. More unneeded manoeuvres than both baselines; more fuel and less warning time than the tuned classical solver (table above).
  • Lead time against the screening cycle. The agent's median decision lead time before closest approach (in the conjunction table) is shorter than one civil screening cycle [88]. In operations that would leave too little time to screen the post-manoeuvre orbit before uplink. As measured, this setting is not ready for operations; shadow mode on your own data exists to tune it.
  • A tuned rule matches zero missed. See the trade-off stated with the conjunction table above.
  • Losses in the other workflows. With an automated stand-in operator, fleet-anomaly diagnosis took about 46 hours and alarm volume showed a small net increase; 1 of 20 fleet-anomaly runs did not replay identically. The incident workflow refused 4 of 20 runs at start, because the seeded outage had no backup gateway in its region or no spacecraft in view at onset; it stopped at open with an input error on those scenarios rather than act on them. No false-halt rate is published for rollouts, because only the bad-build case was run. Fleet anomaly is matched against classical methods and is a loss for the product; the gateway outage is a partial match; rollout and storm surge have no matched comparison.
  • Small storms. The benchmark storm is 10 conjunction events over 24 hours per seed. It says nothing yet about fleet-scale event volumes. Scenarios are synthetic.
  • The product's own storm model, no baselines. On a larger storm generated by the product itself, the agent missed high-risk events: 52 of 265, with a false-manoeuvre rate of 0.146 and a median lead time of 5.45 h. The baselines do not run on these inputs, so there is no comparison.
  • Replay, not live. Conjunction figures on seeds 0–59 and the safety figures replay a recorded live campaign on the release under test; 14 of 25,734 had no recording and stay in n. The held-out conjunction runs (seeds 60–119) were recorded live on v1.0.0 and replayed, 60 of 60.
  • CDM-only feeds, ESA Kelvins collision-avoidance challenge (L score, lower is better). These CDMs are anonymised features with no absolute states and no hard-body radius, so ConstellOps passes through the reported Pc rather than recompute it with an assumed radius, and escalates. On the full set it scores 0.694, the same as the latest-risk baseline by design; the challenge winner scored 0.556, which is better. It has no model of how risk evolves between CDMs, which is what that challenge rewards. No triage gain is claimed.

How to read these numbers

Baselines, replay and labels, and the superseded default-threshold comparison
  • Replay, in one line. We re-ran the released product on the model answers recorded during a live run, at no model cost. The details are below.
  • Best published result: the top published score on a public dataset, where one exists.
  • Solver-only policy: the verified tools with fixed rules, no agent.
  • Threshold runbook: a scripted runbook, the way a team runs it today.
  • Live means the model was called during the run. Replay means the model responses recorded in an earlier live run were served again to the release under test, at zero model cost. Two replays of the conjunction campaign gave identical results on 158 of 158; a small number of model calls had no recording (14 of 25,734), and those episodes stay in n.
  • Every figure shows its mode, model (Amazon Nova Pro), n or seeds, interval where one exists, and baseline.

Superseded: the baselines at their default thresholds.

Before the threshold sweep, the classical policies were run only at their default threshold (Pc 1e-4). These rows are kept for traceability; the sweep above supersedes them as the comparison.

Default-threshold baselines (superseded)
MetricConstellOpsThreshold runbook, default thresholdSolver-only policy, default threshold
High-risk events missed0 of 337 of 337 of 33
Low-risk events given an unneeded manoeuvre (false-manoeuvre rate)0.326 (higher than both baselines)0.0190.009
Hours of warning before closest approach (median decision lead time, higher is better)3.10 h11.68 h (35 of 60 seeds)8.0 h (fixed by the policy)
Fuel used per 1,000 events (Δv)4.05 m/s0.119 m/s0.139 m/s
Operator review minutes per event (modelled assumption, not a result)3.78 min21.23 min0 min (no human review in this policy)
Default-threshold comparison (superseded by the sweep)Conjunctions · benchmark storm · dot = value, whisker = 95% range

High-risk events missed lower is better

ConstellOpsReplay0 of 33
Threshold runbook (default threshold)classical baseline (simulated)7 of 33
Solver-only policy (default threshold)classical baseline (simulated)7 of 33

Low-risk events given an unneeded manoeuvre lower is better

ConstellOpsReplay0.326
Threshold runbook (default threshold)classical baseline (simulated)0.019
Solver-only policy (default threshold)classical baseline (simulated)0.009

Hours of warning before closest approach (median) higher is better

ConstellOpsReplay3.10 h
Threshold runbook (default threshold)classical baseline (simulated)11.68 h
Solver-only policy (default threshold)classical baseline (simulated)8.0 h

Fuel used per 1,000 events (Δv) lower is better

ConstellOpsReplay4.05 m/s
Threshold runbook (default threshold)classical baseline (simulated)0.119 m/s
Solver-only policy (default threshold)classical baseline (simulated)0.139 m/s
Missed high-risk share against false-manoeuvre rateConjunctions · 95% ranges on both axes
0.000.100.200.310.410.000.110.210.320.42missed high-risk share → (lower is better)false-manoeuvre rate → (lower is better)ConstellOpsThreshold runbook (default threshold)Solver-only policy (default threshold)

On a narrow screen the plot is replaced by its values:

  • ConstellOps Replaymissed 0 of 33 · false-manoeuvre rate 0.326
  • Threshold runbook (default threshold) classical baseline (simulated)missed 7 of 33 · false-manoeuvre rate 0.019
  • Solver-only policy (default threshold) classical baseline (simulated)missed 7 of 33 · false-manoeuvre rate 0.009
Each policy sits somewhere on the trade-off between missing a dangerous conjunction and manoeuvring when it was not needed. In this storm ConstellOps sits at zero missed; classical rules sit at fewer manoeuvres. The operator sets where to be.

Lead time and fuel, the other two sides of the trade-off, are in the conjunction table above.

Method, in brief.

Seeds, baselines, intervals and how failed runs are counted

Seeded synthetic scenarios, with public datasets where noted. Baselines run on the same seeds and inputs. 95% ranges are computed over seeds; zero counts carry an exact 95% upper bound where one is published. 33 actionable events is a small sample: 0 of 33 missed has an exact 95% upper bound of 0.106, so it cannot support claims about rare-event behaviour at fleet scale. Failed or incomplete episodes stay in n and are scored as the product behaved. Numbers are never rounded in a way that changes their meaning.

Ask us to walk through any number on this page.

§ · NEXT PASS

Request a briefing.

A 45-minute session with the evidence behind any figure. Operators can send a week of their queue.

This address is used only to reply to you. The site sets no cookies and loads no trackers.

Briefing requests

Sources and labels for every figure on this page (103)

Why two release numbers: the conjunction and safety figures were measured on release v1.0.0rc3 by replaying the model calls recorded during the live campaign on v1.0.0rc2. The demonstration clips were captured on v1.0.0rc4; v1.0.0 ships identical code.

  1. 0 of 1,952 Unsafe commands in the scored runs of the released code, with an operator that approved everything: 0 of 1,952

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3 (conjunction replay) and v1.0.0rc4 (other workflows); v1.0.0 ships the same code as rc4 · n: 1,952 commands sent to spacecraft: 738 conjunction (v1.0.0rc3 replay of recorded model responses) and 1,214 on v1.0.0rc4 recorded live (fleet anomaly 777, rollout 306, space weather 131); the incident workflow sends no command to spacecraft · 95% range: 0–0.002 · baseline: target 0 (safety floor) · scenario: scored runs of the released code; in every run the operator approved every proposal the gates left pending (an automated stand-in for conjunction and fleet anomaly, the product's scripted operator for the others), so the result was held by the automated gate chain and the agents' proposals, not by human review

  2. 0 of 11 High-risk events missed during the surge: 0 of 11

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 19 runs recorded live, 1 replayed · n: 20 seeds (11 actionable events pooled) · scenario: space-weather surge (recorded geomagnetic storm replayed)

  3. 20 of 20 Bad-build rollouts halted before the full fleet: 20 of 20

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 5 runs recorded live, 15 replayed · n: 20 seeds (20 runs, 5 recorded live and 15 replayed) · 95% range: 0.832–1 · scenario: staged rollout with a canary regression

  4. 1.0 min About 1.0 min to detect (mean 1.008 min, 16 of 20 runs; event-driven, while the matched classical solver checks on a 5-minute polling interval and restores service faster)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 6 runs recorded live, 10 replayed (4 refused at start) · n: 16 16 of 20 runs (4 refused by the product at start) · scenario: gateway outage across space, ground and network

  5. 0 of 33 Missed high-risk conjunctions: 0 of 33

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds (33 actionable events pooled) · exact 95% upper bound: 0.106 (about 10.6%) · baseline: scripted threshold runbook 7/33; classical solver-only policy 7/33 · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  6. 0.326 False-manoeuvre rate: 0.326 (higher than both baselines)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds (567 below-threshold events pooled) · 95% range: 0.288–0.368 · baseline: scripted threshold runbook 0.019; classical solver-only policy 0.009 · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  7. 0 of 33 Classical solver tuned to a lower threshold: 0 of 33 missed

    mode: Simulated · n: 60 seeds (33 actionable events pooled) · 95% range: 0–0.106 · row: classical solver-only policy at a lower Pc threshold, 2e-5 · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  8. 3 of 33 Scripted runbook, best swept threshold (Pc 5e-5): 3 of 33 missed

    mode: Simulated · n: 60 seeds (33 actionable events pooled) · 95% range: 0–0.2 · row: scripted threshold runbook at its best swept threshold, Pc 5e-5; analyst capacity binds · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  9. 0.106 Classical solver tuned to a lower threshold: false-manoeuvre rate 0.106

    mode: Simulated · n: 60 seeds (567 below-threshold events pooled) · 95% range: 0.08–0.133 · row: classical solver-only policy at a lower Pc threshold, 2e-5 · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  10. 0.051 Scripted runbook, best swept threshold (Pc 5e-5): false-manoeuvre rate 0.051

    mode: Simulated · n: 60 seeds (567 below-threshold events pooled) · 95% range: 0.032–0.071 · row: scripted threshold runbook at its best swept threshold, Pc 5e-5 · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  11. 4.05 m/s Delta-v spent: 4.05 m/s per 1000 events (higher than both baselines)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds · 95% range: 3.558–4.55 · baseline: scripted threshold runbook 0.119; classical solver-only policy 0.139 · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  12. 0.273 m/s Classical solver tuned to a lower threshold: 0.273 m/s per 1000 events

    mode: Simulated · n: 60 seeds · 95% range: 0.216–0.329 · row: classical solver-only policy at a lower Pc threshold, 2e-5 · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  13. 3.10 h Median decision lead time: 3.10 h before TCA (shorter than both baselines)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds · 95% range: 2.558–3.694 · baseline: scripted threshold runbook 11.68 h; classical solver-only policy 8.0 h · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  14. 8.0 h Classical solver tuned to a lower threshold: 8.0 h before closest approach (fixed by the policy)

    mode: Simulated · n: 60 seeds · 95% range: no spread (the same value on every seed) · row: classical solver-only policy at Pc 2e-5; lead time fixed by the policy · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  15. 0 of 40 ConstellOps on held-out runs: 0 of 40 high-risk missed

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 (40 high-risk events pooled) · 95% range: 0–0.088 · baseline: classical solver-only policy at Pc 2e-5 (threshold chosen on seeds 0-59) and scripted analyst runbook at its default threshold (Pc 1e-4), same held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  16. 0 of 40 Tuned classical solver on held-out runs: 0 of 40 missed

    mode: Simulated · n: 60 held-out seeds 60-119 (40 actionable events pooled) · 95% range: 0–0.088 · row: classical solver-only policy at Pc 2e-5, threshold chosen on seeds 0-59, checked on held-out seeds · scenario: held-out confirmation, same storm generator, seeds 60-119

  17. 6 of 40 Scripted analyst runbook at its default threshold, held-out runs: 6 of 40 missed

    mode: Simulated · n: 60 held-out seeds 60-119 (40 high-risk events pooled) · 95% range: 0.05–0.27 · row: scripted analyst runbook at its default threshold, Pc 1e-4, held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  18. 0.295 ConstellOps on held-out runs: false-manoeuvre rate 0.295 (165/560)

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 (560 below-threshold events pooled) · 95% range: 0.262–0.329 · baseline: classical solver-only policy at Pc 2e-5 (threshold chosen on seeds 0-59) and scripted analyst runbook at its default threshold (Pc 1e-4), same held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  19. 0.089 Tuned classical solver on held-out runs: false-manoeuvre rate 0.089

    mode: Simulated · n: 60 held-out seeds 60-119 (560 below-threshold events pooled) · 95% range: 0.067–0.113 · row: classical solver-only policy at Pc 2e-5 on held-out seeds · scenario: held-out confirmation, same storm generator, seeds 60-119

  20. 3.93 m/s ConstellOps on held-out runs: 3.93 m/s per 1000 events

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 · 95% range: 3.458–4.4 · baseline: classical solver-only policy at Pc 2e-5 (threshold chosen on seeds 0-59) and scripted analyst runbook at its default threshold (Pc 1e-4), same held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  21. 0.30 m/s Tuned classical solver on held-out runs: 0.30 m/s per 1000 events

    mode: Simulated · n: 60 held-out seeds 60-119 · 95% range: 0.242–0.345 · row: classical solver-only policy at Pc 2e-5, threshold chosen on seeds 0-59, held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  22. 2.726 h ConstellOps on held-out runs: 2.7 h before closest approach (median)

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 · 95% range: 2.401–3.09 · baseline: classical solver-only policy at Pc 2e-5 (threshold chosen on seeds 0-59) and scripted analyst runbook at its default threshold (Pc 1e-4), same held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  23. 8.0 h Tuned classical solver on held-out runs: 8.0 h before closest approach (fixed by the policy)

    mode: Simulated · n: 60 held-out seeds 60-119 · 95% range: no spread (the same value on every seed) · row: classical solver-only policy at Pc 2e-5; lead time fixed by the policy · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  24. 0 of 205 ConstellOps on held-out runs: 0 of 205 commands unsafe, with an operator that approved everything the gates passed

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 (205 uplinked commands) · 95% range: 0–0.018 · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  25. 0.106 Missed high-risk conjunctions, exact 95% upper bound: 0.106 (about 10.6%)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds (33 actionable events pooled) · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  26. 5 of 33 Scripted runbook below Pc 5e-5 (2e-5, 1e-5, 5e-6): 5 of 33 missed

    mode: Simulated · n: 60 seeds (33 actionable events pooled) · 95% range: 0.04–0.276 · row: scripted threshold runbook at every swept threshold below Pc 5e-5 · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  27. 52 of 265 Product-native storm: 52 of 265 high-risk events without a timely decision (no baseline)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 38 seeds (265 actionable events pooled) · 95% range: 0.171–0.222 · baseline: none (the baselines run only on the benchmark-generated storm) · scenario: product-native conjunction storm, seeds 0-37

  28. 0 of 33 Missed high-risk conjunctions under CDM prompt injection: 0 of 33

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds (33 actionable events pooled) · exact 95% upper bound: 0.106 (about 10.6%) · baseline: scripted threshold runbook 7/33; classical solver-only policy 7/33 · scenario: same storm with prompt-injection text in CDM remarks, seeds 0-59

  29. 0.333 False-manoeuvre rate under CDM prompt injection: 0.333

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds (567 below-threshold events pooled) · 95% range: 0.292–0.375 · baseline: scripted threshold runbook 0.019; classical solver-only policy 0.009 · scenario: same storm with prompt-injection text in CDM remarks, seeds 0-59

  30. 1,281 of 2,434 Earlier build's two-person defect: 1,281 of 2,434 commands unsafe under the stricter rules (found by the separate benchmark checks, fixed in rc4)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc2 (pre-fix); fixed in v1.0.0rc4 · n: 2,434 commands in the rc2 staged-rollout recording · 95% range: 0.506–0.546 · baseline: target 0 (safety floor) · scenario: an earlier build had a defect: two software commands needed only one approver; the separate benchmark checks found it and rc4 fixed it

  31. 0 of 306 Unsafe uplinks under the two-person rule: 0 of 306

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 5 runs recorded live, 15 replayed · n: 306 uplinked commands over 20 rollout runs, re-verified against the pinned rulebook (two distinct approvers for software activate and rollback) · 95% range: 0–0.012 · scenario: staged rollout with a canary regression

  32. 0 of 3,900 Gate escapes: 0 of 3,900

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 3,900 gate evaluations over 158 episodes · baseline: target 0 (safety floor) · scenario: conjunction storm, CDM prompt injection, product-native storm (158 episodes)

  33. 0 of 738 Commands executed after their deadline: 0 of 738

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 738 uplinked commands over 158 episodes · baseline: target 0 (safety floor) · scenario: conjunction storm, CDM prompt injection, product-native storm (158 episodes)

  34. 0 of 441 Prompt-injection attempts that caused an action: 0 of 441

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 441 injection attempts in CDM remarks over 60 episodes · 95% range: 0–0.008 · baseline: target 0 (safety floor) · scenario: CDM prompt injection, seeds 0-59

  35. 738 of 738 Executed commands with a complete signed decision record: 738 of 738

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 738 uplinked commands over 158 episodes · baseline: target 100% (safety floor) · scenario: conjunction storm, CDM prompt injection, product-native storm (158 episodes)

  36. 158 of 158 Two replays of the recorded campaign gave identical results: 158 of 158 episodes

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 158 seeds · baseline: target 100% · scenario: conjunction storm, CDM prompt injection, product-native storm (158 episodes)

  37. 14 of 25,734 Model calls with no recorded answer: 14 of 25,734 (3 seeds, kept in n)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 25,734 model-call lookups over 158 seeds (3 seeds affected, kept in n) · scenario: conjunction storm, CDM prompt injection, product-native storm (158 episodes)

  38. 158 Episodes checked by deterministic verifiers

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 158 episodes (60 storm + 60 injection + 38 product-native) · scenario: conjunction storm, CDM prompt injection, product-native storm (158 episodes)

  39. $0.468 Model cost per 10-event storm episode: $0.468 at list price

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds · 95% range: 0.451–0.486 · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  40. $0 Re-running the conjunction benchmark from recorded model calls: $0 model cost

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 158 seeds · scenario: conjunction storm, CDM prompt injection, product-native storm (158 episodes)

  41. 19 of 20 Correct bad build identified: 19 of 20 runs (automated stand-in operator)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 8 runs recorded live, 12 replayed · n: 20 seeds (20 runs, 8 recorded live and 12 replayed; automated stand-in operator that approves every proposal) · 95% range: 0.751–0.999 · scenario: common-mode anomaly from a bad software build

  42. 0.95 Related alarms grouped with the right cause: 0.95 (automated stand-in operator)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 8 runs recorded live, 12 replayed · n: 20 seeds (20 runs, 8 recorded live and 12 replayed; automated stand-in operator that approves every proposal) · 95% range: 0.85–1 · scenario: common-mode anomaly from a bad software build

  43. 46.25 h Time to identify the common cause: 46.25 h (about 46 h, automated stand-in operator)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 8 runs recorded live, 12 replayed · n: 19 seeds with a value (1 of 20 without); automated stand-in operator · 95% range: 45.329–47.108 · scenario: common-mode anomaly from a bad software build

  44. +0.06% ConstellOps alarm volume: +0.06%, a slight increase; reduction -0.000583

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 8 runs recorded live, 12 replayed · n: 20 seeds (automated stand-in operator) · scenario: common-mode anomaly from a bad software build

  45. 19 of 20 Runs that replay identically: 19 of 20 (misses the 100% target)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 8 runs recorded live, 12 replayed · n: 20 seeds · 95% range: 0.751–0.999 · scenario: common-mode anomaly from a bad software build

  46. 43.55 ConstellOps approval records per episode under its A1 policy: 43.55

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 8 runs recorded live, 12 replayed · n: 20 seeds (automated stand-in operator that approves every proposal) · 95% range: 38.6–47.651 · baseline: policy count; classical solver automated (0), runbook one sign-off per proposal · scenario: common-mode anomaly from a bad software build (matched worlds)

  47. 16 of 20 Runs completed: 16 of 20 (4 refused at start)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 6 runs recorded live, 10 replayed (4 refused at start) · n: 20 seeds (4 refused by the product at start) · scenario: gateway outage across space, ground and network

  48. 15 min About 15 minutes to restore (mean 14.57 min, 16 of 20 runs)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 6 runs recorded live, 10 replayed (4 refused at start) · n: 16 16 of 20 runs (4 refused by the product at start) · scenario: gateway outage across space, ground and network

  49. 0.824 Capacity kept during the incident: 0.824 (about 82%, synthetic network model)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 6 runs recorded live, 10 replayed (4 refused at start) · n: 16 16 of 20 runs (4 refused by the product at start); 6 recorded live, 10 replayed · 95% range: 0.734–0.873 · scenario: gateway outage across space, ground and network

  50. 270,000 Subscriber-minutes lost per incident: about 270,000; synthetic network model

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 6 runs recorded live, 10 replayed (4 refused at start) · n: 16 16 of 20 runs (4 refused by the product at start); 6 recorded live, 10 replayed · 95% range: 226,000–314,000 · scenario: gateway outage across space, ground and network

  51. 0 Commands sent to spacecraft: 0 (ground-only remediation)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 6 runs recorded live, 10 replayed (4 refused at start) · n: 16 16 of 20 runs (4 refused by the product at start); 6 recorded live, 10 replayed · scenario: gateway outage across space, ground and network

  52. 2.38 ConstellOps approval records per episode under its A1 policy: 2.38

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 6 runs recorded live, 10 replayed (4 refused at start) · n: 16 the 16 runs the product ran · 95% range: 2.125–2.688 · baseline: policy count · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  53. 6 Canary wave: 6 spacecraft of a 300-spacecraft fleet

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 5 runs recorded live, 15 replayed · n: 20 seeds (the same in every run) · scenario: staged rollout with a canary regression

  54. 5 About 5 satellites on the bad build before the halt (mean 4.95 over 20 runs)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 5 runs recorded live, 15 replayed · n: 20 seeds (5 recorded live, 15 replayed) · scenario: staged rollout with a canary regression

  55. 0.45 h Time from first install to halt: 0.45 h (about 27 min)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 5 runs recorded live, 15 replayed · n: 20 seeds (20 runs, 5 recorded live and 15 replayed) · 95% range: 0.392–0.517 · scenario: staged rollout with a canary regression

  56. 0 of 20 Bad build beyond the canary: 0 of 20 rollouts

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 5 runs recorded live, 15 replayed · n: 20 seeds (20 runs, 5 recorded live and 15 replayed) · 95% range: 0–0.168 · scenario: staged rollout with a canary regression

  57. 20 of 20 Runs that replay identically: 20 of 20

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 5 runs recorded live, 15 replayed · n: 20 seeds · scenario: staged rollout with a canary regression

  58. 27.4 ConstellOps approval records per episode under its A1 policy (two-person software actions counted twice): 27.4

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 5 runs recorded live, 15 replayed · n: 20 seeds (20 rollout runs) · 95% range: 25.7–28.9 · baseline: policy count · scenario: staged rollout with a canary regression (not matched; approval policy count only)

  59. 0 of 103 Unneeded manoeuvres: 0 of 103 low-risk events

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 19 runs recorded live, 1 replayed · n: 20 seeds (103 below-threshold events pooled) · scenario: space-weather surge (recorded geomagnetic storm replayed)

  60. 0 of 131 Commands executed after their deadline: 0 of 131

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 19 runs recorded live, 1 replayed · n: 131 uplinked commands over 20 surge runs · 95% range: 0–0.028 · scenario: space-weather surge (recorded geomagnetic storm replayed)

  61. 0 Commands that missed their cutoff: 0

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 19 runs recorded live, 1 replayed · n: 20 seeds (20 runs, 19 recorded live and 1 replayed) · scenario: space-weather surge (recorded geomagnetic storm replayed)

  62. 20 of 20 Runs that replay identically: 20 of 20

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 19 runs recorded live, 1 replayed · n: 20 seeds · scenario: space-weather surge (recorded geomagnetic storm replayed)

  63. 4.32 h Tuned classical solver (cohort correlation, threshold chosen on these runs): 4.32 h to the cause

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs; threshold chosen on these runs · 95% range: 3.225–5.549 · row: classical cohort-correlation solver, alpha 1e-2, chosen in-sample · scenario: common-mode anomaly from a bad software build (matched worlds)

  64. 25.55 h Scripted per-satellite triage runbook: 25.55 h to the cause

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · 95% range: 24.759–26.158 · row: scripted per-satellite triage runbook · scenario: common-mode anomaly from a bad software build (matched worlds)

  65. 5.0 min Classical capacity solver: 5.0 min to detect (its polling interval)

    mode: Simulated · n: 16 the 16 runs the product ran (partial match) · 95% range: no spread (the same value on every seed) · row: classical capacity solver · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  66. 5.0 min Classical capacity solver: 5.0 min to restore

    mode: Simulated · n: 16 the 16 runs the product ran (partial match) · 95% range: no spread (the same value on every seed) · row: classical capacity solver · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  67. 5.40 h Tuned classical solver on held-out runs: 5.40 h to the cause

    mode: Simulated · n: 20 held-out seeds 20-39 · 95% range: 3.868–7.156 · row: same solver and threshold, held-out seeds · scenario: common-mode anomaly from a bad software build (matched worlds)

  68. −16% Tuned classical solver alarm volume: −16%, reduction 0.163

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · row: classical cohort-correlation solver, alpha 1e-2 · scenario: common-mode anomaly from a bad software build (matched worlds)

  69. −8% Scripted runbook alarm volume: −8%, reduction 0.083

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · row: scripted per-satellite triage runbook · scenario: common-mode anomaly from a bad software build (matched worlds)

  70. 20 of 20 Tuned classical solver, correct build identified: 20 of 20 runs

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · 95% range: 0.832–1 · row: classical cohort-correlation solver · scenario: common-mode anomaly from a bad software build (matched worlds)

  71. 19 of 20 Scripted runbook, correct build identified: 19 of 20 runs

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · 95% range: 0.751–0.999 · row: scripted per-satellite triage runbook · scenario: common-mode anomaly from a bad software build (matched worlds)

  72. 0 Classical solver approval records per episode: 0 (automated)

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · row: policy count · scenario: common-mode anomaly from a bad software build (matched worlds)

  73. 1 Scripted runbook approval records per episode: 1 (one sign-off)

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · row: policy count · scenario: common-mode anomaly from a bad software build (matched worlds)

  74. 24.63 min Scripted operations-centre runbook: 24.63 min to detect

    mode: Simulated · n: 16 the 16 runs the product ran (partial match) · 95% range: 23.432–26.173 · row: scripted operations-centre runbook · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  75. 117.66 min Scripted operations-centre runbook: 117.66 min to restore

    mode: Simulated · n: 16 the 16 runs the product ran (partial match) · 95% range: 94.882–141.495 · row: scripted operations-centre runbook · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  76. 0 Classical solver approval records per episode: 0 (automated)

    mode: Simulated · n: 16 the 16 runs the product ran · row: policy count · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  77. 2.56 Scripted runbook approval records per episode: 2.56 (one sign-off per proposal)

    mode: Simulated · n: 16 the 16 runs the product ran · 95% range: 2.188–3.125 · row: policy count · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  78. 0 Classical canary solver approval records per episode: 0 (automated)

    mode: Simulated · n: 20 seeds 0-19 (rollout not matched; policy count only) · row: policy count only; rollout outcomes not matched · scenario: staged rollout with a canary regression (not matched; approval policy count only)

  79. 1 Bake-and-review runbook approval records per episode: 1 (one sign-off)

    mode: Simulated · n: 20 seeds 0-19 (rollout not matched; policy count only) · row: policy count only; rollout outcomes not matched · scenario: staged rollout with a canary regression (not matched; approval policy count only)

  80. 16 of 16 Gateway-outage runs in which the product did not model the nuisance alarms: 16 of 16

    mode: Simulated · n: 16 the 16 runs the product ran · scenario: gateway outage, partial match

  81. 3 of 16 Gateway-outage runs with a second outage the product did not model: 3 of 16

    mode: Simulated · n: 16 the 16 runs the product ran · scenario: gateway outage, partial match

  82. 4 h Conjunction screening every four hours; ephemerides exchanged in CCSDS OCM

    Industry fact · US Office of Space Commerce, TraCSS overview, May 2026 · source

  83. 0.146 Product-native storm: false-manoeuvre rate 0.146 (no baseline)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 38 seeds · 95% range: 0.123–0.174 · baseline: none (the baselines run only on the benchmark-generated storm) · scenario: product-native conjunction storm, seeds 0-37

  84. 5.45 h Product-native storm: median decision lead time 5.45 h (no baseline)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 38 seeds · 95% range: 5.163–5.734 · baseline: none (the baselines run only on the benchmark-generated storm) · scenario: product-native conjunction storm, seeds 0-37

  85. 0.694 Kelvins L score 0.694: parity with the latest-risk baseline, behind the challenge winner

    mode: Public dataset · release: ConstellOps (reported-Pc passthrough on CDM-only feeds) · n: 2,167 test events (150 high-risk), full official test set · 95% range: 0.46–0.965 · baseline: latest-risk-prediction baseline 0.694 (identical by construction); challenge winner 0.556 · scenario: ESA Kelvins Collision Avoidance Challenge, full test set

  86. 0.556 Challenge winner (published): 0.556

    Published result · ESA Kelvins Collision Avoidance Challenge, full test set · source

  87. 7 of 33 Scripted threshold runbook: 7 of 33 missed

    mode: Simulated · n: 60 seeds (33 actionable events pooled) · 95% range: 0.088–0.351 · row: this row is the scripted threshold-runbook baseline · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  88. 7 of 33 Classical solver-only policy: 7 of 33 missed

    mode: Simulated · n: 60 seeds (33 actionable events pooled) · 95% range: 0.088–0.355 · row: this row is the classical solver-only baseline · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  89. 0.019 Scripted threshold runbook: 0.019

    mode: Simulated · n: 60 seeds · 95% range: 0.009–0.032 · row: this row is the scripted threshold-runbook baseline · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  90. 0.009 Classical solver-only policy: 0.009

    mode: Simulated · n: 60 seeds · 95% range: 0.002–0.017 · row: this row is the classical solver-only baseline · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  91. 11.68 h Scripted threshold runbook: 11.68 h (35 of 60 seeds)

    mode: Simulated · n: 35 seeds with a decision (25 of 60 seeds had none) · 95% range: 10.237–13.134 · row: this row is the scripted threshold-runbook baseline · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  92. 8.0 h Classical solver-only policy: 8.0 h (fixed by the policy)

    mode: Simulated · n: 60 seeds · 95% range: no spread (the same value on every seed) · row: this row is the classical solver-only baseline · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  93. 0.119 m/s Scripted threshold runbook: 0.119 m/s per 1000 events

    mode: Simulated · n: 60 seeds · 95% range: 0.084–0.155 · row: this row is the scripted threshold-runbook baseline · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  94. 0.139 m/s Classical solver-only policy: 0.139 m/s per 1000 events

    mode: Simulated · n: 60 seeds · 95% range: 0.094–0.187 · row: this row is the classical solver-only baseline · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  95. 3.78 min Operator review time: 3.78 min per event (modelled)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds · 95% range: 3.417–4.183 · baseline: scripted threshold runbook 21.23 min; classical solver-only policy 0 (no human in the loop) · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  96. 21.23 min Scripted threshold runbook: 21.23 min per event (modelled)

    mode: Simulated · n: 60 seeds · 95% range: 18.277–24.51 · row: this row is the scripted threshold-runbook baseline · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  97. 0 min Classical solver-only policy: 0 min (no human review in this policy)

    mode: Simulated · n: 60 seeds · 95% range: no spread (the same value on every seed) · row: this row is the classical solver-only baseline · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59