← All briefings

Briefing · Technical due diligence

Check the gates, the log and the numbers yourself.

The gates, the signed record and replay, and how every figure is produced.

Request a briefing
  1. Six layers; nothing crosses without a clearance
  2. The gate pipeline and the autonomy ladder
  3. The two-person rule
  4. The signed decision log and replay
  5. How every figure is produced
  6. Limits
  7. Security posture
  8. How it is deployed on AWS
  9. How we deploy
  10. Request a technical briefing
Encounter planecovariance ellipse · miss vector · hard-body radius
B-PLANE (ILLUSTRATIVE)misspost-burn
Illustrative · synthetic fleet · no figures

Six layers; nothing crosses without a clearance.

§01 · ARCHITECTURE
LayerContents
L5 GovernanceAutonomy levels, signed PROV-O decision log, record and replay, billing guard
L4 RuntimeOps-agent construction, gate pipeline and intervention on every action tool, reasoning scaffold, agent teams
L3 ToolsParameter, action and alert tools; verified solvers (Pc, manoeuvres, scheduling); read-only knowledge queries
L2 KnowledgeOperations ontology (SOSA/SSN, QUDT, PROV-O, OWL-Time), SHACL flight rules, CDM/OMM/XTCE handling
L1 Assets and fleetSpacecraft drivers, fleet registry, contact-window transport, orbital simulation backends
L0 AdaptersMission-control systems, flight-software simulators, orbit data, CCSDS CDM
Read the full section

ConstellOps on top: a fleet supervisor dispatches work to five function agents (conjunction, triage, FDIR, network incident and release). The supervisor never commands directly. The operations console is the human half of the approval gate. The product is built only on the public API of the Agentic Operations Stack, whose current release is v1.0.2.

Licence and IP. Proprietary, all rights reserved. Earlier releases (the SDK up to v1.0.2; the other three up to v1.0.0) were Apache-2.0 and shared only privately; releases from the relicensing onward are proprietary. Source code, evidence bundles and replay recordings are available to evaluation teams under NDA, and a software bill of materials is available on request. Pricing is not published; it is discussed in a briefing.

Versions. v1.0.0 is the current release. Published figures come from rc2 recordings replayed on rc3 (conjunctions) and from rc4 (the other four workflows). Conjunction figures: ConstellOps v1.0.0rc3, replaying the model calls recorded during the live campaign on v1.0.0rc2. Demonstration clips: captured on v1.0.0rc4; v1.0.0 ships identical code. SDK: Agentic Operations Stack v1.0.2.

Model use. One configured model serves every role: Amazon Nova Pro. The model is configuration. Agents run with shell, file-write and edit tools switched off. Reasoning text is split from the answer and stored only in the signed record; a malformed split rejects the output, and nothing is proposed.

The gate pipeline and the autonomy ladder.

Order (ConstellOps, 7 gates): autonomy level → scope cap → two-person (hazardous) → SHACL flight rules → Cedar authority → digital-twin check → human approval → signed clearance → contact-window transport (which verifies the clearance). The SDK's canonical chain has five gates; ConstellOps adds the scope cap and the two-person gate.

§02 · GATE PIPELINE AND AUTONOMY LEVELS
PropertyBehaviour
Fail closedAn error, a timeout or an unknown in any gate is a deny
DeterminismGates are code, not prompts. A model, judge or steering handler can never turn a deny into an allow
ClearanceIssued only by the full pipeline. The transport refuses a command with no clearance or an invalid one
ScopeAutonomy is set per agent function and per scope (fleet, class, asset); the most specific wins; scopes are canonicalised
AuditEvery autonomy change is a signed decision record
Read the full section

Ladder:

§02 · GATE PIPELINE AND AUTONOMY LEVELS
LevelRoutineCautionHazardous
A0 observe and explaindenydenydeny
A1 proposeconfirmconfirmconfirm
A2 pre-approved proceduresallow if pre-approved, else confirmconfirmconfirm
A3 adapt within envelopeallowallowconfirm (two approvers)
A4 pursue goalsno commanding in v1no commanding in v1no commanding in v1

This is the SDK's ladder (Agentic Operations Stack v1.0.2). ConstellOps v1 caps every agent at A2, so A3 is not offered in the product.

The two-person rule.

Hazardous commands (burns, flight-software activation and rollback, and others marked hazardous in the dictionary) require two distinct approvers. Approver identities are compared in canonical form, so one person approving twice under different spellings counts once. Each approval is its own signed record.

Approver identities are recorded as the console reports them, not authenticated against an identity provider, so the rule today checks reported identities.

Extending the rule to software activation and rollback is the latest release candidate's change; its benchmark result is 0 of 306.

The signed decision log and replay.

  • Record. Every step is a PROV-O record: inputs (with hashes), tool calls and versions, gate verdicts, approvers, the clearance and the transport outcome. Records are hash-chained and signed with Ed25519. Verification checks the chain and each signature.
  • Traceability. 738 of 738.
  • Replay. Every model call can be recorded to a cassette keyed by a hash of the request. Replay serves the recorded answer, so an episode can be re-run offline. What the published figures show:
  • Two separate claims. Determinism means two independent replays give identical episode digests. Equivalence to the live run means record-by-record equality, plus a metadata diff limited to a named allowlist. The figures on this site claim determinism only; they are replays of live recordings; the held-out conjunction runs were recorded live on v1.0.0.

How every figure is produced.

Arms. Each benchmark compares:

  • ConstellOps in a stated mode: live, or replay of recorded live calls;
  • the published best result on a public dataset, where one exists;
  • a solver-only policy: the verified tools with fixed rules and no agent;
  • a scripted runbook: the written procedure a team follows today.

All arms run on the same seeds and inputs.

Data. Seeded synthetic scenarios over a synthetic constellation, plus the public ESA Kelvins collision-avoidance dataset (real CDMs).

Statistics. Benchmark rates and medians use 95% ranges over seeds. Safety zeros are reported with their exact 95% upper bound, for example 0.005. Failed or incomplete episodes stay in n.

Read the full section

The safety headline, and what held it. 0 of 1,952 unsafe commands sent to spacecraft in the scored runs of the released code: conjunctions on ConstellOps v1.0.0rc3 (replay of recorded model responses) and fleet anomalies, staged rollouts and space weather on v1.0.0rc4, recorded live. v1.0.0 has the same code as rc4; cross-domain incidents send no commands to spacecraft. Each command was checked by the evaluation harness against its own pinned flight rules. In every one of these runs the operator approved every proposal the gates left pending (a harness stand-in for conjunctions and fleet anomalies, the product's scripted operator for the other three workflows). So this result was held by the automated gate chain and the agents' proposals, not by human review. A further 37 commands recorded live in one fleet-anomaly episode were also judged safe (0 of 37 unsafe), but that episode does not reproduce on replay, so they are reported separately. An earlier build (rc2) had a defect: two software commands needed only one approver. Under the stricter rules, 1,281 of 2,434 commands in its staged-rollout recording were unsafe. The harness found the defect and rc4 fixed it.

Separation. Scoring runs in a separate conformance and benchmark harness: a different codebase from the product, which installs the released package and scores it from outside. It is built by the same author, so it is separate, not independent. For true independence, we will run it against your own scenarios or hand it to your evaluation team.

Labelling. Every figure carries its mode, model, n, interval and baseline. Figures that are still pending are shown as pending, never as placeholders.

Conjunction results, in full:

§05 · BENCHMARK METHODOLOGY
MeasureConstellOpsClassical solver (tuned)Scripted analyst runbook (best)
Missed high-risk (exact 95% upper bound for ConstellOps: 0.106)0 of 330 of 333 of 33
False-manoeuvre rate0.326 (higher than both baselines)0.106—
Median lead time3.10 h8.0 h—
Δv4.05 m/s0.273 m/s—
Missed high-risk, 60 held-out runs (runbook at its default threshold)0 of 400 of 406 of 40
False-manoeuvre rate, held-out runs0.295 (165/560)0.089—
Median lead time, held-out runs2.726 h8.0 h—
Δv, held-out runs3.93 m/s0.30 m/s—
Unsafe commands, held-out runs0 of 205——
Under CDM prompt injection: missed / false-manoeuvre0 of 33 / 0.333——
Product-native storm (no baselines): missed / false-manoeuvre / lead time52 of 265 / 0.146 (no baseline) / 5.45 h (no baseline)——
ESA Kelvins L score, full set (lower is better)0.694latest-risk: 0.694winner: 0.556
Model cost per storm episode$0.468——

Held-out runs: On held-out seeds (60–119) the agent misses 0 of 40 high-risk conjunctions (95% upper bound 8.8%), the same as a classical solver acting at Pc ≥ 2e-5 (threshold chosen on seeds 0–59), but manoeuvres for 29.5% of low-risk events (165/560) against the solver's 8.9%, uses about 13× the Δv (3.93 vs 0.30 m/s per 1,000 events), and decides later (2.7 h before closest approach vs 8 h). A scripted analyst runbook misses 6 of 40 at its default threshold. 0 unsafe commands of 205, with an operator that approved everything the gates passed (constellops v1.0.0, live-recorded, replayed 60/60). Model cost: about $0.47 per seed at list price for the model calls (the campaign spent more, including failed retries).

The default-threshold baseline rows (Pc 1e-4), including the modelled review minutes, are superseded by the sweep and kept for traceability on the Benchmarks page.

Other workflows. Fleet anomaly, incident, rollout and space-weather results were recorded on v1.0.0rc4 (the code released as v1.0.0), 20 runs each, and are published per workflow (matched comparisons below). Unsafe uplinks across those runs: 0 of 1,214.

Matched comparisons. The classical solver and the scripted runbook were run with the same scenario generators and seeds as the product's recorded runs; how much of each world the product actually used differs by workflow. Fleet anomaly is matched; the gateway outage is a partial match (primary outage only, the 16 runs the product ran); rollout and storm surge are not matched, so no outcome comparison is shown for them. Approval records are a policy count, never a measure of effort: under its A1 policy the product asks for an approval on every command, the classical solver is automated and the runbook asks for one sign-off per proposal.

§05 · BENCHMARK METHODOLOGY
Fleet anomaly (matched, 20 runs)ConstellOps (automated stand-in operator)Classical solver (threshold tuned on these runs)Scripted runbook
VerdictBehind on time to cause, alarm volume and correct build——
Time to identify the common cause46.25 h (about 46 h, automated stand-in operator)4.32 h25.55 h
Time to identify the common cause, held-out runsnot run5.40 h—
Alarm volume+0.06% (slight increase)−16%−8%
Correct build identified19 of 2020 of 2019 of 20
Human sign-offs required per run (by policy)43.550 (automated)1 (one sign-off)
§05 · BENCHMARK METHODOLOGY
Gateway outage (partial match, 16 runs)ConstellOpsClassical solverScripted runbook
VerdictDetects fastest; restores slower than the classical solver——
Mean time to detectabout 1.0 min (event-driven)5.0 min (checks every 5.0 min)24.63 min
Mean time to restoreabout 15 min5.0 min117.66 min
Human sign-offs required per run (by policy)2.380 (automated)2.56 (one sign-off per proposal)
§05 · BENCHMARK METHODOLOGY
Staged rollout (not matched: policy count only)ConstellOpsClassical solverScripted runbook
Human sign-offs required per run (by policy)27.40 (automated)1 (one sign-off)

Sign-offs reflect the product's per-command approval policy (every command approved individually and signed; tunable by autonomy level), not hours of work. Fleet anomaly is a loss for the product: slower to the cause, alarm volume up rather than down, and the right build in 19 of 20 runs against 20 of 20 for the solver, with an automated stand-in operator. On the gateway outage the product detects fastest (event-driven; the classical solver checks every 5.0 min), but only on the primary outage: it did not model the nuisance alarms (16 of 16 runs), a second outage was present in 3 of 16 runs, and capacity and subscriber-minutes are not comparable across systems. The classical solver restores service faster. Rollout and storm-surge outcomes have no matched comparison.

Limits.

Read the full section
  • The escalation policy trades false manoeuvres, Δv and lead time for zero missed high-risk events. The threshold is tunable.
  • There is no temporal model of how risk evolves between CDMs, so Kelvins performance is parity with the latest-risk baseline at best.
  • Pc is recomputed only from complete CDMs (absolute states for both objects and a hard-body radius). Otherwise the reported Pc is used, flagged, and escalated.
  • The median decision lead time in the benchmark storm is shorter than one civil screening cycle, which in operations would leave too little time to screen the post-manoeuvre orbit. As measured, this setting is not ready for operations.
  • A threshold sweep on the same seeds shows a classical solver at a stricter threshold matching the miss count with about a third of the unneeded manoeuvres and about 15 times less fuel; the threshold was tuned on the same runs and held on 40 held-out high-risk events (0 of 40 missed). On the same held-out runs the agent missed 0 of 40, at a false-manoeuvre rate of 0.295 against the solver's 0.089, about 13 times its Δv and later decisions. The scripted analyst runbook's best is 3 of 33 (at Pc 5e-5); it reaches 0 at no threshold swept, and below that threshold its misses rise to 5 of 33, consistent with analyst capacity binding (not separately measured).
  • The traffic, capacity and telemetry models in the scenarios are synthetic; figures describe the models, not a real network or fleet.
  • There is no onboard autonomy; ConstellOps v1 caps every agent at A2 (the SDK supports A3; A4 cannot command in v1); A2 runs at A1 until the veto window ships.
  • The safety bounds are what the sample sizes can show, from replayed scenario episodes, not flight operations.
  • No customers or field deployments yet.

Security posture.

  • Untrusted inputs. CDM remarks, telemetry strings, event text and tool outputs are data, never instructions. They are rendered inert in the operator view.
  • Reduced tool surface. No shell, file-write or edit tools. Role files accept allowlisted keys only.
  • Least privilege. Each agent function has declared permissions, and Cedar policy enforces them per asset.
  • Integrity. Signed, hash-chained decision records. Clearances are verified at the transport.
  • Supply chain. Dependencies are pinned to tagged releases, and an install check confirms where they came from. Releases refuse packaged local state or account identifiers.
  • Billing guard. Live model calls pass a spending guard; replay costs nothing.
  • Secrets. Code, logs, recordings and evidence are designed and checked to carry no credentials or account identifiers.
  • In scope for security reports: gate bypass, prompt injection, audit-integrity attacks, tool privilege escalation, secrets exposure and billing-guard bypass. Report to the contact below.

How it is deployed on AWS.

  • Tenancy. Designed to be deployed into the operator's own AWS account; no customer deployment exists yet. The operator owns the data, the keys and the decision log.
  • Models. Called through Amazon Bedrock in that account. Today every role uses Amazon Nova Pro, and the model is configuration.
  • Offline capability. Recorded episodes replay with no model or cloud access, for audit, training and regression.
  • Command path. It connects to the operator's existing mission-control system and ground stations; it adds no new route to the spacecraft.
  • Per-engagement design. The network placement, identity integration, key management and storage of the signed log are designed with the operator's security team during discovery and reviewed before the gated pilot.
  • Technologies. Built with Strands Agents. Models: Amazon Nova Pro on Amazon Bedrock. Deployed on AWS. These are named as the technologies used; no partnership is implied.

How we deploy.

We deploy it into your operations, not ours.

  1. STAGE 1

    Discovery

    None
  2. STAGE 2

    Shadow mode

    A0 observe and explain
  3. STAGE 3

    Gated pilot

    A1 propose, human approves
  4. STAGE 4

    Production

    A1 by default
How each stage works, and its exit

ConstellOps is an agentic satellite-operations stack that we deploy for constellation operators: into your own AWS account and your ground segment, integrated with your command path and your flight rules, with benchmarking and assurance that continue after go-live.

  • Your account, your data. The stack runs in your AWS account. Models are called through Amazon Bedrock in that account. CDMs, telemetry, command dictionaries and the decision log stay under your control.
  • Your command path. Approved commands leave through your existing mission-control system and ground stations, queued for uplink in your contact windows. ConstellOps adds no new route to the spacecraft.
  • Your flight rules. Your constraints, command dictionaries and authority policy are encoded as gates and reviewed with your flight-operations team. The gates enforce your rules, not ours.
  • Measured, then trusted. The same separate conformance and benchmark harness that produced the figures on this site is run against your scenarios and your baselines before and after each stage, with 95% intervals. Autonomy is raised only on evidence, one function at a time.
Engagement stages
StageWhat happensAutonomy on your fleetExit criterion
1 · DiscoveryWe map your conjunction, anomaly, incident, release and space-weather workflows, your command path, flight rules and data feeds. We agree which workflow goes first and which baselines define success.None: no connection to the command pathAgreed scope, baselines and success metrics
2 · Shadow modeThe stack runs on your live or recorded feeds next to your team. Agents rank, correlate and propose; nothing is sent. Every proposal is scored against what your operators actually did.A0 observe and explain; commanding disabledBenchmarked against your baselines on your data, with intervals
3 · Gated pilotOne workflow, one scope (a shell, a class or a set of spacecraft), connected to the command path through the full gate chain. Every command needs operator approval; hazardous commands need two approvers.A1 propose, human approvesZero unsafe uplinks, every decision traceable, and agreed outcome targets met
4 · ProductionFurther workflows and scopes are added one at a time. Levels are raised only where evidence supports it and your Flight Director signs off. Benchmarking and assurance continue with every release.A1 by default; A2 pre-approved procedures only where agreed; A3 and A4 not offered in ConstellOps v1Ongoing: regression benchmarks per release, signed decision-log audits

Where we are: pre-deployment. No operator runs ConstellOps in production yet, and there is no flight heritage to show. We are looking for a design-partner operator to take one workflow through shadow mode and a gated pilot, measured against its own baselines.


What we do not publish: pricing. Commercial terms are discussed in a briefing.

Request a technical briefing.

We can walk through the gate code paths, verify a signed decision log in front of you, and replay any benchmark episode.

Briefing requests

Sources and labels for every figure on this page (71)

Why two release numbers: the conjunction and safety figures were measured on release v1.0.0rc3 by replaying the model calls recorded during the live campaign on v1.0.0rc2. The demonstration clips were captured on v1.0.0rc4; v1.0.0 ships identical code.

  1. 0 of 1,952 Unsafe commands in the scored runs of the released code, with an operator that approved everything: 0 of 1,952

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3 (conjunction replay) and v1.0.0rc4 (other workflows); v1.0.0 ships the same code as rc4 · n: 1,952 commands sent to spacecraft: 738 conjunction (v1.0.0rc3 replay of recorded model responses) and 1,214 on v1.0.0rc4 recorded live (fleet anomaly 777, rollout 306, space weather 131); the incident workflow sends no command to spacecraft · 95% range: 0–0.002 · baseline: target 0 (safety floor) · scenario: scored runs of the released code; in every run the operator approved every proposal the gates left pending (an automated stand-in for conjunction and fleet anomaly, the product's scripted operator for the others), so the result was held by the automated gate chain and the agents' proposals, not by human review

  2. 1,281 of 2,434 Earlier build's two-person defect: 1,281 of 2,434 commands unsafe under the stricter rules (found by the separate benchmark checks, fixed in rc4)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc2 (pre-fix); fixed in v1.0.0rc4 · n: 2,434 commands in the rc2 staged-rollout recording · 95% range: 0.506–0.546 · baseline: target 0 (safety floor) · scenario: an earlier build had a defect: two software commands needed only one approver; the separate benchmark checks found it and rc4 fixed it

  3. v1.0.2 Agentic Operations Stack v1.0.2

  4. v1.0.0 ConstellOps v1.0.0 (the code of v1.0.0rc4, on which the results were measured)

  5. Amazon Nova Pro Amazon Nova Pro on Amazon Bedrock, prompted reasoning, one model for every role

    model: Amazon Nova Pro (prompted reasoning)

  6. 0 of 306 Unsafe uplinks under the two-person rule: 0 of 306

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 5 runs recorded live, 15 replayed · n: 306 uplinked commands over 20 rollout runs, re-verified against the pinned rulebook (two distinct approvers for software activate and rollback) · 95% range: 0–0.012 · scenario: staged rollout with a canary regression

  7. 738 of 738 Executed commands with a complete signed decision record: 738 of 738

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 738 uplinked commands over 158 episodes · baseline: target 100% (safety floor) · scenario: conjunction storm, CDM prompt injection, product-native storm (158 episodes)

  8. 158 of 158 Two replays of the recorded campaign gave identical results: 158 of 158 episodes

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 158 seeds · baseline: target 100% · scenario: conjunction storm, CDM prompt injection, product-native storm (158 episodes)

  9. 14 of 25,734 Model calls with no recorded answer: 14 of 25,734 (3 seeds, kept in n)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 25,734 model-call lookups over 158 seeds (3 seeds affected, kept in n) · scenario: conjunction storm, CDM prompt injection, product-native storm (158 episodes)

  10. $0 Re-running the conjunction benchmark from recorded model calls: $0 model cost

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 158 seeds · scenario: conjunction storm, CDM prompt injection, product-native storm (158 episodes)

  11. 0.106 Missed high-risk conjunctions, exact 95% upper bound: 0.106 (about 10.6%)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds (33 actionable events pooled) · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  12. 0 of 33 Missed high-risk conjunctions: 0 of 33

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds (33 actionable events pooled) · exact 95% upper bound: 0.106 (about 10.6%) · baseline: scripted threshold runbook 7/33; classical solver-only policy 7/33 · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  13. 0 of 33 Classical solver tuned to a lower threshold: 0 of 33 missed

    mode: Simulated · n: 60 seeds (33 actionable events pooled) · 95% range: 0–0.106 · row: classical solver-only policy at a lower Pc threshold, 2e-5 · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  14. 3 of 33 Scripted runbook, best swept threshold (Pc 5e-5): 3 of 33 missed

    mode: Simulated · n: 60 seeds (33 actionable events pooled) · 95% range: 0–0.2 · row: scripted threshold runbook at its best swept threshold, Pc 5e-5; analyst capacity binds · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  15. 0.326 False-manoeuvre rate: 0.326 (higher than both baselines)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds (567 below-threshold events pooled) · 95% range: 0.288–0.368 · baseline: scripted threshold runbook 0.019; classical solver-only policy 0.009 · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  16. 0.106 Classical solver tuned to a lower threshold: false-manoeuvre rate 0.106

    mode: Simulated · n: 60 seeds (567 below-threshold events pooled) · 95% range: 0.08–0.133 · row: classical solver-only policy at a lower Pc threshold, 2e-5 · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  17. 3.10 h Median decision lead time: 3.10 h before TCA (shorter than both baselines)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds · 95% range: 2.558–3.694 · baseline: scripted threshold runbook 11.68 h; classical solver-only policy 8.0 h · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  18. 8.0 h Classical solver tuned to a lower threshold: 8.0 h before closest approach (fixed by the policy)

    mode: Simulated · n: 60 seeds · 95% range: no spread (the same value on every seed) · row: classical solver-only policy at Pc 2e-5; lead time fixed by the policy · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  19. 4.05 m/s Delta-v spent: 4.05 m/s per 1000 events (higher than both baselines)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds · 95% range: 3.558–4.55 · baseline: scripted threshold runbook 0.119; classical solver-only policy 0.139 · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  20. 0.273 m/s Classical solver tuned to a lower threshold: 0.273 m/s per 1000 events

    mode: Simulated · n: 60 seeds · 95% range: 0.216–0.329 · row: classical solver-only policy at a lower Pc threshold, 2e-5 · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59

  21. 0 of 40 ConstellOps on held-out runs: 0 of 40 high-risk missed

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 (40 high-risk events pooled) · 95% range: 0–0.088 · baseline: classical solver-only policy at Pc 2e-5 (threshold chosen on seeds 0-59) and scripted analyst runbook at its default threshold (Pc 1e-4), same held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  22. 0 of 40 Tuned classical solver on held-out runs: 0 of 40 missed

    mode: Simulated · n: 60 held-out seeds 60-119 (40 actionable events pooled) · 95% range: 0–0.088 · row: classical solver-only policy at Pc 2e-5, threshold chosen on seeds 0-59, checked on held-out seeds · scenario: held-out confirmation, same storm generator, seeds 60-119

  23. 6 of 40 Scripted analyst runbook at its default threshold, held-out runs: 6 of 40 missed

    mode: Simulated · n: 60 held-out seeds 60-119 (40 high-risk events pooled) · 95% range: 0.05–0.27 · row: scripted analyst runbook at its default threshold, Pc 1e-4, held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  24. 0.295 ConstellOps on held-out runs: false-manoeuvre rate 0.295 (165/560)

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 (560 below-threshold events pooled) · 95% range: 0.262–0.329 · baseline: classical solver-only policy at Pc 2e-5 (threshold chosen on seeds 0-59) and scripted analyst runbook at its default threshold (Pc 1e-4), same held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  25. 0.089 Tuned classical solver on held-out runs: false-manoeuvre rate 0.089

    mode: Simulated · n: 60 held-out seeds 60-119 (560 below-threshold events pooled) · 95% range: 0.067–0.113 · row: classical solver-only policy at Pc 2e-5 on held-out seeds · scenario: held-out confirmation, same storm generator, seeds 60-119

  26. 2.726 h ConstellOps on held-out runs: 2.7 h before closest approach (median)

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 · 95% range: 2.401–3.09 · baseline: classical solver-only policy at Pc 2e-5 (threshold chosen on seeds 0-59) and scripted analyst runbook at its default threshold (Pc 1e-4), same held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  27. 8.0 h Tuned classical solver on held-out runs: 8.0 h before closest approach (fixed by the policy)

    mode: Simulated · n: 60 held-out seeds 60-119 · 95% range: no spread (the same value on every seed) · row: classical solver-only policy at Pc 2e-5; lead time fixed by the policy · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  28. 3.93 m/s ConstellOps on held-out runs: 3.93 m/s per 1000 events

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 · 95% range: 3.458–4.4 · baseline: classical solver-only policy at Pc 2e-5 (threshold chosen on seeds 0-59) and scripted analyst runbook at its default threshold (Pc 1e-4), same held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  29. 0.30 m/s Tuned classical solver on held-out runs: 0.30 m/s per 1000 events

    mode: Simulated · n: 60 held-out seeds 60-119 · 95% range: 0.242–0.345 · row: classical solver-only policy at Pc 2e-5, threshold chosen on seeds 0-59, held-out seeds · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  30. 0 of 205 ConstellOps on held-out runs: 0 of 205 commands unsafe, with an operator that approved everything the gates passed

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 (205 uplinked commands) · 95% range: 0–0.018 · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  31. 0 of 33 Missed high-risk conjunctions under CDM prompt injection: 0 of 33

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds (33 actionable events pooled) · exact 95% upper bound: 0.106 (about 10.6%) · baseline: scripted threshold runbook 7/33; classical solver-only policy 7/33 · scenario: same storm with prompt-injection text in CDM remarks, seeds 0-59

  32. 0.333 False-manoeuvre rate under CDM prompt injection: 0.333

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds (567 below-threshold events pooled) · 95% range: 0.292–0.375 · baseline: scripted threshold runbook 0.019; classical solver-only policy 0.009 · scenario: same storm with prompt-injection text in CDM remarks, seeds 0-59

  33. 52 of 265 Product-native storm: 52 of 265 high-risk events without a timely decision (no baseline)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 38 seeds (265 actionable events pooled) · 95% range: 0.171–0.222 · baseline: none (the baselines run only on the benchmark-generated storm) · scenario: product-native conjunction storm, seeds 0-37

  34. 0.146 Product-native storm: false-manoeuvre rate 0.146 (no baseline)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 38 seeds · 95% range: 0.123–0.174 · baseline: none (the baselines run only on the benchmark-generated storm) · scenario: product-native conjunction storm, seeds 0-37

  35. 5.45 h Product-native storm: median decision lead time 5.45 h (no baseline)

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 38 seeds · 95% range: 5.163–5.734 · baseline: none (the baselines run only on the benchmark-generated storm) · scenario: product-native conjunction storm, seeds 0-37

  36. 0.694 Kelvins L score 0.694: parity with the latest-risk baseline, behind the challenge winner

    mode: Public dataset · release: ConstellOps (reported-Pc passthrough on CDM-only feeds) · n: 2,167 test events (150 high-risk), full official test set · 95% range: 0.46–0.965 · baseline: latest-risk-prediction baseline 0.694 (identical by construction); challenge winner 0.556 · scenario: ESA Kelvins Collision Avoidance Challenge, full test set

  37. 0.694 Latest-risk-prediction baseline (published): 0.694

    Published result · ESA Kelvins Collision Avoidance Challenge, full test set · source

  38. 0.556 Challenge winner (published): 0.556

    Published result · ESA Kelvins Collision Avoidance Challenge, full test set · source

  39. $0.468 Model cost per 10-event storm episode: $0.468 at list price

    mode: Replay · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc3, replaying the recorded v1.0.0rc2 live campaign · n: 60 seeds · 95% range: 0.451–0.486 · scenario: benchmark-generated conjunction storm, 10 events / 24 h, seeds 0-59

  40. $0.47 about $0.47 per seed at list price for the model calls (the campaign spent more, including failed retries)

    mode: Replay and recorded-live · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0 · runs: recorded live 2026-10-01 to 2026-10-03, then replayed: 60 of 60 runs gave identical results · n: 60 held-out seeds 60-119 · 95% range: 0.46–0.489 · scenario: held-out runs, same benchmark-generated conjunction storm (10 events / 24 h), seeds 60-119

  41. 46.25 h Time to identify the common cause: 46.25 h (about 46 h, automated stand-in operator)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 8 runs recorded live, 12 replayed · n: 19 seeds with a value (1 of 20 without); automated stand-in operator · 95% range: 45.329–47.108 · scenario: common-mode anomaly from a bad software build

  42. 4.32 h Tuned classical solver (cohort correlation, threshold chosen on these runs): 4.32 h to the cause

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs; threshold chosen on these runs · 95% range: 3.225–5.549 · row: classical cohort-correlation solver, alpha 1e-2, chosen in-sample · scenario: common-mode anomaly from a bad software build (matched worlds)

  43. 25.55 h Scripted per-satellite triage runbook: 25.55 h to the cause

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · 95% range: 24.759–26.158 · row: scripted per-satellite triage runbook · scenario: common-mode anomaly from a bad software build (matched worlds)

  44. 5.40 h Tuned classical solver on held-out runs: 5.40 h to the cause

    mode: Simulated · n: 20 held-out seeds 20-39 · 95% range: 3.868–7.156 · row: same solver and threshold, held-out seeds · scenario: common-mode anomaly from a bad software build (matched worlds)

  45. +0.06% ConstellOps alarm volume: +0.06%, a slight increase; reduction -0.000583

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 8 runs recorded live, 12 replayed · n: 20 seeds (automated stand-in operator) · scenario: common-mode anomaly from a bad software build

  46. −16% Tuned classical solver alarm volume: −16%, reduction 0.163

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · row: classical cohort-correlation solver, alpha 1e-2 · scenario: common-mode anomaly from a bad software build (matched worlds)

  47. −8% Scripted runbook alarm volume: −8%, reduction 0.083

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · row: scripted per-satellite triage runbook · scenario: common-mode anomaly from a bad software build (matched worlds)

  48. 19 of 20 Correct bad build identified: 19 of 20 runs (automated stand-in operator)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 8 runs recorded live, 12 replayed · n: 20 seeds (20 runs, 8 recorded live and 12 replayed; automated stand-in operator that approves every proposal) · 95% range: 0.751–0.999 · scenario: common-mode anomaly from a bad software build

  49. 20 of 20 Tuned classical solver, correct build identified: 20 of 20 runs

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · 95% range: 0.832–1 · row: classical cohort-correlation solver · scenario: common-mode anomaly from a bad software build (matched worlds)

  50. 19 of 20 Scripted runbook, correct build identified: 19 of 20 runs

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · 95% range: 0.751–0.999 · row: scripted per-satellite triage runbook · scenario: common-mode anomaly from a bad software build (matched worlds)

  51. 43.55 ConstellOps approval records per episode under its A1 policy: 43.55

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 8 runs recorded live, 12 replayed · n: 20 seeds (automated stand-in operator that approves every proposal) · 95% range: 38.6–47.651 · baseline: policy count; classical solver automated (0), runbook one sign-off per proposal · scenario: common-mode anomaly from a bad software build (matched worlds)

  52. 0 Classical solver approval records per episode: 0 (automated)

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · row: policy count · scenario: common-mode anomaly from a bad software build (matched worlds)

  53. 1 Scripted runbook approval records per episode: 1 (one sign-off)

    mode: Simulated · n: 20 seeds 0-19, matched to the product runs · row: policy count · scenario: common-mode anomaly from a bad software build (matched worlds)

  54. 1.0 min About 1.0 min to detect (mean 1.008 min, 16 of 20 runs; event-driven, while the matched classical solver checks on a 5-minute polling interval and restores service faster)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 6 runs recorded live, 10 replayed (4 refused at start) · n: 16 16 of 20 runs (4 refused by the product at start) · scenario: gateway outage across space, ground and network

  55. 5.0 min Classical capacity solver: 5.0 min to detect (its polling interval)

    mode: Simulated · n: 16 the 16 runs the product ran (partial match) · 95% range: no spread (the same value on every seed) · row: classical capacity solver · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  56. 24.63 min Scripted operations-centre runbook: 24.63 min to detect

    mode: Simulated · n: 16 the 16 runs the product ran (partial match) · 95% range: 23.432–26.173 · row: scripted operations-centre runbook · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  57. 15 min About 15 minutes to restore (mean 14.57 min, 16 of 20 runs)

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 6 runs recorded live, 10 replayed (4 refused at start) · n: 16 16 of 20 runs (4 refused by the product at start) · scenario: gateway outage across space, ground and network

  58. 5.0 min Classical capacity solver: 5.0 min to restore

    mode: Simulated · n: 16 the 16 runs the product ran (partial match) · 95% range: no spread (the same value on every seed) · row: classical capacity solver · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  59. 117.66 min Scripted operations-centre runbook: 117.66 min to restore

    mode: Simulated · n: 16 the 16 runs the product ran (partial match) · 95% range: 94.882–141.495 · row: scripted operations-centre runbook · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  60. 2.38 ConstellOps approval records per episode under its A1 policy: 2.38

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 6 runs recorded live, 10 replayed (4 refused at start) · n: 16 the 16 runs the product ran · 95% range: 2.125–2.688 · baseline: policy count · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  61. 0 Classical solver approval records per episode: 0 (automated)

    mode: Simulated · n: 16 the 16 runs the product ran · row: policy count · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  62. 2.56 Scripted runbook approval records per episode: 2.56 (one sign-off per proposal)

    mode: Simulated · n: 16 the 16 runs the product ran · 95% range: 2.188–3.125 · row: policy count · scenario: gateway outage, primary outage matched on the 16 runs the product ran (partial match)

  63. 27.4 ConstellOps approval records per episode under its A1 policy (two-person software actions counted twice): 27.4

    mode: Recorded live, with replays · model: Amazon Nova Pro (prompted reasoning) · release: ConstellOps v1.0.0rc4 · runs: 5 runs recorded live, 15 replayed · n: 20 seeds (20 rollout runs) · 95% range: 25.7–28.9 · baseline: policy count · scenario: staged rollout with a canary regression (not matched; approval policy count only)

  64. 0 Classical canary solver approval records per episode: 0 (automated)

    mode: Simulated · n: 20 seeds 0-19 (rollout not matched; policy count only) · row: policy count only; rollout outcomes not matched · scenario: staged rollout with a canary regression (not matched; approval policy count only)

  65. 1 Bake-and-review runbook approval records per episode: 1 (one sign-off)

    mode: Simulated · n: 20 seeds 0-19 (rollout not matched; policy count only) · row: policy count only; rollout outcomes not matched · scenario: staged rollout with a canary regression (not matched; approval policy count only)

  66. 16 of 16 Gateway-outage runs in which the product did not model the nuisance alarms: 16 of 16

    mode: Simulated · n: 16 the 16 runs the product ran · scenario: gateway outage, partial match

  67. 3 of 16 Gateway-outage runs with a second outage the product did not model: 3 of 16

    mode: Simulated · n: 16 the 16 runs the product ran · scenario: gateway outage, partial match

  68. 5 of 33 Scripted runbook below Pc 5e-5 (2e-5, 1e-5, 5e-6): 5 of 33 missed

    mode: Simulated · n: 60 seeds (33 actionable events pooled) · 95% range: 0.04–0.276 · row: scripted threshold runbook at every swept threshold below Pc 5e-5 · scenario: threshold sweep on the benchmark-generated conjunction storm, seeds 0-59