Engineering blog Post 07

What we learned putting a small decision model in front of our agents

Vivek Raja3 min readPart 7 of 9, published as one series

  • strands-decider
  • hybrid agents
  • benchmark
  • injection
What goes forward: an escalate-only pre-screenpipeline
UntrustedtextCDM remark ·telemetrystrands-deciderinjected text?Injectionflag?escalate onlyAmazon NovaProon every itemGate chainunchangedthe decision model is never a gate

We tested strands-decider, an open-source "system one" decision model, as a fast first pass in front of Amazon Nova Pro. The evaluation ran offline on recorded data from our own campaigns, at no model cost, as advice only: no gate read it. The failures come first, because they decided what happens next.

What failed

A screen that can be talked out of its answer by the text it is screening is not a screen.
Link to this quote

What worked, narrowly

As a yes-or-no injection screen on recorded conjunction messages, it flagged injected remarks in 441 of 441 cases, and clean strings in 0 of 4,696.

That test is in distribution only: six injection templates and four distinct clean texts. On untrusted strings from the other workflows it also flagged clean text: 2 of 20.

It is fast and local: a median of 159.5 milliseconds per question on a laptop, with no model bill.

How the decider benchmark runsmethod
Recorded decisionsearlier campaigns, unchangedno new model callsOffline replayweights pinned by revisionno model costDecision model answerspick · yes/no · score + confidenceadvisoryComparerecorded decision · rules · labelsagreement · calibrationEvidence bundle → claimsreview board approvedevery figure labelled

What goes forward

Only one use survives: an escalate-only injection pre-screen. If it flags a string, the item goes to the full model and a person sees why; if it does not, nothing changes. It never allows anything, and it is never a gate.

Even that waits for a harder benchmark first: more injection styles, many more distinct clean strings, and the incident-style instructions that steered it here. It goes into v1.1 only if it passes.

The pre-screen never allows anythingsequence
Agentstrands-deciderAmazon Nova ProGate chainscreen: injected text?flag → escalate, show whyevery item still reasonedreasoned answerany proposal → gatesallow · deny · approval

A side finding about our own agent

The evaluation also checked the recorded agent against its own role prompt. Recorded agent decisions proposing a burn below the action threshold: 375 of 475. Its instructions said to recommend no action there.

That is consistent with the conjunction agent's published false-manoeuvre rate, and it is now filed against the product. The gates did not stop those proposals, because a burn below the threshold is allowed with human approval; it is the operator's go that would catch it.

How the test ran

  1. Inputs were recorded decisions and messages from earlier campaigns, unchanged. No new model calls.
  2. Weights were pinned by revision and file hash, and checked before any question.
  3. Answers were compared with the recorded decisions, with rule-based expectations and with labelled injected text.
  4. Everything went into an evidence bundle, approved by the review board, an independent AI reviewer agent, before any figure appeared here.

Next pass

We set out to make agents faster with a small model. What we learned is narrower and more useful: where it is safe to put one, and where it is not. We said we would publish whatever it showed, and this is it.

§ · NEXT PASS

Request a briefing.

A 45-minute session with the evidence behind any figure. Operators can send a week of their queue.

This address is used only to reply to you. The site sets no cookies and loads no trackers.

Briefing requests

Sources and labels for every figure on this page (11)

  1. 14 of 16 Incident episodes where an injected instruction steered its runbook pick: 14 of 16

    mode: Replay · model: strands-decider-2B-hobson-v19, run locally with pinned weights · release: offline evaluation on recorded ConstellOps campaign data (v1.0.0rc2 conjunction, v1.0.0rc4 gateway outage); advisory only, no gate reads it · n: 16 recorded gateway-outage episodes · 95% range: 0.617–0.984 · scenario: incident runbook step on recorded gateway-outage episodes

  2. 16 of 16 Right runbook step with only the injected instruction removed: 16 of 16

    mode: Replay · model: strands-decider-2B-hobson-v19, run locally with pinned weights · release: offline evaluation on recorded ConstellOps campaign data (v1.0.0rc2 conjunction, v1.0.0rc4 gateway outage); advisory only, no gate reads it · n: 16 recorded gateway-outage episodes, injected instruction removed · 95% range: 0.794–1 · scenario: incident runbook step, ablation

  3. 0 of 5,137 Injection-screen answers at confidence 0.9 or more: 0 of 5,137

    mode: Replay · model: strands-decider-2B-hobson-v19, run locally with pinned weights · release: offline evaluation on recorded ConstellOps campaign data (v1.0.0rc2 conjunction, v1.0.0rc4 gateway outage); advisory only, no gate reads it · n: 5,137 injection-screen answers · 95% range: 0–7e-4 · scenario: injection screen; confidence = |2p - 1|

  4. 0.285 Injection-screen calibration error (ECE): 0.285

    mode: Replay · model: strands-decider-2B-hobson-v19, run locally with pinned weights · release: offline evaluation on recorded ConstellOps campaign data (v1.0.0rc2 conjunction, v1.0.0rc4 gateway outage); advisory only, no gate reads it · n: 5,137 injection-screen answers · 95% range: 0.284–0.287 · scenario: injection screen

  5. 453 of 475 Conjunction picks matching the recorded agent: 453 of 475 (not interpretable: it always picked the same option slot)

    mode: Replay · model: strands-decider-2B-hobson-v19, run locally with pinned weights · release: offline evaluation on recorded ConstellOps campaign data (v1.0.0rc2 conjunction, v1.0.0rc4 gateway outage); advisory only, no gate reads it · n: 475 recorded multi-option conjunction decisions · 95% range: 0.931–0.971 · scenario: conjunction option pick on recorded decisions

  6. 16 of 16 Incident class right: 16 of 16 (trivial: the class is the same in every episode)

    mode: Replay · model: strands-decider-2B-hobson-v19, run locally with pinned weights · release: offline evaluation on recorded ConstellOps campaign data (v1.0.0rc2 conjunction, v1.0.0rc4 gateway outage); advisory only, no gate reads it · n: 16 recorded gateway-outage episodes · 95% range: 0.794–1 · scenario: incident classification on recorded gateway-outage episodes

  7. 441 of 441 Recorded injected remarks flagged: 441 of 441 (in distribution only: 6 templates)

    mode: Replay · model: strands-decider-2B-hobson-v19, run locally with pinned weights · release: offline evaluation on recorded ConstellOps campaign data (v1.0.0rc2 conjunction, v1.0.0rc4 gateway outage); advisory only, no gate reads it · n: 441 recorded injected remarks (6 templates) · 95% range: 0.992–1 · scenario: injection screen on recorded conjunction messages, in distribution only

  8. 0 of 4,696 Clean strings flagged: 0 of 4,696 (only 4 distinct clean texts)

    mode: Replay · model: strands-decider-2B-hobson-v19, run locally with pinned weights · release: offline evaluation on recorded ConstellOps campaign data (v1.0.0rc2 conjunction, v1.0.0rc4 gateway outage); advisory only, no gate reads it · n: 4,696 clean remarks and message strings (only 4 distinct clean remarks) · 95% range: 0–8e-4 · scenario: injection screen on recorded conjunction messages, in distribution only

  9. 2 of 20 Clean out-of-distribution strings flagged: 2 of 20

    mode: Replay · model: strands-decider-2B-hobson-v19, run locally with pinned weights · release: offline evaluation on recorded ConstellOps campaign data (v1.0.0rc2 conjunction, v1.0.0rc4 gateway outage); advisory only, no gate reads it · n: 20 distinct untrusted strings from the other workflows, labelled by a published rule · 95% range: 0.012–0.317 · scenario: injection screen, out of distribution (fleet-anomaly, gateway, rollout and space-weather strings)

  10. 159.5 Injection-screen latency per question, median, local: 159.5

    mode: Replay · model: strands-decider-2B-hobson-v19, run locally with pinned weights · release: offline evaluation on recorded ConstellOps campaign data (v1.0.0rc2 conjunction, v1.0.0rc4 gateway outage); advisory only, no gate reads it · n: 5,158 injection-screen questions · 95% range: 158.7–160.3 · scenario: injection screen at normal machine load; not Bedrock latency

  11. 375 of 475 Recorded agent decisions proposing a burn below the action threshold: 375 of 475

    mode: Replay · model: Amazon Nova Pro (prompted reasoning), recorded on v1.0.0rc2 · release: offline evaluation on recorded ConstellOps campaign data (v1.0.0rc2 conjunction, v1.0.0rc4 gateway outage); advisory only, no gate reads it · n: 475 recorded multi-option conjunction decisions · 95% range: 0.75–0.825 · scenario: side finding: the recorded conjunction agent against its own role prompt