Engineering blog Post 07
What we learned putting a small decision model in front of our agents
We tested strands-decider, an open-source "system one" decision model, as a fast first pass in front of Amazon Nova Pro. The evaluation ran offline on recorded data from our own campaigns, at no model cost, as advice only: no gate read it. The failures come first, because they decided what happens next.
What failed
A screen that can be talked out of its answer by the text it is screening is not a screen.Link to this quote
What worked, narrowly
As a yes-or-no injection screen on recorded conjunction messages, it flagged injected remarks in 441 of 441 cases, and clean strings in 0 of 4,696.
That test is in distribution only: six injection templates and four distinct clean texts. On untrusted strings from the other workflows it also flagged clean text: 2 of 20.
It is fast and local: a median of 159.5 milliseconds per question on a laptop, with no model bill.
What goes forward
Only one use survives: an escalate-only injection pre-screen. If it flags a string, the item goes to the full model and a person sees why; if it does not, nothing changes. It never allows anything, and it is never a gate.
Even that waits for a harder benchmark first: more injection styles, many more distinct clean strings, and the incident-style instructions that steered it here. It goes into v1.1 only if it passes.
A side finding about our own agent
The evaluation also checked the recorded agent against its own role prompt. Recorded agent decisions proposing a burn below the action threshold: 375 of 475. Its instructions said to recommend no action there.
That is consistent with the conjunction agent's published false-manoeuvre rate, and it is now filed against the product. The gates did not stop those proposals, because a burn below the threshold is allowed with human approval; it is the operator's go that would catch it.
How the test ran
- Inputs were recorded decisions and messages from earlier campaigns, unchanged. No new model calls.
- Weights were pinned by revision and file hash, and checked before any question.
- Answers were compared with the recorded decisions, with rule-based expectations and with labelled injected text.
- Everything went into an evidence bundle, approved by the review board, an independent AI reviewer agent, before any figure appeared here.
Next pass
We set out to make agents faster with a small model. What we learned is narrower and more useful: where it is safe to put one, and where it is not. We said we would publish whatever it showed, and this is it.