Both Guards are the same model reading a sandwich-formatted prompt: trusted framing on top, untrusted text bracketed in the middle, and a closing reminder that the bracketed part is data. What separates them is what goes in the filling, and that difference decides which one does the work.
Pick a Guard and a message. The Input Guard wraps one untrusted block. The Output Guard wraps two, because it has to judge the response in light of what was asked.
Gemma 2 9B Instruct at temperature 0 with a 128-token budget. The parser takes the last occurrence of a verdict token, so an earlier mention inside quoted text cannot shadow the decision. UNCLEAR maps to ALLOW by default; a fail-secure variant maps it to BLOCK for deployments that prefer refusing over answering.
This is the most surprising number in the perimeter evaluation. On one benchmark the Input Guard catches almost everything. On the other it catches nothing at all, and every block is taken after the model has already answered.
Why. InjecAgent injections are directly imperative, of the wire-five-thousand-dollars kind, and that reads as an instruction at the perimeter. BIPIA injections are framed as content: a marketing pitch, a phishing email, a dissemination request. The Input Guard finds no direct-injection lexical signal in prose that looks like prose, so nothing trips until the Output Guard sees what the model actually produced.
A blocking gate short-circuits everything downstream. Where the block lands therefore decides whether the security layer costs time or saves it.
The Input Guard refuses most instances before the Domain LLM is ever called, so the median request pays one judge call instead of three model calls. The defended pipeline is faster than no defense at all.
P95 still rises, 11.2 to 15.0 s, because the unblocked minority pays the full cost.
Nothing is caught at the perimeter, so every instance runs the Input Guard, the Domain LLM and the Output Guard in full before anything can be refused. Every catch is paid for at full price.
P95 rises from 13.1 to 17.2 s. The 500 ms per-turn target is far away on both benchmarks.
| Configuration | Block rate | Naive ASR | Judged ASR |
|---|---|---|---|
| BIPIA, no defense | 0.000 | 1.000 | 0.521 |
| BIPIA, rule-based | 0.000 | 1.000 | 0.521 |
| BIPIA, LLM Guards | 0.595 | 0.405 | 0.049 |
| InjecAgent, no defense | 0.000 | 1.000 | 0.659 |
| InjecAgent, rule-based | 0.000 | 1.000 | 0.659 |
| InjecAgent, LLM Guards | 0.987 | 0.013 | 0.009 |
The rule-based rows are identical to no-defense to four decimal places, because under greedy decoding both send the same input to the same model. The rules look for imperative overrides and role-token smuggling, and BIPIA contains neither. That is a defense calibrated for one threat class being measured against another, not a broken implementation.
Judged ASR is scored by a Gemma judge, checked against 147 human labels. On BIPIA that gave 91 percent agreement and a kappa of 0.82, with no under-counting, so the 0.049 residual is an honest upper bound. On InjecAgent agreement collapsed to 45 percent with a kappa of zero and a 100 percent false-complied rate, because the text-only Coordinator never executes a tool and the judge counts the injected instruction merely appearing as compliance. Lead on BIPIA; treat the InjecAgent column as a loose bound.
Gemma 2 9B is both Guards, the judged-ASR judge, the capability action detector and the fragmentation detector. Architectural diversity from the Llama 3.1 target means an attack tuned to one family is less likely to transfer, but it does not make Guard failures independent of measurement failures. The human validation is what anchors the metric.
On the 120-prompt benign set the LLM Guards sit at 0.008. On real multi-turn dialogues they refuse 8 of 30, a false-positive rate of 0.267, concentrated in the longest conversations. The perimeter is well behaved on single prompts and much less so on extended sessions.