Prompt Injection Detection Models: The Three Numbers to Measure Before You Deploy One

Meta shipped two generations of the same prompt injection detector. Between them, the metric on the model card went down and the metric that decides a deployment went up fourfold.

That is the clearest available evidence that a published detection score does not predict production behavior, and it is sitting in Meta’s own comparison table.

The gap matters because the decision waiting on it is not academic. You are being asked to put a probabilistic control in front of agents that already hold credentials, and to do it without breaking the workflows engineering shipped last quarter. Prompt injection has held first place in OWASP’s Top 10 for LLM applications since the list existed, the market has responded with a large number of models that describe themselves identically, and the numbers on their cards are not comparable to each other.

Three numbers settle it, and none of them appear on a model card. Recall at the false positive rate your traffic can absorb. Messages scored per day, which follows from where in the agent loop you place the model. And the residual, which is what gets through and what every control you build after this one has to carry.

The rest of this article is how to measure each of them, starting with why the numbers already published for you are the wrong ones.

Every detection model is one of three architectures, and the architecture sets the ceiling

A prompt injection detection model is a classifier that reads a span of text and returns a score for whether that span is content the agent should process or an instruction the agent should refuse.

Encoder classifiers are the workhorses, running between 22 million and 400 million parameters. Meta’s Llama Prompt Guard 2 ships in 22M and 86M variants built on DeBERTa-xsmall and mDeBERTa-base. Qualifire’s Sentinel is fine-tuned from ModernBERT-large. A 184M DeBERTa-v3 detector appears as the baseline in most published comparisons. They return a probability, you set a threshold, and they run on CPU at single-digit milliseconds. This is where most production deployments land, and for good reason: the cost per call is small enough that you can afford to score messages other than the user’s.

LLM-as-judge guard models run from 0.6B to 8B and reason about the text rather than scoring it. Llama Guard 3, Google’s ShieldGemma at 2B, and Alibaba’s Qwen3-Guard across 0.6B, 4B and 8B sit here. They catch semantic framing a small encoder misses, and they need a GPU.

Embedding plus a classical head takes a pretrained sentence embedding and trains logistic regression or a random forest on top, as NVIDIA’s NeMo jailbreak detector does. You can retrain it on your own labelled data in an afternoon.

All three read text, which means all three are answering a question about where the payload entered rather than about what the agent subsequently does.

One further constraint applies across the category. The guard is smaller than the model it guards, always, because a guard that costs as much as the model doubles your inference bill. That asymmetry is exploitable rather than tunable. Researchers presenting at USENIX Security 2026 built prompts whose payload only becomes legible after the reading model has spent more compute than any bounded filter has budget for, and reported success on all twelve malicious intents against three of four production chat platforms. The measured thresholds sat around 800 prompt tokens for the timed-release variant and roughly 10,000 for the spaced-release variant. No amount of retraining closes a gap that exists because the filter has a smaller compute budget than the model.

One number quadrupled, another collapsed, and the headline score moved by 0.007

Here is the table, from Meta’s LlamaFirewall paper.

Model	AUC (English)	Recall @ 1% FPR	Latency (A100, 512 tokens)
Prompt Guard 1	.987	21.2%	92.4ms
Prompt Guard 2, 86M	.98	97.5%	92.4ms
Prompt Guard 2, 22M	.995	88.7%	19.3ms

Read the first two rows. English AUC went down, from .987 to .98. Recall at a 1% false positive rate went from 21.2% to 97.5%, a fourfold improvement in the model’s ability to catch attacks while staying inside a false positive budget a production system can absorb.

Compare those two models on AUC, the metric almost every model card leads with, and generation two looks like a marginal regression. On the metric that describes a deployment, generation two is a different product.

Now the other direction. The same USENIX Security work evaluated both generations against encoding-based attacks and found that Prompt Guard 2 regressed where Prompt Guard 1 held. Detection scores against those attacks fell to somewhere between 0.02 and 0.10, against 0.9978 to 0.9986 for the previous generation, while benchmark performance between the two generations stayed broadly comparable.

So one number quadrupled. A second fell from near-certain detection to near-total blindness. The headline metric moved by 0.007.

AUC integrates performance across every possible threshold, which makes it a ranking metric. Deploying a classifier is a thresholded decision: you pick one operating point and you live on it. A model can rank inputs beautifully and still be useless at the specific threshold your false positive budget allows. Meta’s engineering notes describe exactly this, including a modified energy-based loss chosen to improve precision on out-of-distribution inputs, which is a threshold-shaping intervention AUC is structurally unable to see.

Record the first two of your three numbers for every model you evaluate: recall at the false positive rate you can afford, and per-classification latency at your token length on your hardware. The rest of the card ranks. Those two deploy.

Score transfer fails between corpora, so reproduce the number on your own traffic

Every detector in this category loses ground on a corpus its authors did not build.

Take the 184M DeBERTa-v3 baseline from the architecture section. One 2025 evaluation scored it at 65.2% on Qualifire’s prompt injection benchmark. A separate reproduction put its F1 at 0.539 on a 2,386-case holdout. A third study measured it against JailbreakHub, a corpus of jailbreaks collected in the wild, and recorded a false positive rate of 0.9596, which the authors called unacceptable for production use. Same weights, three corpora, three answers, none of them wrong. A text classifier’s performance is a joint property of the model and the distribution it is scored against, and your traffic is a fourth distribution nobody has measured.

Over-defense is the same failure wearing a different name. The InjecGuard authors built a benchmark specifically for benign inputs containing attack-adjacent vocabulary, the kind of text your own security team writes all day, and reported beating that baseline by 30.83 points of average accuracy on it. That comparison was published by a team proposing its own detector, which is true of most cross-evaluations in this category and is a fair reason to discount any single figure. The pattern is what survives the discount: no detector has yet been measured on someone else’s corpus without losing ground.

A small number of model cards report this honestly. The prismor/prompt-guard-1.5b card documents a false positive rate of 0.379 on a benchmark of complex jailbreaks and long benign prompts, alongside 70% accuracy on obfuscated inputs and an explicit note that the model was never trained against adaptive attackers. That is the disclosure standard worth asking a vendor for. Where a card reports only in-distribution results, treat the out-of-distribution number as unmeasured rather than absent.

So build your own corpus. Five hundred labelled examples is enough to separate candidate models, and it does not need a research team.

Sample 250 benign examples directly from your own traffic. Real retrieved chunks, real tool responses, real user messages, weighted the way your traffic is actually weighted. Include the awkward ones: the support ticket quoting an attacker’s payload, the internal wiki page about prompt injection, the security runbook. Those are the inputs that generate your false positives, and they are absent from every public benign set.

Write 250 adversarial examples against your own system prompt and your own tool schema. Generic payloads test generic defenses, and the payloads that have actually worked in the wild were shaped around a specific product’s exit path rather than around a benchmark. NVIDIA’s garak automates a large volume of this against your specific configuration, which produces a failure rate belonging to your environment.

Then do the multiplication nobody in the category does. Take the false positive rate you measured, multiply it by the number of messages you will score per day, and read the answer as broken production tasks per day. An agent fleet handling 40,000 tool responses daily, scored at 3% FPR, produces 1,200 failed calls a day. That number decides whether the control survives its first week, and it depends far more on which messages you score than on which model you picked.

Your false positive budget is set by which turn you score, not by which model you pick

A turn is one message in the agent’s loop: the user’s message, a retrieved chunk, a tool response, the model’s own output. An agent loop has four places a detection model can read those turns, and they differ by an order of magnitude in volume.

The input rail scores the user message, one call per request. The retrieval rail scores each chunk before it enters the context, which multiplies volume by your top-k. The tool-output rail scores each tool response as it re-enters the model’s context, and this is where indirect payloads actually arrive. The tool-call rail scores the arguments before a tool executes, which catches the action instead of the text.

Most default deployments stop at the input rail, which is where every framework quickstart puts it. Production payloads arrive somewhere else: inside a RAG chunk, an MCP tool response, a fetched page, or the output of another agent.

Meta’s own evaluation methodology makes the turn dependency explicit. Running PromptGuard against AgentDojo traces, they scored only messages with the user or tool role, because those are the messages that can carry an injected payload, and restricted their alignment auditor to assistant messages, because those carry the agent’s reasoning. Two guardrails, two disjoint sets of turns, one pipeline.

Get the turn wrong and a competent model produces nothing useful. A 2025 evaluation of guardrail models against agentic backdoor data found one general-purpose safety model firing consistently on the wrong turn: it flagged user messages containing credit card numbers and addresses while overlooking the tool-observation turn where the injection payload actually sat.

Volume then sets your latency ceiling. Meta’s table puts the 22M variant at 19.3ms and the 86M at 92.4ms per classification on an A100 at 512 tokens. Classifiers in the 23M range have been measured at roughly 9.5ms per tool response on a single commodity CPU core, batching to under 4ms. Guard models in the 7B to 8B class need a GPU and land at 25 to 50ms. Practitioner guidance converges on sub-100ms p50 for an inline rail, with the observation that past roughly 200ms someone files a ticket to turn the rail off, and a rail that gets turned off has a recall of zero.

So size the model to the turn. Small encoders go on the high-frequency turns, retrieval and tool output, where volume punishes both latency and false positives. The expensive judge model handles escalation, scoring only what the cheap model flagged as ambiguous. That two-tier shape is how the cost stays bounded while coverage extends past the input rail.

Meta narrowed its own model away from the attack you are worried about

Prompt Guard 1 tried to detect goal hijacking. Prompt Guard 2 stopped trying.

Meta documented the reasoning directly. The first generation attempted broader goal hijacking detection, and that generalization produced excessive false positives when the model lacked context about user intent and application-specific goals. The second generation narrowed its scope to explicit jailbreak techniques, which are lexically regular and structurally predictable, and the team recommends pairing it with a separate reasoning-based check for anything subtler.

That is an engineering finding and it is worth reading as one. The team shipping the leading open detector measured that goal hijacking is not separable from legitimate instruction at the text plane without context about what this specific agent is supposed to be doing. A general-purpose classifier does not have that context. It sees a span of text and a training distribution.

Goal hijacking is the coercion case. Coercion describes the situation where the instruction that starts the attack reads as work: nothing gets exploited, nothing gets escalated, and the agent is simply pointed at a different objective and pursues it with authority you granted on purpose. There is no adversarial text for a classifier to score, and no failed permission check for anything downstream to notice. Which control catches it turns out to depend almost entirely on which position that control occupies relative to the model’s decision.

Which is why the layered numbers are worth reading precisely. On AgentDojo, Meta reports a baseline attack success rate of 17.63% with no guardrail. Prompt Guard 2 86M alone brings it to 7.53%. Adding a chain-of-thought alignment auditor running on Llama 4 Maverick brings it to 1.75%, with task utility falling from 47.73% to 42.68%.

That is the strongest published text-plane configuration. It costs five points of utility to reach 1.75% residual on a benchmark whose attack class is narrow, and your residual is larger. That residual is the third of your three numbers, and it is an input to the next design decision rather than a caveat at the bottom of the evaluation.

Size the downstream control to the residual you just measured

A residual becomes actionable when something scores what the agent did rather than what it read.

That means a definition of normal belonging to one specific agent, derived from execution rather than declared in a manifest. Declared permissions describe a ceiling, and the ceiling is the space coercion operates inside. ARMO builds that envelope per agent as Application Profile DNA, assembled from what a kernel-level eBPF sensor observes, and we have previously broken down the four dimensions a coerced agent has to touch along with how quickly each one becomes available during an attack.

Sizing is the part that follows from your measurement. A residual of a few percent against agents that only read is a monitoring problem. The same residual against agents holding write access or reachable credentials is a containment problem with a clock on it, and the response has to execute inside the window the agent needs to finish the objective.

Enforcement runs from the same signal, which is what keeps it survivable. The progression starts in audit: every action the platform would have taken gets logged with the baseline dimension it deviated from, and production traffic stays untouched. You read that log, fix what it exposes, then graduate per workload with no sidecars and no code changes. Credential isolation sits alongside it as the cheapest containment against coercion specifically, because an agent holding substitute keys continues to run, continues to reason, and reaches nothing. The observation window that has to run before enforcement is armed is the constraint worth planning around, because it starts when you deploy the sensor rather than when you finish choosing a classifier.

The classifier still earns its place. It reduces arrival rate, and that work is real. This is the control sized to the number the classifier leaves behind, which is what runtime security for AI workloads is built to carry.

Calibrate the model you have before you buy the model you do not

Three numbers decide whether a prompt injection detection model is worth deploying, and none of them appear on a model card.

Recall at the false positive rate your traffic can absorb, measured on a corpus assembled from your own turns. Messages scored per day, which follows from where in the loop the model sits and sets both your latency ceiling and your broken-task count. And the residual, which is the fraction that gets through and the input to every control you build after this one.

The corpus is the part teams assume they have to construct from scratch. It is already in your traces. Five hundred labelled examples pulled from real retrieval chunks, real tool responses, and adversarial prompts written against your own system prompt will separate candidate models more reliably than any published benchmark, because it is the only distribution that describes your environment.

Measure those three. Then decide what carries the third one.

See how ARMO scores per-agent deviation and contains a coerced agent before the transfer completes.

Frequently Asked Questions

Which prompt injection detection model is best for production? The question resolves to a threshold and a placement rather than a name. Compare candidates on recall at the false positive rate your traffic can absorb, measured on a corpus you assembled, and score the same turns you intend to score in production. Small encoder classifiers in the 22M to 200M range are the right default for high-frequency turns like retrieval and tool output, with a larger judge model reserved for escalating whatever the encoder flags as ambiguous.

What false positive rate is acceptable for a prompt injection classifier? Derive it from volume rather than from a benchmark. Multiply your candidate false positive rate by the number of messages you will score per day and read the product as failed production tasks per day: 40,000 tool responses at 3% FPR is 1,200 broken calls daily. Most teams find the acceptable figure sits below 1% once they score anything past the input rail, which is why recall at 1% FPR is the comparison metric worth building your evaluation around.

Should I run the 22M or the 86M variant? Turn volume and latency ceiling decide it. Meta reports the 22M variant at 19.3ms per classification against 92.4ms for the 86M on an A100 at 512 tokens, with the 86M holding stronger multilingual coverage and the 22M trading some of that away. If you are scoring every tool response on a busy fleet, the smaller model on the hot path with the larger one handling escalations will beat either model used alone.

Can I fine-tune a detection model on my own attack data? Yes, and it moves the number that matters most. Fine-tuning on your own benign traffic is usually a bigger win than fine-tuning on attacks, because most production pain comes from over-defense on domain text containing attack-adjacent vocabulary. It will not close the structural gap: the guard still runs on a smaller compute budget than the model it protects, and an attacker who knows your defense exists can optimize against it.

Do I still need runtime detection if the classifier scores well on my corpus? Yes, and the residual you measured is the argument for it. A classifier scores text at a turn boundary, which leaves the case where the instruction reads as legitimate work and the agent uses authorized capability against its intended purpose. Detecting that requires a definition of normal belonging to one specific agent, scored against what it actually executed, which is a different signal from anything a text classifier can produce.