WebInject: The Web Agent Prompt Injection With No Payload

A browser agent in your cluster opens a supplier portal, screenshots it, and clicks somewhere the task never called for. The page looks exactly like the page the supplier serves, and to the person who checks it later, it still does.

The classifier in front of the agent returned nothing, because the instruction that produced the click does not exist. WebInject, published at EMNLP 2025, perturbs the raw pixel values of a rendered page so the agent’s own screenshot drives it to an action the attacker picked, with no sentence added anywhere.

Every prompt injection control you own inspects text, or something recoverable as text. Here there is none. What follows is how the attack works, why an evasion ledger has no row for it, which agents have a perception layer to attack, why goal-alignment checks pass it, and what the action record still shows.

WebInject Writes the Action and Leaves the Page Readable

WebInject produces an action without producing a sentence.

The attack targets web agents built on multimodal models: agents that take a screenshot of a page and generate their next step from the image. It adds a perturbation to the raw pixel values of the rendered page. Those pixels land in the screenshot the agent captures, and the screenshot drives the agent to emit an action the attacker specified, a click at a chosen coordinate.

Finding that perturbation is an optimization problem with one awkward property. The mapping from raw pixel values to the screenshot the model sees is non-differentiable, so gradients cannot be pushed back to the pixels. The authors trained a neural network to approximate the mapping, then ran projected gradient descent against the approximation.

The numbers are what a design review needs. Against Gemma-3, averaged across ten webpage datasets, the strongest existing webpage-based injection reached a success rate of 0.062 and the optimized attack reached 0.972. Prior screenshot-optimized attacks reached exactly zero, and that zero is the paper’s actual contribution: those perturbations were computed on the screenshot and then written into the page’s pixels, where the monitor’s color transform destroyed them. Approximating that transform is what makes the attack work at all. The full results cover five agent models across real and synthetic pages.

Now be precise about what changes and what does not, because the difference decides which control could have caught it. The page changes. The attack injects code that reads the rendered pixels off a canvas, adds the perturbation, writes them back, and layers the original elements on top at zero opacity so clicks still land where a user expects. That is a structural artifact, and a DOM integrity check or a content security policy would have something to look at.

What does not change is the meaning. There is no override phrasing, no encoding, no zero-width character, no white-on-white text, and nothing for an OCR pass to recover, because nothing was written. The perturbation is bounded so it stays invisible to a person. A vision model asked what the page says reads the supplier’s page.

So the exposed agents are the ones that look at a page rather than parse it. Before naming which of yours those are, there is a prior problem. The frameworks security teams use to sort injection attacks have nowhere to put this one.

The Evasion Ledger Has No Row for an Attack With No Payload

Sorting injection techniques by required evasion produces three tiers, and this attack sits underneath all of them.

The ledger runs like this. Tier 1 techniques defeat the model’s refusal boundary, which is a boundary an agent attacker has little reason to touch. Tier 2 defeats an instruction hierarchy or an input filter, and every entry carries evidence of its own evasion inside the payload: an override has to read as an override, an encoded block has to be encoded, hidden text has to be hidden. Tier 3 defeats nothing at all, because it is a plain sentence asking for an action the agent already holds permission to take.

The closest row is Tier 2’s multimodal entry, an instruction placed inside an image so a text-only inspection walks past it. That row describes a legible instruction with pixels around it. Run the image through OCR, or hand it to a vision model, and the instruction comes back out. The evasion is the placement, and the placement is recoverable.

WebInject is not sitting in that row. There is no instruction inside the image, and there is no instruction anywhere else either. Nor is it Tier 3, where a payload exists and reads as ordinary language. A Tier 3 payload is a sentence a human reviewer would find if they went looking. Here a human reviewer finds the supplier’s page.

That is why the ledger’s axis stops working. Required evasion measures how much work a payload did before its instruction landed, and the measurement assumes an instruction that has to land. WebInject skips the instruction and writes the action directly, so the column has no value to take.

The consequence for coverage is sharper than a missed detection. A control that scores text is not performing badly here in the way it performs badly on a plain authorized request, where the score is real and the threshold is wrong. It is being handed a page whose readable content is untouched. The modification is real and it is structural, which means the control with a chance at the input layer is a DOM or policy check nobody files under AI security. Trace the attacks on the public record from a translation override in 2022 to a poisoned tool description last year and every one of them ends with a sentence arriving somewhere a model could read it. This one ends with a coordinate.

Which Agents in Your Cluster Have a Perception Layer to Attack

The exposed property is a rendered raster handed to a vision model.

That is narrower than “browser agents” and wider than “the browser.” An agent qualifies when it takes a picture of something an attacker can influence and decides its next action from the picture. Four shapes of that live inside ordinary clusters.

Headless browser agents doing portal work qualify, and they are the common case: an agent driving a supplier portal, an invoice system, a carrier tracking site, or any third-party web application your team never wanted to build an integration against. QA and monitoring agents qualify when the pages they exercise carry externally served content, including ad slots and embedded widgets. Scraping and competitive-research agents qualify by definition, since every page they open is attacker-writable by someone. Document pipelines qualify when they rasterize an incoming PDF and hand the image to a vision model, which is how most invoice and claims processing works.

Agents that parse the DOM or the accessibility tree as text and never look at an image are outside this. So is anything where the rendering surface is entirely under your control.

One caveat has to sit here rather than in a footnote. The published threat model is a person viewing a page on a physical monitor, and the perturbation is optimized against that monitor’s color profile. A headless agent in a cluster has no monitor. The reason the attack still concerns you is that the monitor is the hard part: the authors spend the paper approximating a transform that varies across displays, and a containerized browser renders through one fixed, known pipeline with no such variance. That reasoning is ours rather than the paper’s, and nobody has published the cluster version. Treat it as a surface to inventory, not a demonstrated exploit against your stack.

The precondition is that an attacker can influence the pixels your agent renders, which is the same precondition as any indirect injection through a page. What the precondition buys has changed.

Which raises the objection worth answering before it gets raised for you. An attacker who can write the page could write a sentence on it, and writing a sentence is free while running an optimization is not. That is true, and it is why text injection stays the common case. What the perturbation buys is the removal of two dependencies. A sentence has to survive the model deciding whether to follow it, and it has to survive whatever scores the page on the way in. A perturbation is optimized against the action itself, so it does not ask for the action and does not need to be persuasive. The price is compute and model access. The return is determinism, which is why this belongs in a different detectability class than the sentence it replaces.

Three limits belong on that, and they are what separates a threat model from a headline. The attack is white-box: it assumes access to the parameters of the model the agent runs, which is reasonable for the open-weight vision models the authors tested and unevaluated against closed ones, a limitation they state themselves. Generalization across user prompts is bounded, holding up when a user’s phrasing stays semantically close to the prompts used during optimization and untested beyond that. And these are results on collected webpage datasets, not an incident report. An agent running a closed frontier model is not covered by any published number here.

A Goal-Alignment Check Cannot Read a Coordinate

Three controls sit between a web agent’s plan and its click: the classifier that reads the page, the alignment check that judges the proposed action, and the confirmation gate that stops on enumerated rows. The most complete public architecture for an agentic browser ships all five layers including these three, and each of the three fails here differently.

The classifier reads the page and finds the page. Its job is to flag content that has intentionally targeted the model, and the vendor states plainly that it cannot flag everything sent with malicious intent. That limit is usually read as a rate on adversarial text, which is how detection models get compared to each other. Against WebInject it is not a rate at all, because the content the classifier reads is the content the site publishes.

The alignment check judges the proposed action from a description of it, deliberately blind to the page so a malicious page cannot poison it. That blindness is the guarantee and also the ceiling. A click on a task-relevant page, during the task the user asked for, at a coordinate inside a form the task involves, serves the stated goal. The check approves it, and the check is correct. Nothing about the action describes where the action came from.

The confirmation gate fires on an enumerated list: sensitive site categories, credential use, purchases, sent messages. The gate covers what is written on it, and a click at a coordinate is not written on it.

Control position relative to the model’s decision decides what a control can catch. All three of these sit around a decision the model has already made, and each evaluates that decision using either the content that produced it or a description of what it proposes. The residual the browser design names for itself is semantic: an ask shaped like the task, phrased as ordinary site copy. This one is below the semantic layer entirely. There is no ask.

The Action Record Is the Only Evidence This Attack Produces

The good news is that the evidence already exists, and you are already collecting it.

When nothing legible enters the agent, what the agent did is the whole case file. A click at a coordinate becomes a form submission, which becomes an outbound request, which becomes a credential exercised against a destination. Each of those leaves a record at the kernel whether or not anything upstream understood why it happened.

That record is worth something only when it is scoped to one agent. A form submission on a supplier portal is routine for the agent that reconciles invoices and unprecedented for the agent that checks shipment status, and no fleet-wide rule expresses the difference. It is the same comparison that surfaces intent drift when an agent’s authorized actions stop resembling its own.

ARMO records that from the kernel with an eBPF sensor and builds it into a per-agent behavioral profile, Application Profile DNA (APD™), held at the Deployment level so it survives the churn of ephemeral pods. The question it asks about a driven click is narrow and answerable: has this agent performed this action, against this destination, before. A coordinate the attacker chose produces a request the profile has no precedent for, and that stays true regardless of what the classifier saw, what the alignment check approved, or whether any instruction was ever written.

Arming that has to be sequenced or it takes production down. The sensor observes first and records the profile. You review what a policy would have blocked against real traffic. Only then does the policy arm, per agent, which is what progressive enforcement means in practice and why a wrong boundary shows up in the record instead of in an outage.

Credential isolation sets a second bound. When an agent holds placeholder credentials and the real ones are substituted outside its reach, a driven click that needs a key completes with a key that opens nothing. It does not stop the click. It decides what the click is worth.

One residual belongs on the page. A driven action that lands inside the agent’s recorded envelope, a click on a button this agent presses every day, will not surface as deviation. That is the residual every behavioral control carries, and it is the argument for keeping an agent’s reachable actions small enough to reason about. ARMO’s AI workload security is built for the case where the input plane has nothing to offer, and this attack is the cleanest published instance of that case.

Detection Has a Floor and This Attack Is Standing On It

Every control that inspects input inspects a representation of meaning: a string, a string recovered from an image, an embedding of a string. WebInject attacks the mapping from pixels to action, which sits below meaning, and it shows that the input plane has a floor rather than a difficulty curve.

Three things follow for a security team this week. Inventory the agents whose perception step is a rendered image, using the four shapes above, and mark which of them read pages an attacker can write. Ask whether anything in your path would notice a page that repaints its own viewport through a canvas and hides the original elements behind it, because that check exists in web security and almost never gets pointed at agent traffic. Then take each of those agents and ask what it can complete with no human in the path, because testing each consequence is how that list gets read before an attacker reads it, and the list is finite.

Injection has always been an argument about what the model was persuaded to do. This one skips the persuasion and leaves the doing, which is the part your sensors were watching anyway.

FAQ

What is the WebInject attack? WebInject is a prompt injection against web agents built on multimodal models, presented at EMNLP 2025. It adds an optimized perturbation to the raw pixel values of a rendered webpage so that when the agent screenshots the page, the image drives the agent to emit an action the attacker specified. No instruction is placed in the page. The source code is modified, since the perturbation is implemented by injected code that repaints the viewport, but nothing readable as an instruction is added and the change stays invisible to a person.

Is WebInject a prompt injection or an adversarial example? Both descriptions fit, and the distinction decides which control applies to it. It is an adversarial example by mechanism, since it is a gradient-optimized perturbation on model input. It is filed as a prompt injection by outcome, since untrusted content causes an agent to act with the application’s authority. Teams that classify it only as an adversarial example tend to route it to model evaluation, which is the wrong home for something that ends in a credentialed action.

Can a prompt injection classifier be tuned to detect WebInject? Not through better tuning of a text classifier, because there is no text to tune against. Detecting the perturbation itself is an image-domain problem and a different class of control, and research on detecting and localizing webpage injections is active enough that the position is worth rechecking. The control most likely to catch this today is not in the AI security stack at all: the attack repaints the viewport through injected code, which is a structural change a DOM integrity check or a content security policy can see. Everything after that has to come from what the agent does.

Which of my agents are actually exposed? Any agent whose perception step is a rendered image it did not produce: headless browser agents working third-party portals, QA agents exercising pages with externally served content, scraping and research agents, and document pipelines that rasterize incoming files for a vision model. Two honest qualifiers. Agents that read the DOM or accessibility tree as text and never look at a picture are outside this. And the published attack targets a user at a physical monitor running an open-weight vision model, so treat a containerized agent as an unmeasured surface rather than a proven target.

What actually detects it, then? Deviation of the agent’s actions from that agent’s own recorded behavior. The attack produces a click, then a request, then a credential used against a destination, and those are observable even though nothing observable entered. The control has to be per-agent rather than fleet-wide, and it has to be armed through an observe-then-enforce sequence so the boundary is validated against real traffic before it blocks anything.