Prompt Infection: Why LLM-to-LLM Injection Makes Every Agent Your Blast Radius

Self-replicating prompt injection is a containment failure.

The detection part usually works. A sensor fires, an analyst opens the alert, the workload it names gets quarantined, and the ticket closes. The payload is still running on agents the alert never named, because it copied itself into them before anyone opened it.

A conventional injection compromises the agent that read it, and that agent is the one your alert names. A self-replicating one turns every agent it reaches into a carrier, so your alert names one member of a set nobody has measured.

We have described coercion as the attack where a trusted input redirects an agent’s authorized capability toward someone else’s goal. Every step is permitted, and the sequence is the attack. Prompt infection is coercion with a copy instruction attached, and the research from University College London and Stanford that named it is where the mechanics were first measured.

That work ran on GPT-4o and GPT-3.5 Turbo, against simulated multi-agent applications and a simulated society of agents, with payloads planted in synthetic documents rather than live systems. The models are two generations old, so the absolute success rates below will not transfer to your stack. The structural findings will, and those are what this piece is built on.

Replication changes the size of the incident your runbook has to scope

The infection prompt carries four parts: an instruction that makes the agent abandon its role, a payload assigning work based on the tools that agent holds, a data field, and the self-replication instruction that forwards the whole package onward.

The data field is the part worth sitting with. It is a running note that accumulates as the prompt crosses each hop. One agent appends the tool inventory it discovered. The next appends the records it pulled. By the time the package reaches an agent with code execution, the sensitive material is already inside the prompt.

Those agents are making network calls throughout, of course: retrieving documents, querying databases, hitting APIs. The point is that none of that traffic looks like exfiltration, because the collected data is riding inside a prompt rather than inside a payload leaving your environment. The first egress event that carries anything worth stopping is the last one in the chain.

Spread through a connected population follows a logistic curve rather than a linear one. In the paper’s agent-society simulation, a ten-agent population reached full infection around turn 4.7 and a twenty-agent population around turn 6.3. At thirty, forty and fifty agents, saturation arrived in a proportionally smaller share of the simulated turns, in the range of roughly a fifth to a quarter. Read that carefully rather than as a clean monotonic law: the reported fractions do not decrease strictly with size, and the runs that produced them included the memory manipulation described further down. The directional finding is the useful one. More agents means more transmission opportunities per turn, so larger fleets do not dilute an infection.

Workload-scoped containment was the correct design for every threat that came before this one. A compromised container is a compromised container, and quarantining it ends the exposure. Replication is the first agent-native attack where the pod you caught is a sample rather than a boundary.

The defenses that work require unanimity your fleet does not have

The research proposed LLM Tagging as a defense: prepend a marker identifying the speaking agent, so a downstream agent can tell an upstream agent’s output from a user instruction. Deployed alone, it reduced attack success by roughly 5%.

Marking, which inserts a distinguishing symbol between user and agent content, initially held at 0% attack success. Then the researchers wrote a counterattack that neutralized the marker by interleaving underbars through the payload, and marking on its own permitted 76% of attacks.

The combinations are where the numbers get good. Marking paired with LLM Tagging prevented every attack in that evaluation. Instruction Defense, which adds a standing instruction never to modify the user’s original request, paired with Tagging dropped attack success to 3%. Sandwich, which repeats the user instruction after the prior agent’s output, paired with Tagging landed at 16%.

Now read that table as an operations requirement rather than a research result. Each of those defenses is a convention about message format. A convention only functions when every participant follows it. One agent that does not prepend the tag is a clean channel through the middle of your fleet, and the infection only has to find it once.

That matters because of who owns the agents. Count how many of the agents in your production graph run prompt-handling code your security team can change. The support workflow belongs to one team, the internal research agent to another, the coding agent in the remote development environment to a third. Somewhere in the graph there is likely a vendor agent or a third-party MCP tool server, and neither ships with your message-format convention.

None of this argues against deploying tagging. Inside an orchestrator you control it is close to free, it reduces the arrival rate of payloads that reach the rest of the fleet, and every infection stopped there is one you never have to scope. It just cannot be the thing holding the line, because its guarantee is only as strong as the least cooperative agent in the chain.

The attacker trades reach for stealth and pays in runtime evidence either way

Self-replication is not uniformly better for the attacker, and the exception is the interesting part.

Under global messaging, where agents share full message history, replication won clearly on scams, malware distribution and content manipulation: about 13.92% higher success on GPT-4o and 209% higher on GPT-3.5 Turbo. Data theft behaved differently. Self-replication still won at three agents, but as the agent count grew the non-replicating variant overtook it by an average of 8.48%, because data theft demands tight coordination across at least three specific roles and copying the payload at every hop gets in the way.

There is a countervailing constraint. Under local messaging, where each agent sees only its predecessor’s output, the non-replicating variant struggled to compromise more than two agents at all. Replication was the only method that scaled further, which is what makes it the only viable approach against a system that limits what each agent can see.

So the attacker picks. Optimize for reach and the payload crosses more agents than the objective needs, most of them irrelevant, each one departing from its own normal behavior. Optimize for precision and the chain stays short and quiet, straight to the objective.

Those two choices produce different evidence. Reach looks like a cluster of agents departing from their baselines inside the same window. Precision looks like a short causal chain with an abrupt terminus.

Neither branch removes the terminus. To get anything out, one agent has to make an outbound request, and the design of the attack means that agent is making a request it has no history of making. Prompt text is cheap to vary and cheap to retry. Execution is neither.

A stronger model is a better carrier once it is carrying

In the paper’s failure analysis, GPT-4o ignored 66% of self-replicating infection attempts outright. GPT-3.5 Turbo ignored 9%. On resistance, the stronger model is clearly better, and if you are choosing a model for an agent facing untrusted input, that gap is worth paying for.

The rest of the analysis complicates the picture in two directions, and both are worth carrying.

First, some of GPT-3.5’s apparent resistance was incompetence wearing the costume of judgment. A large share of its failures fell into agent errors and no-action outcomes, meaning the attack failed because the model fumbled the task. Nobody should want that sitting in a risk register as a control.

Second, when GPT-4o did get compromised, it produced fewer deformed replications and fewer cases of confusing the user’s task with the attacker’s. It copied the payload more faithfully and executed the assigned step more precisely. The capability that makes a model good at following your instructions makes it good at following someone else’s.

The practical consequence is narrow and worth writing down verbatim. A model upgrade moves your arrival rate. It leaves your blast radius where it was.

There is a second consequence that surfaces during incident response. Mixed-model fleets have mixed susceptibility, so the infection front stops wherever it meets a resistant agent. Those agents produce no alert, and they may sit between two agents that do. Scoping an incident by counting which agents fired therefore undercounts reach in a predictable direction, and it is the direction that leaves carriers running.

Memory can keep the infection after the pods are gone

Whether the infection survives your response depends on how your agents remember things, and this is the question most runbooks never ask.

Some agent memory architectures rank stored entries for retrieval on a combination of importance, recency and relevance, with the importance score assigned by a model. That design comes out of the generative-agents line of research and shows up in agent frameworks that need long-running memory. Plenty of production RAG stacks do nothing of the kind and rank on similarity alone. Find out which one you are running, because the answer determines whether a pod restart is a fix.

Where importance scoring is in play, it is manipulable by the same mechanism as everything else here. Embedding a single line instructing the rating model to assign the maximum score moved GPT-4o’s rating of the infection prompt from 1.94 to 10.00, and GPT-3.5’s from 1.00 to 9.84, averaged across a hundred runs. Without that line the infection could not compete with genuinely important memories and dropped out of the retrieval window within a few turns, at the top-three retrieval setting the simulation used.

With it, the entry sits at the top of the ranking. Retrieval then resets recency, which pushes it further up. The payload has bought a durable seat in the context of every agent that reads from that store.

That store is usually a vector database or a session backend, and it is usually instrumented for cost and latency rather than for security. So the containment step nobody has in their runbook is the one that matters most here: identify the writes made during the infection window and remove them. Restart every pod in the namespace and the first retrieval after the agents come back can reintroduce the payload from a store the restart never touched.

Contain the reachable graph, then reopen agent by agent

The signal you need mostly exists already. What has to change is the unit the response operates on.

A per-agent behavioral baseline is what makes fleet-scale infection legible. ARMO builds this as Application Profile DNA (APD™), derived from runtime observation rather than from a declared manifest: kernel-level eBPF telemetry covering processes, syscalls, file access, network destinations and identity use, with application-layer signals adding the tool invocations and execution chains the kernel cannot see on its own. Each agent is measured against its own history rather than against a namespace-wide policy.

Measured that way, replication produces a signature that is easier to act on than a single-agent compromise. Several agents departing from several independent baselines inside the same window is a high-confidence pattern, and it arrives while the chain is still running.

It is not self-interpreting, though, and pretending otherwise would set up a false positive. Legitimate change does move several agents at once: a shared base image rolls out, a model endpoint version changes, a common RAG corpus gets refreshed. Those shifts arrive alongside infrastructure events. A correlated behavioral shift with no deployment, image digest change or configuration event behind it is the one that scores as deviation. Correlating the two is the discriminator, and it is why this needs Kubernetes context rather than syscall telemetry alone.

Credential handling closes the terminus. An agent operating with substituted credentials rather than live ones reaches nothing when it tries to use them, no matter how many carriers are upstream of it. The infection can spread widely and still come away empty, because the one action it cannot avoid is the one action that does not work.

The containment sequence that follows runs in four steps.

Scope by reachability. Start from the agent that alerted and take everything it could have delegated to, written a shared store for, or handed context to, transitively. Contain that set. Do not start from the list of agents that fired, because resistant agents in the middle of the graph produce no alert and pass the payload anyway.

Neutralize credentials across the scoped set before triaging any individual agent. This is the cheap, reversible action, and a false positive costs a retried call. It stops the blast radius expanding while the rest of the work happens.

Purge the memory writes inside the infection window. Identify what each contained agent wrote to shared vector stores, scratchpads and session state during the window, and remove it. Skipping this step risks reinfection on the first retrieval after recovery.

Reopen per agent in Audit. Bring agents back one at a time with enforcement logging rather than blocking, confirm the baseline is clean against live traffic, then graduate to enforcement. This is the same progressive enforcement path that keeps generated policies from breaking production, applied to recovery instead of rollout.

Replication is a scoping problem before it is a detection problem

Most teams running agents in production already collect a good share of what this needs. Process execution, network connections and identity usage come from kernel-level instrumentation. Tool invocation and agent execution chains need application-layer visibility on top. What is usually missing is a per-agent definition of normal to score all of it against, and a response whose unit is the agent graph.

Measure one number in your own environment: the time between the first scored baseline deviation and containment of the full reachable set, not just the containing pod. If that number is expressed in workloads rather than graphs, adding another detector produces a more detailed account of the same breach.

The infection is cheap for the attacker to write and cheap to retry. Making every agent do something is not.

See how runtime behavioral security for AI workloads scores per-agent deviation across a fleet and contains a coerced agent graph before the transfer completes. Book a demo to walk an infection chain on your own clusters.

Frequently Asked Questions

How do I tell a prompt infection from several unrelated agent alerts?

Check whether the correlated deviations line up with an infrastructure event. Unrelated alerts across a fleet usually share a cause you can point at: a deployment, a base image update, a model version change, a corpus refresh. An infection produces simultaneous departures from multiple per-agent baselines with no corresponding change event behind them. That correlation, rather than the raw count of alerting agents, is what separates the two, and it requires Kubernetes and deployment context alongside runtime telemetry.

Does LLM Tagging work well enough to be worth deploying?

Deploy it inside orchestrators you control, and do not count on it outside them. On its own it reduced attack success by roughly 5% in testing, which is real but small. Combined with marking or instruction defense it performed far better, and those results are worth having. The limit is structural rather than technical: the guarantee depends on every agent in the chain implementing the same convention, and most production fleets include agents whose message handling the security team does not own.

If I contained the compromised pod, why might the infection still be live?

Because the pod was a carrier rather than the boundary. Other agents in the graph may have received the replicated prompt before you contained the first one, and each runs independently. Separately, if the payload reached a shared memory or vector store that ranks retrievals by a model-assigned importance score, it can survive every pod in the namespace restarting and re-enter context on the next retrieval. Containment has to cover the reachable agent set and the stores those agents read from.

Are stronger models a mitigation for prompt infection?

They reduce how often an injection lands and they do not reduce what happens when one does. In the study, GPT-4o ignored 66% of self-replicating attempts against GPT-3.5’s 9%, so the resistance gain is real. Once compromised, though, the stronger model replicated the payload more faithfully and executed the malicious step more cleanly. Treat a model upgrade as a reduction in arrival rate and keep your blast-radius controls exactly where they were.

What telemetry do I need before per-agent baselines are useful here?

Process and syscall activity, file access, network destinations and identity usage attributable to the specific agent workload, which kernel-level eBPF collection supplies without code changes. On top of that you need tool invocation records and execution chains, which sit above the kernel and require application-layer visibility. Finally you need the delegation relationships between agents, so scoping an incident can follow reachability instead of guesswork. That last surface is the one we have covered in depth for multi-agent orchestration deployments.