Detecting Prompt Injection in Enterprise AI Copilots

Your company runs two kinds of AI copilot, and only one of them can be watched for prompt injection.

The first kind is a vendor product running in the vendor’s cloud. When it reads a document or answers a question, the only record you get is the vendor’s audit log. That log tells you a document was touched. It does not tell you what process ran, which credential was used, or where the data went. No security product can add that, because the runtime is not yours.

The second kind is a copilot your own team built and runs on Kubernetes. It usually has a service account and read access to the wiki, the ticket system and the data warehouse. Because it runs on your infrastructure, every action it takes can be observed at the kernel.

The security effort usually goes to the first kind, where nothing more can be seen, and the second kind runs unwatched.

This article splits the inventory into two security problems, states what a vendor-operated copilot actually lets you detect, shows why a copilot is the hardest coercion case you own, gives a four-question ceiling test per copilot, and shows how to raise the ceiling on the copilots you run yourself.

Your Copilot Inventory Is Two Security Problems

One inventory holds two kinds of copilot, vendor-operated and self-operated, and the line between them is who runs the process that reads your data.

That line sets what this article calls the detection ceiling. The ceiling is the most your security team can ever learn about what a copilot did, and it is fixed by tenancy: whether the copilot’s runtime lives in the vendor’s tenant or in yours. Tooling moves you toward the ceiling. Tooling cannot move the ceiling.

The vendor-operated rows are the ones the rollout memo was written about: the productivity assistant licensed per seat, the chat assistant embedded in the ticketing product, and the coding agents running in a vendor’s cloud dev environment with an organization token. Each of them reads your content inside a process you do not own, on a host you cannot instrument.

The self-operated rows are quieter. A retrieval copilot the platform team built in a quarter, deployed on the same clusters as everything else, with a service account, a vector store fed from the wiki, and connectors to the ticket system and the warehouse. It answers questions for internal teams and nobody outside engineering knows its name. It reads more sensitive data than the vendor assistant, through the same delivery path, and it produces the same attack chains when an injected instruction lands.

Both kinds get one policy, because the inventory has one column for “AI assistant” and the policy was written against the vendor rows. That default is an instinct. Nobody decided that the copilot holding the warehouse credential deserved less attention than the one summarizing email; the rollout memo simply never listed it.

Split the column. Every row below is either a vendor-tenant copilot or a self-tenant copilot, and the two halves have different ceilings, different evidence and different budgets.

What a Vendor-Operated Copilot Actually Lets You Detect

The vendor’s audit log is the whole detection surface for a vendor-operated copilot.

The log records who prompted, when, in which host application, and which resources the copilot touched to build the response. Microsoft’s documentation for Copilot audit records lists the user, the time, the host application, the plugins invoked, and each accessed resource with its action (read, create, modify) and sensitivity label, retained 180 days by default. It answers “did this copilot read the board pack” and “which user’s session pulled the customer list.”

The vendor also adds detection of its own inside the tenant, because the vendor can see its own runtime. The same audit record carries a per-resource flag for a detected cross-prompt injection and a per-message flag for a detected jailbreak. Those detections, the retention window and the export format are the vendor’s to set, and yours to configure.

Four signals never reach the log. It does not record the process that assembled the response. It does not record which credential the copilot presented to a connector outside the tenant. It does not record the outbound destination when a response includes a link the client fetches; web grounding appears as a plugin name, without the host. And it does not order the actions inside an interaction, since the accessed resources arrive as a set with one timestamp for the whole exchange. Detection through this log is retrospective, shaped by a schema you did not write, and blind to sequence below the interaction.

EchoLeak made the shape of that blindness concrete. CVE-2025-32711 was a zero-click exfiltration from Microsoft 365 Copilot through a crafted email; the Microsoft advisory rated it critical and fixed it server-side. Ask the four-signal question against it. While the exfiltration ran, the tenant log could show that a Copilot interaction referenced the email and the internal files. It could not show the fetch to the attacker’s endpoint, the credential context of the retrieval, or the order in which the reference and the fetch occurred. Microsoft could see those, in Microsoft’s runtime, and Microsoft shipped the fix. The customer’s view ended at the log.

The vendor can raise the ceiling on a vendor-operated copilot, because the vendor owns the runtime. No third party can, and that includes ARMO. Anything sold as runtime detection for a copilot in someone else’s tenant reads the same log you already have, or a copy of it, and the attribution ceiling it hits is the schema.

Spend on the vendor rows accordingly. Turn auditing on, set retention to the maximum the license allows, export to your SIEM, and enable whatever the vendor’s own detections cover. Then stop, because the ceiling has been reached.

Why a Copilot Is the Hardest Coercion Case You Own

A copilot exists to read broadly, so the one control that shrinks most agent incidents is unavailable to it.

Coercion is the attack that fits this shape. A trusted prompt, arriving through a retrieved document or a ticket, redirects capability the copilot already holds. No control breaks. Every step is permitted. The sequence is the attack.

For most agents the first response to coercion is to cut scope: fewer tools, a narrower service account, an egress allowlist. A copilot cannot take that cut. Its value is read access across the wiki, the ticket history, the shared drives and the warehouse, and a copilot that reads less is a worse product by exactly that much. The permission set is the feature, and so is the blast radius.

So the borrowed authority a copilot lends an injected instruction is the widest of any agent you run, and it is the authority you least want to reduce. What it lends is exactly what the instruction asks for: a summary of a customer table, included in the answer. That request is tool misuse executed with a valid identity, and nothing in its text is unsafe on its own. A filter reads a request for a summary. The copilot reads the same thing.

With scope fixed and text clean, one signal is left. The copilot has a history. It reads the wiki, calls the vector store, answers, and repeats, and the shape of that repeats day after day. A read of a table it has never touched, a connection to a host it has never contacted, a credential path it has never opened, stand out against that history even though each one was allowed.

Behavior is the only signal that survives the copilot’s own permissions. The rest of this article is about whether you can see it.

A Four-Question Ceiling Test Grades Every Copilot in an Afternoon

Four questions, asked of each inventory row, produce the row’s ceiling grade: attribution, action record, sequence, and latency.

Answer them from the records you already have, and write the grade next to the row. The test takes an afternoon for an inventory of a few dozen copilots, and the grade decides the budget that follows.

Attribution: does any record name this copilot?

The first question is whether any record you hold names the copilot as its subject. A vendor log names the user and the application. A Kubernetes audit log names a service account that three other workloads share. A pod log names a pod that was replaced this morning. If the best you can do is a shared identity or an ephemeral name, then any evidence you gather cannot be assigned to this copilot, and detection stops at “something in that namespace did this.”

Attribution depends on which of the instrumentation tiers the record comes from. Framework callbacks name the agent and can be stripped by the agent. Kernel records name the process and cannot be. Grade the row on the strongest record that survives the copilot being compromised.

Action record: is there a record of what it did?

The second question is whether the record describes what the copilot did outside the vendor’s own surface, or only what it touched inside it. “Read file X in SharePoint” is inside. “Opened a connection to host Y with credential Z” is outside. Vendor logs stop at the tenant boundary, because that is the boundary of what the vendor’s product performs on the vendor’s side, and the actions that carry data out happen wherever the copilot’s connectors and links reach. A self-hosted copilot produces actions in your cluster, and the question is only whether you are recording them below the framework.

Sequence: can two actions be ordered from one clock?

The third question is whether two events can be ordered. Coercion is a sequence, so evidence that cannot be ordered cannot show coercion. A retrieval and an egress logged by two systems with two clocks and two retention windows prove that each happened. They do not prove that one caused the other. A single sensor with one timeline proves both.

Latency: how old is the record when you read it?

The fourth question is how long after the action the record is readable. A vendor export that lands nightly grades the copilot as retrospective. A SIEM ingest that lags an hour grades it as slow. A kernel stream read within seconds grades it as live. The grade sets what response is possible: a nightly export supports a scope review, and only a live stream supports stopping the sequence before the last step.

Write four cells per row and read the pattern. Vendor-tenant rows grade out at “user-level attribution, in-tenant actions only, ordering by interaction only, retrospective,” and that grade is the honest ceiling for those rows. Self-tenant rows grade wherever your instrumentation puts them, and for most companies that is the same grade as the vendor rows, because nothing was ever installed under the copilot. The difference is that the self-tenant grade can change.

Raising the Ceiling on the Copilots You Run Yourself

A copilot running on your Kubernetes clusters can be observed at the kernel, below the framework, without a line of its code changing.

A kernel-level sensor records what the copilot’s process does in execution: files opened, processes spawned, connections made, credentials read. The record does not depend on the framework emitting anything, so an injected instruction that drives the model to bypass the framework and call a tool directly still produces evidence. The overhead sits at 1 to 2.5 percent of CPU and 1 percent of memory, with no sidecar.

From that record ARMO builds a per-agent behavioral profile, Application Profile DNA (APD™), and evaluates each new action against that copilot’s own history. A read of the customer table is routine for a reporting agent and unprecedented for a wiki copilot, and a fleet-wide rule cannot express that difference. The profile can, because it was built from this copilot and no other.

The profile reads at the level where coercion shows. It does not stop at the verb, since “query the warehouse” is what the copilot does all day. The profile holds the argument and the destination: which tables, which hosts, which credential paths. A query against a table outside the profile, or a connection to a host outside it, is the deviation, and the eight-stage chain an injected agent runs lands as one Attack Story on one timeline, which answers the sequence question from the test.

Turning the profile into a bound follows progressive enforcement: Audit, then Enforce. The sensor records first. The team reviews what a per-copilot policy would have blocked against a week of real traffic. The policy arms for this copilot once the record shows the boundary is right, and no query is blocked before that.

Credential isolation adds an outcome on top. The copilot works with placeholder credentials, and the real ones are substituted outside its reach. An injected instruction that asks the copilot to read its warehouse credential and include it in a response completes with a credential that opens nothing. The coercion still happens. What it borrows is smaller.

AI workload security from ARMO covers the self-tenant half of the inventory: what the copilot did, measured against what it does. Bring one self-hosted copilot and its inventory row to a working session and the four cells get filled in from its actual runtime record.

Assign Every Copilot a Ceiling Before You Assign It a Policy

Every copilot in the inventory has a ceiling before it has a policy, and the ceiling should decide the policy.

For a vendor-tenant copilot, the ceiling is the vendor’s log. Configure it fully and budget nothing past it. For a self-tenant copilot, the ceiling is the kernel, and the grade you wrote in the test is the distance between what you could see and what you do see.

Do the split this week. Take the inventory, add the tenancy column, grade four questions per row, and move the budget toward the rows whose grade can change. The copilot holding the warehouse credential has been running unwatched for as long as it has been running. The vendor assistant was never the harder problem.

FAQ

Can I detect prompt injection in Microsoft 365 Copilot? Within the ceiling the tenant gives you. Purview audit records show which user prompted, when, in which application and which files were referenced, and Microsoft’s own tenant-side detections cover part of what the log cannot. What no customer can see is the process, the credential context, the outbound fetch or the order of events, because those happen in Microsoft’s runtime. Enable auditing, set maximum retention, export to your SIEM, and treat anything sold as deeper visibility on this copilot as a re-reading of the same log.

What telemetry do I need for an internal copilot? Four streams, and attribution comes first. You need a record that names the copilot’s process rather than a shared service account, a record of actions outside the copilot’s own surface (process starts, file opens, connections, credential reads), a single clock across all of them so two actions can be ordered, and a read path that delivers the record in seconds. A kernel-level sensor produces all four from one source. Confirm the copilot obeys instructions in its retrieved content with one plain ask per permission it holds before trusting any of it.

Does a prompt filter in front of the copilot count as detection? It counts as a rate reduction on the attacks that announce themselves, and it should stay on. A filter scores text, and the instruction that coerces a copilot is a grammatical request for an action the copilot is allowed to take, so the filter passes it as ordinary content. The ceiling does not move, because the filter sees what the copilot read and nothing about what the copilot then did. Keep the filter, and grade the row on its action record.

How long before a behavioral baseline is usable on a copilot? As long as it takes the copilot’s traffic to show its full range, which for an internal retrieval copilot is usually a week or two of ordinary use. Run the sensor in audit mode for that window and review what a policy would have blocked against real queries before arming anything. A copilot whose connectors or corpus change materially needs the window reopened, so tie the baseline review to the change process rather than to the calendar.

My copilot is read-only, do I still need this? Yes, because read scope is the exfiltration path. A copilot that only reads can still be instructed to read a table it never touches and place the contents in a response, a link or a summary that leaves the tenant with the user. Nothing in that sequence needs write access. The action record for a read-only copilot is which sources it read and where the response went, and both are observable on a copilot you run.