Databricks published a demonstration this week that is worth taking seriously on its own terms. It shows a realistic agent — a vendor-review assistant with three tools: read the runbook, read a document, send the report — quietly steered into leaking confidential commercial terms to an outside address, with every individual step looking like ordinary work [1] [2]. This is a slow-burn attack: the malicious goal is split into small legitimate actions, and only the combination is harmful. No single-action guardrail catches it, because no single action is dangerous [1].
Then they defend against it. A single built-in contextual policy — a session risk score — keeps one number in session memory. Every document read adds thirty to it. The outbound email tool is watched, and once the score crosses a threshold of fifty, that tool flips from allowed to denied [1]. Run the attack again: the agent reads the runbook, opens two documents — the score climbs zero, thirty, sixty — writes the summary, tries to send, and is stopped at the gate [1]. The demo works. The confidential terms stay inside.
It is a clean result, and the mechanism deserves the credit the authors give it. It also proves, more precisely than any critic could, what the governing layer is and is not.
What the score caught
The score caught a symptom. It did not see the cause.
The cause was a single tampered line in the runbook — a records-retention instruction, phrased as routine compliance, telling the agent to send the finished summary to an external archive [1]. That line was an indirect prompt injection: it arrived inside content the agent read, not in the user’s request, and the authors are explicit that the model’s own safety training did not catch it because, phrased as ordinary process, it read as legitimate work [1].
Here is the part the demo does not dwell on. When the injection arrived, the policy layer saw it — policies monitor tool calls and their results, which is how the read events incremented the score in the first place [3]. The layer read the poisoned runbook, passed it to the model, and kept a number. What it retained was reads-before-send equals sixty. What it discarded was the instruction that made the send malicious. The score is a true statement about the session’s shape and a silent one about the session’s cause.
That is not a defect in the policy. It is the definition of the policy. A risk score is a scalar the authors themselves describe as a single number that tracks how much risk has built up as the agent works [7]. It is designed to be legible precisely by throwing away everything that is not the count. The demo succeeds because the attack it was built against happened to be expressible as a count: reads accumulate, and the outbound gate trips. The retention that makes it work is the retention that makes it blind to anything that is not a counting problem.
The tamper-resistance is real — and it is still enforcement
The strongest section of the demo is the one that anticipates the obvious objection: can the agent just turn the policy off? The answer is no, and the authors give three properties, all verifiable in the source, that make it so. There is no tool to remove, edit, or disable a policy — the agent is given exactly two policy-related tools, one to browse and one to add [1]. Adding a policy requires explicit human approval [1]. And when policies combine, a single denial wins, so no added permissive rule can lift an existing block [1]. The control sits outside the agent, in the runtime, not inside the reasoning that can be argued with [1].
Every one of those properties is real, and every one of them is a property of enforcement. They harden the gate. They guarantee that when the threshold is crossed, the deny holds — that a compromised agent cannot lower the number or drop the guard. What none of them does is retain the basis. Tamper-resistance answers can the verdict be overturned. It does not answer why was the verdict correct. The demo makes the enforcement unarguable and leaves the attestation exactly where it was: absent.
Even the sophisticated builtins are flow control
It would be a weak reading to rest this on the risk score alone, because the risk score is the simplest of the builtins. The others are more sophisticated, and they make the same point more sharply.
The Google Drive policy implements a no-write-down rule: mark a set of documents confidential, and the moment the agent opens one, writes are confined to that set, so it cannot copy privileged content into a document it is allowed to share [3] [7]. That is the Bell-LaPadula model from 1973 — a classification control that governs the flow of data between sensitivity levels. It is a genuinely good primitive. It is also, structurally, flow control: it governs where data may move, and keeps no account of why a given movement was reasoned to be safe.
The intent-based authorization policy is sharper still against this very demo. It locks each session to only the tools relevant to the user’s original prompt [7]. But the slow-burn works precisely because the malicious step was made to look like the original task — a records-retention line inside a vendor-review runbook, where sending the report is the job [1]. Intent-lock keys off the prompt; the attack keys off content read after the prompt. Two builtins, and the injection lives in the seam between what each one watches.
None of these is attestation. Every one governs a flow — of risk, of classified data, of tool scope — and a flow control that keeps no record of its reasoning is a gate, not a witness.
Who, whether, and the question no surface answers
The managed corridor adds a second governing layer, and it is worth reading because it fails in the same direction by a different mechanism. On Databricks, every request carries the user’s workspace identity, and Omnigent attributes each session and each turn to that identity [8]. This is real governance: authentication through the workspace identity provider, sessions private to their owner until shared, each turn’s author shown by workspace email [8].
But notice what identity resolves. It answers who ran the session. The scalar answers whether the action could proceed. Neither answers why the action was permitted — on what basis, under whose reasoning, with what knowledge of the tampered instruction that made a routine send a leak. The identity layer records the actor; the policy layer records the verdict; the reasoning that connects them is retained by nothing. And the sharing model shows the same grain: an Edit grant lets a collaborator run shell commands and modify files on the host through the agent, and view the owner’s other session chats — coordination-grade access, recorded as whose identity holds it, not what basis justified each act [8]. And on the managed corridor the enterprise cannot answer the gap by writing a better policy: only built-in contextual policies are supported, and custom policies are not available [4].
Two layers, two honest answers — who, and whether. The auditor’s question is the third one, and it is the one the architecture is built not to keep.
The gate governed the exit. The injection was already inside when it fired — read by the layer, passed to the model, and discarded in favor of a number. A control that hardens the door data leaves by, while keeping no account of the reasoning that arrived through the door it came in by, is coordination-grade by construction: it can prove the send was stopped and never why the send was wrong.
The demonstration is the strongest available evidence for that reading, because it is the architecture performing at its best and still leaving the auditor a scalar and an identity where a decision should be. Enforcement was never the open question. Every gate in the demo holds. The open question is retention, and the demo answers it in the negative the way only a success can.
